Image segmentation method, device and electronic equipment

Abstract

This application provides an image segmentation method, apparatus, and electronic device that can be applied to the field of medical image processing technology. The method includes: extracting multi-scale visual features from multiple image patches to obtain global visual features and multiple sub-visual features; using a graph neural network module to aggregate neighborhood features of node features in an initial graph structure to obtain aggregated features. The initial graph structure includes node and edge relationships; node features represent pathological features of the image patches, and aggregated features represent the contextual information of multiple neighboring nodes of the target node within the aggregated neighborhood; performing attention fusion on the global visual features and aggregated features based on an attention mechanism to obtain fused features; and upsampling and decoding the fused features, global visual features, and multiple sub-visual features to obtain an image segmentation result. The image segmentation result represents the maturity level of the three-level lymphatic structure within the tissue region of the target object.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, specifically to an image segmentation method, apparatus, and electronic device. Background Technology

[0002] Tertiary Lymphoid Structures (TLSs) are ectopic lymphoid organs found in non-lymphoid tissues, and are organized clusters of immune cells. The maturity and area of ​​TLSs are significant in cancer prognostic assessment and immunotherapy response prediction. To quantify maturity characteristics, pathologists typically need to annotate numerous TLS regions at different maturity stages in whole-slice images. However, relevant image segmentation methods usually only retain coarse-grained contextual information, resulting in low segmentation accuracy. Summary of the Invention

[0003] In view of the above problems, this application provides an image segmentation method, apparatus and electronic device.

[0004] According to a first aspect of this application, an image segmentation method is provided, comprising: extracting multi-scale visual features from multiple image patches to obtain global visual features and multiple sub-visual features, wherein the multiple image patches are obtained based on the segmentation of a pathological image of a target object; using a graph neural network module to aggregate neighborhood features of node features of multiple nodes in an initial graph structure to obtain aggregated features, wherein the initial graph structure includes nodes and edge relationships, the node features characterize the pathological features of the image patches, the edge relationships are determined based on the positional relationships of multiple image patches in the pathological image, and the aggregated features characterize the contextual information of multiple neighboring nodes of the target node in the aggregated neighborhood; performing attention fusion on the global visual features and aggregated features based on an attention mechanism to obtain fused features; and upsampling and decoding the fused features, global visual features, and multiple sub-visual features to obtain an image segmentation result, wherein the image segmentation result characterizes the maturity level of the three-level lymphatic structure within the tissue region of the target object.

[0005] According to an embodiment of this application, the initial graph structure is determined based on the following operations: determining the adjacent image blocks corresponding to the image block according to the preset connectivity rules and the position of the image block in the pathological image; generating the initial graph structure based on the target node represented by the image block and the edge relationship between the adjacent nodes represented by the adjacent image blocks.

[0006] According to an embodiment of this application, the graph neural network module includes K cascaded graph convolutional sub-modules; wherein, the graph neural network module is used to aggregate neighborhood features of multiple nodes in the initial graph structure to obtain aggregated features, including: using the weight matrix of the k-th layer graph convolutional sub-module to perform a linear transformation on the (k-1)-th sub-aggregated features of the target node to obtain the k-th transformed feature of the target node, 1 < k < K; fusing the k-th transformed feature of the target node according to the k-th transformed features of the neighboring nodes corresponding to the target node to obtain the k-th sub-aggregated feature of the target node; and fusing the K sub-aggregated features of the target node using a multilayer perceptron to obtain the aggregated feature of the target node.

[0007] According to embodiments of this application, attention fusion is performed on global visual features and aggregated features based on an attention mechanism to obtain fused features, including: concatenating global visual features and aggregated features to obtain concatenated features; using a self-attention mechanism to fuse query features, key features, and value features determined based on the concatenated features to obtain attention weights; and using the attention weights to weight the value features to obtain fused features.

[0008] According to embodiments of this application, upsampling decoding is performed on fused features, global visual features, and multiple sub-visual features to obtain image segmentation results, including: decoding the fused features to obtain initial decoded features; upsampling decoding is performed on the initial decoded features, global visual features, and multiple sub-visual features to obtain target decoded features; the target decoded features are processed using a multilayer perceptron to obtain probability results; and the probability results are normalized to obtain image segmentation results.

[0009] According to an embodiment of this application, upsampling decoding is performed on initial decoding features, global visual features, and multiple sub-visual features to obtain target decoding features, including: concatenating the (i-1)th decoding feature and sub-visual features to obtain the (i-1)th concatenated feature, where the first decoding feature is the initial decoding feature and the first concatenated feature is obtained by concatenating the initial decoding feature and global visual features; upsampling the (i-1)th concatenated feature to obtain the (i-1)th upsampled feature; decoding the (i-1)th upsampled feature to obtain the ith decoding feature; and determining the target decoding feature based on the ith decoding feature.

[0010] The second aspect of this application is an image segmentation apparatus, characterized in that the apparatus comprises: a feature extraction module, used to extract multi-scale visual features from multiple image blocks to obtain global visual features and multiple sub-visual features, wherein the multiple image blocks are obtained based on the segmentation of a pathological image of a target object; a feature aggregation module, used to aggregate neighborhood features of node features of multiple nodes in an initial graph structure using a graph neural network module to obtain aggregated features, wherein the initial graph structure includes nodes and edge relationships, node features characterize the pathological features of the image blocks, edge relationships are determined based on the positional relationships of multiple image blocks in the pathological image, and aggregated features characterize the contextual information of multiple adjacent nodes of the target node in the aggregated neighborhood; a feature fusion module, used to perform attention fusion of global visual features and aggregated features based on an attention mechanism to obtain fused features; and a segmentation module, used to upsample and decode the fused features, global visual features, and multiple sub-visual features to obtain an image segmentation result, wherein the image segmentation result characterizes the maturity level of the three-level lymphatic structure within the tissue region of the target object.

[0011] A third aspect of this application provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0012] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program or instructions are stored, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0013] According to a fifth aspect of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0014] The image segmentation method provided in this application aggregates the neighborhood features of nodes in the initial graph structure based on the topological relationship between image blocks using a graph neural module, and fuses the global visual features and aggregated features through an attention mechanism. By introducing a graph neural network module to construct the contextual relationship between image blocks, it can capture long-distance dependencies within a larger field of view in the aggregated neighborhood, thus solving the bottleneck technical problem of limited field of view in segmentation based on individual image blocks. It can effectively utilize the macroscopic tissue structure information outside the image blocks to assist in segmentation, effectively ensuring the regional consistency of the segmentation results, thereby significantly improving the accuracy and consistency of pixel-level semantic segmentation of the maturity level of the three-level lymphoid structure. It has good model versatility and task scalability, and can assist pathologists in conducting more accurate prognostic analysis. Attached Figure Description

[0015] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0016] Figure 1 A flowchart of an image segmentation method according to an embodiment of this application is shown.

[0017] Figure 2A A schematic diagram of a segmentation model according to an embodiment of this application is shown.

[0018] Figure 2B A schematic diagram of the initial diagram structure according to an embodiment of this application is shown.

[0019] Figure 2C A schematic diagram of a graph neural network module according to an embodiment of this application is shown.

[0020] Figure 2D A schematic diagram of a decoder according to an embodiment of this application is shown.

[0021] Figure 3 A schematic diagram showing a visualized image segmentation result according to an embodiment of this application is provided.

[0022] Figure 4 A structural block diagram of an image segmentation apparatus according to an embodiment of this application is shown.

[0023] Figure 5 A block diagram of an electronic device suitable for implementing an image segmentation method according to an embodiment of this application is shown. Detailed Implementation

[0024] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0027] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0028] In implementing this application, it was discovered that WSI images typically reach sizes in the billions of pixels. Related deep learning segmentation methods usually employ a patch-based strategy, cutting the WSI image into small image patches for independent processing. However, the model cannot capture the contextual information outside the target image patch, resulting in a lack of overall consistency in the segmentation results and difficulty in distinguishing morphologically similar but semantically different tissue regions. Although some related techniques attempt to incorporate context using multi-resolution methods, this often limits the model's scalability and struggles to capture long-distance dependencies across a larger visual range, typically retaining only coarse-grained contextual information. Therefore, how to effectively utilize the neighborhood contextual information outside the image patch to guide the refined segmentation of the target image patch is a pressing technical problem to be solved.

[0029] In view of this, embodiments of this application provide an image segmentation method. The method includes: extracting multi-scale visual features from multiple image patches to obtain global visual features and multiple sub-visual features; using a graph neural network module to aggregate neighborhood features of node features in an initial graph structure to obtain aggregated features, where the initial graph structure includes node and edge relationships, node features characterize the pathological features of the image patch, and aggregated features characterize the contextual information of multiple neighboring nodes of the target node within the aggregated neighborhood; performing attention fusion on the global visual features and aggregated features based on an attention mechanism to obtain fused features; and upsampling and decoding the fused features, global visual features, and multiple sub-visual features to obtain an image segmentation result, where the image segmentation result characterizes the maturity level of the tertiary lymphoid structures within the tissue region of the target object.

[0030] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0031] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0032] Figure 1 A flowchart of an image segmentation method according to an embodiment of this application is shown.

[0033] like Figure 1 As shown, the method includes operations S110 to S140.

[0034] In operation S110, multi-scale visual feature extraction is performed on multiple image patches to obtain global visual features and multiple sub-visual features.

[0035] In operation S120, the graph neural network module is used to aggregate the neighborhood features of multiple nodes in the initial graph structure to obtain aggregated features.

[0036] In operation S130, attention fusion is performed on global visual features and aggregated features based on the attention mechanism to obtain fused features.

[0037] In operation S140, the fused features, global visual features, and multiple sub-visual features are upsampled and decoded to obtain the image segmentation result.

[0038] The target object is the object to be detected.

[0039] Pathological images are high-resolution whole-section images obtained by digitizing multiple consecutive pathological sections related to the region of interest of the target object through a digital pathology scanner after fixation, staining, and other processes.

[0040] The pathological image is filtered for background and the foreground region is cut into multiple non-overlapping image blocks of a fixed size, for example, the image block size is 256×256 pixels.

[0041] Divide the foreground region into different regions to be segmented. ,in, The set of regions to be segmented. This represents the nth region to be segmented.

[0042] For each region to be segmented, it is divided into non-overlapping image blocks by a convolutional neural network, resulting in a set of image blocks.

[0043] Figure 2A A schematic diagram of a segmentation model according to an embodiment of this application is shown.

[0044] like Figure 2A As shown, the segmentation model includes a hybrid feature encoder, a graph neural network module, a feature fusion module, a decoder, and a segmentation head.

[0045] The hybrid feature encoder employs a hybrid architecture. First, it utilizes a convolutional neural network (Residual Network Version 2, ResNetV2) as the backbone to extract intermediate layer feature maps at different resolutions. These feature maps are preserved for skip connections in the subsequent decoder. The output of this layer is further processed and flattened by convolutional layers, then superimposed with learnable positional codes before being input into the Transformer encoder. The Transformer encoder consists of multiple stacked blocks, each containing layer normalization (LayerNorm), multi-head attention, and a multilayer perceptron (MLP). Through layer-by-layer processing, it outputs sub-visual features of local texture and global visual features of global dependencies.

[0046] Features of image patches can be extracted using multiple cascaded convolutional layers in a Residual Network Version 2 (ResNetV2) while preserving skip connection features to obtain global visual features. and multiple sub-visual features .

[0047] Global visual features represent the deep features of the overall output of the convolutional neural network, preserving the global contextual dependencies and high-order pathological patterns of the image; sub-visual features represent the features output by each of the multi-layered cascaded convolutional layers, preserving multi-scale local visual patterns and hierarchical spatial details, which facilitates the recovery of image resolution during the decoding process.

[0048] An initial graph structure is constructed based on the positional relationships of multiple image patches in the pathological image. .

[0049] The initial graph structure includes nodes And edge relationship Nodes represent image blocks.

[0050] Edge connections are constructed based on the spatial connectivity and positional relationships of image patches in pathological images, for example, adjacent image patches in a predefined adjacency relationship.

[0051] Each image patch is encoded using a pre-trained Universal Pathology Model (UNI) to obtain the pathological features of the image patch. Node features characterize the pathological features of the image patch.

[0052] The graph neural network module can be constructed based on Graph Attention Network (GAT), Graph Convolutional Network (GCN), or Knowledge-Aware Attention-Based Dynamic Graph Network (WiKG). The network consists of N graph convolutional layers, taking an initial graph structure as input, containing node features and edge relationship indices. Each graph convolutional layer receives node features from the previous layer, aggregates information from its neighboring nodes, and processes this information through normalization and activation functions. After multiple propagations, the node features corresponding to the target graph tile are extracted and used as a global context embedding. This feature contains multi-hop neighborhood information surrounding the target graph tile.

[0053] A graph neural network module can be used to progressively aggregate multi-hop neighborhood features of target image patches, enabling information propagation on the constructed context graph. For each target node, multiple graph convolutional layers iteratively aggregate the node features of multiple neighboring nodes within the aggregated neighborhood to generate aggregated features. .

[0054] Aggregation features characterize the contextual information of a target node within its aggregated neighborhood of multiple neighboring nodes. Aggregation features contain rich contextual information about the neighborhood.

[0055] For example, if the aggregated neighborhood is a 4-neighborhood, then the neighboring nodes of target node 1 include the left neighbor, right neighbor, upper neighbor, and lower neighbor. Based on the left neighbor, right neighbor, upper neighbor, and lower neighbor, the graph neural network module aggregates the node features of the left neighbor, right neighbor, upper neighbor, and lower neighbor layer by layer to obtain the aggregated features of target node 1.

[0056] A self-attention mechanism can be used to dynamically fuse aggregated features with global visual features to obtain fused features. The fused features represent a context-enhanced joint representation after fusing rich neighborhood information.

[0057] A U-shaped decoder can be used to perform feature fusion while gradually restoring image resolution. After initial decoding, the fused features are concatenated with global visual features and then injected into the decoding path via mapping and projection, using a residual connection. The decoding path has multiple decoding blocks, which add sub-visual features while receiving fused feature information from the upper layer to restore image details. The segmentation information after layer-by-layer decoding is projected into pixel-level prediction results to obtain the image segmentation result.

[0058] Image segmentation results characterize the maturity level of the three-tiered lymphatic structures within the tissue region of the target object. For example, the image segmentation result for a pixel can be classified as level one, two, or three.

[0059] The embodiments of this application, by aggregating the neighborhood features of nodes in the initial graph structure based on the topological relationship between image patches using a graph neural module, and by fusing global visual features and aggregated features through an attention mechanism, and by introducing a graph neural network module to construct the contextual relationship between image patches, can capture long-distance dependencies within a larger field of view in the aggregated neighborhood. This solves the bottleneck technical problem of limited field of view in segmentation based on individual image patches, effectively utilizes macroscopic tissue structure information outside the image patches to assist in segmentation, effectively ensures the regional consistency of the segmentation results, and thus significantly improves the accuracy and consistency of pixel-level semantic segmentation of the maturity level of the three-level lymphoid structure. It has good model versatility and task scalability, and can assist pathologists in conducting more accurate prognostic analysis.

[0060] According to an embodiment of this application, the initial graph structure is determined based on the following operations: determining the adjacent image blocks corresponding to the image block according to the preset connectivity rules and the position of the image block in the pathological image; generating the initial graph structure based on the target node represented by the image block and the edge relationship between the adjacent nodes represented by the adjacent image blocks.

[0061] The feature vector of each image patch is extracted using a deep learning feature extractor and used as a graph node. Based on the spatial adjacency of the image patches in the pathological image and the preset connectivity rules, the edges between nodes are established using connectivity, an adjacency matrix is ​​constructed, and an initial graph structure is generated.

[0062] For example, a pathological image is divided into 4 image blocks. Based on the positional relationship of multiple image blocks in the pathological image, the position of image block 1 is determined to be (1,1), the position of image block 2 is (1,2), the position of image block 3 is (2,1), and the position of image block 4 is (2,2).

[0063] Figure 2B A schematic diagram of the initial diagram structure according to an embodiment of this application is shown.

[0064] like Figure 2BAs shown, the preset connectivity rule is 4-aggregate neighborhood, then for the image patch position is ( The target node 210 needs to be connected to the following location: (), (), (), The adjacent nodes of ).

[0065] By utilizing connectivity, edge relationships are established between the target node and multiple adjacent nodes to generate the initial graph structure.

[0066] According to an embodiment of this application, the graph neural network module includes K cascaded graph convolutional sub-modules; wherein, the graph neural network module is used to aggregate neighborhood features of multiple nodes in the initial graph structure to obtain aggregated features, including: using the weight matrix of the k-th layer graph convolutional sub-module to perform a linear transformation on the (k-1)-th sub-aggregated features of the target node to obtain the k-th transformed feature of the target node, 1 < k < K; fusing the k-th transformed feature of the target node according to the k-th transformed features of the neighboring nodes corresponding to the target node to obtain the k-th sub-aggregated feature of the target node; and fusing the K sub-aggregated features of the target node using a multilayer perceptron to obtain the aggregated feature of the target node.

[0067] The target node is the node that is being computed and updated in the graph neural network.

[0068] The weight matrix is ​​a learnable parameter in a graph convolutional layer. It linearly projects the features of nodes onto a new feature space to better extract and combine information. Each layer has its own weight matrix.

[0069] Graph neural networks typically consist of multiple (or deep) stacked graph convolutional modules. The number of layers, k, also represents the "hop count" of information propagation. The features of nodes in the k-th layer contain information about their neighboring nodes within their aggregate neighborhood.

[0070] Neighbor nodes are nodes in the graph structure that are directly connected to the target node. The representation of the target node is enriched by aggregating information from neighbor nodes.

[0071] The (k-1)th sub-aggregated feature is the feature representation obtained after the target node has completed the calculation at the (k-1)th layer. The target node contains information from its k-hop neighbor nodes after the information aggregation of the first k-1 layers.

[0072] In one embodiment, the k-th sub-aggregation feature is as shown in formula (1):

[0073] (1).

[0074] in, A matrix representing a relationship based on edge relationships. To introduce a self-loop adjacency matrix, for The degree matrix, Let I be the learnable weight matrix in the k-th layer graph convolutional submodule, and let I be the identity matrix. It is a non-linear activation function. This represents the (k-1)th sub-aggregation feature.

[0075] By stacking the sub-aggregation features output by the k-layer graph convolutional submodules, each target node can aggregate the context information within its k-hop aggregation neighborhood.

[0076] Figure 2C A schematic diagram of a graph neural network module according to an embodiment of this application is shown.

[0077] As shown in Figure C, the graph neural network module includes K-layer cascaded graph convolutional sub-modules. The K-layer graph convolutional sub-modules are used to aggregate the multi-hop neighborhood features of the target graph patch layer by layer. Each layer outputs a sub-aggregated feature, and then multiple sub-aggregated features are concatenated. Finally, the global aggregated feature is obtained by mapping and concatenating the features through a multilayer perceptron (MLP).

[0078] According to the embodiments of this application, by introducing a graph neural network module to construct the contextual relationship between image blocks, the technical problem of limited field of view based on segmentation of individual image blocks is solved, and the macroscopic organizational structure information outside the image blocks can be effectively used to assist segmentation.

[0079] According to embodiments of this application, attention fusion is performed on global visual features and aggregated features based on an attention mechanism to obtain fused features, including: concatenating global visual features and aggregated features to obtain concatenated features; using a self-attention mechanism to fuse query features, key features, and value features determined based on the concatenated features to obtain attention weights; and using the attention weights to weight the value features to obtain fused features.

[0080] Using splicing functions ( ) for global visual features and aggregation features By splicing the pieces together, we can obtain the splicing features. .

[0081] Multi-head self-attention (MSA) modules can be used for fusion to obtain fused features. Multi-head self-attention modules include... Layer self-attention module.

[0082] splicing features The first fusion feature is obtained by inputting it into the first layer self-attention module; the first fusion feature is input into the second layer self-attention module to obtain the second fusion feature; the (l-1)th fusion feature is input into the l-th layer self-attention module to obtain the l-th fusion feature; the l fusion features are fused again to obtain the global fusion feature.

[0083] In one embodiment, the l-th layer self-attention module is shown in equations (2), (3), (4), (5), (6), and (7):

[0084] (2);

[0085] (3);

[0086] (4);

[0087] (5);

[0088] (6);

[0089] (7).

[0090] in, For the (l-1)th fusion feature, To query features, N is the key feature, and N is the value feature. , , All of these are model parameter matrices. For attention weights, To scale the attention score of the dot product, For the l-th fusion feature, is the scaling factor, and softmax is the normalization function.

[0091] The scaling factor can prevent the dot product value from becoming too large and entering the softmax saturation region during the calculation of attention weights.

[0092] According to embodiments of this application, the spliced ​​features obtained based on global visual features and aggregated features can dynamically adjust the attention weights of local features according to neighborhood context information, thereby enhancing the ability to distinguish the boundaries and maturity levels of tertiary lymphoid structures.

[0093] According to embodiments of this application, upsampling decoding is performed on fused features, global visual features, and multiple sub-visual features to obtain image segmentation results, including: decoding the fused features to obtain initial decoded features; upsampling decoding is performed on the initial decoded features, global visual features, and multiple sub-visual features to obtain target decoded features; the target decoded features are processed using a multilayer perceptron to obtain probability results; and the probability results are normalized to obtain image segmentation results.

[0094] The decoder comprises multiple cascaded encoding layers. First, the sub-visual features and global visual features output by the Transformer are reconstructed back into a two-dimensional feature map and then bilinearly upsampled. Next, skip connections are implemented, where at each decoding stage, the upsampled features are concatenated with the sub-visual features retained in the hybrid encoder along the channel dimension to supplement lost spatial details. Then, context injection is employed. The aggregated graph context embedding undergoes dimensionality mapping and nonlinear transformation via a feedforward network, and is directly injected into the decoder's feature stream through residual addition, allowing macroscopic organizational information to directly correct the microscopic pixel feature representations.

[0095] In one embodiment, the initial decoding features are as shown in formula (8):

[0096] (8).

[0097] in, For initial decoding features, As a feature of fusion, This is the first decoding layer.

[0098] The target decoding features are obtained by upsampling and decoding the initial decoding features, global visual features, and multiple sub-visual features layer by layer.

[0099] The segmentation head is implemented by the SegmentationHead class and contains convolutional layers. Its function is to map the high-dimensional feature map output by the decoder to the number of target classification categories, and output pixel-level predicted logical values.

[0100] Through multilayer perceptron The images are mapped and flattened into logical values ​​that reflect the segmentation results, resulting in pixel-level image segmentation.

[0101] In one embodiment, the initial decoding features are as shown in formula (9):

[0102] (9).

[0103] in, Represents the image segmentation result. The target decoding feature is the output of the I-th decoding layer.

[0104] According to an embodiment of this application, upsampling decoding is performed on initial decoding features, global visual features, and multiple sub-visual features to obtain target decoding features, including: concatenating the (i-1)th decoding feature and sub-visual features to obtain the (i-1)th concatenated feature, where the first decoding feature is the initial decoding feature and the first concatenated feature is obtained by concatenating the initial decoding feature and global visual features; upsampling the (i-1)th concatenated feature to obtain the (i-1)th upsampled feature; decoding the (i-1)th upsampled feature to obtain the ith decoding feature; and determining the target decoding feature based on the ith decoding feature.

[0105] The upsampled features are the recovered detailed information.

[0106] The decoding and encoding layers are symmetrical in structure.

[0107] The first decoded feature and the global visual feature are fused by skipping steps to obtain the first concatenated feature. The first concatenated feature is then upsampled to obtain the first upsampled feature. The first upsampled feature is then input into the second decoding layer to obtain the second decoded feature. Here, the global visual feature is the feature output by the encoding layer corresponding to the first decoding layer.

[0108] Skip fusion is performed on the (i-1)th decoded feature and the sub-visual feature output by the corresponding encoding layer of the (i-1)th decoded layer to obtain the (i-1)th stitched feature. The (i-1)th stitched feature is then upsampled to obtain the (i-1)th upsampled feature. The (i-1)th upsampled feature is then input into the i-th decoded layer to obtain the i-th decoded feature.

[0109] Skip-fusion is performed on the (I-1)th decoded feature and the sub-visual feature output from the corresponding encoding layer of the (I-1)th decoded layer to obtain the (I-1)th concatenated feature. The (I-1)th concatenated feature is then upsampled to obtain the (I-1)th upsampled feature. This upsampled feature is then input into the I-th decoded layer to obtain the I-th decoded feature. Skip-fusion and upsampling are then performed on the I-th decoded feature and the sub-visual feature output from the I-th encoding layer to obtain the target decoded feature.

[0110] Figure 2D A schematic diagram of a decoder according to an embodiment of this application is shown.

[0111] like Figure 2DAs shown, the decoder includes three decoding layers. Decoding layer 1 decodes the fused features to obtain initial decoded features. Skip fusion and upsampling are performed on the initial decoded features and the global visual features output by encoding layer 3 to obtain the first upsampled features. The first upsampled features are input to decoding layer 2 to obtain the second decoded features. Skip fusion and upsampling are performed on the second decoded features and the sub-visual features output by encoding layer 2 to obtain the second upsampled features. The second upsampled features are input to decoding layer 3 to obtain the third decoded features. Skip fusion and upsampling are performed on the third decoded features and the sub-visual features output by encoding layer 1 to obtain the target decoded features. The target decoded features are processed by the segmentation head to obtain the image segmentation result.

[0112] All samples used to train the segmentation model are divided into training, validation, and test sets. During training, online data augmentation can be performed on the training set data, including random flipping and rotation, to enhance the model's robustness.

[0113] The samples are image patches and their corresponding initial graph structures, and the labels are the pixel-level segmentation masks corresponding to the image patches. Supervised training of the model uses an optimizer (Adam with Decoupled Weight Decay, AdamW) and a cross-entropy loss function to achieve more effective regularization, more stable convergence, and better generalization performance.

[0114] The specific training process includes: loading the initialized TLSGNN model, setting the total number of training epochs and batch size; reading a batch of data from the training set, including pathological images, initial graph structure, and node position indices; performing data augmentation preprocessing on the pathological images, such as random rotation and flipping; inputting the processed data into the model, first extracting image features through a Transformer, and simultaneously extracting graph context features through a GNN, then fusing and decoding in the decoder, finally outputting the image segmentation result with a predicted mask; comparing the image segmentation result output by the model with the true label mask, and calculating the loss function value; performing backpropagation based on the calculated loss, calculating the gradient, and updating the model parameters using an optimizer; evaluating the performance of the current model on the validation set; if the current model's performance on the validation set is better than the historical best, saving the current model parameters as the best model; determining whether the preset number of training epochs has been reached. If not, repeating the above steps; until the iteration stopping condition is met.

[0115] The model testing configuration is designed to verify the model's denoising and segmentation capabilities. First, the optimal model parameters obtained during training are loaded. Then, pathological images from the test set and their corresponding initial image structures are input into the segmentation model. The model outputs pixel-level prediction results, which are then processed using a softmax operation to obtain the final binarized or multi-class segmentation mask. The image segmentation results are compared with the ground truth annotations, and metrics such as the macro-average F1 score (mF1) and average intersection-over-union (mIoU) are calculated to evaluate model performance. Visualized image segmentation results can also be generated to assist doctors in the identification and quantification of TLS (Throughput Lesions).

[0116] Figure 3 A schematic diagram showing a visualized image segmentation result according to an embodiment of this application is provided.

[0117] like Figure 3 As shown, the horizontal axis represents different related segmentation methods or models, including methods 1 to 7 and the image segmentation method of this application. TIS1 represents level one, TIS2 represents level two, and TIS3 represents level three. The vertical axis represents different image patches or test samples, each with a corresponding label. TIS2 is marked in red, TIS1 in blue, and TIS3 in green. Each TLS contains two rows of images: the top row displays the global view, and the bottom row displays the detailed view of the highlighted area segmentation results. Each cell shows the segmentation mask of the corresponding method on a specific image patch. By comparing the segmentation masks of each method with the leftmost label, the performance of different methods, such as segmentation accuracy and edge detail preservation, can be intuitively evaluated.

[0118] Experiments show that, compared with the benchmark model TransUNet, the method in this application improves the mF1 score by 22.08% and the mIoU score by 26.57%. The generated image segmentation results are superior to the benchmark model in terms of boundary integrity and class consistency.

[0119] Figure 4 A schematic block diagram of an image segmentation apparatus according to an embodiment of this application is shown.

[0120] like Figure 4 As shown, the image segmentation device 400 of this embodiment includes a feature extraction module 410, a feature aggregation module 420, a feature fusion module 430, and a segmentation module 440.

[0121] The feature extraction module 410 is used to extract multi-scale visual features from multiple image blocks to obtain global visual features and multiple sub-visual features. The multiple image blocks are obtained by segmenting the pathological image of the target object.

[0122] The feature aggregation module 420 is used to aggregate neighborhood features of multiple nodes in the initial graph structure using the graph neural network module to obtain aggregated features. The initial graph structure includes nodes and edge relationships. The node features represent the pathological features of the image blocks. The edge relationships are determined based on the positional relationships of multiple image blocks in the pathological image. The aggregated features represent the contextual information of multiple neighboring nodes of the target node in the aggregated neighborhood.

[0123] The feature fusion module 430 is used to perform attention fusion on global visual features and aggregated features based on the attention mechanism to obtain fused features.

[0124] The segmentation module 440 is used to upsample and decode the fused features, global visual features and multiple sub-visual features to obtain the image segmentation result. The image segmentation result represents the maturity level of the three-level lymphatic structure in the target object's tissue region.

[0125] According to an embodiment of this application, the feature aggregation module 420 includes a first aggregation submodule, a second aggregation submodule, and a third aggregation submodule.

[0126] The first aggregation submodule is used to perform a linear transformation on the (k-1)th sub-aggregation feature of the target node using the weight matrix of the k-th layer graph convolution submodule, to obtain the k-th transformed feature of the target node, where 1 < k < K.

[0127] The second aggregation submodule is used to fuse the k-th transformation feature of the target node based on the k-th transformation feature of the neighboring nodes corresponding to the target node, so as to obtain the k-th sub-aggregated feature of the target node.

[0128] The third aggregation submodule is used to fuse the K sub-aggregation features of the target node using a multilayer perceptron to obtain the aggregated features of the target node.

[0129] According to an embodiment of this application, the feature fusion module 430 includes a first fusion submodule, a second fusion submodule, and a third submodule.

[0130] The first fusion submodule is used to stitch together global visual features and aggregated features to obtain stitched features.

[0131] The second fusion submodule is used to fuse the query features, key features, and value features determined based on the concatenation features using a self-attention mechanism to obtain attention weights.

[0132] The third fusion submodule is used to weight the value features using attention weights to obtain fused features.

[0133] According to an embodiment of this application, the segmentation module 440 includes a first decoding submodule, a second decoding submodule, a third decoding submodule, and a fourth decoding submodule.

[0134] The first decoding submodule is used to decode the fused features to obtain the initial decoded features.

[0135] The second decoding submodule is used to upsample and decode the initial decoding features, global visual features, and multiple sub-visual features to obtain the target decoding features.

[0136] The third decoding submodule is used to process the target decoding features using a multilayer perceptron to obtain probability results.

[0137] The fourth decoding submodule is used to normalize the probability results to obtain the image segmentation results.

[0138] According to an embodiment of this application, the second decoding submodule includes a first decoding unit, a second decoding unit, a third decoding unit, and a fourth decoding unit.

[0139] The first decoding unit is used to concatenate the (i-1)th decoded feature and the sub-visual feature to obtain the (i-1)th concatenated feature. The first decoded feature is the initial decoded feature, and the first concatenated feature is obtained by concatenating the initial decoded feature and the global visual feature.

[0140] The second decoding unit is used to upsample the (i-1)th concatenated feature to obtain the (i-1)th upsampled feature.

[0141] The third decoding unit is used to decode the (i-1)th upsampled feature to obtain the i-th decoded feature.

[0142] The fourth decoding unit is used to determine the target decoding feature based on the I-th decoding feature.

[0143] Figure 5 A block diagram of an electronic device suitable for implementing an image segmentation method according to an embodiment of this application is shown.

[0144] Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0145] like Figure 5As shown, a computer electronic device 500 according to an embodiment of this application includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a ROM 502 (read-only memory) or a program loaded from a storage portion 508 into a RAM 503 (random access memory). The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0146] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 502 and / or RAM 503. It should be noted that programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0147] In embodiments of this application, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0148] The embodiments of this application, and the method flow according to the embodiments of this application, can be implemented as a computer software program. For example, an embodiment of this application includes a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of the embodiments of this application. In the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0149] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the image segmentation method according to the embodiments of this application.

[0150] In embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0151] For example, in embodiments of this application, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0152] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the image segmentation method provided in the embodiments of this application.

[0153] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this application embodiment. In the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0154] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0155] In embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments of this application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of this application.

[0157] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. An image segmentation method, characterized in that, The method includes: Multi-scale visual feature extraction is performed on multiple image patches to obtain global visual features and multiple sub-visual features. The multiple image patches are obtained based on the segmentation of the pathological image of the target object. The graph neural network module is used to aggregate neighborhood features of the node features of multiple nodes in the initial graph structure to obtain aggregate features. The initial graph structure includes the nodes and edge relationships. The node features represent the pathological features of the image blocks. The edge relationships are determined based on the positional relationships of multiple image blocks in the pathological image. The aggregate features represent the context information of multiple neighboring nodes of the target node in the aggregate neighborhood. The global visual features and the aggregated features are fused based on an attention mechanism to obtain fused features; The fused features, the global visual features, and the multiple sub-visual features are upsampled and decoded to obtain an image segmentation result, which characterizes the maturity level of the three-level lymphatic structure within the tissue region of the target object.

2. The method according to claim 1, characterized in that, The initial graph structure is determined based on the following operations: Based on the preset connectivity rules and the position of the image block in the pathological image, determine the adjacent image blocks corresponding to the image block; The initial graph structure is generated based on the target node represented by the image patch and the edge relationships between adjacent nodes represented by the adjacent image patches.

3. The method according to claim 2, characterized in that, The graph neural network module includes K layers of cascaded graph convolutional sub-modules; The step of using a graph neural network module to aggregate neighborhood features of multiple nodes in the initial graph structure to obtain aggregated features includes: The (k-1)th sub-aggregation feature of the target node is linearly transformed using the weight matrix of the k-th layer graph convolution sub-module to obtain the k-th transformed feature of the target node, where 1 < k < K; The k-th transformation feature of the target node is fused with the k-th transformation feature of the neighboring nodes corresponding to the target node to obtain the k-th sub-aggregation feature of the target node; The K sub-aggregated features of the target node are fused using a multilayer perceptron to obtain the aggregated features of the target node.

4. The method according to claim 1, characterized in that, The attention-based fusion of the global visual features and the aggregated features to obtain fused features includes: The global visual features and the aggregated features are concatenated to obtain the concatenated features; The query features, key features, and value features determined based on the concatenated features are fused using a self-attention mechanism to obtain attention weights; The fused features are obtained by weighting the value features using the attention weights.

5. The method according to claim 1, characterized in that, The upsampling and decoding of the fused features, the global visual features, and the multiple sub-visual features to obtain the image segmentation result includes: The fused features are decoded to obtain the initial decoded features; Upsample and decode the initial decoded features, the global visual features, and the multiple sub-visual features to obtain the target decoded features; The target decoding features are processed using a multilayer perceptron to obtain probability results; The probability results are normalized to obtain the image segmentation results.

6. The method according to claim 5, characterized in that, The upsampling and decoding of the initial decoded features, the global visual features, and multiple sub-visual features to obtain the target decoded features includes: The (i-1)th decoded feature and the sub-visual feature are concatenated to obtain the (i-1)th concatenated feature. The first decoded feature is the initial decoded feature, and the first concatenated feature is obtained by concatenating the initial decoded feature and the global visual feature. The (i-1)th spliced ​​feature is upsampled to obtain the (i-1)th upsampled feature; The (i-1)th upsampled feature is decoded to obtain the i-th decoded feature; The target decoding feature is determined based on the I-th decoding feature.

7. An image segmentation apparatus, characterized in that, The device includes: The feature extraction module is used to extract multi-scale visual features from multiple image blocks to obtain global visual features and multiple sub-visual features. The multiple image blocks are obtained by segmenting the pathological image of the target object. The feature aggregation module is used to aggregate neighborhood features of multiple nodes in the initial graph structure using the graph neural network module to obtain aggregated features. The initial graph structure includes the nodes and edge relationships. The node features characterize the pathological features of the image blocks. The edge relationships are determined based on the positional relationships of multiple image blocks in the pathological image. The aggregated features characterize the context information of multiple neighboring nodes of the target node in the aggregated neighborhood. The feature fusion module is used to perform attention fusion on the global visual features and the aggregated features based on an attention mechanism to obtain fused features; The segmentation module is used to upsample and decode the fused features, the global visual features, and multiple sub-visual features to obtain an image segmentation result, which represents the maturity level of the three-level lymphatic structure within the target object's tissue region.

8. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images