Self-supervised medical image segmentation method and system based on structural relationship modeling

Abstract

The application provides a self-supervised medical image segmentation method and system based on structural relationship modeling, the application models the complex relationship among nodes, subgraphs and the whole graph, so as to comprehensively capture the structural information of the graph; by constructing and optimizing the structural relationship model of the graph, the performance of graph representation learning is improved, especially for graph data with complex topological structure and non-uniform distribution, the structural relationship in the medical image is modeled, and the segmentation effect of the medical image is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of machine learning, and particularly relates to a self-supervised medical image segmentation method and system based on structural relationship modeling. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the rapid development of medical imaging technology, a large amount of medical image data is used for clinical diagnosis and treatment. Medical image segmentation is a key step to automatically extract the region of interest (such as tumor, organ) in these image data. However, traditional medical image segmentation methods usually rely on a large amount of labeled data, and the acquisition of these data is costly and time-consuming.

[0004] In order to better cope with these challenges, recent research has begun to explore the improvement of representation learning through multi-level graph structure modeling. These methods not only consider the local interaction between nodes, but also introduce the fusion of global topological information, thereby improving the shortcomings of traditional methods to some extent. However, most of the existing methods still face the problems of high computational complexity and low information transmission efficiency when dealing with large-scale graph data. Therefore, there is an urgent need for a new method to effectively balance the local and global information of the graph to achieve more robust graph representation learning, thereby effectively improving the performance of medical image segmentation. SUMMARY

[0005] In order to solve the above problems, the present application proposes a self-supervised medical image segmentation method and system based on structural relationship modeling. The present application models the complex relationship between nodes, subgraphs and the whole graph, thereby comprehensively capturing the structural information of the graph. By constructing and optimizing the structural relationship model of the graph, the performance of graph representation learning is improved, especially for graph data with complex topological structure and non-uniform distribution. By modeling the organizational structure relationship in medical images, the segmentation effect of medical images is improved.

[0006] According to some embodiments, the present application adopts the following technical solution:

[0007] A self-supervised medical image segmentation method based on structural relationship modeling, comprising the following steps:

[0008] Obtain a medical image, pre-process the medical image, divide the pre-processed medical image, take the divided blocks as nodes, and construct a graph structure;

[0009] Convert the graph structure, calculate the similarity between nodes and fold similar nodes to realize node merging;

[0010] partitioning the graph structure after node merging to form a plurality of sub-graphs;

[0011] performing feature sampling, starting from selected starting points, sampling a fixed number of nodes based on similarity measure, extracting local features of the sub-graphs from the nodes;

[0012] performing global feature extraction on the entire graph structure;

[0013] fusing the extracted global features and local features, normalizing the fused features, and decoding based on the normalized fused features to obtain reconstructed medical images and sub-graphs.

[0014] As an optional implementation, the preprocessing includes denoising, graying, or / and normalization processing.

[0015] As an optional implementation, the process of constructing the graph structure with the divided blocks as nodes includes dividing the medical image into blocks, each block being regarded as a node in the graph structure, edges between nodes being calculated according to similarity or adjacency relationship between blocks, similarity between adjacent blocks being used as edge weight, and the graph structure being constructed to obtain the graph G.

[0016] As an optional implementation, the process of calculating similarity between nodes and folding similar nodes includes:

[0017]

[0018] T is a generalized aggregation matrix, used to represent the aggregation effect of multi-hop neighborhood; A is an adjacency matrix, representing the connection relationship between nodes; K is a degree matrix, diagonal elements K ii represent the degree of node i; W i is a weight coefficient, used to adjust the influence of different hop number neighborhoods; t is the running time, and satisfies the condition: n represents the maximum hop number of aggregation;

[0019] By calculating T, the similarity between node u and node v is captured, and the calculation formula is:

[0020]

[0021] A∈R N×N is the adjacency matrix of the graph G=(V,E), where N=|V| represents the total number of nodes, X∈R N×F is the node feature matrix of the graph G;

[0022] μ is a smoothing parameter, used to adjust the influence of local structure features;

[0023] After L times of coarsening iteration, the smallest coarsened graph G LL is an integer.

[0024] As an alternative implementation, the process of partitioning the graph structure after node merging to form a plurality of sub-graphs includes partitioning the coarsened graph into a number of groups, each group representing a sub-graph in the graph, representing the intra-group edge connections and inter-group edge connections using an adjacency matrix, and projecting each group back into the original medical image using a greedy algorithm and boundary adjustment.

[0025] As an alternative implementation, a partition is randomly selected as a starting point and a fixed number of nodes are sampled from it based on a similarity measure.

[0026] As an alternative implementation, the process of extracting local features of the sub-graph from the nodes includes, for each node in the sub-graph, extracting image block features of its neighborhood nodes, obtaining an initial feature representation of each node through a local convolution operation; and fusing the information of the center node and its adjacent nodes through a neighborhood aggregation operation between nodes to generate a feature vector of each sub-graph.

[0027] As an alternative implementation, the process of extracting global features of the entire graph structure includes applying a self-attention mechanism on the global graph to capture global dependency relationships between all nodes in the graph and extract feature interactions between long-distance nodes; and globally aggregating the features of all nodes to generate a global feature vector.

[0028] As an alternative implementation, the process of fusing the extracted global features and local features and normalizing the fused features includes using a vector concatenation operation to concatenate the local and global features together; and in the feature normalization stage, introducing smoothness, connectivity and sparsity constraints.

[0029] A self-supervised medical image segmentation system based on structure relationship modeling, comprising:

[0030] A graph structure construction module configured to obtain a medical image, pre-process the medical image, divide the pre-processed medical image, and construct a graph structure taking the divided blocks as nodes;

[0031] A graph structure processing module configured to convert the graph structure, calculate the similarity between nodes and fold similar nodes to achieve node merging;

[0032] A partition module configured to partition the graph structure after node merging to form a plurality of sub-graphs;

[0033] A local feature extraction module configured to perform feature sampling, start from a selected starting point, sample a fixed number of nodes based on a similarity measure, and extract local features of the sub-graph from the nodes;

[0034] The global feature extraction module is configured to perform global feature extraction on the entire graph structure.

[0035] The feature fusion and normalization processing module is configured to fuse the extracted global features and local features, and normalize the fused features, and decode based on the normalized fused features to obtain a reconstructed medical image and subgraph.

[0036] Compared with the prior art, the present application has the following advantages:

[0037] The present application can better capture the overall structure information of the graph by modeling the complex relationship between the nodes, subgraphs and global graph. Compared with the traditional method, the present application not only considers the local interaction between nodes, but also introduces the fusion of global topological information, thereby realizing the comprehensive understanding of the graph structure.

[0038] The present application is based on the local node subset sampling mechanism based on region division, which can alleviate the problems of over-aggregation and topological decay caused by topological imbalance through small batch training, and can realize efficient information fusion and transmission in small batch training by using node merging, region division, boundary optimization and feature sampling processes, thereby significantly improving the training efficiency and generalization ability of the model.

[0039] The present application can fully capture and model local and global information by fusing local and global branches, ensure the integrity of the structure information, and enable the decoder to effectively combine local and global features. Specifically, the present application adopts a hierarchical feature extraction method, which extracts local node subset features at the local level and fuses them with global node features, thereby realizing comprehensive capture and understanding of the graph structure, and effectively solving the information loss problem caused by only focusing on local or global in the traditional method.

[0040] The present application ensures the robustness of the model during training by constraining the smoothness, connectivity and sparsity between nodes. Specifically, the present application constrains the feature similarity of adjacent nodes to avoid overfitting or underfitting during training, and further improves the accuracy of graph representation by reasonably constraining the connectivity and sparsity of nodes.

[0041] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are used for explanation. BRIEF DESCRIPTION OF DRAWINGS

[0042] The drawings accompanying the specification of the present application form part of the present application and are used to provide a further understanding of the present application, the illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation on the present application.

[0043] Figure 1 This is a framework diagram of a self-supervised graph representation learning method as an example. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0046] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0047] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0048] Example 1

[0049] First, for the obtained graph data, a partition-based sub-node set sampling mechanism is adopted to divide the original graph into multiple subgraphs, and independent training is performed on these subgraphs.

[0050] Secondly, by fusing local and global information, a more robust graph representation can be constructed.

[0051] Furthermore, to ensure the stability of the graph representation, this invention introduces a feature normalization strategy, including smoothness constraints, connectivity constraints, and sparsity constraints, to address the complexity of the graph structure.

[0052] Through these steps, the present invention can significantly improve the performance of node classification and graph classification tasks on multiple public datasets.

[0053] The following is a detailed description:

[0054] like Figure 1 As shown, the input medical image is first preprocessed, including steps such as denoising, grayscale conversion, or normalization, to ensure that the image quality is suitable for subsequent processing.

[0055] The pre-processed medical image is divided into several small blocks (e.g. 8x8 or NxN size), each of which is regarded as a node in the graph structure, and the edges between nodes are calculated according to the similarity or adjacency relationship between small blocks. The similarity between adjacent small blocks is used as the edge weight, so as to construct the graph structure, and then obtain the graph G, and G0 is the untreated graph G.

[0056] S2: Node merging

[0057] Convert the graph G into a series of gradually simplified coarse graphs G L , calculate the similarity between nodes by formula and fold similar nodes:

[0058]

[0059] T is a generalized aggregation matrix, which is used to represent the aggregation effect of multi-hop neighborhood. A is the adjacency matrix, which represents the connection relationship between nodes. K is the degree matrix, and the diagonal element K ii represents the degree of node i. W i is the weight coefficient, which adjusts the influence of different hop number neighborhoods. t is the running time, and satisfies the condition: n represents the maximum hop number of aggregation.

[0060] By calculating T, the similarity between node u and node v can be captured, and the calculation formula is:

[0061]

[0062] μ is the smoothing parameter, which adjusts the influence of local structure features.

[0063] After L times of coarse iteration, the smallest coarse graph G L is obtained.

[0064] S3: Region division and boundary optimization

[0065] Divide the coarse graph G L into several groups, and each group represents a subgraph in the graph.

[0066] The adjacency matrix after partitioning can be represented as:

[0067]

[0068] represents the edge connection within the partition . represents the edge connection between partitions and . Each partition is projected back to the original graph G0 through a greedy algorithm and boundary adjustment.

[0069] S4: Feature Sampling

[0070] In the sampling stage, the algorithm randomly selects a partition as the starting point and samples a fixed number of nodes from it based on similarity measures. The final subgraph contains sufficient information.

[0071] S5: Feature Extraction

[0072] After feature sampling, the sampled nodes are further processed through feature extraction.

[0073] Local Feature Extraction: For each subgraph obtained by sampling, local feature extraction is performed. This process uses a convolutional neural network (CNN) or graph convolution network (GCN) to extract features from the nodes in the subgraph. First, for each node in the subgraph, the image block features of its neighboring nodes are extracted. Through local convolution operation, the initial feature representation of each node is obtained. Second, the neighborhood aggregation operation between nodes is used to fuse the information of the center node and its neighboring nodes, further enriching the local feature representation of the node. Finally, the feature vector of each subgraph is generated, which is used for subsequent local and global feature fusion.

[0074] Global Feature Extraction: For the coarse graph structure G L , a global feature extraction method is used for processing. This process captures long-range dependencies in the global graph through self-attention mechanism (Transformer) or graph neural network. First, the self-attention mechanism is applied to the global graph to capture the global dependencies between all nodes in the graph, extracting the feature interaction between long-range nodes. Then, the features of all nodes are globally aggregated to generate a global feature vector, ensuring that the overall structure information of the graph is preserved for subsequent local and global feature fusion.

[0075] S6: Feature Fusion

[0076] In the local and global information fusion stage, the features of the subgraph hierarchy and the features of the global graph structure are fused to enhance the robustness of the graph representation.

[0077] The calculation formula of the fused features is:

[0078]

[0079] F is the fused feature vector. w is a learnable parameter used to weight the local and global features. Cat is the vector concatenation operation that concatenates the local and global features. represents the input features of the subgraph. represents the input features of the global graph.

[0080] The local and global input features are combined by the above operations to generate fused output features.

[0081] S7: Feature normalization

[0082] In the feature normalization stage, smoothness, connectivity and sparsity constraints are introduced to improve the stability and performance of the model. The calculation formula of the normalization loss is:

[0083] L R = aL S + bL CS

[0084] L R is the total normalization loss. L S is the smoothness loss, which ensures the smoothness of the features of adjacent nodes. L CS is the connectivity and sparsity loss, which ensures the reasonable connection and sparsity between nodes. a and b are weight parameters of the loss terms.

[0085] The calculation of L S is as follows:

[0086]

[0087] N is the number of nodes. denotes the global feature of node j in the i-th iteration. denotes the corresponding global feature in the previous iteration. tr(·) denotes the trace of the matrix. L is the Laplacian matrix associated with the adjacency matrix of the graph.

[0088] The calculation of L CS is as follows:

[0089]

[0090] 1 denotes the all-1 vector. A is the adjacency matrix. λ is a non-negative parameter that controls sparsity. ||·|| F is the Frobenius norm of the adjacency matrix.

[0091] Finally, the reconstructed graph and subgraph are output by decoding.

[0092] The method provided in this embodiment has shown excellent performance in multiple node classification and graph classification tasks, demonstrating its wide applicability and strong generalization ability on complex data structures. This is particularly important for the field of medical image segmentation.

[0093] The method provided in this embodiment has achieved high accuracy on the OCT dataset disclosed by Drive and kermany, as shown in Tables 1 and 2. In particular, in the case of limited data labels, it still shows excellent classification effect.

[0094] Table 1 DRIVE dataset experiment result comparison

[0095]

[0096]

[0097] Table 2 OCT dataset experiment result comparison

[0098]

[0099] This shows that the method provided by the embodiment can effectively capture local and global information in the graph structure, thereby improving the classification performance. For the pixel or region classification task in the medical image segmentation of eye diseases, such ability is particularly crucial because the labeling of medical data is often expensive and limited.

[0100] Embodiment Two

[0101] A self-supervised medical image segmentation system based on structure relationship modeling, comprising:

[0102] A graph structure construction module configured to obtain a medical image, pre-process the medical image, divide the pre-processed medical image, take the divided blocks as nodes, and construct a graph structure;

[0103] A graph structure processing module configured to convert the graph structure, calculate the similarity between nodes and fold similar nodes, and realize node merging;

[0104] A partition module configured to partition the graph structure after node merging to form a plurality of subgraphs;

[0105] A local feature extraction module configured to perform feature sampling, start from a selected starting point, sample a fixed number of nodes based on similarity measurement, and extract local features of the subgraph from the nodes;

[0106] A global feature extraction module configured to perform global feature extraction on the entire graph structure;

[0107] A feature fusion and normalization processing module configured to fuse the extracted global features and local features, normalize the fused features, decode based on the normalized fused features, and obtain a reconstructed medical image and subgraph.

[0108] It can be understood that the above various units / modules can be respectively or all combined into one or several other units / modules to constitute, or some of the units can be further split into a plurality of units with smaller functions to constitute, which can realize the same operation without affecting the realization of the technical effects of the embodiments of the present application.

[0109] The above modules of the system are divided based on logical functions. In actual applications, the functions of one module can also be implemented by multiple modules, or the functions of multiple modules can be implemented by one module. For example, the feature fusion and normalization processing module in this embodiment can include the following modules in other embodiments:

[0110] The feature fusion is configured to fuse the extracted global features and local features.

[0111] The normalization processing module is configured to normalize the fused features, and decode based on the normalized fused features to obtain a reconstructed medical image and a sub-image.

[0112] Similarly, in other embodiments of the present application, the system can also include other units / modules. In actual applications, these functions can also be assisted by other units, and can be implemented in cooperation with multiple units.

[0113] Those skilled in the art should understand that embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0114] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 An apparatus for performing each flow or multiple flows and / or blocks Figure 1 The functions specified in one or more flows or blocks.

[0115] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 An apparatus for performing each flow or multiple flows and / or blocks Figure 1 The functions specified in one or more flows or blocks.

[0116] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, so that the instructions executed on the computer or other programmable data processing devices provide processes for implementing the functions specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0117] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Various modifications and changes can be made to the present application by those skilled in the art without departing from the spirit and principles of the present application. Any modifications, equivalent replacements, improvements, etc. made by those skilled in the art without creative labor within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A self-supervised medical image segmentation method based on structural relationship modeling, characterized in that, Includes the following steps: Acquire medical images, preprocess the medical images, divide the preprocessed medical images into blocks, and construct a graph structure using the blocks as nodes; The graph structure is transformed, the similarity between nodes is calculated, and similar nodes are collapsed to achieve node merging. The process of calculating the similarity between nodes and collapsing similar nodes includes: It is a generalized aggregation matrix used to represent the aggregation effect of multi-hop neighborhoods; It is an adjacency matrix, representing the connection relationships between nodes; It is a degree matrix; These are weighting coefficients used to adjust the influence of neighborhoods with different hop counts; , The runtime is [time], and the following conditions are met: ; Indicates the maximum number of hops in the aggregation; Through calculation Capture node and nodes The similarity between them is calculated using the following formula: It is a picture The adjacency matrix, where Indicates the total number of nodes. This is the node feature matrix of graph G; To smooth out parameters and adjust for the influence of local structural features; After several coarsening iterations, the minimum coarsened map is obtained. , L It is an integer; The graph structure after node merging is partitioned to form multiple subgraphs; Feature sampling is performed starting from a selected starting point. A fixed number of nodes are sampled based on a similarity metric. Local features of the subgraph are extracted from these nodes. The process includes extracting image patch features of the neighboring nodes for each node in the subgraph, obtaining the initial feature representation of each node through local convolution operations, and using a neighborhood aggregation operation between nodes to fuse the information of the central node and its neighboring nodes to generate a feature vector for each subgraph. Global feature extraction of the entire graph structure includes applying a self-attention mechanism to the global graph to capture the global dependencies between all nodes in the graph and extracting the feature interactions between long-distance nodes; and globally aggregating the features of all nodes to generate a global feature vector. The extracted global and local features are fused and the fused features are normalized. The process includes using vector concatenation to concatenate the local and global features together. In the feature normalization stage, smoothness, connectivity and sparsity constraints are introduced. Decoding is performed based on the fusion features after normalization to obtain the reconstructed medical image and sub-image.

2. The self-supervised medical image segmentation method based on structural relationship modeling as described in claim 1, characterized in that, The preprocessing includes denoising, grayscale conversion, and / or normalization.

3. The self-supervised medical image segmentation method based on structural relationship modeling as described in claim 1, characterized in that, The process of constructing a graph structure using the partitioned blocks as nodes involves dividing the medical image into several blocks, with each block considered as a node in the graph structure. The edges between nodes are calculated based on the similarity or adjacency relationship between the blocks, and the similarity between adjacent blocks is used as the edge weight. This process constructs the graph structure, resulting in graph G.

4. The self-supervised medical image segmentation method based on structural relationship modeling as described in claim 1, characterized in that, The process of partitioning the graph structure after node merging to form multiple subgraphs includes partitioning the coarsened graph into several groups, each group representing a subgraph in the graph. An adjacency matrix is ​​used to represent the edge connections within the partitions and the edge connections between the partitions. Each partition is projected back into the original medical image through a greedy algorithm and boundary adjustment.

5. The self-supervised medical image segmentation method based on structural relationship modeling as described in claim 1, characterized in that, Randomly select a partition as the starting point, and sample a fixed number of nodes from it based on a similarity metric.

6. A self-supervised medical image segmentation system based on structural relationship modeling, characterized in that, include: The graph structure construction module is configured to acquire medical images, preprocess the medical images, divide the preprocessed medical images into blocks, and construct a graph structure using the blocks as nodes. The graph structure processing module is configured to transform the graph structure, calculate the similarity between nodes and collapse similar nodes to achieve node merging. The process of calculating the similarity between nodes and collapsing similar nodes includes: It is a generalized aggregation matrix used to represent the aggregation effect of multi-hop neighborhoods; It is an adjacency matrix, representing the connection relationships between nodes; It is a degree matrix; These are weighting coefficients used to adjust the influence of neighborhoods with different hop counts; , The runtime is [time], and the following conditions are met: ; Indicates the maximum number of hops in the aggregation; Through calculation Capture node and nodes The similarity between them is calculated using the following formula: It is a picture The adjacency matrix, where Indicates the total number of nodes. This is the node feature matrix of graph G; To smooth out parameters and adjust for the influence of local structural features; After several coarsening iterations, the minimum coarsened map is obtained. , L It is an integer; The partitioning module is configured to partition the graph structure after node merging, forming multiple subgraphs. The local feature extraction module is configured to perform feature sampling, starting from a selected starting point, sampling a fixed number of nodes based on a similarity metric, and extracting local features of the subgraph from the nodes. The process includes extracting image patch features of the neighboring nodes for each node in the subgraph, obtaining the initial feature representation of each node through a local convolution operation, and using a neighborhood aggregation operation between nodes to fuse the information of the central node and its neighboring nodes to generate a feature vector for each subgraph. The global feature extraction module is configured to perform global feature extraction on the entire graph structure. The process includes applying a self-attention mechanism on the global graph to capture the global dependencies between all nodes in the graph and extracting the feature interactions between long-distance nodes; and globally aggregating the features of all nodes to generate a global feature vector. The feature fusion and normalization module is configured to fuse extracted global and local features and normalize the fused features. The process includes using vector concatenation to concatenate local and global features together. In the feature normalization stage, smoothness, connectivity and sparsity constraints are introduced. Decoding is performed based on the fusion features after normalization to obtain the reconstructed medical image and sub-image.

Images