Self-supervised medical image segmentation method and system based on structural relation modeling

Abstract

The invention provides a self-supervised medical image segmentation method and system based on structural relation modeling, and the method comprises the steps: carrying out the modeling of a complex relation among nodes, sub-graphs and a whole graph, thereby comprehensively capturing the structural information of the graph; by constructing and optimizing the structural relation model of the graph, the performance of graph representation learning is improved, the method is particularly suitable for graph data with a complex topological structure and non-uniform distribution, and the segmentation effect of the medical image is improved by modeling the organization structure relation in the medical image.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning, and in particular relates to a self-supervised medical image segmentation method and system based on structural relationship modeling. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of medical imaging technology, a large amount of medical image data is used in clinical diagnosis and treatment. Medical image segmentation is a key step to automatically extract regions of interest (such as tumors and organs) from these image data. However, traditional medical image segmentation methods usually rely on a large amount of labeled data, which is expensive and time-consuming to obtain.

[0004] In order to better cope with these challenges, recent studies have begun to explore improving the effect of representation learning through multi-level graph structure modeling. These methods not only consider the local interactions between nodes, but also introduce the fusion of global topological information, thereby improving the shortcomings of traditional methods to a certain extent. However, most existing methods still face problems such as high computational complexity and low information transmission efficiency when processing large-scale graph data. Therefore, there is an urgent need for a new method to effectively balance the local and global information of the graph to achieve more robust graph representation learning, thereby effectively improving the performance of medical image segmentation. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a self-supervised medical image segmentation method and system based on structural relationship modeling. The present invention comprehensively captures the structural information of the graph by modeling the complex relationships between nodes, subgraphs and the entire graph; improves the performance of graph representation learning by constructing and optimizing the structural relationship model of the graph, which is particularly suitable for graph data with complex topological structures and non-uniform distribution, and improves the segmentation effect of medical images by modeling the organizational structure relationships in medical images.

[0006] According to some embodiments, the present invention adopts the following technical solutions:

[0007] A self-supervised medical image segmentation method based on structural relationship modeling comprises the following steps:

[0008] Acquire a medical image, preprocess the medical image, divide the preprocessed medical image, and construct a graph structure using the divided blocks as nodes;

[0009] Transform the graph structure, calculate the similarity between nodes and collapse similar nodes to achieve node merging;

[0010] Partition the graph structure after merging nodes to form multiple subgraphs;

[0011] Perform feature sampling, starting from the selected starting point, sampling a fixed number of nodes based on a similarity metric, and extracting local features of the subgraph from the nodes;

[0012] Perform global feature extraction on the entire graph structure;

[0013] The extracted global features and local features are fused and normalized, and decoding is performed based on the normalized fused features to obtain the reconstructed medical image and sub-image.

[0014] As an optional implementation, the preprocessing includes denoising, grayscale and / or normalization processing.

[0015] As an optional implementation, the process of constructing a graph structure with the divided blocks as nodes includes dividing the medical image into several blocks, each block is regarded as a node in the graph structure, the edges between the nodes are calculated based on the similarity or adjacency relationship between the blocks, the similarity between adjacent blocks is used as the edge weight, and the graph structure is constructed to obtain the graph G.

[0016] As an optional implementation, the process of calculating similarities between nodes and collapsing similar nodes includes:

[0017]

[0018] T is the generalized aggregation matrix, which is used to represent the aggregation effect of multi-hop neighborhoods; A is the adjacency matrix, which represents the connection relationship between nodes; K is the degree matrix, and the diagonal element K ii represents the degree of node i; W i is the weight coefficient, which is used to adjust the influence of neighborhoods with different hop counts; t is the running time and satisfies the following conditions: n represents the maximum number of hops for aggregation;

[0019] By calculating T, we can capture the similarity between node u and node v. The calculation formula is:

[0020]

[0021] A∈R N×N is the adjacency matrix of the graph G = (V, E), where N = |V| represents the total number of nodes, X∈R N×F Then it is the node feature matrix of graph G;

[0022] μ is a smoothing parameter to adjust the influence of local structural features;

[0023] After L coarsening iterations, the minimum coarsening graph G is obtained L, L is an integer.

[0024] As an optional implementation, the process of partitioning the graph structure after the nodes are merged to form multiple subgraphs includes partitioning the coarsened graph into several groups, each group representing a subgraph in the graph, using an adjacency matrix to represent the edge connections within the partition and the edge connections between the partitions, and each partition is projected back to the original medical image through a greedy algorithm and boundary adjustment.

[0025] As an alternative implementation, a partition is randomly selected as a starting point, and a fixed number of nodes are sampled from it based on a similarity measure.

[0026] As an optional implementation, the process of extracting local features of the subgraph from the node includes extracting image block features of the neighborhood nodes of each node in the subgraph, obtaining the initial feature representation of each node through a local convolution operation; and using a neighborhood aggregation operation between nodes to fuse the information of the central node and its neighboring nodes to generate a feature vector for each subgraph.

[0027] As an optional implementation method, the process of extracting global features for the entire graph structure includes applying a self-attention mechanism on the global graph to capture the global dependencies between all nodes in the graph and extract feature interactions between long-distance nodes; and globally aggregating the features of all nodes to generate a global feature vector.

[0028] As an optional implementation, the process of fusing the extracted global features and local features and normalizing the fused features includes using a vector concatenation operation to concatenate the local and global features together; in the feature normalization stage, smoothness, connectivity, and sparsity constraints are introduced.

[0029] A self-supervised medical image segmentation system based on structural relationship modeling, comprising:

[0030] A graph structure building module is configured to acquire a medical image, preprocess the medical image, divide the preprocessed medical image, and use the divided blocks as nodes to build a graph structure;

[0031] A graph structure processing module, configured to convert the graph structure, calculate the similarity between nodes and collapse similar nodes to achieve node merging;

[0032] A partitioning module, configured to partition the graph structure after the nodes are merged to form multiple subgraphs;

[0033] A local feature extraction module is configured to perform feature sampling, starting from a selected starting point, sampling a fixed number of nodes based on a similarity metric, and extracting local features of the subgraph from the nodes;

[0034] A global feature extraction module is configured to perform global feature extraction on the entire graph structure;

[0035] The feature fusion and normalization processing module is configured to fuse the extracted global features and local features, normalize the fused features, and decode based on the normalized fused features to obtain the reconstructed medical image and sub-image.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention can better capture the overall structural information of the graph by modeling the complex relationships between nodes, subgraphs and the global graph. Compared with traditional methods, the present invention not only considers the local interactions between nodes, but also introduces the fusion of global topological information, thereby achieving a comprehensive understanding of the graph structure.

[0038] The present invention is based on a local node subset sampling mechanism of region partition. Through small-batch training, the problems of over-aggregation and topological decay caused by topological imbalance are alleviated. The local node subset sampling mechanism is constructed by utilizing processes such as node merging, region partition, boundary optimization and feature sampling. It can realize efficient information fusion and transmission in small-batch training, and significantly improve the training efficiency and generalization ability of the model.

[0039] The present invention comprehensively captures and models local and global information by fusing local and global branches, ensuring the integrity of structural information and enabling the decoder to effectively combine local and global features. Specifically, the present invention adopts a hierarchical feature extraction method, which extracts local node subset features at the local level and fuses them with global node features, thereby achieving comprehensive capture and understanding of the graph structure; effectively solving the problem of information loss caused by focusing only on the local or global aspects in traditional methods.

[0040] The present invention ensures the robustness of the model during the training process by constraining the smoothness, connectivity and sparsity between nodes. Specifically, the present invention avoids overfitting or underfitting of the model during the training process by constraining the feature similarity of adjacent nodes; at the same time, by reasonably constraining the connectivity and sparsity of nodes, the accuracy of the graph representation is further improved.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0043] Figure 1 The framework diagram of a self-supervised graph representation learning method of an embodiment. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0046] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0047] In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.

[0048] Embodiment 1

[0049] Firstly, for the obtained graph data, a partition-based sub-node set sampling mechanism is adopted to divide the original graph into multiple sub-graphs, and independent training is performed on these sub-graphs.

[0050] Secondly, by fusing local and global information, a more robust graph representation is constructed.

[0051] In addition, in order to ensure the stability of graph representation, the present invention introduces a feature normalization strategy, including smoothness constraints, connectivity constraints and sparsity constraints, to cope with the complexity of the graph structure.

[0052] Through these steps, the present invention can significantly improve the performance of node classification and graph classification tasks on multiple public datasets.

[0053] The following is a detailed description:

[0054] like Figure 1 As shown, the input medical image is first preprocessed, including steps such as denoising, grayscale or normalization, to ensure that the image quality is suitable for subsequent processing.

[0055] The preprocessed medical image is divided into several small blocks (e.g., 8×8 or N×N in size), each of which is regarded as a node in the graph structure, and the edges between nodes are calculated based on the similarity or adjacency between the small blocks. The similarity between adjacent small blocks is used as the edge weight to construct the graph structure, and then the graph G is obtained. G0 is the unprocessed graph G.

[0056] S2: Node Merger

[0057] Convert the graph G into a series of gradually simplified coarse graphs G L , calculate the similarity between nodes and collapse similar nodes by the formula:

[0058]

[0059] T is the generalized aggregation matrix, which is used to represent the aggregation effect of multi-hop neighborhoods. A is the adjacency matrix, which represents the connection relationship between nodes. K is the degree matrix, and the diagonal element K ii W represents the degree of node i. i is the weight coefficient, which adjusts the influence of neighborhoods with different hop counts. t is the running time and satisfies the following conditions: n indicates the maximum number of hops in the aggregation.

[0060] By calculating T, we can capture the similarity between node u and node v. The calculation formula is:

[0061]

[0062] μ is a smoothing parameter to adjust the influence of local structural features.

[0063] After L coarsening iterations, the minimum coarsening graph G is obtained L .

[0064] S3: Region Division and Boundary Optimization

[0065] For the coarsened graph G L Partition it into several groups, each group represents a subgraph in the graph.

[0066] The adjacency matrix after partitioning can be expressed as:

[0067]

[0068] Represents a partition The inner edges are connected. Represents a partition and Each partition is projected back to the original graph G0 through a greedy algorithm and boundary adjustment.

[0069] S4: Feature sampling

[0070] In the sampling phase, the algorithm randomly selects a partition as the starting point and samples a fixed number of nodes from it based on the similarity metric. The resulting subgraph contains enough information.

[0071] S5: Feature extraction

[0072] After feature sampling, the sampled nodes are further processed by feature extraction.

[0073] Local feature extraction: Each subgraph obtained by sampling is processed by local feature extraction. This process uses a convolutional neural network (CNN) or a graph convolutional network (GCN) to extract features from the nodes in the subgraph. First, for each node in the subgraph, the image block features of its neighborhood nodes are extracted. The initial feature representation of each node is obtained through local convolution operations. Secondly, the neighborhood aggregation operation between nodes is used to fuse the information of the central node and its neighboring nodes to further enrich the local feature representation of the node. Finally, the feature vector of each subgraph is generated for subsequent local and global feature fusion.

[0074] Global feature extraction: For the entire graph structure G after coarsening L , and a global feature extraction method is used for processing. This process captures long-distance dependencies in the global graph through a self-attention mechanism (Transformer) or a graph neural network. First, a self-attention mechanism is applied to the global graph to capture the global dependencies between all nodes in the graph and extract feature interactions between long-distance nodes. Subsequently, the features of all nodes are globally aggregated to generate a global feature vector, ensuring that the overall structural information of the graph is preserved for subsequent local and global feature fusion.

[0075] S6: Feature Fusion

[0076] In the local and global information fusion stage, the features at the sub-graph level are fused with the features of the global graph structure to enhance the robustness of the graph representation.

[0077] The calculation formula of fusion features is:

[0078]

[0079] F is the fused feature vector. w is a learnable parameter used to weight local and global features. Cat is a vector concatenation operation that concatenates local and global features together. Represents the input features of the subgraph. Represents the input features of the global graph.

[0080] The local and global input features are combined through the above operations to generate fused output features.

[0081] S7: Feature Normalization

[0082] In the feature normalization stage, smoothness, connectivity, and sparsity constraints are introduced to improve the stability and performance of the model. The calculation formula of the normalization loss is:

[0083] L R =aL S +bL CS

[0084] L R is the total normalized loss. L S is the smoothness loss, ensuring the smoothness of adjacent node features. CS is the connectivity and sparsity loss, ensuring the reasonable connection and sparsity between nodes. a and b are the weight parameters of the loss term.

[0085] Where L S The calculation of is as follows:

[0086]

[0087] N is the number of nodes. represents the global feature of node j in iteration i. denotes the corresponding global feature in the previous iteration. tr(·) denotes the trace of the matrix. L is the Laplacian matrix associated with the adjacency matrix of the graph.

[0088] L CS The calculation of is as follows:

[0089]

[0090] 1 represents a vector of all 1s. A is the adjacency matrix. λ is a non-negative parameter that controls sparsity. ||·|| F is the Frobenius norm of the adjacency matrix.

[0091] Finally, the reconstructed graph and subgraph are output after decoding.

[0092] The method provided in this embodiment has shown excellent performance in multiple node classification and graph classification tasks, demonstrating its wide applicability and strong generalization ability on complex data structures. This is particularly important in the field of medical image segmentation.

[0093] The method provided in this embodiment achieved a high accuracy rate on the OCT datasets published by Drive and Kermany, as shown in Tables 1 and 2, and especially showed an excellent classification effect when the data labels were limited.

[0094] Table 1 Comparison of experimental results of DRIVE dataset

[0095]

[0096]

[0097] Table 2 Comparison of experimental results of OCT dataset

[0098]

[0099] This shows that the method provided in this embodiment can effectively capture local and global information in the graph structure, thereby improving classification performance. This capability is particularly critical for pixel or region classification tasks in medical image segmentation of eye diseases, because the annotation of medical data is often expensive and limited.

[0100] Embodiment 2

[0101] A self-supervised medical image segmentation system based on structural relationship modeling, comprising:

[0102] A graph structure building module is configured to acquire a medical image, preprocess the medical image, divide the preprocessed medical image, and use the divided blocks as nodes to build a graph structure;

[0103] A graph structure processing module, configured to convert the graph structure, calculate the similarity between nodes and collapse similar nodes to achieve node merging;

[0104] A partitioning module, configured to partition the graph structure after the nodes are merged to form multiple subgraphs;

[0105] A local feature extraction module is configured to perform feature sampling, starting from a selected starting point, sampling a fixed number of nodes based on a similarity metric, and extracting local features of the subgraph from the nodes;

[0106] A global feature extraction module is configured to perform global feature extraction on the entire graph structure;

[0107] The feature fusion and normalization processing module is configured to fuse the extracted global features and local features, normalize the fused features, and decode based on the normalized fused features to obtain the reconstructed medical image and sub-image.

[0108] It can be understood that the above-mentioned units / modules can be separately or completely combined into one or several other units / modules, or one (some) of the units can be further divided into multiple functionally smaller units, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application.

[0109] The above modules of the system are divided based on logical functions. In practical applications, the function of one module can also be implemented by multiple modules, or the functions of multiple modules can be implemented by one module. For example, the feature fusion and normalization processing module of this embodiment may include:

[0110] Feature fusion, which is configured to fuse the extracted global features and local features;

[0111] The normalization processing module is configured to normalize the fused features, perform decoding based on the normalized fused features, and obtain the reconstructed medical image and sub-image.

[0112] Similarly, in other embodiments of the present application, the system may also include other units / modules. In actual applications, these functions may also be implemented with the assistance of other units, and may be implemented by collaboration of multiple units.

[0113] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.

Claims

1. A self-supervised medical image segmentation method based on structural relationship modeling, characterized in that: The following steps are involved: Acquire a medical image, preprocess the medical image, divide the preprocessed medical image, and construct a graph structure using the divided blocks as nodes; Transform the graph structure, calculate the similarity between nodes and collapse similar nodes to achieve node merging; Partition the graph structure after merging nodes to form multiple subgraphs; Perform feature sampling, starting from the selected starting point, sampling a fixed number of nodes based on a similarity metric, and extracting local features of the subgraph from the nodes; Perform global feature extraction on the entire graph structure; The extracted global features and local features are fused and normalized, and decoding is performed based on the normalized fused features to obtain the reconstructed medical image and sub-image.

2. A self-supervised medical image segmentation method based on structural relationship modeling as claimed in claim 1, characterized in that: The preprocessing includes denoising, grayscale and / or normalization processing.

3. The self-supervised medical image segmentation method based on structural relationship modeling as claimed in claim 1, characterized in that: Taking the divided blocks as nodes, the process of constructing the graph structure includes dividing the medical image into several blocks, each block is regarded as a node in the graph structure, the edges between the nodes are calculated according to the similarity or adjacency relationship between the blocks, and the similarity between adjacent blocks is used as the edge weight to construct the graph structure and obtain the graph G.

4. The self-supervised medical image segmentation method based on structural relationship modeling as claimed in claim 1, characterized in that: The process of calculating the similarity between nodes and collapsing similar nodes includes: T is the generalized aggregation matrix, which is used to represent the aggregation effect of multi-hop neighborhoods; A is the adjacency matrix, which represents the connection relationship between nodes; K is the degree matrix, and the diagonal element K ii represents the degree of node i; W i is the weight coefficient, which is used to adjust the influence of different hop count neighborhoods; W i =e - t t i / i ！ , t is the running time, and satisfies the conditions: n represents the maximum number of hops for aggregation; By calculating T, we can capture the similarity between node u and node v. The calculation formula is: A∈R N×N is the adjacency matrix of the graph G = (V, E), where N = |V| represents the total number of nodes, X∈R N×F Then it is the node feature matrix of graph G; μ is a smoothing parameter to adjust the influence of local structural features; After L coarsening iterations, the minimum coarsening graph G is obtained L , L is an integer.

5. The self-supervised medical image segmentation method based on structural relationship modeling as claimed in claim 1, characterized in that: The process of partitioning the graph structure after merging nodes to form multiple subgraphs includes partitioning the coarsened graph into several groups, each group represents a subgraph in the graph, and using the adjacency matrix to represent the edge connections within the partition and the edge connections between the partitions. Each partition is projected back to the original medical image through a greedy algorithm and boundary adjustment.

6. The self-supervised medical image segmentation method based on structural relationship modeling as claimed in claim 1, characterized in that: A partition is randomly chosen as the starting point and a fixed number of nodes are sampled from it based on a similarity measure.

7. The self-supervised medical image segmentation method based on structural relationship modeling as claimed in claim 1, characterized in that: The process of extracting local features of the subgraph from the nodes includes extracting the image block features of the neighborhood nodes of each node in the subgraph, obtaining the initial feature representation of each node through local convolution operation; and fusing the information of the central node and its neighboring nodes through neighborhood aggregation operation between nodes to generate a feature vector of each subgraph.

8. The self-supervised medical image segmentation method based on structural relationship modeling as claimed in claim 1, characterized in that: The process of global feature extraction for the entire graph structure includes applying the self-attention mechanism on the global graph to capture the global dependencies between all nodes in the graph and extract feature interactions between long-distance nodes; globally aggregating the features of all nodes to generate a global feature vector.

9. The self-supervised medical image segmentation method based on structural relationship modeling as claimed in claim 1, characterized in that: The process of fusing the extracted global features and local features and normalizing the fused features includes using a vector concatenation operation to concatenate the local and global features together; in the feature normalization stage, smoothness, connectivity, and sparsity constraints are introduced.

10. A self-supervised medical image segmentation system based on structural relationship modeling, characterized in that: include: A graph structure building module is configured to acquire a medical image, preprocess the medical image, divide the preprocessed medical image, and use the divided blocks as nodes to build a graph structure; A graph structure processing module, configured to convert the graph structure, calculate the similarity between nodes and collapse similar nodes to achieve node merging; A partitioning module, configured to partition the graph structure after the nodes are merged to form multiple subgraphs; A local feature extraction module is configured to perform feature sampling, starting from a selected starting point, sampling a fixed number of nodes based on a similarity metric, and extracting local features of the subgraph from the nodes; A global feature extraction module is configured to perform global feature extraction on the entire graph structure; The feature fusion and normalization processing module is configured to fuse the extracted global features and local features, normalize the fused features, and decode based on the normalized fused features to obtain the reconstructed medical image and sub-image.

Images