Anatomical plausibility detection method and system based on self-clustering graph convolution

Abstract

The application discloses an anatomic reasonableness detection method and system based on self-clustering graph convolution, extracts a feature map through full convolution operation, then carries out M times of independent random clustering on nodes in the feature map to obtain M groups of clustering results, selects any one of the M groups of clustering results as a benchmark, calculates the intersection and union ratio of each clustering set in the selected clustering result and clustering sets in other groups of clustering results, selects a clustering set with the maximum intersection and union ratio in each clustering result and merges the clustering set to obtain a final clustering result, takes the final clustering result as an adjacency matrix, and carries out feature fusion on the adjacency matrix and the feature map extracted through the full convolution operation through graph convolution operation to obtain an anatomic fusion feature, carries out multi-layer convolution operation, and outputs a reasonableness judgment result. The application carries out information fusion between the same categories, and gives a reasonableness evaluation on anatomic characteristics of organs or tissues.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing, and particularly relates to an anatomical rationality detection method and system based on self-clustering graph convolution. BACKGROUND

[0002] In recent years, with the significant improvement of medical image acquisition technology, medical devices can acquire a large amount of medical images and sensor data in real time at a faster image frame rate, higher image resolution and communication technology. The medical images contain rich information of human organs or tissues, and normal or diseased organs or tissues have their own anatomical characteristics. Evaluating the rationality of the anatomical characteristics of organs or tissues is a hot research topic.

[0003] Traditional convolutional neural networks can only process Euclidean space data (such as matrix form), but many real-life scenarios, such as traffic networks, social networks, and citation networks, exist in the form of graph data. Anatomical characteristics are also a non-Euclidean space feature, so graph convolution and other technologies are needed to better handle them.

[0004] For medical images, each tissue or organ has its own texture and characteristics, and the distinction between them must be preserved. However, there are many types of tissues or organs involved in medical images, and if they are not distinguished, graph convolution processing using only one class of adjacency matrix can easily cause excessive averaging of anatomical characteristics between different tissues or organs, which does not have distinguishing characteristics. Although there are related research results in the field of natural image segmentation that can preserve or even enhance the distinction, this method requires the assistance of semantic labels.

[0005] The anatomical characteristics involved in the anatomical rationality evaluation method are separated in space, and the equal-scale convolution kernel used by traditional convolutional neural networks is not suitable for processing this information. As an alternative technology, graph convolution relies on the assistance of anatomical level standard information, which can cause excessive information sharing and averaging between characteristics of different tissues or organs, ultimately leading to ineffective extraction of anatomical characteristics. SUMMARY

[0006] The purpose of the present application is to provide an anatomical rationality detection method and system based on self-clustering graph convolution, which overcomes the problem of excessive information sharing and averaging between different class characteristics in graph convolution operation and solves the problem of excessive fusion of different class characteristics.

[0007] To achieve the above purpose, the technical scheme of the present application is as follows:

[0008] An anatomical rationality detection method based on self-clustering graph convolution, comprising:

[0009] The original medical image is acquired, a feature map is extracted through a full convolution operation, and M independent random clustering operations are performed on nodes in the feature map to obtain M groups of clustering results, each of which includes K clustering sets, and each clustering set represents a tissue or organ in the image;

[0010] Any one of the M groups of clustering results is selected as a reference, the intersection-over-union of each clustering set in the selected clustering result and the clustering sets in the other groups of clustering results is calculated, and for any clustering set in the selected clustering result, the clustering set with the maximum intersection-over-union in each clustering result is selected and merged with the clustering set, to obtain K final clustering sets as the final clustering result;

[0011] The final clustering result is taken as an adjacency matrix, and the feature map extracted through the full convolution operation is fused through a graph convolution operation to obtain an anatomical fusion feature;

[0012] The anatomical fusion feature is input into an anatomical reasonableness decision maker for multi-layer convolution operation, and a reasonableness decision result is output.

[0013] Further, the feature fusion by taking the final clustering result as an adjacency matrix and the feature map extracted through the full convolution operation through the graph convolution operation further includes:

[0014] For the final clustering result, the shape is K×H×W, the H×W node matrix is flattened into a one-dimensional vector, and is adjusted to K×N, N=H×W, as an adjacency matrix;

[0015] Wherein, H represents the length of the feature map, and W represents the width of the feature map;

[0016] For the feature map extracted through the full convolution operation, the shape is C×H×W, and is adjusted to N×C, wherein C represents the number of channels of the feature map.

[0017] Further, during training of the anatomical reasonableness decision maker, the loss function is:

[0018]

[0019] Wherein Loss diff represents a difference loss function, D(Y) represents the score of the positive sample Y given by the anatomical reasonableness decision maker D, D(Y') represents the score of the negative sample Y' given by the anatomical reasonableness decision maker D, represents the gradient, wherein Y represents the residual image corresponding to the positive sample, Y' represents the residual image corresponding to the negative sample, and ‖.‖1 and ‖.‖2 represent the L1 norm and the L2 norm, respectively.

[0020] The application also provides an anatomical rationality detection system based on self-clustering graph convolution, comprising a feature extractor, a random clusterer, a difficult communication device, a graph convolution module and an anatomical rationality decision device.

[0021] The feature extractor is configured to acquire an original medical image and extract a feature map through full convolution operation.

[0022] The random clusterer is configured to perform M times of independent random clustering on nodes in the feature map to obtain M groups of clustering results, each of which comprises K clustering sets, and each of the clustering sets represents an organization or an organ in the image.

[0023] The difficult communication device is configured to select any one of the M groups of clustering results as a reference, calculate the intersection-over-union of each clustering set in the selected clustering result and clustering sets in other groups of clustering results, and select the clustering set with the largest intersection-over-union in each of the clustering results to merge with the clustering set, to obtain K final clustering sets as a final clustering result.

[0024] The graph convolution module is configured to take the final clustering result as an adjacency matrix, and perform feature fusion on the feature map extracted through the full convolution operation through graph convolution operation, to obtain anatomical fusion features.

[0025] The anatomical rationality decision device is configured to perform multi-layer convolution operation on the input anatomical fusion features, and output a rationality decision result.

[0026] Further, the graph convolution module takes the final clustering result as an adjacency matrix, and performs feature fusion on the feature map extracted through the full convolution operation through graph convolution operation, and further performs the following operations:

[0027] For the final clustering result with a shape of KxHxW, the HxW node matrix is flattened into a one-dimensional vector, adjusted to KxN (N=HxW) as an adjacency matrix.

[0028] Wherein, H represents the length of the feature map, and W represents the width of the feature map.

[0029] For the feature map extracted through the full convolution operation with a shape of CxHxW, it is adjusted to NxC, wherein C represents the channel number of the feature map.

[0030] Further, the loss function of the anatomical rationality decision device during training is:

[0031]

[0032] Wherein, Loss diffdenotes the difference loss function, D(Y) denotes the score of the anatomical plausibility judge D on the positive sample Y, and D(Y') denotes the score of the anatomical plausibility judge D on the negative sample Y', denotes the gradient, where Y denotes the residual image corresponding to the positive sample, Y' denotes the residual image corresponding to the negative sample, and ||.||1 and ||.||2 respectively denote the L1 norm and the L2 norm.

[0033] The application provides an anatomical plausibility detection method and system based on self-clustering graph convolution. An unsupervised, difference-preserving feature extraction method in a non-Euclidean space is provided. In the absence of anatomical labels, the feature extraction method is suitable for a non-Euclidean space. A graph convolution network is used for feature fusion, which avoids excessive information sharing and averaging between features. The application can fuse information between the same category (tissue organs) and give a reasonable evaluation of the anatomical features of organs or tissues. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A flowchart of the anatomical plausibility detection method based on self-clustering graph convolution is provided.

[0035] Figure 2 A structural schematic diagram of the anatomical plausibility detection system based on self-clustering graph convolution is provided. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0037] In one embodiment, as shown in Figure 1 , an anatomical plausibility detection method based on self-clustering graph convolution is provided, comprising:

[0038] Step S1, obtaining an original medical image, extracting a feature map through a full convolution operation, and then performing M times of independent random clustering on nodes in the feature map to obtain M groups of clustering results, each group of clustering results including K clustering sets, and each clustering set representing a tissue or organ in the image.

[0039] The embodiment constructs an anatomical plausibility detection network model, as shown in Figure 2 , including a feature extractor, a random clusterer, a difficult communication device, a graph convolution module and an anatomical plausibility judge.

[0040] This step uses a fully convolutional network (as a feature extractor) to extract features; classic convolutional neural networks such as ResNet101 and VGG16 can be used. After acquiring the original medical image (such as a human mammogram), the fully convolutional network performs convolution, average pooling, and normalization operations on the medical image. The feature map size decreases layer by layer, and the information contained is abstracted and enriched layer by layer, finally obtaining a feature map F (with dimensions C×H×W), where H represents the length of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map.

[0041] This embodiment performs M independent (random clustering methods can use the KMeans algorithm, clustering based on the C channels of each node, with multiple random clusterings not interfering with each other. The cluster centers are randomly initialized, so there will be no identical results; in this embodiment, M=10 ensures sufficient randomness) random clustering on the N=H×W nodes in the feature map F. Figure 2 The random clusterer in the model contains K cluster sets for each cluster result (here K=10, which is the data finally calculated by statistically analyzing the number of tissues and organs in a large number of medical images).

[0042] Each clustering result contains K cluster sets. The number of feature points in each cluster set is not fixed (they may be the same or different), but the total number of feature points in each cluster set is equal to N. Each cluster set represents a tissue or organ in the image. Let i represent the j-th group and the i-th cluster.

[0043] Step S2: Select any one of the M clustering results as the benchmark, calculate the intersection-union ratio (IUR) of each cluster set in the selected clustering results with the cluster sets in other clustering results, and for any cluster set in the selected clustering results, select the cluster set with the largest IUR in each clustering result and merge it with it to obtain the final K cluster sets as the final clustering results.

[0044] Specifically, corresponding to Figure 2 The complex communication mechanism in the middle. For example, selecting the first group of cluster results, its included cluster sets include: first right Calculate the Intersection over Union (IOU), and then select the cluster set with the largest IOU from each clustering result set. To merge, such as This results in a merged cluster set.

[0045] Similarly, for Repeat the above steps to merge the clusters with the largest IOU, thus obtaining a new set of clustering results, which will be used as the final clustering result.

[0046] Step S3, the final clustering result is taken as an adjacency matrix, and the feature map extracted by the full convolution operation is subjected to feature fusion by a graph convolution operation to obtain an anatomical fusion feature.

[0047] The anatomical feature fusion in this embodiment adopts a classical graph convolution form, formula Y = A x F x W, A is an adjacency matrix of graph nodes (records the attribution relationship between each tissue organ pixel on the feature map), that is, the final clustering result obtained in step S2 (the shape is adjusted from K x H x W to K x N, that is, the node matrix of H x W is flattened into a one-dimensional vector), F is the feature map F obtained in step S1 (the shape is adjusted from C x H x W to N x C), W is a learnable weight matrix (the shape is C x C), and Y is a feature map obtained by matrix multiplication (the shape is K x C).

[0048] In the matrix operation, the convolution kernel of each graph convolution only extracts features for the corresponding tissue organ, avoiding information sharing between different parts.

[0049] Step S4, inputting the anatomical fusion feature into an anatomical reasonableness decision maker to perform a multi-layer convolution operation and outputting a reasonableness decision result.

[0050] In this step, the anatomical reasonableness evaluation decision maker performs a multi-layer convolution operation to score the anatomical fusion feature, and the higher the score, the higher the reasonableness.

[0051] This application inputs a medical image with unknown anatomical feature reasonableness (which may be taken by a professional instrument or artificially forged), first extracts features by full convolution, generates multiple clustering results by random clustering, selects a set of clustering results as a reference, calculates the intersection and union ratio of each clustering set in the selected clustering results and the clustering sets in other sets of clustering results, selects the clustering set with the largest intersection and union ratio in each clustering result and merges it, and obtains K final clustering sets as the final clustering result. Through such an operation, a more reasonable clustering result is generated. The clustering result is taken as a multi-class adjacency matrix and the extracted features are subjected to a graph convolution operation to fuse and generate an anatomical fusion feature. Finally, the reasonableness decision maker scores the feature, and the higher the score, the higher the reasonableness.

[0052] In another embodiment, as shown in Figure 2 , a system for detecting anatomical reasonableness based on self-clustering graph convolution is provided, including a feature extractor, a random clustering device, a difficult communication device, a graph convolution module, and an anatomical reasonableness decision maker.

[0053] Wherein, wherein, the feature extractor is used to acquire an original medical image, and a feature map is extracted by a full convolution operation;

[0054] The random clusterer is configured to perform M independent random clustering on the nodes in the feature map to obtain M sets of clustering results, each set of clustering results including K cluster sets, and each cluster set representing an organ or tissue in the image.

[0055] The difficult communication device is configured to select any one of the M sets of clustering results as a reference, calculate the intersection-over-union of each cluster set in the selected clustering result and the cluster sets in other sets of clustering results, and select the cluster set with the largest intersection-over-union in each set of clustering results to merge with the cluster set, to obtain K final cluster sets as a final clustering result.

[0056] The graph convolution module is configured to perform feature fusion on the final clustering result as an adjacency matrix and the feature map extracted by the full convolution operation through graph convolution operation to obtain an anatomical fusion feature.

[0057] The anatomical rationality decision device is configured to perform multi-layer convolution operation on the input anatomical fusion feature to output a rationality decision result.

[0058] It is easy to understand that the embodiment of the present application is an anatomical rationality detection system based on self-clustering graph convolution, which is a network model. After the network model is constructed, it needs to be trained. After the network model is trained, the trained network model is used to process the medical image to be detected to obtain the anatomical rationality detection result.

[0059] During training, the original medical image is first obtained as a positive sample, and the positive sample is disturbed to generate a corresponding negative sample as a positive and negative sample. That is, the positive sample x is the original medical image (such as a human breast molybdenum target image), and the negative sample y is a texture discontinuous (unreasonable) image generated by disturbing the positive sample x. The positive sample x and the negative sample y are input into the network model as input sample images in batches to train the network model.

[0060] During training, the positive sample and the negative sample are first extracted by the feature extractor, the feature extractor extracts features from the input image, and the size is compressed to generate a feature map, each node of which represents a graph node.

[0061] Then, the random clusterer is used to perform random clustering to obtain M sets of clustering results. The random clusterer classifies the graph nodes on the feature map, and the M sets of clustering results are roughly the same, but the clustering results of some difficult graph nodes have great randomness. This randomness usually occurs in the intersection of two tissues, and the clustering cannot completely determine the category of the point. Forcedly dividing the point into a certain category will affect the final result. Therefore, the randomness of the point needs to be retained and merged into all categories similar to it (merge the cluster sets by intersection-over-union), so as to minimize the loss as much as possible.

[0062] Then the final clustering result is obtained by using the difficult communication device, and the difficult communication device merges the M groups of clustering results according to the maximum IOU between different clusters to obtain a result in which there are repeated nodes between clusters. Some difficult graph nodes may belong to different clusters at the same time.

[0063] Then the feature fusion is performed through the graph convolution module to obtain the anatomical fusion feature. The graph convolution module takes the clustering label generated by the difficult communication device as an adjacency matrix, and only performs graph convolution operation between the clusters to output the final feature map. The graph convolution module adopts the CDGC network based on unsupervised modification, and the input is the feature map and the adjacency matrix formed by the union of multiple clustering, and the output is the feature map after information fusion.

[0064] Finally, the anatomical features of the positive sample and the negative sample are input into the anatomical rationality decision maker, and the input features are scored. The higher the score is, the higher the rationality is. The anatomical rationality decision maker adopts a multi-layer convolutional network, inputs a feature vector with a dimension of KxC, and outputs a score after convolution operation. The higher the score is, the more reasonable the anatomical feature of the image is.

[0065] During the training of the present application, the training is performed through the following loss function:

[0066]

[0067] Where Loss diff represents the difference loss function, D(Y) represents the score of the anatomical rationality decision maker D on the positive sample Y, D(Y') represents the score of the anatomical rationality decision maker D on the negative sample Y', represents the gradient, where Y represents the residual image corresponding to the positive sample, Y ′ represents the residual image corresponding to the negative sample. ‖.‖1 and ‖.‖2 represent L1 norm and L2 norm respectively.

[0068] The second term on the right side of the above loss function equation is a regularization penalty term, which can form a relatively flat hyperplane around the positive sample for the decision maker. On the one hand, it requires the decision maker D to give a score close to the positive sample for similar positive sample data, so that the score of the decision maker D on the positive sample has high cohesion. On the other hand, it can regularize the decision maker to prevent it from diverging.

[0069] The present application calculates the loss according to the preset loss function, and supervises the training of the feature extractor and the anatomical rationality decision maker by using the anatomical features of the positive sample and the negative sample to obtain a trained network model. During the training, the feature map Y generated by the positive sample x is given a higher score, and the feature map Y ′ generated by the negative sample y is given a lower score, which can cooperate with the regularization method to obtain a relatively stable training process.

[0070] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a more specific and detailed manner, but should not be construed as limiting the scope of the patent. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An anatomical plausibility detection method based on self-clustering graph convolution, characterized in that, The self-clustering graph convolution-based anatomical reasonableness detection method comprises the following steps: An original medical image is acquired, a feature map is extracted through full convolution operation, and M groups of clustering results are obtained by performing M times of independent random clustering on nodes in the feature map, each group of clustering results including K clustering sets, and each clustering set representing an organ or tissue in the image; Any one of the M groups of clustering results is selected as a reference, the intersection-over-union of each clustering set in the selected clustering result and the clustering sets in the other groups of clustering results is calculated, and for any clustering set in the selected clustering result, the clustering set with the largest intersection-over-union in each clustering result is selected and merged with the clustering set, to obtain K final clustering sets as the final clustering result; The final clustering result is taken as an adjacency matrix, and feature fusion is performed on the final clustering result and the feature map extracted through the full convolution operation by graph convolution operation, to obtain anatomical fusion features; The anatomical fusion features are input into an anatomical reasonableness decision maker for multi-layer convolution operation, and a reasonableness decision result is output. 2.The self-clustering graph convolution-based anatomical plausibility detection method of claim 1, wherein, The self-clustering graph convolution-based anatomical reasonableness detection method comprises the following steps: For the final clustering result, the shape is K×H×W, the H×W node matrix is flattened into a one-dimensional vector, and is adjusted to K×N, N=H×W, as an adjacency matrix; wherein H represents the length of the feature map, and W represents the width of the feature map; For the feature map extracted through the full convolution operation, the shape is C×H×W, and is adjusted to N×C, wherein C represents the channel number of the feature map. 3.The self-clustering graph convolution-based anatomical plausibility detection method of claim 1, wherein, During training of the anatomical reasonableness decision maker, the loss function is: where Loss diff represents the difference loss function, D(Y) represents the score of the anatomical plausibility judge D on the positive sample Y, D(Y') represents the score of the anatomical plausibility judge D on the negative sample Y', represents the gradient, where Y represents the residual image corresponding to the positive sample, Y' represents the residual image corresponding to the negative sample, and ||.||1 and ||.||2 represent the L1 norm and the L2 norm, respectively.

4. An anatomical plausibility detection system based on self-clustering graph convolution, characterized in that, The self-clustering graph convolution-based anatomical reasonableness detection system comprises a feature extractor, a random clustering device, a difficult communication device, a graph convolution module, and an anatomical reasonableness decision maker; The feature extractor is configured to acquire an original medical image and extract a feature map through full convolution operation. The random clustering device is configured to perform M times of independent random clustering on nodes in the feature map to obtain M groups of clustering results, each group of clustering results including K clustering sets, and each clustering set representing an organ or tissue in the image. The difficult communication device is configured to select any one of the M groups of clustering results as a reference, calculate the intersection-over-union of each clustering set in the selected clustering result and the clustering sets in the other groups of clustering results, select, for any clustering set in the selected clustering result, the clustering set with the largest intersection-over-union in each clustering result, and merge the clustering sets to obtain K final clustering sets as the final clustering result. The graph convolution module is configured to take the final clustering result as an adjacency matrix, perform feature fusion on the final clustering result and the feature map extracted through the full convolution operation by graph convolution operation, and obtain anatomical fusion features. The anatomical reasonableness decision maker is configured to perform multi-layer convolution operation on the input anatomical fusion features and output a reasonableness decision result.

5. The self-clustering graph convolution-based anatomical plausibility detection system of claim 4, wherein, The graph convolution module takes the final clustering result as an adjacency matrix, performs feature fusion on the final clustering result and the feature map extracted through the full convolution operation by graph convolution operation, and further performs the following operations: For the final clustering result, the shape is KxHxW, the HxW node matrix is flattened into a one-dimensional vector, adjusted to KxN, N=HxW, as an adjacency matrix; Where H represents the length of the feature map, and W represents the width of the feature map; For the feature map extracted by the full convolution operation, the shape is CxHxW, and it is adjusted to NxC, where C represents the number of channels of the feature map. 6.The self-clustering graph convolution-based anatomical plausibility detection method of claim 4, wherein, When the anatomical rationality decision device is trained, the loss function is: where Loss diff represents the difference loss function, D(Y) represents the score of the anatomical plausibility judge D on the positive sample Y, D(Y') represents the score of the anatomical plausibility judge D on the negative sample Y', represents the gradient, where Y represents the residual image corresponding to the positive sample, Y ′ represents the residual image corresponding to the negative sample, ‖.‖1 and ‖.‖2 represent the L1 norm and the L2 norm, respectively.

Images