A method, device, and program product based on thrombus image reconstruction map data

Abstract

This application relates to the field of intelligent healthcare, and specifically relates to a method, device, and program product based on thrombus image reconstruction data. It includes obtaining a thrombus image of a patient, where the image contains a thrombus and surrounding tissues; extracting features from the image to obtain feature vectors; obtaining the spatial relationship between the thrombus and surrounding tissue vectors based on the feature vectors of the thrombus and surrounding tissues; constructing a graph structure based on the spatial relationship and feature vectors to obtain graph data. The present invention can extract morphological features and spatial relationships in multiple dimensions of the image, can effectively identify subtle differences, and has good clinical value.

Description

Technical Field

[0001] This application relates to the field of intelligent medicine, and specifically to a method, device, program product, and computer-readable storage medium based on thrombus image reconstruction map data. Background Art

[0002] Deep Vein Thrombosis (DVT), as a potentially fatal vascular disease, forms in the deep veins, especially in the venous system of the lower extremities, and may trigger serious complications such as pulmonary embolism, posing a direct threat to the patient's life. The accurate identification and staging of DVT are crucial for formulating an appropriate treatment path, including the choice of anticoagulant therapy, the timing of thrombolytic therapy, and even mechanical thrombectomy surgery when necessary. In clinical practice, the diagnosis of DVT often combines the patient's clinical manifestations (such as leg swelling and pain), hematological examinations (such as elevated D-dimer levels), and imaging evidence. Although hematological examinations can provide indirect evidence, they are not sufficient for a definite diagnosis; while clinical symptoms are not specific and are easily confused with other diseases. Therefore, imaging examinations have become an indispensable part of the definite diagnosis of DVT.

[0003] Recent research has explored the application of deep learning technology in DVT diagnosis, aiming to automate and precisely identify and stage thrombi. One is to implement a machine learning (ML) model on edge devices to propose a more efficient and reliable method for diagnosing deep vein thrombosis (DVT), and its system has the characteristics of high performance, portability, open source, and low cost. The other is to use deep learning technology to precisely segment venous thrombi in magnetic resonance images, especially optimizing the model's ability to identify low-signal thrombus regions and reducing the interference of background noise and artifacts. However, the above algorithms for DVT diagnosis do not delve into staging classification, especially in dealing with the subtle differences in different development stages of thrombi. In the daily process of reading images and traditional diagnostic algorithms, it is difficult to directly combine the overall characteristics of various tissues in the image and the relationships between different levels for comprehensive judgment. The image contains a large amount of complex soft tissue information, and the relationships and interactions between different levels are very important for accurate diagnosis. However, traditional methods are often limited to a planar perspective and are difficult to make full use of the rich information and structural connections in three-dimensional space, resulting in incomplete and inaccurate diagnostic results. Summary of the Invention

[0004] In view of the above problems, the present invention provides a method based on thrombus image reconstruction map data. By reconstructing thrombus image data, this method can effectively capture these complex blood flow patterns and tissue structure information. Especially for subacute non-fully organized thrombi and fully organized thrombi in the chronic stage, it can well obtain the distinguishable feature information, thus having good performance in subsequent staging tasks, specifically including:

[0005] Obtain the thrombus image of the patient, where the image contains the thrombus and surrounding tissues;

[0006] Extract features from the image to obtain feature vectors;

[0007] Based on the feature vectors of the thrombus and surrounding tissues, obtain the spatial relationship between the thrombus and surrounding tissue vectors;

[0008] Based on the spatial relationship and feature vectors, construct a graph structure to obtain graph data.

[0009] Furthermore, the graph structure is obtained by cross - layer connecting feature vectors at different levels through the spatial relationship between feature vectors;

[0010] Optionally, the spatial relationship includes the spatial relationship between adjacent vectors in the same dimension, the spatial relationship between adjacent vectors in different dimensions, and the global three - dimensional spatial information relationship;

[0011] Optionally, the image includes one or more of the following: MRI, CT.

[0012] Optionally, the construction process of the graph structure is specifically as follows:

[0013] S1: Convert the spatial coordinates of the feature vectors into original nodes;

[0014] S2: Based on the original nodes, traverse all their adjacent position nodes to obtain the spatial coordinates of the adjacent nodes and the spatial relationship between the nodes;

[0015] S3: Connect the original nodes and adjacent nodes with edges and update the original nodes with the spatial coordinates of the adjacent nodes. Repeat the above steps until the graph structure is obtained;

[0016] Optionally, the traversal adopts one or more of the following: depth - first traversal, breadth - first traversal, iterative deepening depth - first search;

[0017] Optionally, the construction of the graph structure also includes dynamic construction through a dynamic adjustment mechanism;

[0018] Optionally, the dynamic construction dynamically changes the connection mode of nodes and edges through the characteristics of the thrombus.

[0019] The feature extraction is to extract feature vectors from the thrombus and surrounding tissues in the image data through a pre - trained model;

[0020] Optionally, the pre - trained model includes any one or more of the following: ResNet18, ResNet34, ResNet50, Vision Transformer, DenseNet, VGGNet, Inception ResNet;

[0021] Optionally, the pre-trained model extracts features based on the ResNet18 model. First, the deep vein thrombosis image data is obtained and input into the pre-trained ResNet18 model for adjustment to obtain an adjusted pre-trained ResNet18 model. Feature vectors are obtained by extracting features based on the adjusted pre-trained ResNet18 model.

[0022] The method further includes data preprocessing. The image data of deep vein thrombosis patients is preprocessed to obtain processed data, and feature vectors are obtained by extracting features of thrombus and surrounding tissues in the processed data.

[0023] Optionally, the data preprocessing includes one or more of the following: image size standardization, image equalization, image background removal, and image enhancement.

[0024] Optionally, the image size standardization expands the image size by padding with zero values or interpolation.

[0025] Optionally, the image background removal removes the background and / or non-interested regions by the global threshold method.

[0026] The object of the present invention is to provide a method for staging deep vein thrombosis based on a graph convolutional neural network, including:

[0027] The method obtains graph data based on the above method of reconstructing graph data from thrombus images.

[0028] The graph data is input into a graph neural network for staging to obtain a staging result.

[0029] Furthermore, the graph neural network includes an input layer, a graph convolutional layer, a graph attention layer, a pooling layer, and an output layer. The graph data passes through the input layer, the graph convolutional layer, the graph attention layer, the pooling layer, and the output layer in sequence to obtain a staging result.

[0030] Optionally, the graph convolutional layer performs convolution on the weights of nodes and edges in the graph data to obtain feature information between different positions and different levels. The feature information is sorted by the graph attention layer to obtain important node features and edge features. The node features and edge features are input into the pooling layer and the output layer to obtain a staging result.

[0031] Optionally, the pooling layer adopts one or more of the following: average pooling, max pooling, differentiable pooling.

[0032] Optionally, the differentiable pooling aggregates each node feature to obtain a graph feature, and the graph feature is staged by the output layer to obtain a staging result.

[0033] Optionally, the output layer includes one or more of the following: softmax, sigmoid, multi-layer perceptron;

[0034] Optionally, the multi-layer perceptron determines the stage of deep vein thrombosis by identifying the characteristics of the deep vein thrombosis map structure;

[0035] Optionally, the graph neural network is obtained by fusing N graph neural networks with different architectures or parameters, where N is a natural number greater than 1;

[0036] Optionally, the fusion process includes one or more of the following: voting, averaging, weighted averaging.

[0037] The multi-layer perceptron is a trained multi-layer perceptron. By obtaining deep vein thrombosis images of different stages and then converting the deep vein thrombosis images of different stages into graph structure data of different stages, the multi-layer perceptron is trained through the graph structure data of different stages to obtain a trained multi-layer perceptron for morphological features and spatial hierarchical features.

[0038] The purpose of the present invention is to provide a computer program product with a computer program or instruction thereon, and the computer program or instruction is executed by a processor to implement the above method based on reconstructed graph data of thrombus images or the above method for staging deep vein thrombosis based on a graph neural network.

[0039] The purpose of the present invention is to provide a computer device, including a memory, a processor, and a computer program or instruction stored on the memory, including:

[0040] The processor executes the computer program or instruction to implement the above method based on reconstructed graph data of thrombus images or the above method for staging deep vein thrombosis based on a graph neural network.

[0041] The purpose of the present invention is to provide a computer-readable storage medium with a computer program or instruction stored thereon, and the computer program or instruction is executed by a processor to implement the above method based on reconstructed graph data of thrombus images or the above method for staging deep vein thrombosis based on a graph neural network.

[0042] Advantages of the present invention:

[0043] 1. A graph structure method with cross-layer connections is introduced. By constructing a graph structure that comprehensively considers local spatial proximity and global three-dimensional spatial hierarchical information, it can more comprehensively integrate and understand the spatial distribution and three-dimensional overall situation of thrombi and their surrounding tissues in magnetic resonance images. Specifically, the graph structure method with cross-layer connections can capture the complex relationships and interactions between different levels while retaining detailed information, thereby improving the accuracy and reliability of the model for thrombus staging. This method not only improves the sensitivity of the model to local features but also enhances the ability to understand the overall spatial structure, making the staging diagnosis of deep vein thrombosis more accurate and comprehensive.

[0044] 2. In the graph convolutional neural network structure, the pooling layer performs differentiable pooling. The differentiable pooling method can aggregate the feature information of each part, thereby retaining local key features that are easily overlooked by the average pooling method, more precisely capturing small but important soft tissue features in the image, adapting to the complex contrast of different soft tissues, and thus improving the accuracy of feature extraction. And it allows the effective propagation of gradients, which helps the network optimize parameters during training and enhances the sensitivity and recognition ability to the complex contrast of different soft tissues.

[0045] 3. In feature extraction, a pre-trained model is adopted and fine-tuned through deep vein thrombosis images, which has good feature processing capabilities and can effectively identify thrombi and surrounding tissues in deep vein thrombosis images; at the output layer of the graph convolutional neural network, staging is performed through a trained multi-layer perceptron to improve the recognition of thrombus features in different periods and the staging accuracy.

[0046] 4. The present invention forms an organic whole for deep vein thrombosis staging by feature extraction, constructing a graph structure, and then inputting the graph structure data into the graph convolutional neural network. The output of the previous step serves as the input of the next step, and their functions support each other. Based on the patient's image data, specific deep vein thrombosis staging results can be obtained, which has practical clinical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0048] Figure 1 It is a schematic flowchart of a method based on reconstructed graph data of thrombus images provided by an embodiment of the present invention;

[0049] Figure 2 It is a schematic diagram of a system based on reconstructed graph data of thrombus images provided by an embodiment of the present invention;

[0050] Figure 3 Schematic diagram of a device based on thrombus image reconstruction map data provided by an embodiment of the present invention;

[0051] Figure 4 Schematic diagram of the process for constructing map data provided by an embodiment of the present invention;

[0052] Figure 5 Schematic diagram of the process of the graph neural network structure provided by an embodiment of the present invention. Detailed implementation manners

[0053] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0054] In some processes described in the specification, claims and the above-mentioned drawings of the present invention, a plurality of operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear herein or may be executed in parallel. The serial numbers of the operations, such as S101, S102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit that "first" and "second" are of different types.

[0055] Figure 1 Schematic diagram of a method based on thrombus image reconstruction map data provided by an embodiment of the present invention, specifically including:

[0056] S101: Obtain the thrombus image of the patient, and the image contains the thrombus and surrounding tissues;

[0057] In one embodiment, the image includes one or more of the following: MRI, CT.

[0058] In one embodiment, the method further includes data preprocessing, performing data preprocessing on the image data of the deep vein thrombosis patient to obtain processed data, and extracting features of the thrombus and surrounding tissues in the processed data to obtain feature vectors;

[0059] Optionally, the data preprocessing includes one or more of the following: image size normalization, image equalization, image background removal, image enhancement;

[0060] Optionally, the image size normalization expands the image size by filling zeros or interpolation;

[0061] Optionally, the image background removal removes the background and / or non - interested regions by the global threshold method.

[0062] In one embodiment, Magnetic Resonance Imaging (MRI), with its non - invasive nature, high resolution, and excellent soft - tissue contrast, provides unique advantages for the diagnosis of DVT. MRI can clearly display venous structures and identify direct images of thrombi, which is crucial for understanding the location, size, and impact on surrounding tissues of the thrombi. However, the development of DVT is a dynamic process. From the acute phase to the chronic phase, significant changes occur in the physical properties, morphology, and signal characteristics of thrombi on magnetic resonance images.

[0063] In a specific embodiment, pre - processing of the data is performed, including standardizing the size of magnetic resonance image data. Specifically:

[0064] Statistical information on the sizes of all collected magnetic resonance images is obtained, and the maximum image matrix size among them is selected as the standard size after pre - processing. For example, if the maximum size is 960×320, then all images smaller than this size will be expanded by padding zeros around them or using high - quality interpolation techniques (such as cubic spline interpolation) to match the standard size.

[0065] It also includes equalization processing of the content of magnetic resonance images: To address the differences in brightness and contrast between different images, a standardized data normalization technique is used to map the pixel values of all images into a common specified interval. This process includes calculating the mean and variance of each image, and then performing a linear transformation on the images based on these statistics to ensure that the pixel value distributions of all images have similar statistical properties.

[0066]

[0067] Equation 1 is a method for equalization processing of image content P, where P nom is the image content after standardization. Among them, I i represents the gray - scale value of the i - th pixel, and N is the total number of pixels in the image. This design can eliminate these non - biological variations for all input magnetic resonance images.

[0068] It also includes background removal to adapt to memory limitations: To address the issue of GPU memory overflow when processing large magnetic resonance images, the entire magnetic resonance image is first segmented into multiple small blocks, called Patches. The size of each Patch is set to 100 pixels wide and 80 pixels high. Such a size can not only adapt to common GPU memory capacities but also retain sufficient local image information for analysis. Subsequently, before analyzing the segmented Patches, the global thresholding method is used to remove background noise and non - interesting regions in the magnetic resonance image. This step helps to focus on analyzing the thrombus area and reduce interference from irrelevant information.

[0069]

[0070] Equation 2 is for removing background noise and non - interesting regions in the magnetic resonance image by the global thresholding method. Among them, T * The optimal threshold required for threshold segmentation is calculated by obtaining the average gray level through the frequency p(i) corresponding to the gray level i in the gray - level histogram, so that the square of the difference between the gray - level means of the background region and the foreground region multiplied by the product of the weights of the two regions reaches the maximum, thereby achieving the optimal binary segmentation of the image. L is the maximum value of the pixel values in the image. Wb(T) is the weight of the class below the threshold T, that is, the weight of the background class. Ww(T) is the weight of the class at and above the threshold T, that is, the weight of the foreground class.

[0071] S102: Extract features from the said image to obtain a feature vector;

[0072] In one embodiment, the feature extraction is performed by a pre - trained model to extract feature vectors from thrombus and surrounding tissues in the image data;

[0073] Optionally, the pre - trained model includes any one or several of the following: ResNet18, ResNet34, ResNet50, Vision Transformer, DenseNet, VGGNet, Inception ResNet;

[0074] Optionally, the pre - trained model performs feature extraction based on the ResNet18 model. First, deep vein thrombosis image data is obtained and input into the pre - trained ResNet18 model for adjustment to obtain an adjusted pre - trained ResNet18 model, and then feature vectors are extracted based on the adjusted pre - trained ResNet18 model.

[0075] In one embodiment, in addition to using ResNet18 as a feature extractor, other pre-trained models, such as Vision Transformer, etc., can be explored, or combined with a transfer learning strategy, models pre-trained on other medical imaging tasks can be used to improve the model's ability to understand DVT images.

[0076] In one embodiment, the data construction process of the present invention is as follows Figure 4 As shown, a magnetic resonance image is input and image preprocessing is performed to obtain processed data, and the image preprocessing includes image size standardization, image content equalization, and image background removal; the processed data is subjected to node feature extraction by a deep learning method (node feature extraction is performed by a fine-tuned pre-trained model) to obtain a feature vector, and the feature vector constructs a graph structure based on the spatial relationship between features to obtain graph data.

[0077] In a specific embodiment, node feature extraction is performed by a deep learning method, specifically:

[0078] Pre-preparation: Fine-tune the pre-trained ResNet18 model using deep vein thrombosis MRI images to obtain a fine-tuned ResNet18 model;

[0079] Training or application: Use the fine-tuned ResNet18 model to extract features for each patch.

[0080] ResNet18 is a deep learning model pre-trained on the ImageNet large-scale image database, which can effectively extract high-level features from images. The fine-tuned model can focus on the effective features of deep vein thrombosis. The feature vector of each patch is the node feature in the graph structure. These features contain rich information related to thrombosis, which is helpful for subsequent analysis and identification.

[0081]

[0082] Equation 3 is the fine-tuning process of the pre-trained model. In MRI data The model focused on the characteristics of deep vein thrombosis was obtained by adjusting Using the Model When the last classification layer is removed, the MRI image patch that constructs the graph structure is input into the model. And get its eigenvector.

[0083] S103: obtaining a spatial relationship between the thrombus and the surrounding tissue vectors based on the feature vectors of the thrombus and the surrounding tissues;

[0084] In one embodiment, the spatial relationship includes the spatial relationship between adjacent vectors in the same dimension, the spatial relationship between adjacent vectors in different dimensions, and the global three-dimensional spatial information relationship.

[0085] S104: Construct a graph structure based on the spatial relationship and feature vectors to obtain graph data.

[0086] In one embodiment, the graph structure is obtained by cross-layer connecting feature vectors at different levels through the spatial relationship between feature vectors.

[0087] In one embodiment, the construction process of the graph structure is specifically as follows:

[0088] S1: Convert the spatial coordinates of the feature vectors into original nodes;

[0089] S2: Based on the original nodes, traverse all their adjacent position nodes to obtain the spatial coordinates of the adjacent nodes and the spatial relationship between nodes;

[0090] S3: Connect the original nodes and the adjacent nodes with edges and update the original nodes with the spatial coordinates of the adjacent nodes, and repeat the above steps until the graph structure is obtained;

[0091] Optionally, the traversal adopts one or several of the following: depth-first traversal, breadth-first traversal, iterative deepening depth-first search;

[0092] In one embodiment, the construction of the graph structure further includes dynamic construction through a dynamic adjustment mechanism;

[0093] Optionally, the dynamic construction dynamically changes the connection mode of nodes and edges through the characteristics of thrombus.

[0094] In a specific embodiment, construct a graph structure based on the spatial relationship: After feature extraction, construct a graph structure according to the spatial relationship between Patches. Specifically, the feature vector representation points N expressed by Patches with an adjacent distance of 1 are connected by edges to form a spatially adjacent node relationship. Considering the three-dimensional characteristics of magnetic resonance images, in addition to adjacent Patches on the same layer, Patches at the same position on adjacent layers are also connected by edges to form a cross-layer connection structure. The graph structure constructed in this way not only considers local spatial information but also incorporates context information in the three-dimensional space, which helps to more comprehensively understand and analyze the distribution and characteristics of deep vein thrombosis. The specific process is as follows:

[0095] Let G = V, E be the constructed graph, where V is the set of nodes and E is the set of edges, and P x,y,z represents the image Patch located at the three-dimensional coordinates (x, y, z). For all (x, y, z) within the valid range, perform the following steps:

[0096] 1. Convert P x,y,z into node v x,y,z , and add it to V:

[0097] V ← V ∪ {v x,y,z}} (4)

[0098] Traverse the neighboring positions (x′, y′, z′), where x' ∈ {x - 1, x, x + 1}, y' ∈ {y - 1, y, y + 1}, z' ∈ {z - 1, z, z + 1}, and (x′, y′, z′) are all within the image boundaries. Also consider cross - layer adjacency when (x' = x, y' = y, z' = z ± 1):

[0099] If v x′,y′,z′ does not yet exist in V, then add it:

[0100] V ← V ∪ {v x′,y′,z′}} (5)

[0101] Add an edge to E to represent the spatial adjacency relationship:

[0102]

[0103] Through the above steps, the present invention constructs a graph structure G that takes into account local spatial information and three - dimensional context information, which helps to deeply analyze the distribution and characteristics of deep vein thrombosis.

[0104] The embodiments of the present disclosure also provide a computer program product or system, including a computer program that, when executed by a processor, implements the steps of the above - mentioned method based on thrombus image reconstruction map data.

[0105] Figure 2 A schematic diagram of a system based on thrombus image reconstruction map data provided by an embodiment of the present invention specifically includes:

[0106] An acquisition unit: acquires the thrombus image of a patient, and the image contains thrombus and surrounding tissues;

[0107] A feature extraction unit: extracts features from the image to obtain feature vectors;

[0108] A spatial relationship unit: obtains the spatial relationship between thrombus and surrounding tissue vectors based on the feature vectors of thrombus and surrounding tissues;

[0109] A construction unit: constructs a graph structure based on the spatial relationship and feature vectors to obtain graph data.

[0110] Figure 3 A schematic diagram of a device based on thrombus image reconstruction map data provided by an embodiment of the present invention specifically includes:

[0111] A memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions, and when the program instructions are executed, any of the above methods based on thrombus image reconstruction map data.

[0112] The disclosed embodiments of the present invention also provide a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, any of the above methods based on thrombus image reconstruction map data.

[0113] The disclosed embodiments of the present invention also provide a method for staging deep vein thrombosis based on a graph neural network, including:

[0114] The method obtains graph data based on the above method based on thrombus image reconstruction map data;

[0115] The graph data is input into a graph neural network for staging to obtain a staging result.

[0116] In one embodiment, deep vein thrombosis can be divided into the following three stages according to the onset time: acute stage, subacute stage, and chronic stage. Among the three stages of deep vein thrombosis (DVT), the distinction between the subacute stage and the chronic stage is sometimes difficult, mainly because the process from the subacute stage to the chronic stage is a continuous pathological process, and the thrombus gradually fibroses and may calcify. During this process, the physical and biochemical characteristics of the thrombus change gradually, and there is no clear time point to strictly distinguish them. Therefore, it may be challenging to accurately define the end of the subacute stage and the beginning of the chronic stage through imaging at a certain time point. At the same time, at the end of the subacute stage and in the chronic stage, the venous system may adapt to the blood flow disorder caused by the thrombus through collateral circulation, or the thrombus may partially recanalize. These changes may be manifested as complex blood flow patterns in the imaging, making it more difficult to directly distinguish the two stages based on the signal characteristics of the thrombus.

[0117] Therefore, the present invention performs staging through a method for staging deep vein thrombosis based on a graph neural network. Among them, the graph data of the image is constructed through three-dimensional relationships, comprehensively capturing spatial structure information. It not only considers local spatial proximity but also integrates three-dimensional spatial context information, enabling the model to more comprehensively understand the spatial distribution and morphological characteristics of the thrombus, enhancing the recognition ability of the complex structure of the thrombus; secondly, the introduction of the graph convolution layer design and the graph attention mechanism, the graph convolution layer promotes the effective propagation of information in the graph structure, while the graph attention mechanism enables the model to dynamically focus on important nodes and edges, increasing the robustness and adaptability of the model in the face of complex image changes, and further improving the expression ability of the model. Finally, differentiable pooling is used to aggregate graph node information, enhancing the model's recognition ability for deep vein thrombosis.

[0118] In one embodiment, the structure of the graph neural network is as follows. The spatial relationship graph data reconstructed from the nuclear magnetic resonance image is input into the network. Feature learning is performed through the graph convolutional layer, and network attention is allocated through the graph attention mechanism, enabling the model to pay more attention to important nodes and edges, obtaining the learned node features. Then, through the differentiable pooling layer, the graph node information is aggregated, and through the output layer, staging classification is performed to obtain the staging result. Figure 5 As shown in Figure 5 , the spatial relationship graph data reconstructed from the nuclear magnetic resonance image is input into the network. Feature learning is performed through the graph convolutional layer, and network attention is allocated through the graph attention mechanism, enabling the model to pay more attention to important nodes and edges, obtaining the learned node features. Then, through the differentiable pooling layer, the graph node information is aggregated, and through the output layer, staging classification is performed to obtain the staging result.

[0119] In one embodiment, the graph data is input into the graph neural network for staging to obtain the staging result.

[0120] Furthermore, the graph neural network includes an input layer, a graph convolutional layer, a graph attention layer, a pooling layer, and an output layer. The graph data passes through the input layer, the graph convolutional layer, the graph attention layer, the pooling layer, and the output layer in sequence to obtain the staging result.

[0121] Optionally, the graph convolutional layer performs convolution on the weights of the nodes and edges in the graph data to obtain feature information between different positions and different levels. The feature information is sorted by feature weights through the graph attention layer to obtain important node features and edge features. The node features and edge features are input into the pooling layer and the output layer to obtain the staging result.

[0122] Optionally, the pooling layer adopts one or more of the following: average pooling, max pooling, differentiable pooling.

[0123] Optionally, the differentiable pooling aggregates the respective node features to obtain the graph feature, and the graph feature passes through the output layer for staging to obtain the staging result.

[0124] Optionally, the output layer includes one or more of the following: softmax, sigmoid, multi-layer perceptron.

[0125] Optionally, the multi-layer perceptron determines the stage of the deep vein thrombosis by recognizing the features of the deep vein thrombosis graph structure.

[0126] In one embodiment, the multi-layer perceptron is a trained multi-layer perceptron. By obtaining deep vein thrombosis images of different stages, and then converting the deep vein thrombosis images of different stages into graph structure data of different stages, the multi-layer perceptron is trained through the graph structure data of different stages for morphological features and spatial hierarchical features to obtain the trained multi-layer perceptron.

[0127] In one embodiment, the graph neural network is obtained by performing fusion processing on N graph neural networks with different architectures or parameters, where N is a natural number greater than 1.

[0128] Optionally, the fusion processing includes one or more of the following: voting, averaging, weighted averaging.

[0129] In one embodiment, the graph data is input into a graph neural network for training, where the graph data includes acute-phase data, sub-acute-phase data, and chronic-phase data of deep vein thrombosis.

[0130] The trained graph neural network model is obtained by training the graph neural network with graph data of deep vein thrombosis in different periods.

[0131] In a specific embodiment, the soft tissue in magnetic resonance images has high contrast and rich information. Therefore, the soft tissue in magnetic resonance images (including blood vessels, thrombus, blood, fat, muscle, etc.) is very complex. This complex and rich soft tissue contrast provides sufficient information for the deep learning method to stage thrombus. The traditional average pooling method averages most of the information in the whole image, resulting in a large reduction of these contrast information, thus ignoring local key features. The differentiable pooling method can aggregate the feature information of each part, thereby retaining local key features that are easily ignored by the average pooling method, more precisely capturing tiny but important soft tissue features in magnetic resonance images, adapting to the complex contrast of different soft tissues, and thus improving the accuracy of feature extraction. And it allows the effective propagation of gradients, which helps the network optimize parameters during training and enhance the sensitivity and recognition ability to different soft tissue contrasts.

[0132] In a specific embodiment, the design of the graph convolutional layer: In the graph neural network, the present invention adopts a graph convolutional layer (GCN), which is based on node features and edge weights to realize the diffusion and aggregation of information in the graph. The propagation of feature information between different positions and hierarchical patches is achieved through graph convolution. Since there are a large number of low-information images in the nuclear magnetic resonance images of deep vein thrombosis, the present invention introduces a graph attention mechanism to enhance the model's attention to important nodes and edges. The calculation formula for the attention coefficient in the graph attention layer is:

[0133]

[0134] where a ij is the attention vector, which is the dot product of the vector after concatenating the feature vectors of two nodes and is used to calculate the attention score; LeakyReLU is the activation function to increase non-linearity. e ij This represents the measure of the correlation or similarity between node i and node j, which is the weight of the edge between the two. In the formula, i, j, and k represent different nodes respectively. e ik This represents the measure of the correlation or similarity between node i and node k, which is the weight of the edge between the two. Ni is the neighbor set of node i.

[0135]

[0136] Equation 9, in combination with Equation 8, performs three node feature updates on node i. Through the influence of the attention weight a ij , the node can pay more attention to thrombus-related information during the propagation of feature information.

[0137] Graph node information aggregation: After the information propagation of graph convolution, the present invention aggregates the feature information of each node by using differentiable pooling to replace traditional average pooling to form the feature representation of the entire graph, and aggregates the node features into cluster features to represent the specific information of different tissues in the magnetic resonance image of deep vein thrombosis.

[0138] H ′(l) = softmax(H (l) W (l) )H (l) (10)

[0139] Equation 10 uses differentiable pooling to aggregate the node features input by the last layer of graph convolution into different clusters to obtain the cluster feature H ′(l) . Among them, W (l) is a learnable weight matrix, and the softmax operation ensures that the sum of each row is 1, forming an effective probability distribution.

[0140] Output layer: Finally, the present invention adds a multi-layer perceptron to output the diagnostic result. After training, the multi-layer perceptron can comprehensively identify the special image features in the specific disease development of deep vein thrombosis in the target magnetic resonance image based on the morphological features and spatial hierarchical features in the graph structure, so as to be used to judge the development stage of deep vein thrombosis.

[0141] In one embodiment, after reconstructing the image data into graph data, the graph data is input into a graph convolutional network for feature learning to obtain a graph feature vector, and then attention distribution is performed on the graph feature vector through graph attention. More attention is allocated to the nodes and edges with large weights to obtain graph attention features. Then, the graph attention features are input to the differentiable pooling layer for information aggregation of each node to obtain aggregation features. Finally, the aggregation features are input to the multi-layer perceptron for feature classification to obtain the classification and staging results. The staging results include: acute stage, subacute stage, and chronic stage.

[0142] In one embodiment, after reconstructing the image data into graph data, the present invention inputs the graph data into a graph convolutional neural network for deep vein thrombosis staging classification to obtain the staging result. The graph convolutional neural network is a commonly used network structure.

[0143] In another embodiment, after reconstructing the image data into graph data, the present invention inputs the graph data into a graph convolutional neural network with differentiable pooling for deep vein thrombosis staging classification to obtain the staging result.

[0144] In another embodiment, after reconstructing the image data into graph data, the present invention inputs the graph data into a graph convolutional neural network for graph attention allocation to perform deep vein thrombosis staging classification to obtain a staging result.

[0145] In another embodiment, after reconstructing the image data into graph data, the present invention inputs the graph data into a graph convolutional neural network that performs graph attention allocation and differentiable pooling to perform deep vein thrombosis staging classification to obtain a staging result.

[0146] In one embodiment, the present invention constructs graph data by extracting the three-dimensional spatial relationship of the image data for cross-layer connection, and inputs the graph data into a graph convolutional neural network for differentiable pooling to perform deep vein thrombosis staging classification to obtain a staging result.

[0147] In another embodiment, the present invention constructs graph data by extracting the three-dimensional spatial relationship of the image data for cross-layer connection, and inputs the graph data into a graph convolutional neural network that performs graph attention allocation and differentiable pooling to perform deep vein thrombosis staging classification to obtain a staging result.

[0148] The disclosed embodiments of the present invention also provide a computer device, including:

[0149] A memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions, and when the program instructions are executed, any of the above-mentioned deep vein thrombosis staging methods based on graph neural networks.

[0150] The disclosed embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned deep vein thrombosis staging methods based on graph neural networks.

[0151] The verification results of this verification embodiment show that assigning inherent weights to the indications can improve the performance of the present method compared to the default settings. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways.

[0152] For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings, direct couplings, or communication connections shown or discussed among each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units. Those of ordinary skill in the art can understand that all or part of the steps in the above-described methods of the embodiments can be completed by instructing relevant hardware through a program, and this program can be stored in a computer-readable storage medium. The storage medium can include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk, or optical disk, etc.

[0153] Those of ordinary skill in the art can understand that all or part of the steps in the above-described methods of the embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The above-mentioned medium storage can be read-only memory, magnetic disk, or optical disk, etc.

[0154] The above has introduced in detail a computer device provided by the present invention. For those of ordinary skill in the art, according to the idea of the embodiments of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A deep vein thrombosis staging method based on a graph convolutional neural network, characterized in that, Including: Obtain the thrombus image of the patient, where the image contains the thrombus and surrounding tissues; Extract features from the image to obtain feature vectors; Based on the feature vectors of the thrombus and surrounding tissues, obtain the spatial relationship between the thrombus and surrounding tissue vectors; Based on the spatial relationship and feature vectors, construct a graph structure to obtain graph data; The graph structure is obtained by cross-layer connecting feature vectors of different levels through the spatial relationship between feature vectors; the spatial relationship includes the spatial relationship between adjacent vectors of the same dimension, the spatial relationship between adjacent vectors of different dimensions, and the global three-dimensional spatial information relationship; The vertices of the graph structure represent three-dimensional coordinates, and the edges represent spatial adjacency relationships; The graph data is input into a graph neural network for staging to obtain a staging result; The graph neural network includes an input layer, a graph convolutional layer, a graph attention layer, a pooling layer, and an output layer, and the graph data sequentially passes through the input layer, the graph convolutional layer, the graph attention layer, the pooling layer, and the output layer; The graph convolutional layer performs convolution on the weights of the nodes and edges in the graph data to obtain feature information between different positions and different levels; the feature information is sorted by feature weights through the graph attention layer to obtain important node features and edge features; the node features and edge features are input into the pooling layer; The pooling layer uses differentiable pooling, and the differentiable pooling aggregates each node feature to obtain a graph feature, and the graph feature is input into the output layer; The output layer is a multi-layer perceptron, and the multi-layer perceptron determines the stage of deep vein thrombosis by identifying the features of the deep vein thrombosis graph structure; The multi-layer perceptron is a trained multi-layer perceptron. By obtaining deep vein thrombosis images of different stages, and then converting the deep vein thrombosis images of different stages into graph structure data of different stages, the multi-layer perceptron is trained through the graph structure data of different stages to obtain a trained multi-layer perceptron.

2. The deep vein thrombosis staging method based on a graph convolutional neural network according to claim 1, wherein The image includes one or more of the following: MRI, CT.

3. The deep vein thrombosis staging method based on a graph convolutional neural network according to claim 1, wherein The specific process of constructing the graph structure is as follows: S1: Convert the spatial coordinates of the feature vector into original nodes; S2: Based on the original nodes, traverse all adjacent position nodes to obtain the spatial coordinates of the adjacent nodes and the spatial relationship between the nodes; S3: Connect the original nodes and adjacent nodes with edges and update the original nodes with the spatial coordinates of the adjacent nodes, and repeat the above steps until the graph structure is obtained.

4. The deep vein thrombosis staging method based on a graph convolutional neural network according to claim 3, wherein, The traversal adopts one or more of the following: depth-first traversal, breadth-first traversal, iterative deepening depth-first search.

5. The method for staging deep vein thrombosis based on a graph convolutional neural network according to claim 3, wherein The construction of the graph structure also includes dynamic construction through a dynamic adjustment mechanism.

6. The method for staging deep vein thrombosis based on a graph convolutional neural network according to claim 5, wherein The dynamic construction dynamically changes the connection method of nodes and edges through the characteristics of the thrombus.

7. The deep vein thrombosis staging method based on a graph convolutional neural network according to claim 1, wherein The feature extraction obtains feature vectors by using a pre-trained model to extract features of the thrombus and surrounding tissues in the image data.

8. The method for staging deep vein thrombosis based on a graph convolutional neural network according to claim 7, wherein The pre-trained model includes any one or more of the following: ResNet18, ResNet34, ResNet50, VisionTransformer, DenseNet, VGGNet, Inception ResNet.

9. The method for staging deep vein thrombosis based on a graph convolutional neural network according to claim 8, wherein The pre-trained model performs feature extraction based on the ResNet18 model. First, deep vein thrombosis image data is obtained and input into the pre-trained ResNet18 model for adjustment to obtain an adjusted pre-trained ResNet18 model. Feature vectors are obtained based on the adjusted pre-trained ResNet18 model for feature extraction.

10. The method for staging deep vein thrombosis based on a graph convolutional neural network according to claim 1, wherein The method further includes data preprocessing. The image data of deep vein thrombosis patients is preprocessed to obtain processed data, and feature vectors are obtained by performing feature extraction on the thrombus and surrounding tissues in the processed data.

11. The method for staging deep vein thrombosis based on a graph convolutional neural network according to claim 10, wherein The data preprocessing includes one or more of the following: image size normalization, image equalization, image background removal, and image enhancement.

12. The deep vein thrombosis staging method based on a graph convolutional neural network according to claim 11, wherein The image size normalization expands the image size by padding with zero values or interpolation.

13. The deep vein thrombosis staging method based on a graph convolutional neural network according to claim 11, wherein The image background removal removes the background and / or non-interested regions through a global threshold method.

14. The deep vein thrombosis staging method based on a graph convolutional neural network according to claim 1, wherein The pooling layer adopts one or more of the following: average pooling, max pooling.

15. The deep vein thrombosis staging method based on a graph convolutional neural network according to claim 1, wherein The output layer includes one or more of the following: softmax, sigmoid.

16. The method for staging deep vein thrombosis based on a graph convolutional neural network according to claim 1, wherein The graph neural network is obtained by performing fusion processing on N graph neural networks with different architectures or parameters, where N is a natural number greater than 1.

17. The deep vein thrombosis staging method based on a graph convolutional neural network according to claim 16, wherein The fusion processing includes one or more of the following: voting, averaging, weighted averaging.

18. A computer program product having a computer program or instructions thereon, characterized in that, The computer program or instruction is executed by a processor to implement the deep vein thrombosis staging method based on a graph neural network according to any one of claims 1-17.

19. A computer device, comprising a memory, a processor, and a computer program or instruction stored on the memory, characterized in that, Including: The processor executes the computer program or instruction to implement the deep vein thrombosis staging method based on a graph neural network according to any one of claims 1-17.

20. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, The computer program or instruction is executed by a processor to implement the deep vein thrombosis staging method based on a graph neural network according to any one of claims 1-17.

Images