An image graph representation and graph neural network-based bone joint health status evaluation method and system
By combining image pixel clustering and graph neural networks, the problems of subjectivity and poor interpretability of deep learning models in the assessment of knee osteoarthritis are solved, achieving high-precision and rapid assessment of joint health status and improving the interpretability of the model.
Patent Information
- Application Number
- CN202211265611.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-17
- Publication Date
- 2025-12-05
- Estimated Expiration
- 2042-10-17
AI Technical Summary
Existing methods for assessing knee osteoarthritis suffer from high subjectivity, poor interpretability of deep learning models, vanishing gradients, and overfitting, which limit the representational capabilities of graph neural networks on large-scale graphs.
An image pixel clustering algorithm is used to convert images into rectangular nodes. Combined with a graph neural network, the health status of bones and joints is assessed. Node information and adjacency matrix are obtained through the image pixel clustering algorithm. A six-layer graph attention network with residual structure is used for evaluation, and a heat map of the region of interest is visualized.
It improves the accuracy and speed of evaluation, reduces computational complexity, enhances the interpretability and credibility of the model, and enables a better understanding of the model's working mechanism.
Smart Images
Figure CN115527677B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of graph neural networks, and particularly relates to a knee joint health state evaluation method and system based on image graph representation and graph neural networks. BACKGROUND
[0002] Knee osteoarthritis (OA) is the most common musculoskeletal disease. The knee joint ranks 11th in the global disability factor, thus causing a great economic burden to the society. It is reported that the total cost of OA treatment of each patient reaches 19000 euros / year, or even more. The health of knee joint osteoarthritis is also increasingly concerned by people, and the evaluation of bone joint health is currently mainly performed by means of question and answer and X-ray image, and the evaluation process has a certain subjectivity.
[0003] From the current point of view, a solution based on convolutional neural network image classification has been proposed to evaluate the severity of knee osteoarthritis. And the graph neural network (GNN) is a powerful tool for machine learning on graphs. GNN has been successfully applied to graph classification, semantic segmentation and machine translation, etc. Meanwhile, many variant networks have been developed on the basis of GNN, such as graph convolution network (GCN) and graph attention network (GAT), wherein the GAT effectively solves the problem that the GCN cannot process dynamic graphs by performing attention operation on the neighbor nodes of each node. The graph neural network also has many problems, such as the problems of gradient disappearance, over-smoothing and over-fitting of deep graph neural networks. These challenges limit the representation ability of GCN on large-scale graphs. It has been found through research that the problems can be effectively solved by residual and dense connection modules.
[0004] The graph neural network such as graph convolutional neural network and graph attention network recursively passes neural messages along the edges of the input graph, combines the node feature information with the graph structure, and the combination of the graph structure and the feature information will lead to a complex model. The explainability of the deep learning model provides human understandable reasoning for its prediction. If the reasons behind the prediction are not explained, the deep learning algorithm is like a black box and cannot be trusted for some scenarios. Through a reasonable interpreter, it can better understand how the model works behind. SUMMARY
[0005] In view of the deficiencies of the prior art, the present application provides a bone joint health state evaluation method and system based on image graph representation and graph neural network, which comprises the following steps: obtaining a bone joint image and performing graph representation on the bone joint image to obtain node information, edge information and an adjacency matrix of the bone joint image; inputting the node information, edge information and adjacency matrix of the bone joint image into a trained bone joint health state evaluation model based on a graph neural network to obtain an optimal bone joint health state evaluation result; and when the trained model is used to evaluate the bone joint image, a heat map of a bone joint image attention area of the graph neural network can be visualized.
[0006] The process of training the bone joint health state evaluation model based on the graph neural network comprises the following steps:
[0007] S1: obtaining bone joint images with labels, wherein the labels are description labels of bone joint health grades; and dividing the bone joint images with labels into a training set and a test set;
[0008] S2: positioning a knee part in the bone joint image, and dividing each image into a right knee image and a left knee image, wherein each divided image has a label;
[0009] S3: performing graph representation on each image to obtain graph structure information, including node information, edge information and an adjacency matrix;
[0010] S4: inputting the graph structure information of the bone joint images in the training set into the graph neural network for training; inputting the graph structure information of the bone joint images in the test set into the model for evaluation to obtain an optimal bone joint health state evaluation result and an optimal bone joint health state evaluation model.
[0011] Preferably, the process of performing graph representation on each image comprises the following steps:
[0012] performing image pixel clustering on the image to cluster the image into rectangular structures of different sizes, wherein each rectangle is represented as a node of a graph, and if two rectangles have an intersection, it means that the two nodes have a connected edge;
[0013] calculating node information of all nodes in each image, wherein the node information includes a center coordinate, a width, an average pixel, a maximum pixel, a minimum pixel value, a pixel variance and a center point gradient;
[0014] calculating edge information in each image, wherein the edge information includes a total sum of pixels of two rectangles and a proportion of overlapping pixels of the two rectangles to the total sum of pixels;
[0015] calculating an adjacency matrix of each image according to the in-degree and out-degree of each node in the image.
[0016] Further, the process of clustering the image into rectangular structures of different sizes by using the image pixel clustering algorithm comprises:
[0017] Step A: setting a pixel threshold T, a purity threshold P and a variance threshold V;
[0018] Step B: randomly selecting a pixel point in the image as a center pixel point, calculating the pixel difference t of the center pixel point and its surrounding pixel points, if t is less than T, then the point is taken as a heterogeneous point;
[0019] Step C: calculating the purity p of the center pixel point and its surrounding pixel points according to the ratio of the heterogeneous point to the surrounding pixel points and according to the ratio, calculating the pixel variance v of the center pixel point and its surrounding pixel points;
[0020] Step D: if p is less than P and v is greater than V, the current region is divided into a rectangle and returns to step B; otherwise, the region is expanded around and step C is repeated;
[0021] Step E: judging whether there is a matrix containing all pixel points, if yes, the clustering is completed, otherwise, returning to step B.
[0022] Further, the formula for calculating the purity of the center pixel point and its surrounding pixel points is:
[0023]
[0024] Wherein, p represents purity, s represents the number of heterogeneous points, and n represents the number of surrounding pixel points.
[0025] Preferably, the process of obtaining the best bone joint health status evaluation result comprises:
[0026] Mapping the node information and edge information of the bone joint image to 256 dimensions through a linear layer to obtain a mapping matrix;
[0027] Inputting the mapping matrix and the adjacency matrix into a six-layer graph attention network with residual structure to obtain the feature representation of each node in the graph, calculating the feature representation of the graph according to the feature representation of each node; finally, classifying the graph to obtain the best bone joint health status evaluation result.
[0028] Preferably, the process of visualizing the heat map of the attention area of the bone joint image by the graph neural network comprises:
[0029] Performing graph representation on the bone joint image to obtain the graph structure information of the bone joint image;
[0030] The graph structure information is input into an optimal model for evaluation to obtain heat map information, and an attention heat map is drawn on the original image according to the heat map information; wherein the heat map information includes an attention score of each node, region information on the original image, position coordinates of the node and a size of a rectangular frame.
[0031] An osteoarticular health state evaluation system based on an image graph representation and a graph neural network, comprising: an osteoarticular image acquisition module, a graph representation module, an evaluation module and an interpretation module.
[0032] The osteoarticular image acquisition module is used for acquiring an osteoarticular image to be evaluated.
[0033] The graph representation module is used for graph representation of the osteoarticular image to obtain graph structure information.
[0034] The evaluation module is used for osteoarticular health state evaluation according to the graph structure information to obtain an osteoarticular health state evaluation result.
[0035] The interpretation module is used for interpretation of the osteoarticular image, and visualizes a heat map of an attention region of the osteoarticular image.
[0036] The present application has the beneficial effects that: the present application designs a novel image representation method, which converts an osteoarticular image into a graph structure, and uses a graph neural network to realize image classification, compared with the prior art, the present application can obtain part of semantic information in the image in advance through the pixel value clustering graph representation method, compared with the method of directly taking the pixel value as a node and the adjacent nodes existing an edge, the method can reduce the number of nodes and the complexity of calculation. BRIEF DESCRIPTION OF DRAWINGS
[0037] Figure 1 The present application is based on an image graph representation and a graph neural network osteoarticular health state evaluation method flow chart;
[0038] Figure 2 The present application is based on an image graph representation and a graph neural network osteoarticular health state evaluation method structure schematic view;
[0039] Figure 3 The present application is a residual structure graph attention network structure diagram. DETAILED DESCRIPTION
[0040] With reference to the accompanying drawings: clearly and completely describe the technical solutions in the embodiments of the present application, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.
[0041] The present application proposes a bone joint health state evaluation method and system based on image graph representation and graph neural network, as shown in Figure 1 、 Figure 2 The method comprises: acquiring a bone joint image to be evaluated and performing graph representation on the bone joint image to obtain node information, edge information and an adjacency matrix of the bone joint image; inputting the node information, edge information and adjacency matrix of the bone joint image into a trained bone joint health state evaluation model based on a graph neural network to obtain an optimal bone joint health state evaluation result; and visualizing a heat map of a bone joint image attention area of the graph neural network when the trained model is used to evaluate the bone joint image.
[0042] The process of training the bone joint health state evaluation model based on the graph neural network comprises the following contents:
[0043] S1: acquiring a bone joint image with a label, wherein the label is a description mark of a bone joint health grade; and dividing the bone joint image with the label into a training set and a test set;
[0044] The bone joint image with the label can be a knee joint X-ray picture downloaded from a bone joint database, and the label is a description mark of a bone joint health grade; the bone joint image is labeled according to the bone joint health grade, and each image has two labels, which are respectively a left knee label and a right knee label; preferably, the bone joint health grade can be divided into five grades, which are respectively “0, 1, 2, 3, 4”, and the smaller the number is, the healthier the bone joint is; and all the images are divided into the training set and the test set according to a ratio of 8:2.
[0045] S2: positioning the knee part in the bone joint image, and dividing each image into a right knee image and a left knee image, wherein the divided images each have a label.
[0046] The knee part in the image is positioned by using the BoneFinder software, and for the knee part that cannot be positioned or has poor positioning effect by using the software, the positioning is adjusted manually, so that each image is divided into a left knee image and a right knee image, and the divided images each correspond to a label.
[0047] S3: performing graph representation on each image to obtain graph structure information, including node information, edge information and an adjacency matrix.
[0048] The image is clustered into rectangular structures of different sizes by using an image pixel clustering algorithm, each rectangle is represented as a node of a graph, if two rectangles have an intersection, it means that there is a connection between the two nodes, and a graph is obtained by abstracting the rectangular structure; the process of clustering the image into rectangular structures of different sizes by using the image pixel clustering algorithm includes:
[0049] Step A: set the pixel threshold T, the purity threshold P and the variance threshold V;
[0050] Step B: randomly select a pixel point in the image as the center pixel point, calculate the pixel difference t between the center pixel point and its surrounding pixel points, if t is less than T, the point is regarded as a heterogeneous point;
[0051] Step C: calculate the purity p of the region composed of the center pixel point and its surrounding pixel points according to the ratio of the heterogeneous point to the surrounding pixel points and according to the ratio; calculate the pixel variance v of the center pixel point and its surrounding pixel points;
[0052] The formula for calculating the purity of the region composed of the center pixel point and its surrounding pixel points is:
[0053]
[0054] Wherein, p represents the purity, s represents the number of heterogeneous points, and n represents the number of surrounding pixel points.
[0055] Step D: if p is less than P and v is greater than V, the current region is divided into a rectangle and returns to step B; otherwise, the region is expanded, and step C is repeated;
[0056] Step E: determine whether there is a matrix containing all pixel points, if yes, the clustering is completed, otherwise, return to step B.
[0057] Calculate the node information of all nodes in each image; the node information includes: center coordinates, width, average pixel, maximum pixel, minimum pixel value, pixel variance and center point gradient; wherein the process of calculating the center point gradient includes: using Sobel operator to calculate the gradient graph of the center point x and y direction of the image respectively, and then calculating the gradient mean value of the corresponding position on the gradient graph of x direction and y direction as the gradient graph of the center of the image.
[0058] Calculate the edge information in each image; the edge information includes the total sum of pixels of two rectangles and the proportion of overlapping pixels of two rectangles to the total sum of pixels; save the edge information, the dimension of the edge information of each graph is (k, 2), k represents the number of edges.
[0059] The adjacency matrix of each image can be obtained by calculating the in-degree and out-degree of each node in each image. The adjacency matrix of the graph is then converted into COO data format with dimensions (2, 2*n), which stores the nodes that are connected, where n represents the number of nodes in the graph.
[0060] S4: Input the graph structure information of the bone and joint images in the training set into the graph neural network for training; input the graph structure information of the bone and joint images in the test set into the model for evaluation, and obtain the best bone and joint health status assessment result and the best bone and joint health status assessment model.
[0061] The node information, edge information, and adjacency matrix obtained after representing each image graphically are input into a graph neural network-based bone and joint health status assessment model for training. First, the node information and edge information of the bone and joint images in the training set are mapped to 256 dimensions through a linear layer to obtain a mapping matrix;
[0062] like Figure 3 As shown, the mapping matrix and adjacency matrix are input into a six-layer graph attention network with residual structure to obtain the feature representation of each node in the graph. Based on the feature representation of each node, the feature representation of the graph is calculated. Specifically, a method based on a global attention mechanism is used to calculate the features of all nodes in the graph to obtain the feature representation of the graph. Finally, the graph is classified to obtain the best assessment result of bone and joint health status.
[0063] The graph structure information of bone and joint images in the test set is input into the model for evaluation. If the current evaluation result is the best, the current model is saved; otherwise, training continues until the best bone and joint health status evaluation model is obtained.
[0064] When evaluating bone and joint images using a trained model, the process of visualizing the heatmap of the region of interest in the bone and joint image using a graph neural network includes:
[0065] By training the model, an optimal evaluation model can be obtained. Using this model to infer and predict the graph structure information of bone and joint images, heatmap information can be generated. This includes the attention score of each node in the graph, the region information of each node in the original image, the node's position coordinates, and the size of the bounding box. Finally, an attention heatmap can be drawn on the original image based on the heatmap information. On the attention heatmap, the redder the color, the more attention the model pays to the region during inference; conversely, the bluer the color, the less attention the model pays to the region. The attention heatmap allows for the interpretation of the model, increasing its reliability.
[0066] The bone joint image to be evaluated is acquired by using an X-ray machine, the bone joint image is subjected to the above graph representation processing, and the graph structure information obtained by the graph representation, i.e., the node information and the edge information, is input into the trained bone joint health state evaluation model based on the graph neural network, so that the bone joint health state evaluation result of the bone joint image to be evaluated can be obtained, and at the same time, the most concerned area of the model for the image can be observed through the attention heat map, and the reliability of the bone joint health state evaluation result can be judged according to the attention heat map.
[0067] The bone joint health state of people can be detected by using the application, and at the same time, the bone joint health management can also be performed according to the bone joint health state evaluation result obtained from the application.
[0068] The application further provides a bone joint health state evaluation system based on image graph representation and graph neural network, which is used for executing the above bone joint health state evaluation method based on image graph representation and graph neural network, and includes a bone joint image acquisition module, a graph representation module, an evaluation module and an explanation module.
[0069] The bone joint image acquisition module is used for acquiring the bone joint image to be evaluated, preferably, the bone joint image acquisition module acquires the bone joint image to be evaluated by using an X-ray machine.
[0070] The graph representation module is used for performing graph representation on the bone joint image to obtain graph structure information.
[0071] The evaluation module is used for performing bone joint health state evaluation according to the graph structure information to obtain a bone joint health state evaluation result.
[0072] The explanation module is used for explaining the bone joint image, and visualizes the heat map of the concerned area of the bone joint image.
[0073] The above embodiments further specifically describe the purpose, technical scheme and advantages of the application, and it should be understood that the above embodiments are only preferred embodiments of the application, and are not used to limit the application, and any modification, equivalent replacement, improvement, etc. made to the application within the spirit and principle of the application should be included in the protection scope of the application.
Claims
1. A method for bone joint health status assessment based on image graph representation and graph neural network, characterized in that, The method comprises the following steps: Obtain the bone joint image and perform graph representation to obtain node information, edge information and adjacency matrix of the bone joint image; Input the node information, edge information and adjacency matrix of the bone joint image into the trained bone joint health state evaluation model based on the graph neural network to obtain the best bone joint health state evaluation result; when the trained model is used to evaluate the bone joint image, the heat map of the attention area of the graph neural network to the bone joint image can be visualized; The process of training the bone joint health state evaluation model based on the graph neural network comprises the following steps: S1: Obtain bone joint images with labels, wherein the labels are description labels of the health grades of the bone joints; divide the bone joint images with labels into a training set and a test set; S2: Position the knee part in the bone joint image, and divide each image into a right knee image and a right knee image; the divided images are each labeled; S3: Perform graph representation on each image to obtain graph structure information, including node information, edge information and adjacency matrix; the process of performing graph representation on each image comprises the following steps: Use an image pixel clustering algorithm to cluster the image into rectangular structures of different sizes, represent each rectangle as a node of a graph, and if two rectangles have an intersection, it means that the two nodes have a connected edge; the process of using the image pixel clustering algorithm to cluster the image into rectangular structures of different sizes comprises the following steps: Step A: set a pixel threshold T, a purity threshold P and a variance threshold V; Step B: randomly select a pixel point in the image as a center pixel point, calculate the pixel difference t of the center pixel point and its surrounding pixel points, and if t is less than T, the point is an alien point; Step C: calculate the purity p of the region composed of the center pixel point and its surrounding pixel points according to the ratio of the alien point to the surrounding pixel points, and calculate the pixel variance v of the center pixel point and its surrounding pixel points; the formula for calculating the purity of the region composed of the center pixel point and its surrounding pixel points is: Wherein, p represents the purity, s represents the number of alien points, and n represents the number of surrounding pixel points; Step D: if p is less than P and v is greater than V, divide the current region into a rectangle and return to step B; otherwise, expand the region and repeat step C; Step E: determine whether there is a matrix containing all pixel points, and if yes, the clustering is completed, otherwise, return to step B; Calculate the node information of all nodes in each image; the node information includes: center coordinates, width, average pixel, maximum pixel, minimum pixel value, pixel variance and center point gradient; Calculate the edge information in each image; the edge information includes the total pixel sum of two rectangles and the proportion of the overlapping pixel of the two rectangles to the total pixel sum; Calculate the adjacency matrix of each image according to the in-degree and out-degree of each node in the image; S4: input the graph structure information of the bone joint images in the training set into the graph neural network for training; input the graph structure information of the bone joint images in the test set into the model for evaluation to obtain the best bone joint health state evaluation result and the best bone joint health state evaluation model.
2. The bone joint health status evaluation method based on image graph representation and graph neural network according to claim 1, characterized in that, The process of obtaining the best bone joint health state evaluation result comprises the following steps: The node information and edge information of the bone joint image are mapped to 256 dimensions through a linear layer to obtain a mapping matrix; The mapping matrix and the adjacency matrix are input into a six-layer graph attention network with a residual structure to obtain the feature representation of each node in the graph, the feature representation of the graph is calculated according to the feature representation of each node, and finally the graph is classified to obtain the best bone joint health state evaluation result.
3. The method of claim 1, wherein the method is characterized by: The process of visualizing the heat map of the attention region of the bone joint image by the visual graph neural network includes: The bone joint image is represented as a graph to obtain the graph structure information of the bone joint image; The graph structure information is input into the best model for evaluation to obtain heat map information, and the attention heat map is drawn on the original image according to the heat map information; wherein the heat map information includes the attention score of each node, the region information on the original image, the position coordinates of the node and the size of the rectangular frame.
4. A bone joint health status evaluation system based on image graph representation and graph neural network, used for executing the bone joint health status evaluation method based on image graph representation and graph neural network in any one of claims 1-3. It includes: a bone joint image acquisition module, a graph representation module, an evaluation module and an interpretation module; The bone joint image acquisition module is used to acquire the bone joint image to be evaluated; The graph representation module is used to represent the bone joint image as a graph to obtain the graph structure information; The evaluation module is used to evaluate the bone joint health state according to the graph structure information to obtain the bone joint health state evaluation result; The interpretation module is used to interpret the bone joint image and visualize the heat map of the attention region of the bone joint image.