Femoral head necrosis area determination model acquisition method, necrosis area determination method, device and equipment

Abstract

The invention relates to the technical field of medical image segmentation and machine learning, and discloses a femoral head necrosis area determination model obtaining method, a necrosis area determination method, a device and equipment, and the model obtaining method comprises the steps: obtaining an image feature vector of a magnetic resonance imaging image, and inputting the image feature vector into an image neural network; forming a node set according to the image feature vector corresponding to each magnetic resonance imaging image, and constructing a graph structure based on the node set; wherein the magnetic resonance imaging image is a femoral head necrosis related image; according to the nodes in the node set and the graph structure, obtaining preliminary graph matching features in combination with a second-order graph matching algorithm; and training a graph neural network based on the preliminary graph matching features to obtain a femoral head necrosis region determination model. According to the method, the femoral head necrosis area can be accurately captured in the absence of large-scale manual labeling.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image segmentation and machine learning, and particularly relates to a femoral head necrosis region determination model acquisition method, a necrosis region determination method, device and equipment. BACKGROUND

[0002] Femoral head necrosis is a common orthopedic disorder, which is mainly caused by insufficient blood supply to the femoral head, leading to bone tissue necrosis and further loss of joint function. Early diagnosis and quantitative assessment are particularly critical for timely treatment. At present, although magnetic resonance imaging (MRI) is widely used for the diagnosis of femoral head necrosis, existing image analysis methods mostly rely on subjective judgment of doctors, which not only increases subjectivity but also increases the complexity of operation, thereby limiting the possibility of large-scale automatic analysis and accurate quantitative assessment.

[0003] In addition, although deep learning technology has shown significant advantages in medical image segmentation tasks, there are still many challenges for automatic segmentation and quantitative analysis of femoral head necrosis. These problems include high data labeling cost and poor domain adaptability. SUMMARY

[0004] Therefore, the embodiments of the present application provide a femoral head necrosis region determination model acquisition method, a necrosis region determination method, device and equipment, which can accurately capture the femoral head necrosis region in the absence of large-scale manual labeling.

[0005] In the first aspect, the embodiments of the present application provide a femoral head necrosis region determination model acquisition method, comprising:

[0006] obtaining an image feature vector of a magnetic resonance imaging image and inputting a graph neural network;

[0007] forming a node set according to the image feature vector corresponding to each magnetic resonance imaging image, and constructing a graph structure based on the node set; wherein the magnetic resonance imaging image is a femoral head necrosis related image;

[0008] obtaining a preliminary graph matching feature according to the nodes in the node set and the graph structure, combined with a second-order graph matching algorithm;

[0009] training the graph neural network based on the preliminary graph matching feature to obtain a femoral head necrosis region determination model.

[0010] In some embodiments, before the obtaining an image feature vector of a magnetic resonance imaging image and inputting a graph neural network, the method further comprises:

[0011] augmenting the magnetic resonance imaging images to obtain at least two different augmented magnetic resonance imaging images;

[0012] inputting the augmented magnetic resonance imaging images into a decoder to obtain an image feature vector corresponding to each of the image augmented magnetic resonance imaging images.

[0013] In some embodiments, the nodeset is formed according to the image feature vector corresponding to each magnetic resonance imaging image, and a graph structure is constructed based on the nodeset, including:

[0014] all the image feature vectors corresponding to each magnetic resonance imaging image are taken as a node to constitute the nodeset;

[0015] a first edge set and a second edge set corresponding to each node are constructed; wherein the first edge set is used to represent the feature similarity and spatial adjacency relationship between nodes; and the second edge set is used to represent the structure of adjacent slices or similar lesion distribution;

[0016] the first graph structure and the second graph structure corresponding to the first edge set and the second edge set are respectively formed based on the nodeset and the first edge set and the second edge set.

[0017] In some embodiments, the preliminary graph matching feature is obtained according to the node in the nodeset and the graph structure in combination with a second-order graph matching algorithm, including:

[0018] the node affinity matrix and the edge affinity matrix are obtained based on the node in the nodeset and the graph structure based on an aggregation strategy; and the preliminary graph matching matrix is obtained based on the node affinity matrix and the edge affinity matrix in combination with a graph matching objective function.

[0019] In some embodiments, the node affinity matrix and the edge affinity matrix are obtained based on the node in the nodeset and the graph structure based on an aggregation strategy, including:

[0020] the node affinity matrix is obtained based on the node in the nodeset and the first graph structure based on the aggregation strategy;

[0021] the edge affinity matrix is obtained based on the first graph structure and the second graph structure based on an aggregation strategy.

[0022] In some embodiments, the graph matching objective function is:

[0023]

[0024] wherein: GraphMatch is a graph matching function, which is used to find the best matching matrix; C is a cost function, which is used to measure the cost of the matching matrix; and A is an adjacency matrix of the graph structure.V is a node similarity matrix; C E is an edge similarity matrix; X is a matching matrix; u is a feasible region of the matching matrix; (·, ·) is an inner product operator; arg min is a solution to minimize the objective function; and is a tensor product operator.

[0025] In some embodiments, the training of the graph neural network based on the preliminary graph matching feature to obtain the femoral head necrosis region determination model comprises:

[0026] During forward propagation, after adding random noise to the preliminary graph matching feature, a forward graph matching matrix is obtained through the second-order graph matching algorithm;

[0027] During backward propagation, after adding a disturbance obtained based on a combined loss function to the preliminary graph matching feature, a backward graph matching matrix is obtained through the second-order graph matching algorithm;

[0028] According to the forward graph matching matrix and the backward graph matching matrix, a model update gradient is obtained, and the graph neural network is parameter adjusted based on the model update gradient until the latest model update gradient obtained reaches a preset termination condition, so as to obtain the femoral head necrosis region determination model.

[0029] In a second aspect, the embodiments of the present application provide a femoral head necrosis region determination method, comprising:

[0030] The target magnetic resonance imaging image is input into the femoral head necrosis region determination model obtained through the femoral head necrosis region determination model acquisition method to determine the femoral head necrosis region.

[0031] In a third aspect, the embodiments of the present application provide a femoral head necrosis region determination model acquisition device, comprising:

[0032] An input module is configured to acquire an image feature vector of a magnetic resonance imaging image and input the image feature vector into a graph neural network;

[0033] A graph structure construction module is configured to form a node set according to the image feature vector corresponding to each magnetic resonance imaging image, and construct a graph structure based on the node set; wherein the magnetic resonance imaging image is a femoral head necrosis related image;

[0034] A preliminary graph matching feature acquisition module is configured to obtain a preliminary graph matching feature according to the nodes in the node set and the graph structure, in combination with a second-order graph matching algorithm;

[0035] A model determination module is configured to train the graph neural network based on the preliminary graph matching feature to obtain a femoral head necrosis region determination model.

[0036] In a fourth aspect, an embodiment of the present application provides a terminal device, comprising a processor and a memory, the memory storing a computer program, and the processor being configured to execute the computer program to implement the femoral head necrosis area determination model obtaining method or to implement the femoral head necrosis area determination method.

[0037] The embodiments of the present application have the following beneficial effects: the present application can capture complex local and global features in the image by using the image feature vector to represent the MRI image and further processing by the graph neural network, thereby improving the accuracy of femoral head necrosis area recognition; in addition, the second-order graph matching algorithm is used to find the best match in the graph structure, which can significantly reduce the computational complexity compared with the traditional pixel-by-pixel or region-by-region matching method, thereby speeding up the processing speed and making the method more efficient in practical application. Further, the femoral head necrosis area determination model obtained can accurately identify the femoral head necrosis area, thereby being beneficial to clinical diagnosis and treatment. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0039] Figure 1 A first flowchart of the femoral head necrosis area determination model obtaining method of the embodiments of the present application is shown;

[0040] Figure 2 A second flowchart of the femoral head necrosis area determination model obtaining method of the embodiments of the present application is shown;

[0041] Figure 3 A third flowchart of the femoral head necrosis area determination model obtaining method of the embodiments of the present application is shown;

[0042] Figure 4 A fourth flowchart of the femoral head necrosis area determination model obtaining method of the embodiments of the present application is shown;

[0043] Figure 5 A fifth flowchart of the femoral head necrosis area determination model obtaining method of the embodiments of the present application is shown;

[0044] Figure 6 A sixth flowchart of the femoral head necrosis area determination model obtaining method of the embodiments of the present application is shown;

[0045] Figure 7 A structural schematic diagram of a femoral head necrosis area determination model acquisition device is shown. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application.

[0047] The components of the embodiments of the present application generally described and illustrated herein can be arranged and designed in a wide variety of different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application. All other embodiments obtained from the embodiments of the present application by one of ordinary skill in the art without creative work are within the scope of the present application.

[0048] Hereinafter, the terms "include", "have", and their conjugates used in the various embodiments of the present application are only intended to denote that specific features, numbers, steps, operations, elements, components, or combinations thereof are present, and should not be understood as excluding the presence or addition of one or more other features, numbers, steps, operations, elements, components, or combinations thereof. In addition, the terms "first", "second", "third", and the like are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0049] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the present application belong. The terms (such as those defined in commonly used dictionaries) will be interpreted as having a meaning that is the same as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the present application.

[0050] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0051] There are the following technical problems in determining the femoral head necrosis area through the MRI image of the femoral head necrosis in the prior art:

[0052] High subjectivity: the existing method relies on the subjective judgment of radiologists, resulting in insufficient repeatability and objectivity of the results.

[0053] High cost of data annotation: Deep learning methods require a large amount of manually annotated data for training, increasing the cost and difficulty of data preparation.

[0054] Poor domain adaptability: Existing self-supervised learning methods have limited effectiveness in the medical image field, especially in the application of femoral head necrosis, and are difficult to fully utilize the structure and spatial relationship of medical images.

[0055] Insufficient segmentation accuracy: Traditional self-supervised learning methods perform poorly in complex medical image segmentation tasks and cannot fully capture the subtle features of the femoral head necrosis region.

[0056] Therefore, the present application proposes a femoral head necrosis region determination model acquisition method, a necrosis region determination method, device and equipment, which can accurately capture the femoral head necrosis region in the absence of large-scale manual annotation.

[0057] The femoral head necrosis region determination model acquisition method will be described below in conjunction with some specific embodiments.

[0058] Figure 1 A flowchart of the femoral head necrosis region determination model acquisition method of the present application is shown. Illustratively, the femoral head necrosis region determination model acquisition method includes the following steps:

[0059] Step S100, obtaining the image feature vector of the magnetic resonance imaging image and inputting the graph neural network.

[0060] In this embodiment, the graph neural network is a graph convolutional attention network (GCAT).

[0061] Before step S100, the magnetic resonance imaging images used for training need to be preprocessed, and the preprocessing process includes:

[0062] 1. Resampling, resampling all magnetic resonance imaging images to a uniform spatial resolution, such as (0.55, 0.55, 0.55) mm 3 to reduce the impact of resolution differences, the resampling formula is:

[0063] I resampled = Resample(I, target_resolution);

[0064] In the formula, I is the original image, I resampled is the resampled image, target_resolution = (0.55, 0.55, 0.55) mm 3 .

[0065] 2. Cropping, according to the anatomical position of the femoral head, the image after reusing is centrally cropped to ensure that the image area input by the model is concentrated in the femoral head area. The cropping formula is as follows:

[0066] I cropped = CropCenter(I resampled ,W,H,D);

[0067] where W, H, D are the width, height, and depth of the cropping window, respectively; I cropped is the cropped image.

[0068] 3. Normalization, the cropped image is subjected to grayscale value normalization processing to improve the stability and convergence speed of model training. The specific formula is as follows:

[0069]

[0070] In the formula, μ and σ are the mean and standard deviation of the image, respectively; I normalized is the normalized image.

[0071] In some embodiments, as shown in Figure 2 , in order to improve the robustness and generalization ability of the model and prevent overfitting, the embodiment further includes, before step S100, after the image is normalized:

[0072] Step S500, the magnetic resonance imaging image is subjected to augmentation processing to obtain at least two different augmented magnetic resonance imaging images.

[0073] Exemplarily, each normalized image is subjected to image augmentation, and when the image is augmented, it is augmented by random transformation (such as rotation, translation, flipping, scaling, color disturbance, etc.) to obtain at least two augmented views, and the embodiment is described by obtaining two augmented views. The augmentation formula is:

[0074] I aug1 = Augment(I normalized );

[0075] I aug2 = Augment(I normalized );

[0076] where I aug1 and I aug2 are two different transformed images.

[0077] Step S600, the augmented magnetic resonance imaging image is input into the decoder to obtain an image feature vector corresponding to each image augmented magnetic resonance imaging image.

[0078] In this step, the I aug1 and I aug2 are input into the encoder to generate high-latitude features, through which a deep representation of distinguishable ONFH (Osteonecrosis of the Femoral Head) lesion features is obtained, laying a foundation for subsequent graph construction and matching. In order to improve the efficiency and accuracy of feature extraction, the encoder can be EfficientNet-B7 or ViT (Vision Transformer).

[0079] It can be expressed as:

[0080] F1 = Encoder(I aug1 );

[0081] F2 = Encoder(I aug2 );

[0082] wherein F1 and F2 are image feature vectors of I aug1 and I aug2 after feature extraction.

[0083] It should be noted that the training samples used for neural network training can be T1-weighted MRI images (magnetic resonance imaging images) of ONFH patients stored in multiple public databases and clinical cooperation institutions. In order to ensure the diversity and generalization ability of the data, the magnetic resonance imaging images need to cover different imaging modalities and different stages of disease manifestations.

[0084] In step S200, a node set is formed according to the image feature vectors corresponding to each magnetic resonance imaging image, and a graph structure is constructed based on the node set.

[0085] Wherein, the magnetic resonance imaging image is an ONFH related image.

[0086] Wherein, the graph structure includes a first graph structure and a second graph structure.

[0087] Illustratively, the image feature vectors F1 and F2 obtained from the decoder are input into the graph neural network. In one training batch, the graph feature vectors (F1, F2) corresponding to the augmented view pairs (I aug1 and I aug2 ) generated by each MRI image are regarded as a node, therefore, the multiple graph feature vectors corresponding to the multiple MRI images constitute the node set W; then based on the node set, the first type of edge set and the second type of edge set that can more comprehensively capture the structural information in the medical image are established; and based on the node set and the first type of edge set and the second type of edge set, the corresponding first graph structure and second graph structure are constructed.

[0088] In some embodiments, as shown in FIG. 2, step S200 includes steps S210-S230: Figure 3

[0089] Step S210, all image feature vectors corresponding to each magnetic resonance imaging image are taken as a node to constitute a node set.

[0090] In this step, the image feature vectors (F1, F2) corresponding to each magnetic resonance imaging image are taken as a node. Since multiple magnetic resonance imaging images are input in the current training batch, a node set can be formed according to the multiple image feature vectors (F1, F2) corresponding to the multiple magnetic resonance imaging images, and each node corresponds to an augmented view pair.

[0091] Step S220, a first edge set and a second edge set corresponding to each node are constructed.

[0092] In this step, for each node in the node set, an edge is established between each two nodes; wherein the first edge set E s is used to represent the feature similarity and spatial adjacency relationship between nodes. Specifically, these edges connect two augmented view pairs (I aug1 and I aug2 ) of the same image, and the weight of the edge is determined based on the high-dimensional feature similarity between them. The nodes with high similarity have an edge with high weight, otherwise the edge weight is weak or there is no edge connection; the second edge set E t is used to represent the structure of adjacent slices or similar lesion distribution. Such edges can also connect two augmented view pairs of the same image, but the focus is to capture the spatial and structural association between these images. Through the establishment of edges between nodes, the model can better understand the feature and structural information within the image, thereby more accurately performing segmentation and quantitative analysis.

[0093] Step S230, based on the node set and the first edge set and the second edge set, a first graph structure and a second graph structure corresponding to the first edge set and the second edge set are formed respectively.

[0094] In this step, first, the node set W and the first edge set E s are used to construct a first graph structure G s . The first graph structure G s is a graph, in which the nodes represent image pairs (i.e., feature vectors), and the edges represent the similarity relationship between these image pairs. The first graph structure expresses the correlation and similarity between images, and can be represented as G s = (V, E s ); the node set W and the first edge set E t form a second graph structure G​t The graph structure expresses other structural relationships between different image pairs through a second set of edges. It can be expressed as G t = (V, E t ).

[0095] At step S300, a preliminary graph matching feature is obtained according to the nodes in the node set and the graph structure, in combination with a second-order graph matching algorithm.

[0096] Illustratively, a node affinity matrix and an edge affinity matrix are obtained according to the nodes in the node set and the graph structure, based on an aggregation strategy; and a preliminary graph matching matrix is obtained based on the node affinity matrix and the edge affinity matrix, in combination with a graph matching objective function.

[0097] In some embodiments, as shown in FIG. 3, obtaining the node affinity matrix and the edge affinity matrix according to the nodes in the node set and the graph structure, based on the aggregation strategy, includes steps S310-S320: Figure 4

[0098] At step S310, a node affinity matrix is obtained according to the nodes in the node set and the first graph structure, based on an aggregation strategy.

[0099] At step S320, an edge affinity matrix is obtained according to the first graph structure and the second graph structure, based on an aggregation strategy.

[0100] Illustratively, a node affinity matrix C v is obtained according to the nodes in the node set and the first graph structure, based on an aggregation strategy, which can be expressed as C v = MPN v (G s , F1, F2); and an edge affinity matrix C e is obtained according to the first graph structure and the second graph structure, based on an aggregation strategy, which can be expressed as C e = MPN v (G s , G t ). The node affinity matrix represents the similarity between nodes in the graph, and the edge affinity matrix represents the similarity between edges in the graph. The aggregation strategy can be a graph neural network, or other aggregation strategies such as weighted neighborhood aggregation, mean aggregation, recurrent neural network (RNN) aggregation, etc.

[0101] Among the graph neural networks (GNNs), the message passing neural network (MPN) is a common architecture for processing graph structure data. The core idea of MPN is to update the representation of nodes by passing messages between nodes. Specifically, MPN can be used to calculate the affinity matrix of nodes and edges to capture the similarity between nodes and the importance of edges. ​

[0102] The graph matching objective function in this embodiment is:

[0103]

[0104] In the formula: GraphMatch is a graph matching function, which is used to find the best matching matrix; C V is a node similarity matrix, which represents the similarity between nodes in the graph; C E is an edge similarity matrix, which represents the similarity between edges in the graph; X is a matching matrix, which represents the matching relationship between nodes; u is a constraint set, which represents the feasible region of the matching matrix; (·,·) is an inner product operator; argmin is a variable that minimizes the objective function; is a tensor product operator.

[0105] When obtaining the preliminary matching matrix, C V is a node affinity matrix, C E is an edge affinity matrix, and X is a preliminary matching matrix; u is the feasible region of the preliminary matching matrix.

[0106] This embodiment uses a second-order dependency relationship. Compared with first-order matching or contrast learning, the graph matching objective function additionally considers the similarity between edges, which can capture more subtle structural differences in feature distribution, and is particularly suitable for the situation where the shape of the lesion area in femoral head necrosis MRI is variable and the boundary is complex.

[0107] In addition, since the combined graph matching objective function is discrete, direct derivation is not feasible. This embodiment uses the implicit maximum likelihood estimation (IMLE) method to approximate the gradient by introducing Gumbel noise, and the formula is:

[0108]

[0109] wherein, represents the gradient operator with respect to the parameter Θ, E represents the expectation operator, and represents the expectation of a certain distribution. V ~ p(V; Θ) represents that V follows the probability distribution p according to the parameter Θ. ∈ is a random disturbance, which is usually used as a noise term in gradient estimation, and p(∈) is the probability distribution of the disturbance ∈, is a graph matching loss function.

[0110] wherein, the graph matching loss function is: wherein, represents a loss function, which is used to measure the error between the predicted value and the true value. Θ represents the model parameter or the predicted value V represents the true value or the target variable.

[0111] The optimization process is continuous and differentiable by using the approximate gradient in this embodiment, so that the parameter update can be performed using the standard gradient descent method. Specifically, by introducing Gumbel noise, the originally discrete optimization problem can be made continuous and smooth, which makes the optimization process no longer dependent on the accurate gradient, but iteratively optimized by the approximate gradient. Moreover, the introduction of Gumbel noise makes the output of the model be regarded as part of the continuous probability distribution, so that the gradient descent method can be used for optimization, which makes the optimization process more stable and efficient. Furthermore, by the method of approximate gradient, the model can gradually approach the optimal solution in the iteration process, which can effectively optimize even in discrete optimization problems.

[0112] In step S400, the graph neural network is trained based on the preliminary graph matching features to obtain a femoral head necrosis region determination model.

[0113] As shown in the figure, Figure 5 Step S400 includes steps S410-S430:

[0114] In step S410, after adding random noise to the preliminary graph matching features during forward propagation, a forward graph matching matrix is obtained by a second-order graph matching algorithm.

[0115] In step S410, first, first random noise and second random noise are added to the node affinity matrix and the edge affinity matrix, respectively.

[0116] When adding random noise, Gumbel random noise needs to be sampled, i.e., ∈1, ∈3~Gumbel(0,1); that is, the random noise added to the node affinity matrix C v and the edge affinity matrix C e is drawn from the Gumbel(0,1) distribution.

[0117] Then, the node affinity matrix with the first random noise added is taken as the node similarity matrix, and the edge affinity matrix with the second random noise added is taken as the edge similarity matrix, which are brought into the graph matching objective function to obtain the forward graph matching matrix.

[0118] It can be understood that in this step, the forward graph matching matrix V1=GraphMatch(C v +∈1,C e +∈2), when obtaining the forward matching matrix, in the graph matching objective function, C V is C v +∈1, C E is C e +∈2, and X is the forward matching matrix; u is the feasible region of the forward matching matrix.

[0119] The node affinity matrix C is saved after the forward propagation ends v , the edge affinity matrix C e , the random noise ∈1, ∈2 and the forward graph matching matrix V1 are saved for subsequent backward propagation.

[0120] In step S420, the perturbation obtained based on the combined loss function is added to the preliminary graph matching feature during backward propagation, and a backward graph matching matrix is obtained by a second-order graph matching algorithm.

[0121] As shown in FIG. 4, step S420 includes steps S421-S423: Figure 6

[0122] In step S421, according to the forward propagation result, the model gradient is calculated in combination with the combined loss function, and the perturbation is generated according to the model gradient.

[0123] The perturbation includes a first perturbation and a second perturbation; the first perturbation is a perturbation of the node affinity matrix; and the second perturbation is a perturbation of the edge affinity matrix.

[0124] The combined loss function is a loss function combining the Dice loss and the cross-entropy loss, which can balance the accuracy of region overlap and pixel-level classification and ensure overall optimization of the segmentation effect.

[0125] The combined loss function is:

[0126]

[0127] wherein, is the combined loss function, which is a weighted sum of multiple loss functions. λ is a weight coefficient of the Dice loss. γ is a weight coefficient of the cross-entropy loss. is the Dice loss function, which is used to measure the overlap between the prediction and the true segmentation. is the cross-entropy loss function, which is used for error measurement in the task.

[0128] The Dice loss function is:

[0129]

[0130] wherein, P k is the predicted value of the kth sample, G k is the true value of the kth sample, δ is a table smoothing constant to prevent the denominator from being zero. ∑ k represents a summation operation on all samples.

[0131] The cross-entropy loss function is:

[0132]

[0133] ​Wherein, M is the number of samples, indicating that there are M samples in total, k is an index variable, indicating the kth sample, log is a logarithmic function, and natural logarithm.

[0134] After obtaining the forward graph matching matrix in the forward propagation, the model gradient is calculated through the combined loss function, and then the first perturbation added to the node affinity matrix and the second perturbation added to the edge affinity matrix in the back propagation are determined according to the model gradient.

[0135] Step S422, the first perturbation and the second perturbation are added to the node affinity matrix and the edge affinity matrix respectively.

[0136] Step S423, the node affinity matrix added with the first perturbation is taken as the node similarity matrix, and the node affinity matrix added with the second perturbation is taken as the edge similarity matrix, which are brought into the graph matching objective function to obtain the backward graph matching matrix.

[0137] The first perturbation ∈3 and the second perturbation ∈4 are added to the node affinity matrix and the edge affinity matrix respectively, and then the graph matching matrix is recalculated through the graph matching objective function, and the obtained graph matching matrix is the backward graph matching matrix.

[0138] Then the backward graph matching matrix V2 = GraphMatch(C v + ∈3, C e + ∈4).

[0139] Step S430, according to the forward graph matching matrix and the backward graph matching matrix, the model update gradient is obtained, and the graph neural network is parameter adjusted based on the model update gradient until the latest model update gradient reaches the preset termination condition, so as to obtain the femoral head necrosis area determination model.

[0140] Wherein, the model update gradient grad = V1-V2, after obtaining the model update gradient, the graph neural network and the feature extraction network parameters are updated according to the model update gradient, and when the model update gradient reaches the preset termination condition, the femoral head necrosis area determination model is obtained.

[0141] It should be noted that in this embodiment, the graph is constructed, and the features extracted by the encoder are utilized by the second-order graph matching algorithm part, and the feature matching is further optimized by self-supervised learning, which strengthens the understanding of the model to the internal local and global structure information of the image, and at the same time, the skip connection directly transmits the low-level spatial details in the encoder to the decoding, thereby making up for the information that may be lost in the upsampling process, and ensuring that the segmentation result is more fine in boundary and structure; The decoder uses the fused multi-scale features for upsampling, which can output a segmentation map consistent with the size of the original image.

[0142] In addition, the model of the embodiment can use an Adam optimizer during training, with an initial learning rate of 0.001, combined with a OneCycleLR learning rate scheduler, and the learning rate is gradually adjusted during the training process to improve the training efficiency and model performance. The batch size can be set to 4, which can balance the training speed and memory occupation of the model, and ensure efficient training under limited computing resources. In the medical image segmentation task, the MRI image usually has high resolution, and the network size used in the application is large, so the GPU memory demand for single forward and backward propagation is high. By setting the batch size to 4, a balance between certain parallel computing efficiency and avoiding memory overflow can be achieved under limited memory conditions, thereby ensuring that the model can converge within a reasonable time. If the batch size is too small (such as 1 or 2), the randomness in the gradient update process will increase significantly, and the training will be unstable; if the batch size is too large (such as 8 or 16), the memory occupation will be significantly increased, and in the small-scale medical data scenario, the model may fall into a local optimum or have generalization difficulties. Choosing 4 can strike a balance between “computing efficiency” and “gradient stability”, ensuring a more stable training process. The training device used can be an NVIDIA GeForce RTX 3070 GPU.

[0143] The femoral head necrosis region determination model acquisition method of the embodiment can be trained with a small amount or no annotated data when acquiring the femoral head necrosis region determination model, thereby reducing the cost of data preparation. In addition, the embodiment introduces second-order graph matching and self-supervised learning, and the model can better capture the subtle features of the lesion region, thereby improving the accuracy of image segmentation. Furthermore, the embodiment utilizes data augmentation technology and a combined loss function, thereby enhancing the robustness and generalization ability of the model.

[0144] In addition, the application adds random noise to the preliminary graph matching matrix during the forward propagation process, so that the model can maintain relatively stable performance when facing small changes in input data. This approach of the application helps to improve the robustness and generalization ability of the model. In the backward propagation stage, the preliminary graph matching matrix is adjusted through the disturbance calculated based on the combined loss function, which can more accurately guide the learning direction of the model, thereby not only accelerating the convergence process but also improving the accuracy of the model. Furthermore, the application combines the graph matching matrices generated during the forward propagation and backward propagation processes to obtain more refined model update gradients, which are used to update the parameters of the feature extraction network and the graph neural network, thereby continuously optimizing the model performance and ultimately achieving high recognition accuracy.

[0145] In another aspect, the application provides a femoral head necrosis region determination method, including: inputting a target magnetic resonance imaging image into a femoral head necrosis region determination model obtained by the femoral head necrosis region determination model obtaining method to determine a femoral head necrosis region.

[0146] In this embodiment, by inputting the target magnetic resonance imaging image into the femoral head necrosis region determination model obtained by the above method, the model can automatically segment the femoral head necrosis region, and further obtain an unaffected bone marrow region. Based on the femoral head necrosis region and the unaffected bone marrow region, the necrosis ratio can be quantitatively determined.

[0147] The femoral head necrosis region volume is ONV = ∑M i ×V p ; the unaffected bone marrow region UBMV = ∑((1-M i )×V p ); wherein, M i is a binary mask of whether pixel i belongs to the necrosis region (Mi = 1 represents the necrosis region, and Mi = 0 represents the non-necrosis region), and V p is the volume corresponding to each pixel.

[0148] Then the necrosis ratio ONF = ONV / (ONV+UBMV).

[0149] It can be understood that, based on the femoral head necrosis region determination model, the femoral head necrosis region and the unaffected bone marrow region can be determined, and the necrosis volume can be determined, that is, the embodiment can provide accurate quantitative parameters, which not only help clinicians to evaluate the severity of the disease, but also can be used for monitoring the treatment effect and the disease progression.

[0150] Figure 7 A structural schematic diagram of a femoral head necrosis region determination model obtaining device of an embodiment of the application is shown. Exemplarily, the femoral head necrosis region determination model obtaining device includes:

[0151] An input module 100 is configured to obtain an image feature vector of a magnetic resonance imaging image and input the image feature vector into a graph neural network.

[0152] A graph structure constructing module 200 is configured to form a node set according to the image feature vector corresponding to each magnetic resonance imaging image, and construct a graph structure based on the node set; wherein the magnetic resonance imaging image is a femoral head necrosis related image.

[0153] A preliminary graph matching feature obtaining module 300 is configured to obtain a preliminary graph matching feature according to the nodes in the node set and the graph structure, and in combination with a second-order graph matching algorithm.

[0154] The model determining module 400 is configured to train the graph neural network based on the preliminary graph matching features to obtain the femoral head necrosis region determination model.

[0155] It can be understood that the device of the embodiment corresponds to the femoral head necrosis region determination model acquisition method of the above-mentioned embodiment, and the optional items in the above-mentioned embodiment are also applicable to the embodiment, and therefore will not be described again here.

[0156] The present application also provides a terminal device, which exemplarily comprises a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the femoral head necrosis region determination model acquisition method or executes the computer program to enable the terminal device to perform the femoral head necrosis region determination method.

[0157] The processor can be an integrated circuit chip with a signal processing capability. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), and a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc., and can implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application.

[0158] The memory can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is configured to store a computer program, and the processor can execute the computer program after receiving an execution instruction.

[0159] The application further provides a computer readable storage medium for storing the computer program used in the terminal device. For example, the computer readable storage medium can include, but is not limited to, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage media that can store program codes.

[0160] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus embodiments described above are only schematic, for example, the flow charts and structural diagrams in the drawings show the possible implementation architectures, functions and operations of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flow charts or structural diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that, in alternative implementation manners, the functions noted in the blocks can also occur in different orders from those noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the structural diagram and / or flow chart, and the combination of blocks in the structural diagram and / or flow chart, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0161] In addition, each functional module or unit in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0162] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for causing a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application.

[0163] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A method for acquiring a femoral head necrosis region determination model, characterized by, The method comprises the following steps: obtaining an image feature vector of a magnetic resonance imaging image and inputting the image feature vector into a graph neural network; forming a node set according to the image feature vector corresponding to each magnetic resonance imaging image, and constructing a graph structure based on the node set; wherein the magnetic resonance imaging image is a femoral head necrosis related image; obtaining a preliminary graph matching feature according to the nodes in the node set and the graph structure in combination with a second-order graph matching algorithm; training the graph neural network based on the preliminary graph matching feature to obtain a femoral head necrosis region determination model.

2. The method of claim 1, wherein Before the step of obtaining an image feature vector of a magnetic resonance imaging image and inputting the image feature vector into a graph neural network, the method further comprises the following steps: performing augmentation processing on the magnetic resonance imaging image to obtain at least two different augmented magnetic resonance imaging images; inputting the augmented magnetic resonance imaging images into a decoder to obtain an image feature vector corresponding to each of the augmented magnetic resonance imaging images.

3. The method of claim 1, wherein the method further comprises: determining a region of interest (ROI) of the 3D image data; and determining a region of interest (ROI) of the 3D image data. The step of forming a node set according to the image feature vector corresponding to each magnetic resonance imaging image and constructing a graph structure based on the node set comprises the following steps: taking all the image feature vectors corresponding to each magnetic resonance imaging image as a node to constitute the node set; constructing a first edge set and a second edge set corresponding to each node; wherein the first edge set is used to represent the feature similarity and spatial adjacency relationship between nodes; and the second edge set is used to represent the structure of adjacent slices or similar lesions; forming a first graph structure and a second graph structure corresponding to the first edge set and the second edge set respectively based on the node set and the first edge set and the second edge set.

4. The method of claim 3, wherein the method further comprises: determining the location of the femoral head necrosis region based on the obtained image. The step of obtaining a preliminary graph matching feature according to the nodes in the node set and the graph structure in combination with a second-order graph matching algorithm comprises the following steps: obtaining a node affinity matrix and an edge affinity matrix based on an aggregation strategy according to the nodes in the node set and the graph structure; and obtaining a preliminary graph matching matrix based on the node affinity matrix and the edge affinity matrix in combination with a graph matching objective function.

5. The method of claim 4, wherein the method further comprises: determining the location of the femoral head necrosis region based on the obtained image data. The step of obtaining a node affinity matrix and an edge affinity matrix based on an aggregation strategy according to the nodes in the node set and the graph structure comprises the following steps: obtaining the node affinity matrix based on the aggregation strategy according to the nodes in the node set and the first graph structure; obtaining the edge affinity matrix based on the aggregation strategy according to the first graph structure and the second graph structure.

6. The method of claim 4, wherein the method further comprises: determining a region of interest (ROI) of the 3D image data; and determining a region of interest (ROI) of the 3D image data. The graph matching objective function is: where: GraphMatch is the graph matching function that finds the best matching matrix; C V is the node similarity matrix; C E is the edge similarity matrix; X is the matching matrix; u is the feasible region of the matching matrix; (·, ·) is the inner product operator; arg min is the solution that minimizes the objective function; is the tensor product operator.

7. The method of claim 1, wherein The step of training the graph neural network based on the preliminary graph matching feature to obtain a femoral head necrosis region determination model comprises the following steps: adding random noise to the preliminary graph matching feature during forward propagation, and obtaining a forward graph matching matrix through the second-order graph matching algorithm; adding a disturbance obtained based on a combination loss function to the preliminary graph matching feature during backward propagation, and obtaining a backward graph matching matrix through the second-order graph matching algorithm; According to the forward graph matching matrix and the backward graph matching matrix, a model update gradient is obtained, and the graph neural network is adjusted in parameters based on the model update gradient until a latest model update gradient obtained reaches a preset termination condition, to obtain a femoral head necrosis area determination model.

8. A method of determining an area of necrosis of a femoral head, characterized by, The method comprises the steps of: Inputting a target magnetic resonance imaging image into a femoral head necrosis area determination model obtained by the femoral head necrosis area determination model obtaining method according to any one of claims 1-7, to determine a femoral head necrosis area.

9. A device for obtaining a model to determine the area of ​​femoral head necrosis, characterized in that, The method comprises the steps of: An input module is configured to obtain an image feature vector of a magnetic resonance imaging image and input the image feature vector into a graph neural network; A graph structure construction module is configured to form a node set according to the image feature vector corresponding to each magnetic resonance imaging image, and construct a graph structure based on the node set; wherein the magnetic resonance imaging image is a femoral head necrosis related image; A preliminary graph matching feature obtaining module is configured to obtain a preliminary graph matching feature according to nodes in the node set and the graph structure, in combination with a second-order graph matching algorithm; A model determination module is configured to train the graph neural network based on the preliminary graph matching feature, to obtain a femoral head necrosis area determination model.

10. A terminal device, comprising: The terminal device comprises a processor and a memory, the memory stores a computer program, and the processor is configured to execute the computer program to implement the femoral head necrosis area determination model obtaining method according to any one of claims 1-7 or to execute the computer program to implement the femoral head necrosis area determination method according to claim 8.

Images