Medical image segmentation post-processing method and system based on double-branch evidence fusion and graph convolutional neural network

Through the post-processing method of dual-branch evidence fusion and graph convolutional neural network, the problem of insufficient confidence and robustness in small-objective segmentation tasks is solved, which improves segmentation accuracy and reliability, reduces error predictions, and enhances clinical Application value.

CN118941786AActive Publication Date: 2024-11-12ZHEJIANG UNIV OF TECH

Patent Information

Application Number
CN202410888284.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-04
Publication Date
2024-11-12
Estimated Expiration
2044-07-04

AI Technical Summary

Technical Problem

Existing medical image segmentation models such as MedSAM show lower confidence and robustness in small-objective segmentation tasks, and lack uncertainty evaluation mechanisms, which leads to overconfidence in mispredictions, resulting in false positive and false negative areas, limiting Its application in clinical practice.

Method used

Using the post-processing method of double-branch evidence fusion and graph convolution neural network, the output branches of MedSAM are reconstructed through Dropout technology, and the pixel classification evidence of the two-branch is obtained, evidence fusion is carried out, weighted half-marked graphs are constructed, and graph convolution neural is used. The network classifies unmarked points to improve segmentation accuracy.

Benefits of technology

The segmentation accuracy of MedSAM in small-objective segmentation tasks is improved, the occurrence of false positive and false negative areas is reduced, and the reliability and clinical application value of the model are enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941786A_ABST
    Figure CN118941786A_ABST
Patent Text Reader

Abstract

A medical image segmentation post-processing method based on double-branch evidence fusion and an image convolutional neural network comprises the following steps: on the basis of a pre-trained basic model MedSAM, reconstructing two mutually independent output branches through Dropout operation, respectively obtaining pixel classification evidences of the double branches, and fusing the pixel classification evidences to obtain uncertainty after fusion; pixel points in the working area ROI are used as nodes, marking is carried out according to the uncertainty after fusion, node labels with low uncertainty are used as original segmentation results of a basic model, nodes with high uncertainty have no labels, and a weighted semi-marked graph is constructed through graph connection; and calculating joint optimization loss including cross entropy loss and Dice loss by using labeled node output in the weighted semi-labeled graph, training the graph convolutional neural network, classifying unlabeled points, and fusing node labels with low uncertainty to obtain a final segmentation result. According to the invention, the segmentation precision of the region with high uncertainty in the original segmentation result is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a medical image segmentation post-processing method and system based on dual-branch evidence fusion and graph convolutional neural network. Background Art

[0002] Image segmentation is a fundamental task in medical imaging analysis, which involves identifying and delineating regions of interest (ROIs) in various medical images, such as organs, lesions, and tissues. Accurate segmentation is critical for many clinical applications, including disease diagnosis, treatment planning, and disease progression monitoring. Manual segmentation has long been the gold standard for delineating anatomical structures and pathological regions, but this process is time-consuming, labor-intensive, and often requires a high degree of expertise. In recent years, convolutional neural networks (CNNs) and Transformers have achieved remarkable results in medical image segmentation. However, a significant limitation of many current medical image segmentation models is their task specificity. These models are often designed and trained for specific segmentation tasks, and their performance may degrade significantly when applied to new tasks or different types of imaging data. This lack of generality poses a substantial obstacle to the widespread application of segmentation models in clinical practice.

[0003] As a general model that has made revolutionary progress in the field of artificial intelligence in recent years, the basic model has been fully pre-trained on large-scale datasets and has demonstrated strong zero-sample generalization capabilities. In the field of image segmentation, SAM is a segmentation model proposed by Meta in 2023 that aims to serve as a basic model to unify the entire image segmentation task. In order to better adapt to the image segmentation tasks in the medical field, Jun Ma et al. proposed MedSAM based on SAM. MedSAM uses a dataset of more than one million medical images to surpass the existing state-of-the-art segmentation models in performance, and shows performance comparable to or better than that of professional doctors. However, when facing small target segmentation tasks such as brain tumors, MedSAM's segmentation results still show low confidence and robustness. Because the output of MedSAM can only provide deterministic prediction values, it cannot express the uncertainty of the model for the prediction results. This lack of uncertainty evaluation mechanism often leads to the model being overconfident when making wrong predictions, especially in the boundary areas of small targets, which easily produces many false positive (False Positive, FP) and false negative (False Negative, FN) areas, limiting the application of the model in clinical practice.

[0004] Uncertainty estimation is considered to be the key to reflecting the reliability of the model, which can be used to quantify the probability that the model is wrong in prediction. Common methods for evaluating the uncertainty of neural network models include Bayesian neural networks (BNN), ensemble-based methods, and dropout-based methods. In 2018, Sensoy et al. proposed a new uncertainty assessment method, which regards the output of the neural network as evidence of the Dirichlet distribution parameterized by Dempster-Shafer evidence theory (DST) and subjective logic, and provides each object with a trust value belonging to different categories and an overall uncertainty value, showing impressive performance in quantifying model uncertainty. In addition, as an interpretable technology, evidence fusion plays an important role in the information integration of deep learning classifiers. Evidence-based methods can better utilize trust allocation and uncertainty estimation to obtain information from medical images. Compared with the processing method that relies only on a single network, evidence fusion technology can obtain more accurate uncertainty analysis. Therefore, using evidence theory and subjective logic, the uncertainty of the MedSAM model can be evaluated, and then the accuracy of the segmentation results of the MedSAM model can be further improved through the post-processing method of uncertainty-aware fusion. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a medical image segmentation post-processing method and system based on dual-branch evidence fusion and graph convolutional neural network. The pre-trained basic segmentation model MedSAM is used to reconstruct two independent output branches through the Dropout technology, and the pixel classification evidence of the dual branches is obtained and fused respectively to obtain a more reliable uncertainty assessment and working area ROI. Then, the pixels with low uncertainty in the ROI are used as marked nodes in the graph, and the pixels with high uncertainty are used as unlabeled nodes. Then, a semi-supervised learning graph convolutional neural network (GCN) is trained to classify the unlabeled points, thereby improving the segmentation accuracy of the high-uncertainty area in the original segmentation result of MedSAM.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] A medical image segmentation post-processing method based on dual-branch evidence fusion and graph convolutional neural network includes the following steps:

[0008] Step 1: Medical image preprocessing: First, the intensity value v of each pixel i in the input medical image x is calculated. i Normalize:

[0009]

[0010] where v min is the minimum pixel intensity value, vmax is the maximum pixel intensity value, and the adjusted pixel intensity value v i ∈[0,255], then the bicubic interpolation method is used to resize x to H×W×3, where H and W are the height and width of the processed medical image x respectively;

[0011] Step 2: Get the dual-branch output, input the processed medical image x into the MedSAM encoder Encoder, and then pass the MedSAM decoder mask decoder 1 to get the lower branch output m 1 , through the decoder mask decoder 2 to get the lower branch output m 2 , where mask decoder 2 and mask decoder 1 share network parameters. Mask decoder 2 is obtained by dropout operation on the basis of mask decoder 1, that is, the spatial dot product of the output of the two transposed convolution upsampling in mask decoder 1 and the output of the fully connected layer MLP is obtained. 1 , and the spatial dot product of the output of the two transposed convolutions and the output of the last layer of MLP after the dropout operation is obtained by mask decoder 2. 2 ;

[0012] Step 3: Fusion of evidence from both branches to obtain evidence from the upper and lower branches Calculate the trust value of the upper and lower branches and uncertainty Where K is the total number of categories of the segmentation target, n = 1 represents the upper branch, n = 2 represents the lower branch, and the uncertainty of the upper and lower branches is fused based on the evidence theory to obtain the fused uncertainty

[0013] Step 4: Mark each pixel according to its uncertainty after fusion and determine the working area Ub ROI and Ub ROI Pixel labels with low internal uncertainty;

[0014] Step 5: Take the working area Ub ROI Each pixel in the graph is a node, and a semi-labeled weighted graph is constructed through graph connection to obtain Ub ROI The connection matrix A and feature matrix X of the corresponding graph;

[0015] Step 6: Ub the work area ROI The adjacency matrix A and feature matrix X of the corresponding graph are input into the two-layer graph convolutional neural network GCN to obtain the output prediction probability Z, and the parameters W of GCN are updated by minimizing the joint optimization loss function L. (0),W (1)} until the maximum number of iterations is reached:

[0016]

[0017] L=L CE +L Dice

[0018]

[0019] Where W (0) and W (1) is the weight parameter matrix of the first and second layers of GCN. Sigmoid and ReLU are both activation functions. That is, add a unit matrix I to the adjacency matrix A N , L CE and L Dice They are cross entropy loss and Dice loss respectively. is the label y corresponding to pixel i i =1, N is the predicted probability of the working area Ub ROI The total number of pixels with labels, F = {F i |i∈Ub ROI} is the segmentation result after the output prediction probability Z of GCN is converted into binary value by the sign function sgn (threshold is 0.5), F i ∈{0,1}, 1 is foreground, 0 is background;

[0020] Step 7: Ub the work area ROI The adjacency matrix A and feature matrix X of the corresponding graph are input into the trained two-layer graph convolutional neural network GCN to obtain the binary segmentation result F'={F i '|i∈Ub ROI}, in the working area Ub ROI The categories of the original labeled pixels are kept unchanged, and the categories of the unlabeled pixels are replaced with the segmentation results of GCN, and finally the final segmentation result F is obtained. * ={F i * |i∈Ub ROI}:

[0021]

[0022] Further, the process of step three is as follows:

[0023] 3.1 The output m of the dual branch 1 and m 2 The two-branch evidence e is obtained by Softplus activation function respectively n :

[0024] e n =Softplus(m n ), n=1,2 (2)

[0025] 3.2 Based on the two-branch evidence e n , take uniform distribution, and get the parameters of the two-branch Dirichlet distribution and the Dirichlet distribution intensity in

[0026] α n =e n +1, n=1,2 (3)

[0027] 3.3 Calculate the classification confidence value of each pixel i output by the dual branches and uncertainty for:

[0028]

[0029] 3.4 Fusion of the uncertainty of the upper and lower branches to obtain the fused uncertainty

[0030]

[0031] Furthermore, the process of step 4 is as follows:

[0032] 4.1 Based on the uncertainty after fusion Uncertain level marking Ub = {Ub i |i=1,2,...,H×W}:

[0033]

[0034] Where τ is the uncertainty threshold. If Ub i =1 means that the i-th pixel is a pixel with high uncertainty, and Ub i =0 means that the i-th pixel is a pixel with low uncertainty;

[0035] 4.2 Determine the working area Ub ROI :

[0036] Ub ROI =Ub∪Eb (8)

[0037] P = Sigmoid(m 1 ) (9)

[0038] Where P = {P i,j|i=1,2,...,H×W,j=1,2,...,K} is the output of the lower branch of MedSAM 1 After the Sigmoid activation function, the predicted probability value of pixel classification is obtained, Eb = {Eb i |i=1,2,...,H×W} is the segmentation result after P is converted into binary value by the sign function sgn (threshold is 0.5), where Eb i ∈{0,1}, 1 is foreground, 0 is background, ∪ means adding the two first, and then using the sign function sgn (threshold is 1) to convert the result of the addition into a binary mask, where 0 is the background with low uncertainty, 1 is the pixel with high uncertainty and the foreground with low uncertainty, and finally the area with all 1 is taken as Ub ROI ;

[0039] 4.3 Determine Ub according to the uncertainty level ROI Labels for pixels with low uncertainty:

[0040]

[0041] Unlabeled means there is no label.

[0042] Furthermore, the process of step five is as follows:

[0043] 5.1 For any i-th pixel, create connections with its four neighbors above, below, left and right, and also create a connection in Ub ROI Some nodes are randomly selected to connect with it, and the pixel value v = {v i |i=1,2,...,H×W} and the predicted probability value P corresponding to the output of the lower branch of MedSAM={P i,j |i=1,2,...,H×W,j=1,2,...,K} to calculate the edge weights and get Ub ROI The connection matrix of the corresponding graph is A = {w i,j |i∈Ub ROI ,j∈Ub ROI}:

[0044]

[0045] weight2(x i ,x j )=||v i -v j || 2 (13)

[0046] weight3(x i ,x j )=||x i -xj || 2 (14)

[0047] Among them, x i ,x j Ub ROI The positions of the i-th and j-th pixels in , λ is a balancing factor, weight1(·) is determined by the diversity between the two pixels, and weight2(x i ,x j ) and weight3(x i ,x j ) are x i ,x j The pixel weight and distance weight between two points, σ1 and σ2 are weight2(x i ,x j ) and weight3(x i ,x j ), in formula (12), when the segmentation target is only the foreground and the background, K = 2, where P i,1 The probability value of the i-th pixel output by the lower branch of MedSAM is predicted as the foreground, and the probability value predicted as the background is P i,2 =1-P i,1 ,In formula (13) and formula (14),|||| represents the Euclidean distance;

[0048] 5.2 For Ub ROI Each pixel in the , its characteristic value includes the branch output m of the MedSAM model 1 The standardized weighted eigenvalue M = {M i |i∈Ub ROI}, predicted probability value P = {P i,j |i∈Ub ROI , j=1,2,...K} and the uncertainty after fusion Get Ub ROI The corresponding feature matrix X:

[0049]

[0050] Among them, v μ is the output of the MedSAM branch m 1 The average value, v α is m 1 The variance of .

[0051] A medical image segmentation post-processing system based on dual-branch evidence fusion and graph convolutional neural network includes an image preprocessing module, a dual-branch output module, an evidence fusion module, a ROI marking module, a graph construction module, a GCN training module and a GCN segmentation module, each module sequentially implements the processing of steps 1 to 7 of the method.

[0052] The technical concept of the present invention is: the decoder of the basic segmentation model MedSAM is reconstructed through the Dropout operation to obtain two independent output branches, and the pixel classification evidence of the two branches is obtained respectively according to the evidence theory and subjective logic, and then the uncertainty of each pixel classification is quantified through the evidence, and a more reliable pixel classification uncertainty is obtained through the evidence fusion of the two branches, and then the working area is determined in combination with the segmentation result of MedSAM, and a weighted semi-labeled graph corresponding to the working area is constructed, in which the pixels with low uncertainty directly take the segmentation result of MedSAM as the label, and then train the graph convolutional neural network GCN as the post-processing model of the MedSAM segmentation result, and re-predict the classification of pixels with low uncertainty.

[0053] A medical image segmentation post-processing method based on dual-branch evidence fusion and graph convolutional neural network consists of three parts: a dual-branch evidence fusion unit, a ROI weighted graph construction unit, and a GCN training and segmentation unit. Based on a pre-trained basic model, the dual-branch evidence fusion unit reconstructs two independent output branches through a Dropout operation, obtains pixel classification evidence of the dual branches respectively and fuses them to obtain the uncertainty after fusion; the ROI weighted graph construction unit uses the pixel points in the working area ROI as nodes, marks them according to their fused uncertainty, and the node labels with low uncertainty are the segmentation results of the basic model. The nodes with high uncertainty have no labels, and a weighted semi-labeled graph is constructed through graph connection; the GCN training and segmentation unit uses the labeled node outputs in the weighted semi-labeled graph to calculate the joint optimization loss including the cross entropy loss and the Dice loss, trains the graph convolutional neural network (GCN) and classifies the unlabeled points, and fuses the original node labels with low uncertainty to obtain the final segmentation result. The present invention solves the segmentation error caused by the overconfidence of the basic model MedSAM in medical image segmentation.

[0054] The beneficial effects of the present invention are as follows: the present invention utilizes the pre-trained basic segmentation model MedSAM, combines evidence theory and subjective logic, represents the output of MedSAM as the degree of confidence in different categories of segmentation targets and a total uncertainty, and post-processes the segmentation results of MedSAM through an uncertainty-aware graph convolutional neural network with dual-branch evidence fusion, thereby solving the segmentation errors caused by the overconfidence of the MedSAM model in the segmentation of small medical targets and improving the segmentation accuracy of the regions with high uncertainty in the segmentation results of MedSAM. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is the algorithm block diagram of the present invention.

[0056] Figure 2 It is a flow chart of the present invention.

[0057] Figure 3 It is a comparison diagram of the segmentation results of the present invention, wherein (a) is a brain tumor image, (b) is the gold standard, (c) is the segmentation result of the basic model MedSAM, and (d) is the segmentation result of the method of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with the accompanying drawings.

[0059] Reference Figure 1 to Figure 3 , a medical image segmentation post-processing method based on dual-branch evidence fusion and graph convolutional neural network, taking brain tumor image segmentation as an example, includes the following steps:

[0060] Step 1: Preprocess the 2D slice x of the brain tumor and convert the intensity value v of each pixel i i Normalize:

[0061]

[0062] where v min is the minimum pixel intensity value, v max is the maximum pixel intensity value, and the adjusted pixel intensity value v i ∈[0,255], then the bicubic interpolation method is used to resize x to H×W×3, where H and W are the height and width of the processed medical image x, respectively. Here, the sizes of H and W are set to 1024;

[0063] Step 2: Input the processed medical image x into the encoder of the basic segmentation model MedSAM, and then pass it through the decoder mask decoder 1 of MedSAM to obtain the lower branch output m 1 , through the decoder mask decoder 2 to get the lower branch output m 2 , where mask decoder 2 and mask decoder 1 share network parameters. Mask decoder 2 is obtained by dropout operation on the basis of mask decoder 1, that is, the spatial dot product of the output of the two transposed convolution upsampling in mask decoder 1 and the output of the fully connected layer MLP is obtained. 1, and the spatial dot product of the output of the two transposed convolutions and the output of the last layer of MLP after the dropout operation is obtained by mask decoder 2. 2 ;

[0064] Step 3: Obtain evidence for the upper and lower branches Calculate the trust value of the upper and lower branches and uncertainty Where K is the total number of categories of the segmentation target, n = 1 represents the upper branch, n = 2 represents the lower branch, and the uncertainty of the upper and lower branches is fused based on the evidence theory to obtain the fused uncertainty The process is as follows:

[0065] 3.1 The output m of the dual branch 1 and m 2 The two-branch evidence e is obtained by Softplus activation function respectively n :

[0066] e n =Softplus(m n ), n=1,2 (2)

[0067] 3.2 Based on the two-branch evidence e n , take uniform distribution, and get the parameters of the two-branch Dirichlet distribution and the Dirichlet distribution intensity in

[0068] α n =e n +1, n=1,2 (3)

[0069] 3.3 Calculate the classification confidence value of each pixel i output by the dual branches and uncertainty for:

[0070]

[0071] 3.4 Fusion of the uncertainty of the upper and lower branches to obtain the fused uncertainty

[0072]

[0073] Step 4: Mark each pixel according to its uncertainty after fusion and determine the working area Ub ROI and Ub ROI Pixel labels with low internal uncertainty; the process is as follows:

[0074] 4.1 Based on the uncertainty after fusion Uncertain level marking Ub = {Ub i |i=1,2,...,H×W}:

[0075]

[0076] Where τ is the uncertainty threshold. If Ub i =1 means that the i-th pixel is a pixel with high uncertainty, and Ub i =0 means that the i-th pixel is a pixel with low uncertainty;

[0077] 4.2 Determine the working area Ub ROI :

[0078] Ub ROI =Ub∪Eb (8)

[0079] P = Sigmoid(m 1 ) (9)

[0080] Where P = {P i,j |i=1,2,...,H×W,j=1,2,...,K} is the output of the lower branch of MedSAM 1 After the Sigmoid activation function, the predicted probability value of pixel classification is obtained, Eb = {Eb i |i=1,2,...,H×W} is the segmentation result after P is converted into binary value by the sign function sgn (threshold is 0.5), where Eb i ∈{0,1}, 1 is foreground, 0 is background, ∪ means adding the two first, and then using the sign function sgn (threshold is 1) to convert the result of the addition into a binary mask, where 0 is the background with low uncertainty, 1 is the pixel with high uncertainty and the foreground with low uncertainty, and finally the area with all 1 is taken as Ub ROI ;

[0081] 4.3 Determine Ub according to the uncertainty level ROI Labels for pixels with low uncertainty:

[0082]

[0083] Among them, unlabeled means no label;

[0084] Step 5: Take the working area Ub ROI Each pixel in the graph is a node, and a semi-labeled weighted graph is constructed through graph connection to obtain Ub ROI The connection matrix A and feature matrix X of the corresponding graph; the process is as follows:

[0085] 5.1 For any i-th pixel, create connections with its four neighbors above, below, left and right, and also create a connection in Ub ROI Some nodes are randomly selected to connect with it, and the pixel value v = {v i |i=1,2,...,H×W} and the predicted probability value P corresponding to the output of the lower branch of MedSAM={P i,j |i=1,2,...,H×W,j=1,2,...,K} to calculate the edge weights and get Ub ROI The connection matrix of the corresponding graph is A = {w i,j |i∈Ub ROI ,j∈Ub ROI}:

[0086]

[0087] weight2(x i ,x j )=||v i -v j || 2 (13)

[0088] weight3(x i ,x j )=||x i -x j || 2 (14)

[0089] Among them, x i ,x j Ub ROI The positions of the i-th and j-th pixels in , λ is a balancing factor, weight1(·) is determined by the diversity between the two pixels, and weight2(x i ,x j ) and weight3(x i ,x j ) are x i ,x j The pixel weight and distance weight between two points, σ1 and σ2 are weight2(x i ,x j ) and weight3(x i ,x j ), in formula (12), when the segmentation target is only the foreground and the background, K = 2, where P i,1 The probability value of the i-th pixel output by the lower branch of MedSAM is predicted as the foreground, and the probability value predicted as the background is P i,2 =1-P i,1, In formula (13) and formula (14), || || represents the Euclidean distance;

[0090] 5.2 For Ub ROI Each pixel in the , its characteristic value includes the branch output m of the MedSAM model 1 The standardized weighted eigenvalue M = {M i |i∈Ub ROI}, predicted probability value P = {P i,j |i∈Ub ROI , j=1,2,...K} and the uncertainty after fusion Get Ub ROI The corresponding feature matrix X:

[0091]

[0092] Among them, v μ is the output of the MedSAM branch m 1 The average value, v α is m 1 The variance of

[0093] Step 6: Ub the work area ROI The adjacency matrix A and feature matrix X of the corresponding graph are input into the two-layer graph convolutional neural network GCN to obtain the output prediction probability Z, and the parameters W of GCN are updated by minimizing the joint optimization loss function L. (0) ,W (1)} until the maximum number of iterations is reached:

[0094]

[0095] L=L CE +L Dice

[0096]

[0097] Where W (0) and W (1) is the weight parameter matrix of the first and second layers of GCN. Sigmoid and ReLU are both activation functions. That is, add a unit matrix I to the adjacency matrix A N , L CE and L Dice They are cross entropy loss and Dice loss respectively. is the label y corresponding to pixel i i =1, N is the predicted probability of the working area Ub ROI The total number of pixels with labels, F = {F i |i∈Ub ROI} is the segmentation result after the output prediction probability Z of GCN is converted into binary value by the sign function sgn (threshold is 0.5), F i ∈{0,1}, 1 is foreground, 0 is background;

[0098] Step 7: Ub the work area ROI The adjacency matrix A and feature matrix X of the corresponding graph are input into the trained two-layer graph convolutional neural network GCN to obtain the binary segmentation result F'={F i '|i∈Ub ROI}, in the working area Ub ROI The categories of the original labeled pixels are kept unchanged, and the categories of the unlabeled pixels are replaced with the segmentation results of GCN, and finally the final segmentation result F is obtained. * ={F i * |i∈Ub ROI}:

[0099]

[0100] The present embodiment also provides a medical image segmentation post-processing system based on dual-branch evidence fusion and graph convolutional neural network, including a preprocessing module, a dual-branch output module, an evidence fusion module, a ROI labeling module, a graph construction module, a GCN training module and a GCN segmentation module, and the above modules correspond to the contents of steps 1 to 7 of the method of the present invention respectively.

[0101] The solution of this embodiment utilizes the pre-trained basic segmentation model MedSAM, combines evidence theory and subjective logic, and represents the output of MedSAM as the confidence in different categories of segmentation targets and a total uncertainty. The segmentation results of MedSAM are post-processed through an uncertainty-aware graph convolutional neural network with dual-branch evidence fusion, which solves the segmentation errors caused by MedSAM's overconfidence in medical image segmentation and improves the segmentation accuracy of areas with high uncertainty in MedSAM's segmentation results.

[0102] As described above, the specific implementation steps of the present invention make the present invention clearer. Within the spirit of the present invention and the protection scope of the claims, any modifications and changes made to the present invention fall within the protection scope of the present invention.

Claims

1. A medical image segmentation post-processing method based on dual-branch evidence fusion and graph convolutional neural network, characterized in that: The method comprises the following steps: Step 1: Medical image preprocessing: First, the intensity value v of each pixel i in the input medical image x is calculated. i Normalize: where v min is the minimum pixel intensity value, v max is the maximum pixel intensity value, and the adjusted pixel intensity value v i ∈[0,255], then the bicubic interpolation method is used to resize x to H×W×3, where H and W are the height and width of the processed medical image x respectively; Step 2: Get the dual-branch output, input the processed medical image x into the MedSAM encoder Encoder, and then pass the MedSAM decoder mask decoder 1 to get the lower branch output m 1 , through the decoder mask decoder 2 to get the lower branch output m 2 , where mask decoder 2 and mask decoder 1 share network parameters. Mask decoder 2 is obtained by dropout operation on the basis of mask decoder 1, that is, the spatial dot product of the output of the two transposed convolution upsampling in mask decoder 1 and the output of the fully connected layer MLP is obtained. 1 , and the spatial dot product of the output of the two transposed convolutions and the output of the last layer of MLP after the dropout operation is obtained by mask decoder 2. 2 ; Step 3: Fusion of evidence from both branches to obtain evidence from the upper and lower branches Calculate the trust value of the upper and lower branches and uncertainty Where K is the total number of categories of the segmentation target, n = 1 represents the upper branch, n = 2 represents the lower branch, and the uncertainty of the upper and lower branches is fused based on the evidence theory to obtain the fused uncertainty Step 4: Mark each pixel according to its uncertainty after fusion and determine the working area Ub ROI and Ub ROI Pixel labels with low internal uncertainty; Step 5: Take the working area Ub ROI Each pixel in the graph is a node, and a semi-labeled weighted graph is constructed through graph connection to obtain Ub ROI The connection matrix A and feature matrix X of the corresponding graph; Step 6: Ub the work area ROI The adjacency matrix A and feature matrix X of the corresponding graph are input into the two-layer graph convolutional neural network GCN to obtain the output prediction probability Z, and the parameters W of GCN are updated by minimizing the joint optimization loss function L. (0) ,W (1) } until the maximum number of iterations is reached: L=L CE +L Dice Where W (0) and W (1) is the weight parameter matrix of the first and second layers of GCN. Sigmoid and ReLU are both activation functions. That is, add a unit matrix I to the adjacency matrix A N , L CE and L Dice They are cross entropy loss and Dice loss respectively. is the label y corresponding to pixel i i =1, N is the predicted probability of the working area Ub ROI The total number of pixels with labels, F = {F i |i∈Ub ROI } is the segmentation result after the output prediction probability Z of GCN is converted into binary value by the sign function sgn (threshold is 0.5), F i ∈{0,1}, 1 is foreground, 0 is background; Step 7: Ub the work area ROI The adjacency matrix A and feature matrix X of the corresponding graph are input into the trained two-layer graph convolutional neural network GCN to obtain the binary segmentation result F'={F i '|i∈Ub ROI }, in the working area Ub ROI The categories of the original labeled pixels are kept unchanged, and the categories of the unlabeled pixels are replaced with the segmentation results of GCN, and finally the final segmentation result F is obtained. * ={F i * |i∈Ub ROI }:

2. The medical image segmentation post-processing method based on dual-branch evidence fusion and graph convolutional neural network as claimed in claim 1, characterized in that: The process of step three is as follows: 3.1 The output m of the dual branch 1 and m 2 The two-branch evidence e is obtained by Softplus activation function respectively n : e n =Softplus(m n ), n=1,2 (2) 3.2 Based on the two-branch evidence e n , take uniform distribution, and get the parameters of the two-branch Dirichlet distribution and the Dirichlet distribution intensity in a n =e n +1, n=1.2 (3) 3.3 Calculate the classification confidence value of each pixel i output by the dual branches and uncertainty for: 3.4 Fusion of the uncertainty of the upper and lower branches to obtain the fused uncertainty 3. The medical image segmentation post-processing method based on dual-branch evidence fusion and graph convolutional neural network as described in claim 1 or 2, characterized in that: The process of step 4 is as follows: 4.1 Based on the uncertainty after fusion Uncertain level marking Ub = {Ub i |i=1,2,...,H×W}: Where τ is the uncertainty threshold. If Ub i =1 means that the i-th pixel is a pixel with high uncertainty, and Ub i =0 means that the i-th pixel is a pixel with low uncertainty; 4.2 Determine the working area Ub ROI : Ub ROI =Ub∪Eb (8) P=Sigmoid(m 1 ) (9) Where P = {P i,j |i=1,2,...,H×W,j=1,2,...,K} is the output of the lower branch of MedSAM 1 After the Sigmoid activation function, the predicted probability value of pixel classification is obtained, Eb = {Eb i |i=1,2,...,H×W} is the segmentation result after P is converted into binary by the sign function sgn, where Eb i ∈{0,1}, 1 is foreground, 0 is background, ∪ means adding the two first, and then using the sign function sgn (threshold is 1) to convert the result of the addition into a binary mask, where 0 is the background with low uncertainty, 1 is the pixel with high uncertainty and the foreground with low uncertainty, and finally the area with all 1 is taken as Ub ROI ; 4.3 Determine Ub according to the uncertainty level ROI Labels for pixels with low uncertainty: Unlabeled means there is no label.

4. The medical image segmentation post-processing method based on dual-branch evidence fusion and graph convolutional neural network according to claim 1 or 2, characterized in that the process of step 5 is as follows: 5.1 For any i-th pixel, create connections with its four neighbors above, below, left and right, and also create a connection in Ub ROI Some nodes are randomly selected to connect with it, and the pixel value v = {v i |i=1,2,...,H×W} and the predicted probability value P corresponding to the output of the lower branch of MedSAM={P i,j |i=1,2,...,H×W,j=1,2,...,K} to calculate the edge weights and get Ub ROI The connection matrix of the corresponding graph is A = {w i,j |i∈Ub ROI ,j∈Ub ROI }: weight2(x i ,x j )=||v i -v j || 2 (13) weight3(x i ,x j )=||x i -x j || 2 (14) in, x i ,x j Ub ROI The positions of the i-th and j-th pixels in , λ is a balancing factor, weight1(·) is determined by the diversity between the two pixels, and weight2(x i ,x j ) and weight3(x i ,x j ) are x i ,x j The pixel weight and distance weight between two points, σ1 and σ2 are weight2(x i ,x j ) and weight3(x i ,x j ), in formula (12), when the segmentation target is only the foreground and the background, K = 2, where P i,1 The probability value of the i-th pixel output by the lower branch of MedSAM is predicted as the foreground, and the probability value predicted as the background is P i,2 =1-P i,1 , In formula (13) and formula (14), || || represents the Euclidean distance; 5.2 For Ub ROI Each pixel in the , its characteristic value includes the branch output m of the MedSAM model 1 The standardized weighted eigenvalue M = {M i |i∈Ub ROI }, predicted probability value P = {P i,j |i∈Ub ROI , j=1,2,...K} and the uncertainty after fusion Get Ub ROI The corresponding feature matrix X: Among them, v μ is the output of the MedSAM branch m 1 The average value, v α is m 1 The variance of .

5. A system implementing the medical image segmentation post-processing method based on dual-branch evidence fusion and graph convolutional neural network as claimed in claim 1, characterized in that: The system includes an image preprocessing module, a dual-branch output module, an evidence fusion module, a ROI marking module, a graph construction module, a GCN training module and a GCN segmentation module, and each module sequentially implements the processing of steps 1 to 7 of the method.

Citation Information

Patent Citations

  • Graffiti supervision maxillary sinus segmentation method and device based on superpixel and double-branch graph convolution network

    CN118096812A

  • End-to-end hyperspectral image classification method, system, device and terminal

    CN118262244A

  • Semantic segmentation network model uncertainty quantification method based on evidence inference

    US20230084910A1

Cited By

  • Trusted medical image segmentation method and equipment progressively guided by evidence uncertainty

    CN119625319A

  • Double-branch deep neural network uncertainty evaluation method

    CN121388517A

  • A method for evaluating uncertainty in a dual-branch deep neural network

    CN121388517B