Uncertainty-guided retinal vascular image segmentation method and system

Abstract

The present invention discloses an uncertainty-guided retinal vascular image segmentation method and system, which belongs to the field of medical image processing. The present invention first constructs an uncertainty network to quantify the uncertainty in the fundus image to obtain an uncertainty mapping matrix, and uses the mean value of the uncertainty network output as the prior mapping matrix; then constructs a segmentation network based on UNet, and performs multiple feature fusions on the feature map obtained in the encoding stage and the uncertainty mapping matrix to learn the uncertainty area in the fundus image; the loss function of the segmentation network is composed of a prior loss function and a cross-entropy loss function, wherein the prior loss function is constructed based on the prior mapping matrix; then, the fundus image training set is input into the segmentation network to train the segmentation network; finally, the fundus image is input into the trained segmentation network to obtain a retinal vascular segmentation image. The present invention effectively improves the retinal vascular segmentation performance and has practical clinical value.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and more specifically, relates to an uncertainty-guided retinal vascular image segmentation method and system. Background Art

[0002] Fundus images, as a non-invasive means of vascular imaging, have become a crucial reference for doctors in disease diagnosis due to their readily available availability. The retina contains a vast and complex network of blood vessels, which branch into a complete vascular tree. The branching morphology of this tree, including the caliber of vessels near and far from the optic disc, the angles at which they branch, and the degree of curvature, all reflect the health of the human body. Therefore, accurate vessel extraction or segmentation from captured fundus images is crucial in clinical applications.

[0003] Currently, retinal vessel segmentation primarily relies on traditional methods such as matched filtering, morphological processing, and multi-scale analysis, as well as deep learning-based approaches. While many deep learning methods modify network structures to fully capture contextual information, and some work addresses the issue of noisy labels in datasets, these methods neglect the use of prior knowledge in retinal images and rarely address the medical risks associated with uncertainty in retinal vessel segmentation results. Summary of the Invention

[0004] In response to the above defects or improvement needs of the prior art, the present invention provides an uncertainty-guided retinal vessel image segmentation method and system, which aims to improve the accuracy of retinal vessel segmentation.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for training a retinal vessel image segmentation network guided by uncertainty, the method comprising:

[0006] A Bayesian network is constructed using the MC Dropout method to quantify the uncertainty in the fundus image to obtain an uncertainty mapping matrix, and the mean of the Bayesian network output is used as the prior mapping matrix;

[0007] Constructing a UNet-based segmentation network, in which the feature map obtained in the encoding stage is fused with the uncertainty mapping matrix multiple times to learn the uncertain areas in the fundus image; the loss function of the segmentation network is composed of a prior loss function and a cross-entropy loss function, wherein the prior loss function is constructed based on the prior mapping matrix;

[0008] The fundus image training set is input into the segmentation network for network training until the segmentation network training is completed.

[0009] Preferably, the uncertainty mapping matrix and the priori mapping matrix are respectively:

[0010]

[0011]

[0012] Where T is the number of sampling; is the prediction output of the network; is the model parameter of the network; Sigmoid() is the activation function.

[0013] Preferably, the loss function of the segmentation network is:

[0014] L=α·L prior +(1-α)·L hard -log(α·(1-α))

[0015] Among them, L prior is the prior loss function; L hard is the cross entropy loss function; α is a factor between 0 and 1;

[0016]

[0017]

[0018]

[0019] Among them, μ is the prior mapping matrix; is the prediction of the segmentation network; y is the binary true label; Sigmoid() is the activation function; ε is the adaptive parameter; e is a natural constant.

[0020] Preferably, in the segmentation network based on UNet, between each maximum pooling operation and convolution operation, the feature map obtained by the convolution operation is fused with the uncertainty mapping matrix; the feature map obtained after the feature fusion is spliced in the channel dimension after the maximum pooling operation and the average pooling operation, and then a convolution operation is performed to restore the original number of channels.

[0021] In a second aspect, the present invention provides an uncertainty-guided retinal vessel image segmentation method, the method comprising:

[0022] Inputting the fundus image into the segmentation network;

[0023] The segmentation network outputs a retinal blood vessel segmentation image;

[0024] The segmentation network is trained according to one or more methods in the first aspect.

[0025] In a third aspect, the present invention provides a training device for an uncertainty-guided retinal vessel image segmentation network, the device comprising:

[0026] An uncertainty extraction module is used to construct a Bayesian network using the MC Dropout method to quantify the uncertainty in the fundus image to obtain an uncertainty mapping matrix, and use the mean of the Bayesian network output as a priori mapping matrix;

[0027] A segmentation network construction module is used to construct a segmentation network based on UNet. In the segmentation network, the feature map obtained in the encoding stage is multiple-time feature fused with the uncertainty mapping matrix to learn the uncertainty area in the fundus image; the loss function of the segmentation network is composed of a prior loss function and a cross-entropy loss function, wherein the prior loss function is constructed based on the prior mapping matrix;

[0028] The segmentation network training module is used to input the fundus image training set into the segmentation network for network training until the segmentation network training is completed.

[0029] Preferably, the uncertainty mapping matrix and the priori mapping matrix are respectively:

[0030]

[0031]

[0032] Where T is the number of sampling; is the prediction output of the network; is the model parameter of the network; Sigmoid() is the activation function.

[0033] Preferably, the loss function of the segmentation network is:

[0034] L=α·L prior +(1-α)·L hard -log(α·(1-α))

[0035] Among them, L prior is the prior loss function; L hard is the cross entropy loss function; α is a factor between 0 and 1;

[0036]

[0037]

[0038]

[0039] Among them, μ is the prior mapping matrix; is the prediction of the segmentation network; y is the binary true label; Sigmoid() is the activation function; ε is the adaptive parameter; e is a natural constant.

[0040] Preferably, in the segmentation network based on UNet, between each maximum pooling operation and convolution operation, the feature map obtained by the convolution operation is fused with the uncertainty mapping matrix; the feature map obtained after the feature fusion is spliced in the channel dimension after the maximum pooling operation and the average pooling operation, and then a convolution operation is performed to restore the original number of channels.

[0041] In a fourth aspect, the present invention provides an uncertainty-guided retinal vessel image segmentation system, the system comprising:

[0042] An input module, configured to input the fundus image into the segmentation network;

[0043] a segmentation module, configured to output a retinal vessel segmentation image through the segmentation network;

[0044] The segmentation network is trained according to one or more methods in the first aspect.

[0045] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0046] In the method of the present invention, the uncertainty in the fundus image is first quantified by constructing an uncertainty network to obtain an uncertainty mapping matrix, and the average of multiple outputs of the uncertainty network in this stage is used as prior knowledge; the uncertainty mapping matrix is then integrated into the segmentation network so that it can focus on learning areas with high uncertainty in the fundus image. At the same time, the present invention designs an adaptive loss function, and avoids the influence of erroneous prior knowledge on the algorithm performance by setting the weights of the prior loss and the cross-entropy loss as hyperparameters that can be learned by the algorithm; thus, the method of the present invention effectively alleviates the problem of the influence of uncertainty in the fundus image on the performance of the segmentation algorithm, and solves the problem of the difficulty in designing prior knowledge and how to learn correct prior knowledge in the retinal blood vessel segmentation algorithm; the present invention effectively improves the performance of the retinal blood vessel segmentation method and has practical clinical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 1 is a schematic diagram of the overall process of the retinal blood vessel segmentation method provided by an embodiment of the present invention;

[0048] Figure 2 3 is a schematic diagram of the structure of the retinal blood vessel segmentation network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0050] like Figure 1 As shown, a two-stage retinal vessel segmentation model is proposed in this embodiment:

[0051] In the first stage, we use the MC Dropout method to quantify the uncertainty in the fundus image through approximate Bayesian inference, thereby constructing an uncertainty network to obtain the uncertainty mapping matrix, and taking the mean of multiple outputs of the uncertainty network as prior knowledge.

[0052] In the second stage, we propose an uncertainty fusion module and embed it into the UNet-based segmentation network. By fusing the uncertainty mapping matrix with the feature map of the network encoding stage multiple times, it can focus on learning the highly uncertain areas in the network of the previous stage. At the same time, we use prior knowledge as soft labels and propose a new prior loss function to assist the cross-entropy loss function of the binary label to constrain and optimize the second-stage segmentation network. By combining the prior loss function and the cross-entropy loss function, this embodiment designs an adaptive loss function. By setting the weights of the prior loss and the cross-entropy loss as learnable hyperparameters, it avoids the impact of incorrect prior knowledge on the performance of the segmentation network.

[0053] In this embodiment, the MC Dropout method is used to construct a Bayesian network to model the uncertainty network model. The MCDropout method is equivalent to performing approximate variational inference. This method attempts to find a simple distribution Minimize the KL divergence between the model's true posterior probability p(W|X,Y). During testing, enable the dropout layer to randomly inactivate neurons in the network, thereby jittering the network. The presence of the dropout layer makes the results of multiple forward predictions equivalent to the predictions of multiple different neural networks for the same input. Calculating the differences between these predictions models the uncertainty of the model's predicted values.

[0054] Specifically, in this example, 50 Monte Carlo samples of the approximate segmentation prediction probability distribution were obtained by performing multiple forward predictions. From these samples, the approximate prediction mean and variance can be calculated, where the variance σ represents the uncertainty of the model's prediction and the mean μ serves as prior knowledge.

[0055]

[0056]

[0057] Where T is the number of sampling; is the prediction output of the network; is the model parameter of the network; Sigmoid() is the activation function, the set of all x is X, and the set of all y is Y.

[0058] μ is the prior mapping matrix, which gives the segmentation result of the Bayesian network under high confidence; σ is the uncertainty mapping matrix, which gives the confidence level of the network in predicting each area in the fundus image, that is, which areas the prediction results are highly confident and which areas the prediction results are highly uncertain.

[0059] In the second stage, the prior mapping matrix gives the segmentation results of the approximate Bayesian network at a high confidence level in the form of probability scores. Compared with the binary true labels, using the prior mapping matrix as a soft label can provide the segmentation network with more information in addition to the true labels.

[0060] Based on the binary cross entropy loss function, this paper proposes the following prior loss function:

[0061]

[0062] in, is the prediction of the segmentation network, and μ is the prior mapping matrix obtained from the uncertainty network.

[0063] This example uses a binary cross entropy loss function to measure the difference between the segmentation network's prediction and the hard label. The loss function is as follows:

[0064]

[0065] Same as above. is the prediction of the segmentation network, and y is the binary true label.

[0066] Combining the above two losses, this paper proposes the following new loss function:

[0067] L=α·L prior +(1-α)·L hard -log(α·(1-α))

[0068] Among them, L prior is the prior loss function; L hardis the cross entropy loss function; α is a factor between 0 and 1; log(α·(1-α)) is the regularization term; when α=0, the network only calculates the hard loss, and when α=1, the network only calculates the prior loss. In order to fine-tune the ratio between the two losses, an adaptive parameter ε is set, and:

[0069]

[0070] Thus, the range of α is limited to 0 and 1, and the segmentation network is continuously adjusted during the training process. prior and L hard The proportion of the overall loss is used to avoid the impact of incorrect prior knowledge on model performance; where Sigmoid() is the activation function; ε is the adaptive parameter; and e is a natural constant.

[0071] like Figure 2 , which is a schematic diagram of the structure of the segmentation network based on UNet in this embodiment;

[0072] When building the segmentation network, we proposed an uncertainty fusion module, such as Figure 2 As shown in the box in the lower left corner, we embed the uncertainty fusion module into the UNet-based segmentation network. The feature map obtained in the encoding phase is fused with the uncertainty mapping matrix multiple times. In the UNet-based segmentation network, between each maximum pooling operation and convolution operation, the feature map obtained by the convolution operation is fused with the uncertainty mapping matrix. The feature map obtained after this feature fusion is concatenated in the channel dimension after maximum pooling and average pooling operations, and then a convolution operation is performed to restore it to the original number of channels. To ensure that the resolution of the input feature map of the uncertainty fusion module and the uncertainty mapping matrix are consistent, the uncertainty mapping matrix also requires an optimized pooling operation. That is, it is concatenated in the channel dimension after maximum pooling and average pooling operations, and then a convolution operation is performed to restore it to the original number of channels.

[0073] Finally, the fundus image training set is input into the UNet-based segmentation network for segmentation network training. The segmentation network training is completed when the loss between the prediction and label of the segmentation network no longer decreases and the evaluation index no longer increases.

[0074] Those skilled in the art will readily understand that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A training method for an uncertainty-guided retinal vessel image segmentation network, characterized in that: The method comprises: A Bayesian network is constructed using the MC Dropout method to quantify the uncertainty in the fundus image to obtain an uncertainty mapping matrix, and the mean of the Bayesian network output is used as the prior mapping matrix; A UNet-based segmentation network is constructed. In the segmentation network, the feature map obtained in the encoding stage is fused with the uncertainty mapping matrix multiple times to learn the uncertainty area in the fundus image. The loss function of the segmentation network is composed of a priori loss function and a cross-entropy loss function, wherein the priori loss function is constructed based on the prior mapping matrix. The loss function of the segmentation network is: in, is the prior loss function; is the cross entropy loss function; is a factor between 0 and 1; in, is the prior mapping matrix; is the prediction of the segmentation network; is the binary true label; S is the activation function; is an adaptive parameter; is a natural constant; The fundus image training set is input into the segmentation network for network training until the segmentation network training is completed.

2. The method according to claim 1, characterized in that The uncertainty mapping matrix and the priori mapping matrix are respectively: in, is the number of sampling times; ; is the prediction output of the network; are the model parameters of the network; is the activation function.

3. The method according to claim 1, characterized in that In the segmentation network based on UNet, between each maximum pooling operation and convolution operation, the feature map obtained by the convolution operation is fused with the uncertainty mapping matrix; the feature map obtained after the feature fusion is spliced in the channel dimension after the maximum pooling operation and the average pooling operation, and then a convolution operation is performed to restore the original number of channels.

4. An uncertainty-guided retinal vascular image segmentation method, characterized in that: The method comprises: Inputting the fundus image into the segmentation network; The segmentation network outputs a retinal blood vessel segmentation image; The segmentation network is trained according to the method according to any one of claims 1 to 3.

5. A training device for an uncertainty-guided retinal vessel image segmentation network, characterized in that: The device comprises: An uncertainty extraction module is used to construct a Bayesian network using the MC Dropout method to quantify the uncertainty in the fundus image to obtain an uncertainty mapping matrix, and use the mean of the Bayesian network output as a priori mapping matrix; A segmentation network construction module is used to construct a segmentation network based on UNet. In the segmentation network, the feature map obtained in the encoding stage is fused with the uncertainty mapping matrix multiple times to learn the uncertainty area in the fundus image; the loss function of the segmentation network is composed of a priori loss function and a cross entropy loss function, wherein the priori loss function is constructed based on the prior mapping matrix; the loss function of the segmentation network is: in, is the prior loss function; is the cross entropy loss function; is a factor between 0 and 1; in, is the prior mapping matrix; is the prediction of the segmentation network; is the binary true label; S is the activation function; is an adaptive parameter; is a natural constant; The segmentation network training module is used to input the fundus image training set into the segmentation network for network training until the segmentation network training is completed.

6. The device according to claim 5, characterized in that The uncertainty mapping matrix and the prior mapping matrix are: in, is the number of sampling times; ; is the prediction output of the network; are the model parameters of the network; is the activation function.

7. The device according to claim 5, characterized in that In the segmentation network based on UNet, between each maximum pooling operation and convolution operation, the feature map obtained by the convolution operation is fused with the uncertainty mapping matrix; the feature map obtained after the feature fusion is spliced in the channel dimension after the maximum pooling operation and the average pooling operation, and then a convolution operation is performed to restore the original number of channels.

8. An uncertainty-guided retinal vessel image segmentation system, characterized in that: The system comprises: An input module, configured to input the fundus image into the segmentation network; a segmentation module, configured to output a retinal vessel segmentation image through the segmentation network; The segmentation network is trained according to the method according to any one of claims 1 to 3.

Images