A fundus vessel segmentation method based on rotation and scaling equivariant networks

Abstract

The present invention discloses a fundus blood vessel segmentation method based on a rotation and scaling equivariant network. By utilizing the information losslessness of group theory and Fourier expansion, aiming at the characteristics of small scale and complex local morphology of eyeball blood vessels, the symmetry of local features of eyeball blood vessels is accurately characterized, a high-precision fundus blood vessel segmentation network that is equivariant to rotation and scaling transformations is constructed, and the current optimal segmentation ability is achieved in both intra-dataset and cross-dataset experiments, and the number of parameters is only 13.9% of the basic network. This demonstrates the excellent expression accuracy, outstanding generalization ability and great clinical application potential of the rotation and scaling equivariant network designed by the present invention based on Fourier parameterization.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing and deep learning, and particularly relates to a fundus vascular segmentation method based on a rotation and scaling equivariant network. Background Art

[0002] In clinical diagnosis, the geometric morphological changes of the fundus vascular system are highly correlated with some diseases, such as glaucoma, diabetes, hypertension, etc. Therefore, through fundus vascular segmentation, doctors can better observe the morphological characteristics and structural changes of the fundus vascular system, so as to diagnose and track many diseases in the early stage. However, in clinical practice, due to the complexity of the fundus vascular structure, manual annotation is time-consuming, laborious and subjective. Therefore, an effective automated fundus vascular segmentation algorithm is of great significance for ophthalmologists in the clinical evaluation process.

[0003] However, due to the low contrast of fundus images, it is often difficult to separate the fundus vessels from the background, and some fundus pathological exudates are often misclassified as vessels. Coupled with the diversity of the fundus vascular structure, these greatly increase the difficulty of fundus vascular segmentation and affect the segmentation results. Traditional fundus vascular segmentation methods are mainly based on unsupervised image processing methods, which makes the segmentation process rely on a large number of manual features and prior knowledge, resulting in poor accuracy and generalization of traditional segmentation methods.

[0004] In recent years, convolutional neural networks have achieved great success in medical image processing. Through parameter sharing and sliding windows, convolutional neural networks not only enhance the robustness of the model, but also successfully depict the translational symmetry, a widespread image prior. However, for fundus vascular images, rotational symmetry and scale symmetry often appear simultaneously and widely exist in the morphological structure of fundus vessels. However, traditional convolutional operators do not depict the rotational and scaling symmetries of fundus vascular images as they do with translational symmetry. For such problems, data augmentation is the most widely used method. By rotating and scaling the image as a whole, data augmentation enhances the model's ability to handle global symmetries, but for the symmetry changes of local features, such as the directional diversity of local eye vessels and the scale difference between capillaries and main vessels, the data augmentation method still cannot depict them well. And since the scale of eye vessels usually varies between 1 and 20 pixels and has the characteristics of small scale and complex local morphology, the fitting ability of the network is also crucial for the accuracy performance of the segmentation result.

[0005] Therefore, for the fundus vascular segmentation task, it is very necessary to study how to construct a network that is equivariant to rotation and scaling transformations and has high-precision expression ability. Summary of the Invention

[0006] To overcome the above-mentioned drawbacks of the prior art, the purpose of the present invention is to provide a fundus vascular segmentation method based on a rotation and scaling equivariant network, in order to solve the problem of low segmentation accuracy due to the small scale of the eye blood vessels and the complex local morphology.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A fundus vascular segmentation method based on a rotation and scaling equivariant network, comprising the following steps:

[0009] Step 1: Obtain fundus vascular images, annotate the images to obtain fundus vascular masks, and use the paired fundus vascular images and corresponding masks as a data set;

[0010] Step 2: According to the geometric characteristics of the fundus blood vessels, use the symmetry of the group to design and construct a convolutional framework that is equivariant to rotation and scaling transformations; according to the accuracy requirements of fundus blood vessel segmentation, use the information losslessness of Fourier expansion to design and construct a parameterized convolutional kernel with high accuracy both statically and dynamically; combine the convolutional framework and the parameterized convolutional kernel to obtain a Fourier parameterized rotation and scaling equivariant convolutional kernel;

[0011] Step 3: Replace the convolutional kernels in the existing fundus vascular segmentation network with the Fourier parameterized rotation and scaling equivariant convolutional kernels to obtain a Fourier parameterized rotation and scaling equivariant network;

[0012] Step 4: Load the fundus vascular images in Step 1, perform data augmentation, and input them together with the corresponding hyperparameters into the Fourier parameterized rotation and scaling equivariant network. According to the training loss function, use the backpropagation optimization algorithm to iteratively update the network parameters, so that the output result of the network gradually approaches the fundus vascular mask annotated in Step 1. When the set number of iterations is reached, the training terminates, and the network parameters at this time are saved as the training model;

[0013] Step 5: Prepare the fundus vascular images to be tested and the corresponding fundus vascular masks, load the training model, and input the fundus vascular images to be tested and the corresponding fundus vascular masks into the Fourier parameterized rotation and scaling equivariant network for forward calculation. The output result of the network is the fundus vascular segmentation result of the network.

[0014] Compared with the prior art, the present invention has the following beneficial technical effects:

[0015] The present invention provides a construction method of a rotation and scaling equivariant network based on Fourier parameterization for the task of eyeball blood vessel segmentation. Compared with the existing eyeball blood vessel segmentation methods, the present invention fully utilizes the theory of group theory to characterize the symmetry of the local features of eyeball blood vessels by establishing a mapping from the group domain to the real number domain, and embeds equivariance to rotation and scaling transformations into the network; and based on the information losslessness of Fourier expansion, the present invention constructs a parameterized basis with high precision both in static and dynamic states, so that the problem of low segmentation accuracy caused by the small scale and complex local morphology of fundus blood vessels is better solved. Without changing the existing network structure, the present invention achieves the current optimal segmentation ability in both same-dataset and cross-dataset experiments only by replacing the convolution kernel, and the number of parameters is only 13.9% of the basic network, demonstrating the excellent expression accuracy, excellent generalization ability and great clinical application potential of the rotation and scaling equivariant network based on Fourier parameterization designed by the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention will be further described with reference to the accompanying drawings, but the content in the drawings does not constitute any limitation to the present invention.

[0017] Figure 1 It is a flowchart of a fundus blood vessel segmentation method based on a rotation and scaling equivariant network based on Fourier parameterization of the present invention.

[0018] Figure 2 It is a comparison diagram of the prior schematic diagram of symmetry existing in fundus blood vessel images and the convolution results after random initialization of a rotation and scaling equivariant network with Fourier parameterization and a common convolution network. Among them, (a) shows the translational symmetry in fundus blood vessel images; (b) shows the rotational symmetry and scaling symmetry in fundus blood vessel images; (c) shows the rotational and scaling symmetry in fundus blood vessel images; (d) shows the convolution result after random initialization of a common convolution network; (e) shows the convolution result after random initialization of a rotation and scaling equivariant network with Fourier parameterization.

[0019] Figure 3 It is a comparison schematic diagram of a rotation and scaling equivariant convolution with Fourier parameterization and a common convolution. Among them, (a) is the rotation and scaling equivariant convolution structure of the starting layer; (b) is the rotation and scaling equivariant convolution structure of the middle layer; (c) is the corresponding common convolution structure to (a); (d) is the corresponding common convolution structure to (b).

[0020] Figure 4For training and testing on the same dataset, the following is a comparison chart of the results of several comparison methods on the three datasets of DRIVE, STARE, and CHASE_DB1. The dataset name is marked in the upper left corner, pointing from the training dataset to the test dataset. The enlarged partial results are marked with a red box. The segmentation results of FRS U-Net and FRS Iter-Net are the results of the Fourier-parameterized rotation and scale-equivariant convolutional networks, and the others are the segmentation results of the comparison methods.

[0021] Figure 5 For training and testing on different datasets, the following is a comparison chart of the results of several comparison methods on the three datasets of DRIVE, STARE, and CHASE_DB1. The dataset name is marked in the upper left corner, pointing from the training dataset to the test dataset. The enlarged partial results are marked with a red box. The segmentation results of FRS U-Net and FRS Iter-Net are the results of the Fourier-parameterized rotation and scale-equivariant convolutional networks, and the others are the segmentation results of the comparison methods. Specific implementation manner

[0022] The following will describe the implementation manner of the present invention in detail with reference to the accompanying drawings and embodiments.

[0023] Embodiment 1

[0024] Train and test on the three publicly available datasets of DRIVE, STARE, and CHASE_DB1 respectively; the image resolutions are 584×565, 700×605, and 999×960 respectively; the dataset partitions are (training / test) 20 / 20, 16 / 4, and 20 / 8 respectively; the test metrics include five metrics: sensitivity, specificity, F1-score, accuracy, and area under the ROC curve (AUC); the Adam optimizer is used to update the parameters, with the corresponding learning rate of 0.0002, the batch size of 2, and 200 iterations, and the threshold for final probability Figure 2 value quantization is 0.5; all experiments are completed on an NVIDIA 3090 GPU and implemented using the Pytorch framework.

[0025] Reference Figure 1 , the present invention successively includes the following steps:

[0026] Step 1: Obtain fundus vascular images, annotate the images to obtain fundus vascular masks, and use the paired fundus vascular images and corresponding masks as the dataset.

[0027] In this embodiment, a dataset is selected from the three publicly available datasets of DRIVE, STARE, and CHASE_DB1 to obtain fundus vascular images, and the annotated fundus vascular masks are obtained. The paired fundus vascular images and the corresponding masks are split according to the above experimental settings and used as the training dataset and the test dataset respectively. Exemplarily, a feasible image annotation method is as follows: the blood vessels in the fundus vascular image are labeled as 1, and the remaining parts except the blood vessels are labeled as 0, and finally the fundus vascular mask is obtained.

[0028] Step 2: According to the geometric features of the fundus blood vessels, a convolutional framework that is equivariant to rotation and scaling transformations is designed and constructed using the symmetry of the group, and is expressed as follows:

[0029] In the convolutional framework, the rotation and scaling equivariant convolution of the starting layer is mapped as:

[0030]

[0031] where \(t\in T\), is the input image, is the convolutional kernel of the starting layer, \(\mu_0\) is the initial scale, \(\mu\) is the scaling ratio, and the rotation matrix where \(\sigma\) is the Haar measure on the group, for there is \(d\sigma(t_0)=dt_0\), and the rotation and scaling equivariant convolution mapping \(\varPhi\) of the starting layer R maps the input image defined on to the feature map defined on the group \(H\), where \(R\) is the rotation group, \(S\) is the scale group, \(T\) is the translation group, is the rotation angle of the convolutional kernel, is the scale level of the convolutional kernel, is the plane coordinate of the convolutional kernel, \(\times\) is the direct product, is the semi-direct product.

[0032] The rotation and scaling equivariant convolution mapping of the intermediate layer is:

[0033]

[0034] where, is the feature map, is the convolutional kernel of the intermediate layer, and the rotation and scaling equivariant convolution mapping \(\varPhi\) of the intermediate layer H maps a feature map defined on the group \(H\) to another feature map defined on the group \(H\).

[0035] The above equations (1) and (2) satisfy the following equalities:

[0036]

[0037] where, the group action of the starting layer Group action on the intermediate layer They are respectively:

[0038]

[0039] Among them, r is the rotation angle and s is the scale level. It can be proved from this that the convolution frameworks constructed by the above equations (1) and (2) have equivariance to rotation and scaling transformations.

[0040] In this embodiment, the discrete rotation group R is set as The discrete scale group S is set as That is, μ0 = 3 and μ = 1.25.

[0041] Step 3: According to the accuracy requirements of fundus blood vessel segmentation, design a parameterized convolution kernel with high precision both statically and dynamically by using the information losslessness of Fourier expansion; the expression is as follows:

[0042]

[0043] Among them, ψ(x) is the parameterized convolution kernel, and are respectively the sine and cosine basis functions, is the two-dimensional plane coordinate, p is the convolution kernel size, k, l = 0, 1, …, p - 1, a kl and b kl are the expansion coefficients, h′ is the reparameterized image grid size, Ω(x) ≥ 0 is the radial mask, and when ||x|| ≥ (p + 1 / 2)h′, Ω(x) = 0.

[0044] The above reparameterized image grid size h′ is:

[0045]

[0046] Among them, h is the original image grid size, s is the convolution kernel scale level, s′ is the convolution kernel size. By reparameterizing the grid size, the convolution kernel size and scale can be decoupled.

[0047] Among them, the expanded base frequency p is set to 6.

[0048] Step 4: Replace ψ(t) in equation (1) and ψ(r, s, t) in equation (2) with ψ(x) in equation (5) respectively. Thus, the convolution framework and the parameterized convolution kernel can be combined to obtain a Fourier parameterized rotation and scaling equivariant convolution kernel.

[0049] Step 5: Replace the traditional convolution kernel of the existing fundus blood vessel segmentation network with the above designed Fourier parameterized rotation and scaling equivariant convolution kernel to obtain a Fourier parameterized rotation and scaling equivariant network.

[0050] In an embodiment of the present invention, the existing fundus vascular segmentation network is a U-Net or Iter-Net medical image segmentation network, and the obtained Fourier parameterized rotation and scaling equivariant network is an FRS U-Net or an FRS Iter-Net. The number of unfolding iterations of both Iter-Net and FRS Iter-Net is 2.

[0051] Step 6: Load the fundus vascular image in Step 1, perform data augmentation on it in ways such as rotation, scale scaling, flipping, shearing, brightness change, saturation change, and contrast change, and input it together with the corresponding hyperparameters into the constructed Fourier parameterized rotation and scaling equivariant network (FRS U-Net or FRS Iter-Net); according to the training loss function, iteratively update the network parameters through the backpropagation optimization algorithm, so that the output result of the network gradually approaches the fundus vascular mask labeled in Step 1. When the set number of iterations is reached, the training terminates, and the network parameters at this time are saved, which is the training model.

[0052] The training steps specifically include:

[0053] (1) Determine the hyperparameters, including the discrete rotation group R and scale group S and the base Fourier expansion frequency p, and input the hyperparameters, the data-augmented fundus vascular image, and the fundus vascular mask into the Fourier parameterized rotation and scaling equivariant network.

[0054] (2) Calculate the loss function L of the Fourier parameterized rotation and scaling equivariant network;

[0055] (3) Use the stochastic gradient descent algorithm to update the Fourier parameterized rotation and scaling equivariant network parameters and optimize the loss function L.

[0056] Exemplarily, the loss function L determined in the embodiment of the present invention is:

[0057]

[0058] where y i is the mask label of each point of the fundus vascular image, and q i is the corresponding predicted probability.

[0059] Step 7: Prepare the fundus vascular image to be tested and the corresponding fundus vascular mask, load the training model, input the fundus vascular image to be tested and the corresponding fundus vascular mask into the Fourier parameterized rotation and scaling equivariant network for forward calculation, and the output result of the network is the fundus vascular segmentation result of the network.

[0060] Under the condition that all experimental conditions are the same, the comparative experimental results of the present invention and other 10 eye blood vessel segmentation methods in the same dataset training and testing on three public datasets, namely DRIVE, STARE, and CHASE_DB1, are shown in Table 1. Some segmentation results are as Figure 4 shown. It can be seen that the segmentation results of the present invention far exceed those of other comparative methods, demonstrating higher expression accuracy and application potential of the present invention.

[0061] Table 1 Comparative experimental results of FRS U-Net and FRS Iter-Net with other comparative methods on DRIVE, STARE, and CHASE_DB1 in the same dataset. Bold and italic respectively represent the first and second places.

[0062]

[0063] Example 2

[0064] The experimental conditions and steps of Example 2 are the same as those of Example 1. The only difference lies in Step 1. In Example 2, Step 1 is as follows: Select two datasets from the three public datasets of DRIVE, STARE, and CHASE_DB1 to obtain fundus blood vessel images and obtain labeled fundus blood vessel masks. Split the paired fundus blood vessel images and corresponding masks according to the experimental settings in Example 1. Use one dataset as the training dataset and the other dataset as the test dataset, so as to achieve cross-dataset test evaluation.

[0065] Under the condition that all experimental conditions are the same, the comparative experimental results of the present invention and other 10 eye blood vessel segmentation methods in the cross-dataset training and testing on three public datasets, namely DRIVE, STARE, and CHASE_DB1, are shown in Table 2. Some segmentation results are as Figure 5 shown. It can be seen that the segmentation results of the present invention far exceed those of other comparative methods, demonstrating stronger generalization ability and objective clinical application prospects of the present invention.

[0066] Table 2 Comparative experimental results of FRS U-Net and FRS Iter-Net with other comparative methods on DRIVE, STARE, and CHASE_DB1 in the cross-dataset. Bold and italic respectively represent the first and second places.

[0067]

[0068] To verify the effect of the method shown in the present invention, Figure 2 (a)-(c) respectively show the translational symmetry, rotational symmetry, scaling symmetry, and rotational and scaling symmetry existing in the fundus blood vessel image; Figure 2Figures (d) and (e) show the convolution results of a regular convolutional network and a Fourier-parameterized rotation- and scale-equivariant network without training. It can be seen that, compared with Figure 2 in Figure (d), Figure 2 the convolution result in Figure (e) has stronger structure, indicating that, compared with a regular convolutional network, the Fourier-parameterized rotation- and scale-equivariant network has a stronger ability to depict the structural features of fundus blood vessels. Figure 3 Figures (a)-(d) respectively show the structural differences between the Fourier-parameterized rotation- and scale-equivariant convolution and the regular convolution. It can be seen that, compared with the randomness of the regular convolution, the Fourier-parameterized rotation- and scale-equivariant convolution has more structure, that is, the convolution kernels of the same pattern are cyclically translated in the angular dimension and the scale dimension. Figure 4 This is a partial result display of the in-domain experiment of Example 1. It can be seen that, compared with other comparison methods, the Fourier-parameterized rotation- and scale-equivariant convolutional network has a better depiction of the detailed features of fundus blood vessels, demonstrating the excellent expression accuracy of the method of the present invention. Figure 5 This is a partial result display of the cross-domain experiment of Example 2. It can be seen that, compared with other comparison methods, the Fourier-parameterized rotation- and scale-equivariant convolutional network has a better depiction of the overall structure of fundus blood vessels, demonstrating the excellent generalization ability of the method of the present invention.

[0069] In summary, by utilizing the information losslessness of group theory and Fourier expansion, the present invention accurately depicts the symmetry of the local features of eyeball blood vessels in view of the characteristics of small scale and complex local morphology of eyeball blood vessels, constructs a high-precision fundus blood vessel segmentation network that is equivariant to rotation and scale transformations, and respectively achieves the current optimal segmentation ability in the same-dataset and cross-dataset experiments, with the number of parameters being only 13.9% of the basic network. This demonstrates the excellent expression accuracy, excellent generalization ability and great clinical application potential of the Fourier-parameterized rotation- and scale-equivariant network designed by the present invention.

Claims

1. A fundus vascular segmentation method based on a rotation and scaling equivariant network, characterized in that, It includes the following steps: Step 1: Obtain fundus vascular images, annotate the images to obtain fundus vascular masks, and use the paired fundus vascular images and corresponding masks as a dataset; Step 2: According to the geometric characteristics of the fundus vessels, use the symmetry of the group to design and construct a convolutional framework that is equivariant to rotation and scaling transformations; according to the accuracy requirements of fundus vessel segmentation, use the information losslessness of Fourier expansion to design and construct a parameterized convolutional kernel with high precision both statically and dynamically; combine the convolutional framework and the parameterized convolutional kernel to obtain a Fourier-parameterized rotation and scaling equivariant convolutional kernel; Step 3: Replace the convolutional kernels in the existing fundus vessel segmentation network with the Fourier-parameterized rotation and scaling equivariant convolutional kernels to obtain a Fourier-parameterized rotation and scaling equivariant network; Step 4: Load the fundus vascular images in Step 1, perform data augmentation, and input them together with the corresponding hyperparameters into the Fourier-parameterized rotation and scaling equivariant network. According to the training loss function, use the backpropagation optimization algorithm to iteratively update the network parameters so that the output result of the network gradually approaches the fundus vascular mask annotated in Step 1. When the set number of iterations is reached, the training terminates, and the network parameters at this time are saved, which is the training model; Step 5: Prepare the fundus vascular images to be tested and the corresponding fundus vascular masks, load the training model, input the fundus vascular images to be tested and the corresponding fundus vascular masks into the Fourier-parameterized rotation and scaling equivariant network for forward calculation, and the output result of the network is the fundus vascular segmentation result of the network; In Step 2, in the convolutional framework, the rotation and scaling equivariant convolution mapping of the starting layer is: where \(t\in T\), is the input image, is the convolutional kernel of the starting layer, \(\mu_0\) is the initial scale, \(\mu\) is the scaling ratio, and the rotation matrix where \(\sigma\) is the Haar measure on the group, for there is \(d\sigma(t_0)=dt_0\), and the rotation and scaling equivariant convolutional mapping \(\varPhi\) of the starting layer R maps the input image defined on to the feature map defined on the group \(H\), where \(R\) is the rotation group, \(S\) is the scale group, and \(T\) is the translation group, is the rotation angle of the convolutional kernel, is the scale level of the convolutional kernel, is the planar coordinate of the convolutional kernel, \(\times\) is the direct product, is the semidirect product; The rotation and scaling equivariant convolution mapping of the middle layer is: Among them, is the feature map, is the intermediate layer convolution kernel, and the rotation and scaling equivariant convolution mapping Φ of the intermediate layer H maps a feature map defined on the group H to another feature map defined on the group H; The parameterized convolutional kernel is: where ψ(x) is a parameterized convolutional kernel, and are the sine and cosine basis functions respectively, is the two-dimensional plane coordinate, p is the convolutional kernel size, k, l = 0, 1, …, p - 1, a kl and b kl are the expansion coefficients, h ′ is the reparameterized image grid size, Ω(x) ≥ 0 is the radial mask, and Ω(x) = 0 when ||x|| ≥ (p + 1 / 2)h′.

2. The fundus vascular segmentation method based on the rotation and scaling equivariant network according to claim 1, wherein In Step 1, the image annotation method is: label the blood vessels in the fundus vascular image as 1, and label the rest except the blood vessels as 0, and finally obtain the fundus vascular mask.

3. The method for fundus blood vessel segmentation based on rotation and scaling equivariant network according to claim 1, characterized in that The reparameterized image grid size h ′ is as follows: where h is the original grid size of the image, s is the convolutional kernel scale level, and s ′ is the convolutional kernel size. By reparameterizing the grid size, the convolutional kernel size can be decoupled from the scale.

4. The method for fundus blood vessel segmentation based on rotation and scaling equivariant network according to claim 1, characterized in that, In Step 2, replace ψ(t) in the rotation and scaling equivariant convolution mapping formula of the starting layer and ψ(r, s, t) in the rotation and scaling equivariant convolution mapping formula of the middle layer with the parameterized convolutional kernel ψ(x) respectively, and thus combine the convolutional framework and the parameterized basis to obtain a Fourier-parameterized rotation and scaling equivariant convolutional kernel.

5. The fundus vascular segmentation method based on the rotation and scaling equivariant network according to any one of claims 1 to 4, characterized in that, In Step 3, the existing fundus vessel segmentation network is a U-Net or Iter-Net medical image segmentation network.

6. The method for fundus vascular segmentation based on a rotation and scaling equivariant network according to claim 5, wherein In Step 4, the data augmentation includes: rotation, scale scaling, flipping, shearing, brightness change, saturation change, and contrast change.

7. The fundus vascular segmentation method based on the rotation and scaling equivariant network according to claim 5, wherein In Step 4, the training steps include: (1) Determine the hyperparameters, including the discrete rotation group R and scale group S and the base Fourier expansion frequency p, and input the hyperparameters, the fundus vascular images and fundus vascular masks after data augmentation into the Fourier-parameterized rotation and scaling equivariant network together; (2) Calculate the loss function L of the Fourier-parameterized rotation and scaling equivariant network; (3) Use the stochastic gradient descent algorithm to update the parameters of the Fourier-parameterized rotation and scaling equivariant network and optimize the loss function L.

8. The fundus blood vessel segmentation method based on a rotation and scaling equivariant network according to claim 7, wherein The loss function L is: where y i is the mask label of each point in the fundus vascular image, and q i is the corresponding predicted probability.

Images