Retinal vessel segmentation method based on triple V-shaped network

Abstract

The invention provides a retinal vessel segmentation method based on a triple V-shaped network, and the method comprises the steps: constructing a triple V-shaped network architecture, and carrying out the extraction of different levels of features of an input retinal image through employing each V-shaped structure; wherein the first-layer V-shaped network generates a skeleton diagram of a retinal blood vessel and transmits the skeleton diagram to the second-layer V-shaped network; the second-layer V-shaped network further extracts local features of the blood vessel through a multi-view deformable blood vessel shape extraction unit to generate an auxiliary output result, and the third-layer V-shaped network receives the auxiliary output result, captures the boundary of the fine blood vessel and a complex bifurcation structure, and outputs the auxiliary output result to the multi-view deformable blood vessel shape extraction unit. Carrying out weighted summation on the feature maps processed by the multi-view deformable blood vessel shape extraction unit through an adaptive fusion module; according to the method, by combining a skeleton supervision mechanism, MDVEU and AFB, high-precision automatic segmentation of the retinal blood vessel is realized, and particularly, the method has outstanding performance in treatment of small blood vessels and bifurcation regions and has good clinical application prospects.

Description

Technical Field

[0001] The invention belongs to the technical field of blood vessel segmentation in retinal images, and in particular relates to a retinal blood vessel segmentation method based on a triple V-shaped network. Background Art

[0002] Automatic segmentation of retinal vessels plays an important role in the diagnosis of fundus diseases, especially in the early detection of diseases such as glaucoma and diabetic retinopathy. Although the existing U-shaped network has made significant progress in medical image segmentation, it still faces two major challenges in retinal vessel segmentation. First, it is difficult to capture details and contextual information in the image, especially in areas with poor image quality and low contrast; second, traditional convolutional neural networks rely on convolution kernels of fixed size and shape, which are difficult to adapt to blood vessels with different morphologies and sizes.

[0003] Therefore, it is very necessary to propose a blood vessel segmentation method in retinal images to achieve more accurate retinal blood vessel segmentation. Summary of the invention

[0004] In view of this, the object of the present invention is to provide a retinal vessel segmentation method based on a triple V-shaped network, which is guided by a skeleton supervision mechanism and adopts a multi-view deformable vessel shape extraction unit (MDVEU) to dynamically capture complex vascular topology, thereby achieving more accurate retinal vessel segmentation.

[0005] The technical solution of the present invention is: a retinal blood vessel segmentation method based on a triple V-shaped network, comprising:

[0006] Construct a triple V-shaped network architecture, and use each V-shaped structure to extract and optimize different levels of features of the input retinal image;

[0007] The first layer of V-shaped network segments the main blood vessels in the retinal image through the skeleton supervision mechanism, generates the skeleton map of the retinal blood vessels, and transmits it to the second layer of V-shaped network;

[0008] The second layer of the V-shaped network receives the skeleton image, further extracts the local features of the blood vessels through the multi-view deformable blood vessel shape extraction unit, optimizes the blood vessel branch structure, and generates an auxiliary output result, wherein the auxiliary output result contains more detailed information;

[0009] The third-layer V-shaped network receives the auxiliary output results, captures the boundaries of tiny blood vessels and complex bifurcation structures, and performs weighted summation on the feature maps processed by the multi-view deformable blood vessel shape extraction unit through an adaptive fusion module to generate the final main output, which is a high-precision segmentation result.

[0010] Preferably, the skeleton map of the retinal blood vessels includes the trunk structure, bifurcation structure, connectivity and topological information of the blood vessels.

[0011] Preferably, the first-layer V-shaped network transmits the skeleton information to the second-layer and third-layer V-shaped networks through skip connections.

[0012] Preferably, the skeleton graph information between the first layer V-shaped network and the second layer V-shaped network and the third layer V-shaped network is transmitted via a jump connection.

[0013] Preferably, the multi-view deformable blood vessel shape extraction unit is composed of a fixed shape convolution kernel module and a deformable convolution kernel module;

[0014] The fixed shape convolution kernel module uses a horizontal 1×5 convolution kernel, a vertical 5×1 convolution kernel, a main diagonal convolution kernel, and a sub-diagonal convolution kernel, which captures the linear structure of the blood vessel from different directions and obtains the main direction characteristics of the blood vessel;

[0015] The deformable convolution kernel module dynamically adjusts the sampling point and dynamically adjusts the sampling position of the convolution kernel according to the morphology of the blood vessel. The specific formula is as follows:

[0016]

[0017] Among them, p0 is the center pixel, p n is the position in the standard convolution kernel, Δp n is the deformable offset;

[0018] The formulas of the horizontal 1×5 convolution kernel, vertical 5×1 convolution kernel, main diagonal convolution kernel, sub-diagonal convolution kernel and deformable convolution kernel modules are:

[0019] F1=C 1×5 ×X

[0020] F2=C 5×1 ×X

[0021] F3=C diag ×X

[0022] F4=C adiag ×X

[0023] F5=C def (X,Δ)

[0024] Among them, X represents the input feature map, C 1×5 , C 5×1 , C diag , C adiag , C defThey represent the horizontal 1×5 convolution kernel, the vertical 5×1 convolution kernel, the main diagonal convolution kernel, and the sub-diagonal convolution kernel, respectively, and Δ represents the offset in the variable convolution kernel.

[0025] Preferably, the adaptive fusion module uses the Softmax function to normalize the convolution features in different directions, so as to generate the optimal segmentation result. The specific calculation process is as follows:

[0026] Weight calculation: By learning dynamic weights, the feature maps in different convolution directions are weighted. The weight calculation formula is:

[0027]

[0028] Among them, F k represents the feature map of the \(k\)th direction, α k is the dynamic weight obtained by adaptive weight calculation;

[0029] Weight update: Through the ReLU activation function and Softmax normalization, ensure that the weight remains non-negative and can effectively distinguish features in different directions.

[0030] Preferably, the training process of the triple V-shaped network adopts a composite loss function, including skeleton supervision loss and segmentation loss;

[0031] The skeleton supervision loss function is defined as follows:

[0032]

[0033] Among them, Y SKE is the true skeleton label, Skeleton output predicted by the network;

[0034] The composite loss function formula is as follows:

[0035] L Total =L SKE +αL SEG-2 +βL SEG-3

[0036] Among them, L SKE L is the skeleton supervision loss, which guides the network to learn the skeleton structure of blood vessels; Total is the segmentation loss, ensuring that the network can accurately distinguish blood vessels from the background at the pixel level; L SEG-2 With L SEG-3 Represent the segmentation losses of the second and third stages respectively; α and β are hyper parameters.

[0037] The present invention provides a retinal vessel segmentation method based on a triple V-shaped network. The method realizes high-precision automatic segmentation of retinal vessels by combining a skeleton supervision mechanism, MDVEU and an adaptive fusion module (AFB). The triple V-shaped network architecture consists of three layers of gradually refined V-shaped structures, each of which is responsible for extracting and segmenting vascular features of different scales, and enhancing the capture of vascular details through a skeleton supervision mechanism. MDVEU dynamically adjusts the convolution kernel sampling position to adapt to the complex geometric morphology of the blood vessels, and AFB further improves the segmentation effect through dynamic weight fusion. Optimization is performed through a composite loss function to ensure that the network efficiently learns the vascular topology and details.

[0038] The effectiveness of this method has been verified on multiple datasets such as DRIVE, STARE, and CHASE_DB1, and the segmentation accuracy has been significantly improved, especially in the processing of small blood vessels and bifurcation areas, and has good clinical application prospects. Experimental results show that 3VNet surpasses the existing state-of-the-art segmentation methods on multiple public datasets, especially in terms of sensitivity and accuracy, which have been improved by 1.3% and 1.71% respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A schematic diagram of the difficulty of the problem in the retinal blood vessel segmentation task in the present invention;

[0042] Figure 2 The overall framework diagram of the triple V-shaped network (3VNet) proposed in the present invention;

[0043] Figure 3 Generate a skeleton map of retinal blood vessels for the segmentation results of the first layer of V-shaped network;

[0044] Figure 4 A schematic diagram of the structure of a multi-view deformable vessel shape extraction unit (MDVEU) proposed in the present invention;

[0045] Figure 5 Comparison of segmentation results of different networks on DRIVE, CHASE_DB1 and STARE datasets. DETAILED DESCRIPTION

[0046] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of systems consistent with some aspects of the present invention as detailed in the appended claims.

[0047] In the prior art, traditional U-shaped networks usually have difficulty in capturing the details of blood vessels in segmentation tasks, especially in segmenting small and complex blood vessel structures. For this reason, this embodiment provides a retinal blood vessel segmentation method based on a triple V-shaped network, which adopts a three-layer gradually refined V-shaped structure, and each V-shaped structure is responsible for extracting and optimizing different levels of features of blood vessels; this method is particularly suitable for blood vessels with complex morphology, such as small blood vessels and bifurcated blood vessels.

[0048] The triple V-shaped network architecture adopts a three-layer V-shaped structure and guides the network to gradually refine the segmentation results through a skeleton supervision mechanism.

[0049] Specifically, the first layer of V-shaped network: This layer mainly processes the global vascular topology, segments the vascular trunk through the skeleton supervision mechanism (Skeleton Supervision), and generates a preliminary vascular skeleton output (SkeletonOutput), such as Figure 2 As shown in Figure 2, a preliminary skeleton diagram is generated at this stage and passed to the next layer of the network. The skeleton diagram not only describes the connectivity of the blood vessels, but also contains the overall vascular trunk structure, ensuring that the network understands the global information.

[0050] The second layer of V-shaped network: receives the skeleton image generated by the first layer and further optimizes the details through the multi-view deformable vessel shape extraction unit (MDVEU). By extracting the local features of the blood vessels, the vascular branch structure is optimized. This layer focuses on capturing finer vascular branches and generates an auxiliary output (Auxiliary Output), which contains more detailed information.

[0051] The third layer of V-shaped network: As the last layer, it focuses on capturing the boundaries of fine blood vessels and complex bifurcation structures, and processes more delicate vascular structures, such as curved blood vessels and bifurcation points. By fusing the local and global features of the input image, this layer generates the final high-precision segmentation result (Main Output) to achieve accurate pixel-level segmentation.

[0052] In each of the above stages, the V-shaped network passes the previous skeleton topology information to the next stage through skip connections, ensuring the preservation of local and global information during the segmentation process and avoiding information loss. The loss functions of each stage complement each other to ensure that the network strikes a balance between large structures and details.

[0053] In order to improve the ability to capture vascular details, the 3VNet of the present invention introduces a skeleton supervision mechanism. This mechanism extracts the skeleton information of the blood vessels and guides the network to focus on the topological structure of the blood vessels instead of directly extracting the vascular features from the original image, thus avoiding the problem of detail loss caused by directly extracting features from the original image.

[0054] Preferably, the specific steps of the present invention include the following:

[0055] Skeleton generation: First, the input retinal image is processed preliminarily through the U-shaped network to generate the skeleton map of the retinal blood vessels. The skeleton map extracts the main trunk and bifurcation structure of the blood vessels in the retina through morphological operations, which can better reflect the overall morphology of the blood vessels. Figure 3 As shown in Figure 2, the skeleton diagram not only retains the main structure of the blood vessels, but also contains the connectivity and topological information of the blood vessels.

[0056] Skeleton supervision: The skeleton information is passed to the second and third V-shaped networks through skip connections as a reference for further segmentation. In this way, the network can more accurately segment small blood vessels and complex bifurcations.

[0057] Traditional convolutional neural networks usually rely on fixed-shape convolution kernels, which show limitations when processing blood vessels with complex topological structures. To solve this problem, this paper proposes a multi-view deformable vessel shape extraction unit (MDVEU), which can dynamically adjust the convolution kernel from multiple perspectives to adapt to blood vessels of different shapes and sizes:

[0058] Fixed convolution kernel: This module uses convolution kernels in multiple directions (1×5, 5×1, main diagonal convolution and anti-diagonal convolution) to capture the linear structure of blood vessels from different directions. It is suitable for straight and curved blood vessel features, such as Figure 4 shown.

[0059] Deformable convolution kernel: In order to cope with complex vascular morphology, MDVEU introduces a deformable convolution kernel. The deformable convolution kernel can flexibly adapt to blood vessels of different shapes and directions by dynamically adjusting the sampling points, especially showing excellent results at the bends and bifurcations of blood vessels. The specific formula of the deformable convolution is as follows:

[0060]

[0061] Among them, p0 is the center pixel, p n is the position in the standard convolution kernel, Δp n is the deformable offset. By introducing the offset Δp n , deformable convolution can capture the complex geometric shapes of blood vessels more flexibly. The feature maps after these five convolution steps will be fed into the AFB module. The formulas of these five modules are:

[0062] F1=C 1×5 ×X

[0063] F2=C 5×1 ×X

[0064] F3=C diag ×X

[0065] F4=C adiag ×X

[0066] F5=C def (X,Δ)

[0067] Among them, X represents the input feature map, C 1×5 , C 5×1 , C diag , C adiag , C def They represent the horizontal 1×5 convolution kernel, the vertical 5×1 convolution kernel, the main diagonal convolution kernel, the sub-diagonal convolution kernel and the deformable convolution kernel respectively; Δ represents the offset in the variable convolution kernel.

[0068] MDVEU is used in both the encoding and decoding stages of 3VNet. In the encoding stage, MDVEU extracts vascular shape features from multiple perspectives, improving the network's feature expression capabilities. In the decoding stage, MDVEU further refines the topological structure of the blood vessels to ensure the continuity and integrity of the blood vessels.

[0069] In order to further improve the accuracy of segmentation, the present invention introduces an adaptive fusion module (AFB) after MDVEU. AFB uses a dynamic weight adjustment mechanism to perform weighted summation on the feature maps processed by MDVEU, thereby selectively enhancing the features related to blood vessels and suppressing irrelevant information. The specific calculation process is as follows:

[0070] Weight calculation: By learning dynamic weights, feature maps in different convolution directions are weighted to ensure that the most important features are strengthened. The weight calculation formula is:

[0071]

[0072] Among them, F k represents the feature map of the \(k\)th direction, α kis the dynamic weight obtained by adaptive weight calculation.

[0073] Weight update: Through the ReLU activation function and Softmax normalization, ensure that the weight remains non-negative and can effectively distinguish features in different directions.

[0074] The feature extraction results in MDVEU are weighted and fused through the adaptive fusion module, and the weights are dynamically adjusted according to the importance of the features. The adaptive fusion module uses the Softmax function to normalize the convolution features in different directions to generate the optimal segmentation result.

[0075] Through the adaptive fusion of AFB modules, the network can more accurately segment blood vessels with complex morphology, especially in low-contrast areas and vascular bifurcations.

[0076] The method of the present invention uses multiple public retinal blood vessel segmentation datasets for training and verification, including DRIVE, STARE and CHASE_DB1 datasets. The images in each dataset have been subjected to data enhancement processing, such as random flipping, rotation and Gaussian noise addition, to improve the generalization ability of the model.

[0077] In order to ensure the accuracy of the segmentation results, a composite loss function is used in the training process, including skeleton supervision loss and segmentation loss.

[0078] Skeleton supervision loss function: The skeleton supervision loss function is defined as follows

[0079]

[0080] Among them, Y SKE is the true skeleton label, The loss function ensures that the network can accurately learn the skeleton information of blood vessels during training and incorporate this information in the final segmentation.

[0081] The skeleton supervision loss function is combined with the segmentation loss function to ensure that the segmentation process not only captures the topological structure of the blood vessels, but also accurately identifies the boundaries of the blood vessels. The loss function formula is as follows:

[0082] L Total =L SKE +αL SEG-2 +βL SEG-3

[0083] Among them, L SKE L is the skeleton supervision loss, which guides the network to learn the skeleton structure of blood vessels; Total is the segmentation loss, ensuring that the network can accurately distinguish blood vessels from the background at the pixel level. SEG-2 With LSEG-3 They represent the segmentation losses of the second and third stages respectively. α and β are hyper parameters.

[0084] During the training process, 3VNet adopts a step-by-step refinement approach to continuously optimize the network's segmentation performance through skeleton supervision and multi-view feature extraction. The training data comes from retinal vascular segmentation datasets (such as DRIVE, STARE, and CHASE_DB1), and each image undergoes data enhancement processing (such as Gaussian noise addition, rotation, flipping, etc.) to improve the generalization ability of the model. The optimization algorithm uses Adam, with an initial learning rate of 0.001, a momentum parameter of α=0.9β=0.9, and a polynomial decay strategy to gradually reduce the learning rate. To ensure the stability of the training and the convergence of the network. The model is trained on an NVIDIA4090 GPU with a batch size of 2;

[0085] In order to verify the effectiveness of the method of the present invention, multiple sets of experiments were conducted and the results were compared with the most advanced methods. The experiments show that the method of the present invention outperforms the existing segmentation methods on multiple data sets, as shown in the following:

[0086] Improved accuracy: The segmentation accuracy of the 3VNet of the present invention on the DRIVE, STARE and CHASE_DB1 datasets reached 96.74%, 97.39% and 97.52% respectively, as shown in Table 1, significantly surpassing the results of other methods.

[0087] Table 1 Comparison of performance indicators of different methods on retinal vessel segmentation tasks

[0088]

[0089] Sensitivity and specificity: In terms of sensitivity (Sensitivity) and specificity (Specificity) of vascular details, the method of the present invention also performs well, especially on the CHASE_DB1 dataset, where the sensitivity is improved by 1.3% compared with the SOTA method, and the specificity remains at a high level.

[0090] Visual effects: Figure 4 As shown in the figure, 3VNet can well capture the bifurcations and small blood vessels, especially in the low-contrast areas.

[0091] Ablation experiment: In order to verify the contribution of each component to the model performance, an ablation experiment was conducted. The results show that the skeleton supervision mechanism, multi-view deformable unit (MDVEU) and adaptive fusion module (AFB) all significantly improve the overall performance of the model, as shown in Table 2:

[0092] Table 2 Experimental results of ablation of each module

[0093]

[0094] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These changes and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A retinal blood vessel segmentation method based on a triple V-shaped network, characterized in that: include: Construct a triple V-shaped network architecture, and use each V-shaped structure to extract and optimize different levels of features of the input retinal image; The first layer of V-shaped network segments the main blood vessels in the retinal image through the skeleton supervision mechanism, generates the skeleton map of the retinal blood vessels, and transmits it to the second layer of V-shaped network; The second layer V-shaped network receives the skeleton image, further extracts the local features of the blood vessels through the multi-view deformable blood vessel shape extraction unit, optimizes the blood vessel branch structure, and generates an auxiliary output result; The third-layer V-shaped network receives the auxiliary output results, captures the boundaries of tiny blood vessels and complex bifurcation structures, and performs weighted summation on the feature maps processed by the multi-view deformable blood vessel shape extraction unit through an adaptive fusion module to generate the final main output, which is a high-precision segmentation result.

2. The retinal blood vessel segmentation method based on triple V-shaped network according to claim 1, characterized in that: The skeleton image of the retinal blood vessels includes the trunk structure, bifurcation structure, connectivity and topological information of the blood vessels.

3. The retinal blood vessel segmentation method based on triple V-shaped network according to claim 1, characterized in that: The first-layer V-shaped network transmits the skeleton information to the second-layer and third-layer V-shaped networks through skip connections.

4. The retinal blood vessel segmentation method based on triple V-shaped network according to claim 1, characterized in that: The skeleton graph information between the first layer V-shaped network and the second layer V-shaped network and the third layer V-shaped network is transmitted through jump connections.

5. The retinal blood vessel segmentation method based on triple V-shaped network according to claim 1, characterized in that: The multi-view deformable blood vessel shape extraction unit is composed of a fixed shape convolution kernel module and a deformable convolution kernel module; The fixed shape convolution kernel module uses a horizontal 1×5 convolution kernel, a vertical 5×1 convolution kernel, a main diagonal convolution kernel, and a sub-diagonal convolution kernel, which captures the linear structure of the blood vessel from different directions and obtains the main direction characteristics of the blood vessel; The deformable convolution kernel module dynamically adjusts the sampling point and dynamically adjusts the sampling position of the convolution kernel according to the morphology of the blood vessel. The specific formula is as follows: Among them, p0 is the center pixel, p n is the position in the standard convolution kernel, Δp n is the deformable offset; The formulas of the horizontal 1×5 convolution kernel, vertical 5×1 convolution kernel, main diagonal convolution kernel, sub-diagonal convolution kernel and deformable convolution kernel modules are: F1=C 1×5 ×X F2=C 5×1 ×X <h2 style=";text-align:left;direction:ltr">F3 = C<h2 style=";text-align:left;direction:ltr"> diag <h2 style=";text-align:left;direction:ltr"> ×X F4=C adiag ×X F5=C def (X,Δ) Among them, X represents the input feature map, C 1×5 , C 5×1 , C diag , C adiag , C def They represent the horizontal 1×5 convolution kernel, the vertical 5×1 convolution kernel, the main diagonal convolution kernel, and the sub-diagonal convolution kernel, respectively, and Δ represents the offset in the variable convolution kernel.

6. The retinal blood vessel segmentation method based on triple V-shaped network according to claim 1, characterized in that: The adaptive fusion module uses the Softmax function to normalize the convolution features in different directions to generate the optimal segmentation result. The specific calculation process is as follows: Weight calculation: By learning dynamic weights, the feature maps in different convolution directions are weighted. The weight calculation formula is: Among them, F k represents the feature map of the \(k\)th direction, α k is the dynamic weight obtained by adaptive weight calculation; Weight update: Through the ReLU activation function and Softmax normalization, ensure that the weight remains non-negative and can effectively distinguish features in different directions.

7. The retinal blood vessel segmentation method based on triple V-shaped network according to claim 1, characterized in that: The training process of the triple V-shaped network adopts a composite loss function, including skeleton supervision loss and segmentation loss; The skeleton supervision loss function is defined as follows: Among them, Y SKE is the true skeleton label, Skeleton output predicted by the network; The composite loss function formula is as follows: L Total =L SKE +αL SEG-2 +βL SEG-3 Among them, L SKE L is the skeleton supervision loss, which guides the network to learn the skeleton structure of blood vessels; Total is the segmentation loss, ensuring that the network can accurately distinguish blood vessels from the background at the pixel level; L SEG-2 With L SEG-3 Represent the segmentation losses of the second and third stages respectively; α and β are hyper parameters.

Images