A blood vessel segmentation method based on OCTA images
Through the end-to-end U-shaped encoder-decoder network and the Swin Transformer module, combining the hidden vector feature alignment loss function and convolutional block to fuse OCTA images, the accuracy of automatic vascular segmentation in OCTA images is solved, and complex vascularization is achieved The precise segmentation and noise interference mitigation of the system improve segmentation performance.
Patent Information
- Application Number
- CN202210704945.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-21
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2042-06-21
AI Technical Summary
It is difficult to achieve accurate segmentation of the vascular automatic segmentation task based on OCTA images. Due to speckle noise, changes in vascular appearance, movement and shadow artifacts in different depth layers, existing deep learning methods are difficult to meet clinical needs.
Using an end-to-end U-shaped encoder-decoder network, combined with the Swin Transformer module and the residual module, the cross-sectional images of three OCTA corresponding levels are fused through hidden vector feature alignment loss function and convolutional blocks, extract rich features and optimize segmentation. Model.
It realizes accurate and automatic segmentation of complex vascular systems in OCTA images, alleviates speckle noise and artifact interference, and improves segmentation performance and accuracy.
Smart Images
Figure CN115082677B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a blood vessel segmentation method based on OCTA images. Background Art
[0002] The vascular system is one of the most important biological tissue structures in the retina. Its morphological changes are closely related to systemic, metabolic and hematological diseases, which can help ophthalmologists understand the development of the disease and evaluate the effects.
[0003] At present, the work based on OCTA images has received widespread attention, including image artifact removal, image denoising, arteriovenous classification and disease classification. However, there are few studies on the automatic segmentation of blood vessels based on OCTA images, especially the deep learning-based OCTA image blood vessel segmentation is still a very challenging task. The difficulties mainly include: first, in the OCTA imaging process, it is inevitably disturbed by the inherent speckle noise of the OCT system, thus generating images with low signal-to-noise ratio, which has a great adverse effect on the automatic segmentation of retinal blood vessels; second, the changes in the appearance of blood vessels at different depth layers, motion and shadow artifacts, and potential pathological structures significantly increase the difficulty of accurate segmentation of retinal blood vessels. Most deep learning methods are based on region-based learning. This technology can easily produce discontinuous and inaccurate segmentation results for those tightly connected capillaries, which means that when it is necessary to detect vascular structures with different thicknesses and imaging depths, these methods are difficult to meet the current clinical needs. Summary of the invention
[0004] The problem solved by the present invention is how to achieve accurate automatic segmentation of complex vascular systems in OCTA images.
[0005] To solve the above problems, the present invention provides a blood vessel segmentation method based on OCTA images, comprising the steps of:
[0006] S1: Encode the label image through the pre-trained autoencoder network to obtain the label latent vector feature;
[0007] S2: The Swin Transformer module is used to extract latent vector features from the input image, and the latent space of the original image and the latent space of the segmentation label are aligned through the least squares generative adversarial network;
[0008] S3: The aligned latent vector features are upsampled by the decoder network and the segmentation result is obtained.
[0009] In the above method, an end-to-end U-shaped encoder-decoder network for OCTA vascular segmentation is constructed, which effectively integrates the cross-sectional images of three corresponding OCTA layers for automatic and accurate extraction of blood vessels. The Swin Transformer module and the residual module are integrated as the encoder layer of the OCTA vascular segmentation network, which effectively improves the feature extraction ability of the model. The segmentation model is optimized at the latent space feature level through the feature-level latent vector alignment loss function. The Swin Transformer encoder is used to extract rich features from the input image, and a feature alignment loss function is designed from the latent space level to obtain a more accurate overall segmentation result.
[0010] Furthermore, the pre-trained autoencoder network in step S1 includes: an encoder structure and a decoder structure;
[0011] Each layer of encoder and decoder structure uses residual module and maximum pooling layer as the basic architecture; the autoencoder network encodes the features of the input segmentation label image into the latent vector space, and then restores it through the decoder;
[0012] Among them, the reconstruction loss L of the autoencoder network recon is the mean square error loss.
[0013] In the above method, the autoencoder network is first trained to encode the label image to obtain the label latent vector feature, which only includes the encoder structure and the decoder structure. Each layer of the encoder and decoder uses the residual module and the maximum pooling layer as the basic architecture. Unlike the common segmentation network, the autoencoder network abandons the jump connection between the decoder and the encoder, but instead encodes the features of the input segmentation label image into the latent vector space and then directly restores it through the decoder. In this way, the label latent vector has rich label feature information.
[0014] Furthermore, the latent space feature in step S2 includes two loss functions: a pixel-level Dice coefficient loss and a feature-level latent vector alignment loss function;
[0015] The pixel-level DICE coefficient loss function is a metric function for calculating the overlapping area, defined as:
[0016]
[0017] Where N represents the number of pixels; p(k, i)∈[0,1] and g(k, i)∈[0,1] represent the predicted probability and the actual pixel category corresponding to the kth category, respectively; K represents the number of categories to be segmented; ω k represents the weight of the k-th category.
[0018] In the above method, the latent vector space extracted by the trained autoencoder contains rich segmentation label features and can successfully restore the label image through the decoder.
[0019] Furthermore, the least squares method replaces the cross entropy loss function with the least squares loss function, and the objective function of the least squares loss function is:
[0020]
[0021]
[0022] in, and P represent the expected and probability distributions respectively, D and G represent the encoders of the discriminator and segmentation network respectively, x represents the latent vector feature of the label encoding, and z represents the original input image.
[0023] In the above method, in order to obtain rich latent space information related to blood vessels from the original input image, the present invention uses a least squares generative adversarial network to align the features of the latent space of the original image and the latent space of the segmentation label.
[0024] Furthermore, the step S3 comprises:
[0025] S31: fusing the input cross-sectional images of the three corresponding OCTA layers through a convolution block to obtain image feature information;
[0026] S32: The encoder extracts shallow and deep features of the fused image, and combines it with the decoder through jump connections, so as to fuse the features of each level extracted by the encoder with the features obtained by upsampling the decoder to obtain the final blood vessel segmentation result.
[0027] In the above method, the cross-sectional images of the three corresponding layers of the input OCTA, namely SVC, DVC and SVC combined with DVC, are fused through the convolution block, thereby providing rich image information for the encoder and alleviating the interference of speckle noise and artifacts in the OCTA image on the segmentation to a certain extent.
[0028] Furthermore, the ResU-Net network is used as the basic network, and the Swin Transformer is introduced into the ResU-Net to form a new encoder structure. The encoding step of the feature encoder includes four different stages, each of which is composed of a cascade of Transformer layers and residual modules composed of multiple stacked Swin Transformer modules.
[0029] Furthermore, the Swin Transformer module is composed of a window-based multi-head self-attention module and a shifted window-based multi-head self-attention module and two multi-layer perceptrons;
[0030] A layer normalization layer is applied before each multi-head self-attention module and multi-layer perceptron, and a residual connection is applied after the corresponding module.
[0031] In the above method, a residual connection is applied after the corresponding module, so as to effectively improve the representation ability of the encoding module.
[0032] Furthermore, the cross sections of the three OCTA corresponding layers include SVC, DVC and SVC combined with DVC, respectively.
[0033] In the above method, the cross-sectional images of the three OCTA corresponding layers, namely SVC, DVC and SVC combined with DVC, are integrated to automatically and accurately extract blood vessels.
[0034] The present invention adopts the above technical solution to achieve the following beneficial effects:
[0035] In response to the problems encountered in the accurate segmentation of the retinal vascular system in OCTA images, the present invention proposes an end-to-end retinal vascular segmentation network, including a convolution block for fusing cross-sectional images of three corresponding levels of OCTA, a feature encoder based on Swin Tranformer, and a latent vector alignment module. First, the cross-sectional images of the three corresponding levels of OCTA, namely SVC, DVC, and SVC combined with DVC, are fused and input into the segmentation network to enrich the image feature information of the input network and alleviate the interference of speckle noise and artifacts in OCTA images on segmentation to a certain extent. Then, the SwinTransformer encoder is used to extract rich features from the input image, and a feature alignment loss function is designed from the latent space level to obtain a more accurate overall segmentation result. The vascular features are obtained through the Swin Transformer module, and the features are aligned and optimized at the latent space level to achieve improved segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 The blood vessel segmentation method based on OCTA images provided by the embodiment of the present invention Figure 1 ;
[0037] Figure 2 The blood vessel segmentation method based on OCTA images provided by the embodiment of the present invention Figure 2 ;
[0038] Figure 3A schematic diagram of an original OCTA image of a blood vessel segmentation method based on OCTA images provided in an embodiment of the present invention;
[0039] Figure 4 The encoder structure and decoder structure diagram of the blood vessel segmentation method based on OCTA images provided by an embodiment of the present invention;
[0040] Figure 5 A schematic diagram of the structure of a Swin Transformer module of a blood vessel segmentation method based on OCTA images provided in an embodiment of the present invention;
[0041] Figure 6 A schematic diagram of segmentation results of the OCTA image-based blood vessel segmentation method provided in an embodiment of the present invention on two OCTA data sets. DETAILED DESCRIPTION
[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0043] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.
[0044] Example
[0045] This embodiment provides a blood vessel segmentation method based on OCTA images. Figures 1 to 6 As shown, the method comprises the steps of:
[0046] S1: Encode the label image through the pre-trained autoencoder network to obtain the label latent vector feature;
[0047] S2: The Swin Transformer module is used to extract latent vector features from the input image, and the latent space of the original image and the latent space of the segmentation label are aligned through the least squares generative adversarial network;
[0048] S3: The aligned latent vector features are upsampled by the decoder network and the segmentation result is obtained.
[0049] Specifically, this method constructs an end-to-end U-shaped encoder-decoder network for OCTA vascular segmentation, which effectively integrates the cross-sectional (en face) images of three corresponding OCTA layers for automatic and accurate extraction of blood vessels. The Swin Transformer module and the residual module are integrated as the encoder layer of the OCTA vascular segmentation network, which effectively improves the feature extraction ability of the model. The segmentation model is optimized at the latent space feature level through the feature-level latent vector alignment loss function. The Swin Transformer module is used to extract rich features from the input image, and a feature alignment loss function is designed from the latent space level to obtain more accurate overall segmentation results.
[0050] The pre-trained autoencoder network in step S1 includes: an encoder structure and a decoder structure;
[0051] Each layer of encoder and decoder structure uses residual module and maximum pooling layer as the basic architecture; the autoencoder network encodes the features of the input segmentation label image into the latent vector space, and then restores it through the decoder;
[0052] Among them, the reconstruction loss L of the autoencoder network recon is the mean square error loss.
[0053] See also Figure 4 Specifically, first train the autoencoder network to encode the label image and obtain the label latent vector features, which only includes the encoder structure and the decoder structure. Each layer of the encoder and decoder uses the residual module and the maximum pooling layer as the basic architecture. Unlike the common segmentation network, the autoencoder network abandons the jump connection between the decoder and the encoder, but instead encodes the features of the input segmentation label image into the latent vector space and then directly restores it through the decoder. In this way, the label latent vector has rich label feature information.
[0054] Among them, the latent space features in step S2 include two loss functions: pixel-level Dice coefficient loss and feature-level latent vector alignment loss function;
[0055] The pixel-level DICE coefficient loss function is a metric function for calculating the overlapping area, defined as:
[0056]
[0057] Where N represents the number of pixels; p(k, i)∈[0,1] and g(k, i)∈[0,1] represent the predicted probability and the actual pixel category corresponding to the kth category, respectively; K represents the number of categories to be segmented; ω k represents the weight of the k-th category.
[0058] Specifically, the latent vector space extracted by the trained autoencoder contains rich segmentation label features and can successfully restore the label image through the decoder.
[0059] Among them, the least squares replaces the cross entropy loss function with the least squares loss function, and the objective function of the least squares loss function is:
[0060]
[0061]
[0062] in, and P represent the expected and probability distributions respectively, D and G represent the encoders of the discriminator and segmentation network respectively, x represents the latent vector features of the label encoding, and z represents the original input image.
[0063] Specifically, in order to obtain rich latent space information related to blood vessels from the original input image, the present invention uses a least squares generative adversarial network to perform feature alignment on the latent space of the original image and the latent space of the segmentation label.
[0064] See also Figure 2 , wherein step S3 comprises:
[0065] S31: fusing the input cross-sectional images of the three corresponding OCTA layers through a convolution block to obtain image feature information;
[0066] S32: The encoder extracts shallow and deep features of the fused image, and combines it with the decoder through jump connections, so as to fuse the features of each level extracted by the encoder with the features obtained by upsampling the decoder to obtain the final blood vessel segmentation result.
[0067] Specifically, the cross-sectional images of the three corresponding layers of the input OCTA, namely the superficial vascular complexes (SVC), the deep vascular complexes (DVC) and the inner retina vascular plexus (the inner retina vascular plexus including both SVC and DVC, SVC+DVC), are fused through the convolution block, thereby providing rich image information for the encoder and alleviating the interference of speckle noise and artifacts in the OCTA images on the segmentation to a certain extent.
[0068] See also Figure 3, the cross-sectional image of the corresponding layer of OCTA input into the network, A is the corresponding fundus image, the original OCTA image is the central box area, B represents the image under SVC, C represents the image under DVC, and D represents the image under SVC combined with DVC.
[0069] Among them, the ResU-Net network is used as the basic network, and the Swin Transformer is introduced into the ResU-Net to form a new encoder structure. The encoding step of the feature encoder includes four different stages, each of which is composed of a Transformer layer composed of multiple stacked Swin Transformer modules and a cascade of residual modules.
[0070] Among them, the Swin Transformer module consists of a window-based multi-head self-attention module and a shifted window-based multi-head self-attention module and two multi-layer perceptrons;
[0071] A layer normalization layer is applied before each multi-head self-attention module and multi-layer perceptron, and a residual connection is applied after the corresponding module.
[0072] Specific encoding steps: In each stage of the encoder, the input fused image is first divided into non-overlapping local regions of size 4×4, which are used as markers and input into the Transformer layer. The linear embedding layer in the Transformer layer can effectively change the feature dimension of the marker to meet the encoding requirements of each stage. At the same time, the window attention block with linear computational complexity and the cross-window connection for processing non-overlapping local regions can also effectively improve the ability to encode image features.
[0073] In order to achieve hierarchical representation, the feature map is first transformed from Zoom out to Then the next stage of local region merging is performed. In the subsequent stages, local region merging is performed iteratively to obtain a resolution of A hierarchical feature map of , where i∈{2, 3, 4}.
[0074] After the local regions at each stage are merged, the feature map is added to the feature map before being input to the Transformer layer through a residual structure, and the result is combined with the feature map upsampled by the decoder through a skip connection.
[0075] See also Figure 5,Specifically, the local region merging,Swin Transformer module consists of a window-based multi-head self-attention module (Window-based Muti-head Self Attention,W-MSA) and a shifted window-based multi-head self-attention module (Shifted Window-based Muti-head Self Attention,SW-MSA) and two multilayer perceptrons (Multilayer Perceptron,MLP), and a layer normalization (LayerNorm,LN) layer is applied before each MSA module and MLP module.
[0076] Among them, the cross-sections of the three OCTA corresponding levels include SVC, DVC and SVC combined with DVC.
[0077] See also Figure 6 ,Qualitative and quantitative evaluations were performed on OCTA dataset 1 and OCTA dataset 2, and the segmentation results on the two OCTA datasets were presented, indicating that this method can achieve automatic and accurate segmentation of OCTA blood vessels. The first column represents the segmented example image, the second column represents the segmentation result of this method, and the third column represents the segmentation annotation of the corresponding image.
[0078] Aiming at the problems encountered in the accurate segmentation of the retinal vascular system in OCTA images, this method proposes an end-to-end retinal vascular segmentation network, including a convolution block for fusing cross-sectional images of three corresponding OCTA layers, a feature encoder based on Swin Tranformer, and a latent vector alignment module. First, the cross-sectional images of the three corresponding OCTA layers (SVC, DVC, and SVC combined with DVC) are fused and input into the segmentation network to enrich the image feature information of the input network and alleviate the interference of speckle noise and artifacts in OCTA images on segmentation to a certain extent. Then, the SwinTransformer encoder is used to extract rich features from the input image, and a feature alignment loss function is designed from the latent space level to obtain a more accurate overall segmentation result. The vascular features are obtained through the Swin Transformer module, and the features are aligned and optimized at the latent space level to improve the segmentation performance.
[0079] Although the disclosure is disclosed as above, the protection scope of the disclosure is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the disclosure, and these changes and modifications will fall within the protection scope of the present invention.
Claims
1. A blood vessel segmentation method based on OCTA images, characterized in that: Includes steps: S1: Encode the label image through the pre-trained autoencoder network to obtain the label latent vector feature; S2: The Swin Transformer module is used to extract latent vector features from the input image, and the latent space of the original image and the latent space of the segmentation label are aligned through the least squares generative adversarial network; the latent space features include two loss functions: pixel-level Dice coefficient loss and feature-level latent vector alignment loss function; The pixel-level DICE coefficient loss function is a metric function for calculating the overlapping area, defined as: Where N represents the number of pixels; p(k, i)∈[0,1] and g(k, i)∈[0,1] represent the predicted probability and the actual pixel category corresponding to the kth category, respectively; K represents the number of categories to be segmented; ω k represents the weight of the kth category; The least square method replaces the cross entropy loss function with the least square loss function. The objective function of the least square loss function is: in, and P represent the expected and probability distributions respectively, D and G represent the encoders of the discriminator and segmentation network respectively, x represents the latent vector feature of the label encoding, and z represents the original input image; S3: The aligned latent vector features are upsampled by the decoder network and the segmentation result is obtained.
2. The blood vessel segmentation method based on OCTA images according to claim 1, characterized in that: The pre-trained autoencoder network in step S1 includes: an encoder structure and a decoder structure; Each layer of encoder and decoder structure uses residual module and maximum pooling layer as the basic architecture; the autoencoder network encodes the features of the input segmentation label image into the latent vector space, and then restores it through the decoder; Among them, the reconstruction loss L of the autoencoder network recon is the mean square error loss.
3. The blood vessel segmentation method based on OCTA images according to claim 1, characterized in that: The step S3 comprises: S31: fusing the input cross-sectional images of the three corresponding OCTA layers through a convolution block to obtain image feature information; S32: The encoder extracts shallow and deep features of the fused image, and combines it with the decoder through jump connections, so as to fuse the features of each level extracted by the encoder with the features obtained by upsampling the decoder to obtain the final blood vessel segmentation result.
4. The blood vessel segmentation method based on OCTA images according to claim 1, characterized in that: The ResU-Net network is used as the basic network, and the Swin Transformer is introduced into the ResU-Net to form a new encoder structure. The encoding step of the feature encoder includes four different stages, each of which is composed of a Transformer layer composed of multiple stacked Swin Transformer modules and a cascade of residual modules.
5. The blood vessel segmentation method based on OCTA images according to claim 4, characterized in that: The SwinTransformer module consists of a window-based multi-head self-attention module, a shifted window-based multi-head self-attention module, and two multi-layer perceptrons; A layer normalization layer is applied before each multi-head self-attention module and multi-layer perceptron, and a residual connection is applied after the corresponding module.
6. The blood vessel segmentation method based on OCTA images according to claim 5, characterized in that: The cross sections of the three OCTA corresponding levels include SVC, DVC and SVC combined with DVC respectively.