An OCTA image microvessel segmentation method fusing multi-view features

Abstract

The application belongs to the field of deep learning image segmentation, and discloses an OCTA image microvessel segmentation method fusing multi-view features, which comprises the following steps: splitting input 3D data and 2D image data as training set, verification set and test set respectively; meanwhile, obtaining the blood vessel label map of the 2D image; pre-processing and data enhancement are performed on the input data; the pre-processed 3D data is input into a 3D feature extraction network to extract multi-scale spatial features of the 3D image; the pre-processed 2D data is input into a 2D feature extraction network to extract multi-scale features of the 2D image; the 3D features and the 2D features are input into a cross-dimension feature fusion network to generate fusion features; the fusion features are input into a decoder to generate a prediction map of retinal blood vessels; the application can provide a more objective and accurate evaluation method for the diagnosis and research of diseases related to retinal blood vessels, and can quickly and accurately segment out retinal microvessels.

Description

Technical Field

[0001] This invention belongs to the field of deep learning image segmentation, specifically relating to a microvascular segmentation method for OCTA images that integrates multi-view features. Technical Background

[0002] Optical coherence tomography (OCTA) is an extension of optical coherence tomography (OCT) technology. It is a non-invasive imaging method that, compared to traditional angiography, can better measure the boundaries and area of ​​capillaries and neovascularizations. OCTA provides images with detailed retinal and choroidal vascular structures by extracting OCT signal changes caused by erythrocyte movement, overcoming the limitation of OCT in providing detailed vascular information. OCTA imaging can differentiate vascular abnormalities at different levels, including superficial, deep retinal, and choroidal circulatory abnormalities. Physicians can accurately extract retinal vessels, thus OCTA is increasingly used for the diagnosis and evaluation of retinal vascular-related diseases.

[0003] Currently, projection maps are the mainstream method for OCTA image segmentation, but projection maps themselves rely on layer segmentation results. In retinal patients, the fundus structure is damaged due to disease, leading to decreased layer segmentation accuracy and adversely affecting subsequent fundus segmentation. Combining OCTA with projection maps as input to the network is another mainstream approach, attempting to avoid the influence of layer segmentation through image projection networks. However, using OCTA volume data as input increases computational load and, due to simple dimensionality reduction, loses important three-dimensional spatial information while neglecting minute vascular structures. Summary of the Invention

[0004] The purpose of this invention is to solve the technical problems mentioned above, and to provide a microvessel segmentation method for OCTA images that integrates multi-view features, comprising the following steps:

[0005] S1. Split the input 3D data and 2D image data into training set, validation set and test set respectively; at the same time, obtain the blood vessel label map of the 2D image;

[0006] S2. Preprocess and augment the input data;

[0007] S3. Input the preprocessed 3D data into the 3D feature extraction network to extract the multi-scale spatial features of the 3D image and obtain the 3D features.

[0008] S4. Input the preprocessed 2D data into the 2D feature extraction network to extract the multi-scale features of the 2D image and obtain the 2D features;

[0009] S5. Input the 3D features and 2D features into a cross-dimensional feature fusion network to generate fused features;

[0010] S6. Input the fused features into the encoder to generate a prediction map of the retina;

[0011] S7. Calculate Dice loss and structure-aware loss based on the prediction map and the blood vessel label map. Iteratively optimize the 2D blood vessel segmentation network according to the calculated total loss. Test and verify the segmentation results using the optimized network. Retain the network parameters with the minimum total loss as the final prediction network.

[0012] S8. Use the final prediction network to segment the retinal vessels in the OCTA image to be processed.

[0013] In the preferred embodiment, the preprocessing and data augmentation in step S2 specifically include:

[0014] S11. Crop the original 2D and 3D images to a centered size of 256×256 pixels;

[0015] S12. Randomly rotate the original 2D image by an angle of -10 to 10 degrees, and randomly change the brightness and contrast of the image.

[0016] S13. Convert the pixel value range of 3D data and 2D data from [0, 255] to [0, 1].

[0017] In the preferred embodiment, the 3D feature extraction network described in step S3 is implemented based on a pre-trained dynamic 3D UNet, and the extraction of multi-scale 3D spatial features specifically includes:

[0018] S21. Input the preprocessed 3D data into the 3D convolutional neural network DynUNet, the network comprising an input layer, an encoder, a bottleneck layer, a decoder, and an output layer;

[0019] S22. The encoder consists of five levels of 3D downsampling blocks. Each downsampling block uses a residual convolutional block with a stride of 2×2×2 for feature extraction. Each convolutional layer contains two consecutive 3×3×3 convolutional kernels. Each convolutional layer is followed by an instance normalization layer and a LeakyReLU activation layer. The number of channels in the output feature map increases sequentially for each level, namely 32, 64, 128, 256, and 320.

[0020] S23. The decoder consists of five levels of 3D upsampling blocks. Each upsampling block uses a transposed convolution with a stride of 2×2×2 and a kernel size of 2×2×2 to double the feature map size. After skip connections, multi-scale fusion is formed. The fused features are passed through two consecutive 3×3×3 convolution kernels. Each convolutional layer is followed by an instance normalization layer and a LeakyReLU activation layer.

[0021] S24. The skip connection described in step S23 aligns the number of channels of the encoder and decoder feature maps using a 1×1×1 convolutional kernel.

[0022] In a preferred embodiment, step S4, extracting multi-scale 2D texture features in the 2D feature extraction network, specifically includes:

[0023] S31. Input the preprocessed 2D data into a 2D convolutional neural network, the network comprising an encoder, a decoder and dense skip connections;

[0024] S32. The encoder consists of five levels of 2D downsampling modules. Each downsampling block uses a convolutional module with a stride of 2×2 and uses double convolutional blocks for feature extraction. Each convolutional layer is followed by batch normalization and ReLU activation function. The feature map space size is halved by a 2×2 max pooling layer. The number of channels of the output feature map at each level doubles sequentially, namely 64, 128, 256, and 320.

[0025] S33. The decoder consists of four nested upsampling modules. Each upsampling block uses a 2×2 transposed convolution to upsample the feature map. The upsampled features are then concatenated with the following three types of features along the channel dimension: skip connection features from the same level encoder, same-scale features from the previous level decoder, and upsampled features from the deep decoder. Two consecutive 3×3 convolution operations are then performed on the concatenated feature map.

[0026] S34. The dense skip connection adjusts the number of channels in the feature maps of each level through a 1×1 convolution kernel; it fuses features of the same scale from different paths by adding elements one by one; and it establishes cross-level feature reuse channels to form a dense feature pyramid.

[0027] In the preferred embodiment, the cross-dimensional feature fusion module in step S5 specifically includes the following steps:

[0028] S51. Map the input three-dimensional features to a two-dimensional feature space using a dynamic projection method to obtain a multi-view feature set;

[0029] S52. The multi-view feature set is input to the multi-view attention module. Global average pooling is performed on each view feature map. Weight coefficients for each view are generated through a fully connected layer and a Sigmoid activation function. The features of the multi-view are weighted and fused according to the weight coefficients to obtain the attention map.

[0030] S53. Multiply the original two-dimensional feature map element-wise with the attention map obtained in step 51 to achieve feature selection enhancement. Then, add the enhanced features to the original features through residual connection. The output features include the attention focus area and low-level detail features.

[0031] In the preferred embodiment, the dynamic multi-view projection method in step S51 includes the following steps:

[0032] S511, Projection of the maximum value along the first dimension;

[0033] S512, Learnable slice selection along the second and third dimensions, which calculates slice saliency scores through dynamically initialized 1D convolutional kernels;

[0034] S513. Select the optimal slice based on the score;

[0035] In the preferred embodiment, the formula for the maximum projection of the first dimension in step S51 is:

[0036]

[0037] Where xy represents the xy direction in the input V. , represents the maximum projection along the depth direction, b represents the block size, h represents the height, w represents the width, and c represents the number of channels. This represents the maximum value function.

[0038] Furthermore, in step S52, the selection of learnable slices in the second and third dimensions first involves feature compression and dimensional rearrangement, averaging the 3D data along the height dimension, using the following formula:

[0039]

[0040]

[0041] in, , This represents the data after dimensionality averaging. xz represents slices along the X and Z axes in the 3D volume data, yz represents slices along the Y and Z axes in the 3D volume data, H represents the height dimension, W represents the width dimension, and V(b,c,d,h,w) represents the original 4D volume data after dimensionality rearrangement. ;

[0042] Furthermore, in step S53, the importance of the slice is scored using dynamic convolution, and the formula is as follows:

[0043] 2)

[0044] 2)

[0045] in This represents the importance score of the XZ slice calculated for a given batch index b, width index w, channel index c, and depth index d. Denotes the one-dimensional convolution kernel weights used for XZ slices. Let represent the weights of the one-dimensional convolution kernel used for the YZ slice, where c represents the channel index, 1 represents the one-dimensional convolution, and k represents the k-th element in the convolution kernel. This represents the XZ slice data after dimensional rearrangement and reshaping. This represents the YZ slice data after dimensional rearrangement and reshaping, where b represents the batch index, w represents the width index, h represents the height index, c represents the channel index, and d+k−2 represents the depth index plus the convolution kernel offset.

[0046] Finally, the optimal depth index for each W location is selected by averaging along the channel dimension and aggregating along the spatial dimension, using the following formula:

[0047]

[0048]

[0049] in This indicates the optimal depth index selected given the batch index b and the width index w. This indicates the optimal depth index selected given the batch index b and the height index h. This indicates selecting the depth value d with the highest importance among all depths. This represents the importance score of the XZ slices on batch b, width w, channel c, and depth d. The importance score of the XZ slices at batch b, height h, channel c, and depth d is indicated.

[0050] Collect slices and rearrange their dimensions to match the output shape, using the following formula:

[0051] ,(0,2,3,1)))

[0052] ,(0,2,3,1)))

[0053] in This represents the collected XZ slices. This represents the collected YZ slices. The `Permute` function is used to adjust the dimension order of the tensor, and the `Reshape` function is used to change the shape of the tensor. Here, `b` is the batch index, `c` is the channel index, `d` is the depth index, and `w` is the width index. It is the optimal depth index along the height direction. This represents the optimal depth index along the width direction.

[0054] In the preferred embodiment, the multi-view attention mechanism described in step S52 integrates three-view features with the multi-head attention mechanism, and its specific steps include:

[0055] S521. Perform feature reshaping on the input three views to generate spatially flattened features. , , ;

[0056] S522. Generate a trainable parameter matrix based on the mapping relationship between the input feature channel dimension and the preset embedding dimension E. , , The initial values ​​of each matrix element follow a normal distribution with a mean of 0 and a standard deviation of 0.02. (This is from the previous step.) , , Projection features are generated after a dynamic projection layer. , , ;

[0057] S523. Use the projection features from the previous step as the query vector, key vector, and value vector of the multi-head attention mechanism to calculate multi-head attention;

[0058] S524. Generate a spatial attention map based on the multi-head attention calculated in the previous step;

[0059] In the preferred embodiment, the specific steps of feature space flattening in step S521 include:

[0060] S5211. Perform dimension permutation and flattening on the largest projected feature to generate a query matrix. ;

[0061] S5212. Perform dimensionality permutation and flattening on the coronal projection features to generate the key matrix. ;

[0062] S5213. Perform dimension permutation and flattening on the sagittal projection features to generate a value matrix. ;

[0063] In the preferred embodiment, the multi-head attention formula in step S523 is:

[0064]

[0065] in, , , Indicates from projection features Extract the query matrix corresponding to the i-th head. Indicates the current attention head index. This indicates the feature dimension of a single head.

[0066] E represents the total embedding dimension after projection. Indicates the number of attention heads It is the dimension of the key vector. The key matrix of the i-th head This represents the value matrix of the i-th head. The function is used to convert attention scores into a probability distribution.

[0067] Finally, a spatial attention map is generated, and its formula is:

[0068]

[0069] Where, reshape means reshaping into a spatial graph, and mean represents the average value along dimension D. Indicates batch, Indicates altitude, Indicates the width.

[0070] Because the present invention adopts the above-described solution, it has the following beneficial effects:

[0071] 1. This invention solves the problem of spatial information loss during 3D data segmentation through the cross-dimensional feature fusion technology in step S5. By using a dynamic multi-view projection method, high-dimensional features are mapped to a 2D space, generating a multi-view feature set. This assists the model in capturing data features from different angles, obtaining a more comprehensive perspective. A multi-view attention module is used to perform weighted fusion of the multi-view feature set. Through global average pooling and the weight coefficients generated by the sigmoid activation function, the model can identify and strengthen the features most helpful to the task. Through feature selection enhancement and residual connections, the features output by the model not only include the key regions focused on by attention but also retain the underlying detailed features, thereby improving the expressive power of the features and the overall performance of the model.

[0072] 2. This invention addresses the problem of insufficient balance between global information and local details during feature extraction by employing multi-view attention technology. Traditional feature extraction methods may fail to fully capture the multi-dimensional characteristics of data, leading to the loss of important information during feature fusion. The multi-view attention mechanism introduced in this invention enables the model to adaptively weight features from different perspectives, paying more attention to feature regions that significantly impact the task. This method not only enhances the expressive power of features but also strengthens the model's identification and utilization of key features, thereby significantly improving the model's recognition accuracy and generalization ability when processing complex and high-dimensional data.

[0073] 3. This invention recognizes that a single fixed projection (such as MIP) not only loses the depth information of 3D blood vessels but also causes projection artifacts, thus affecting the segmentation of deep blood vessels. To address this problem, this invention does not simply attempt conventional fixed projection but proposes a dynamic scheme that "allows the network to learn which slices are most important." Specifically, it innovatively applies the idea of ​​dynamic convolution to calculate the "slice saliency score," a highly creative technical concept. It transforms feature extraction from a passive "acceptance" to an active "selection," making the 3D to 2D information dimensionality reduction process more intelligent and efficient. Attached Figure Description

[0074] Figure 1 This is a flowchart of the method of the present invention;

[0075] Figure 2 This is a structural diagram of the multi-scale aggregation convolution module in this invention;

[0076] Figure 3 This is a diagram of the segmentation model of the present invention;

[0077] Figure 4 This is a schematic diagram of the segmentation result of the present invention. Detailed Implementation

[0078] This invention proposes an OCTA retinal vessel segmentation method that integrates multi-view features. The network uses encoding and decoding as its structure, and extracts feature information of different dimensions of the image through convolutional networks of different dimensions. The fusion module realizes the aggregation of cross-dimensional information. This cross-dimensional fusion module extracts features from different perspectives of 3D features, improves the utilization rate of 3D information, and alleviates the loss of microvascular information caused by continuous convolution and pooling in 2D networks. This feature that integrates high-dimensional information is helpful for improving the segmentation of small blood vessels.

[0079] The present invention will be described in more detail below with reference to the accompanying drawings of the embodiments of the present invention. The following are merely exemplary and do not limit the scope of protection of the present invention. The present invention provides a method for microvessel segmentation of OCTA images that integrates multi-view features. The specific implementation process is described below with reference to the accompanying drawings. Step 1: The input 3D data and 2D image data are split into training set, validation set and test set, respectively; at the same time, the blood vessel label map of the 2D image is obtained.

[0080] In this embodiment, the image data used comes from the OCTA-500 dataset, which includes subsets of 6mm×6mm and 3mm×3mm images. The first subset, called OCTA_6M, consists of OCTA images of 300 normal individuals or patients with a 6mm×6mm FOV, with a volume size of 400 pixels (px) × 400px × 640px. Its label image size is 400px × 400px. The second subset, called OCTA_3M, consists of OCTA images of 200 normal individuals with a 3mm×3mm FOV, with a volume size of 304px × 304px × 640px. Its label image size is 304px × 304px. Furthermore, OCTA-500 provides six different projections. Among these six projections, the two-dimensional image tested in this invention is the maximum projection between the inner confinement membrane layer and the outer plexus layer of OCTA (ILM_OPL_OCTA).

[0081] Step 2: Preprocess and augment the input data.

[0082] The specific steps include:

[0083] Step S21: Crop the original 2D and 3D images to a centered size of 256×256 pixels;

[0084] Step S22: Randomly rotate the original 2D image by an angle of -10 to 10 degrees, and randomly change the brightness and contrast of the image.

[0085] Step S23 converts the pixel value range of the input 3D and 2D data from [0, 255] to [0, 1].

[0086] Step 3: Input the preprocessed 3D data into the 3D feature extraction network to extract the multi-scale spatial features of the 3D image, thus obtaining the 3D features. The specific steps are as follows:

[0087] Step S31 inputs the preprocessed 3D data into the 3D convolutional neural network DynUNet, which includes an input layer, an encoder, a bottleneck layer, a decoder, and an output layer;

[0088] The encoder part in step S32 consists of five levels of 3D downsampling blocks. Each downsampling block uses a residual convolutional block with a stride of 2×2×2 for feature extraction. Each convolutional layer contains two consecutive 3×3×3 convolutional kernels. Each convolutional layer is followed by an instance normalization layer and a LeakyReLU activation layer. The number of channels in the output feature map increases sequentially for each level, namely 32, 64, 128, 256, and 320.

[0089] The decoder part in step S33 consists of five levels of 3D upsampling blocks. Each upsampling block uses a transposed convolution with a stride of 2×2×2 and a kernel size of 2×2×2 to double the feature map size. After skip connections, multi-scale fusion is formed. The fused features are passed through two consecutive 3×3×3 convolution kernels. Each convolutional layer is followed by an instance normalization layer and a LeakyReLU activation layer.

[0090] The skip connection described in step S34 aligns the number of channels in the encoder and decoder feature maps using a 1×1×1 convolutional kernel.

[0091] In this example, the pre-trained weights of the 3D convolutional neural network are the weights publicly available from the VesselFM model. The model is frozen throughout the training process. Because it is trained on a large amount of labeled 3D data and synthetic data, it achieves better results than unsupervised training, helping to solve the difficulty of lacking 3D labeled data in training. This is hereby stated.

[0092] Step 4: Input the preprocessed 2D data into the 2D feature extraction network to extract the multi-scale features of the 2D image and obtain the 2D features.

[0093] Step S41 inputs the preprocessed 2D data into a 2D convolutional neural network, the network comprising an encoder, a decoder and dense skip connections;

[0094] The encoder described in step S42 consists of five 2D downsampling modules. Each downsampling block uses a convolutional module with a stride of 2×2 and employs double convolutional blocks for feature extraction. Each convolutional layer is followed by batch normalization and ReLU activation functions. The feature map space size is halved by a 2×2 max pooling layer. The number of channels in the output feature map of each level doubles sequentially, namely 64, 128, 256, and 320.

[0095] The decoder described in step S43 consists of four nested upsampling modules. Each upsampling block uses a 2×2 transposed convolution to upsample the feature map. The upsampled features are then concatenated with the following three types of features along the channel dimension: skip connection features from the same level encoder, same-scale features from the previous level decoder, and upsampled features from the deep decoder. Two consecutive 3×3 convolution operations are then performed on the concatenated feature map.

[0096] The dense skip connection described in step S44 adjusts the number of channels in the feature maps of each level through a 1×1 convolution kernel, and fuses the same-scale features of different paths by adding elements one by one, thereby establishing cross-level feature reuse channels and forming a dense feature pyramid.

[0097] In this example, the weights are first obtained by training a 2D network with an appropriate amount of 2D data. The number of downsampling layers in the model is set to be consistent with the downsampling of the 3D model. The feature extraction module extracts the features after each downsampling and only performs this extraction on the four layers that match the number of channels in the 3D model.

[0098] Step 5: Input the 3D features and 2D features into a cross-dimensional feature fusion network to generate fused features. This specifically includes the following steps:

[0099] Step S51 maps the input 3D features to a 2D feature space using a dynamic projection method to obtain a multi-view feature set;

[0100] The multi-view feature set mentioned in step S52 is input to the multi-view attention module. Global average pooling is performed on the feature map of each view. Weight coefficients of each view are generated through a fully connected layer and a Sigmoid activation function. The features of the multi-view are weighted and fused according to the weight coefficients to obtain the attention map.

[0101] Step S53 multiplies the original two-dimensional feature map element-wise with the attention map obtained in step 51 to achieve feature selection enhancement. Then, the enhanced features are added to the original features through residual connections, and the output features include the attention focus area and low-level detail features.

[0102] The dynamic multi-view projection method includes the following steps:

[0103] Step S511 Project the maximum value along the first dimension;

[0104] Step S512 is the learnable slice selection along the second and third dimensions, which calculates the slice saliency score through dynamically initialized 1D convolutional kernels;

[0105] Step S513 selects the optimal slice based on the score;

[0106] The formula for the maximum projection of the first dimension in step S511 is:

[0107]

[0108] Where xy represents the xy direction in the input V. , represents the maximum projection along the depth direction, b represents the block size, h represents the height, w represents the width, and c represents the number of channels. This represents the maximum value function.

[0109] Furthermore, in step S512, the selection of learnable slices in the second and third dimensions first involves feature compression and dimensional rearrangement, averaging the 3D data along the height dimension, using the following formula:

[0110]

[0111]

[0112] in, , This represents the data after dimensionality averaging. xz represents slices along the X and Z axes in the 3D volume data, yz represents slices along the Y and Z axes in the 3D volume data, H represents the height dimension, W represents the width dimension, and V(b,c,d,h,w) represents the original 4D volume data after dimensionality rearrangement. .

[0113] Furthermore, in step S513, the importance of the slice is scored using dynamic convolution, and the formula is as follows:

[0114] 2)

[0115] 2)

[0116] in This represents the importance score of the XZ slice calculated for a given batch index b, width index w, channel index c, and depth index d. Denotes the one-dimensional convolution kernel weights used for XZ slices. Let represent the weights of the one-dimensional convolution kernel used for the YZ slice, where c represents the channel index, 1 represents the one-dimensional convolution, and k represents the k-th element in the convolution kernel. This represents the XZ slice data after dimensional rearrangement and reshaping. This represents the YZ slice data after dimensional rearrangement and reshaping, where b represents the batch index, w represents the width index, h represents the height index, c represents the channel index, and d+k−2 represents the depth index plus the convolution kernel offset.

[0117] Finally, the optimal depth index for each W location is selected by averaging along the channel dimension and aggregating along the spatial dimension, using the following formula:

[0118]

[0119]

[0120] in This represents the optimal depth index selected given the batch index b and the width index w. This indicates the optimal depth index selected for a given batch index b and height index h. This indicates selecting the depth value d with the highest importance among all depths. This represents the importance score of the XZ slices on batch b, width w, channel c, and depth d. The importance score of the XZ slices at batch b, height h, channel c, and depth d is indicated.

[0121] Collect slices and rearrange their dimensions to match the output shape, using the following formula:

[0122]

[0123]

[0124] in This represents the collected XZ slices. This represents the collected YZ slices. The `Permute` function is used to adjust the dimension order of the tensor, and the `Reshape` function is used to change the shape of the tensor. Here, `b` is the batch index, `c` is the channel index, `d` is the depth index, and `w` is the width index. It is the optimal depth index along the height direction. This represents the optimal depth index along the width direction.

[0125] The multi-view attention mechanism described in step S52 fuses three-view features on top of the multi-head attention mechanism. Its specific steps include:

[0126] S521 performs feature reshaping on the input three views to generate spatially flattened features. , , ;

[0127] S522 generates a trainable parameter matrix based on the mapping relationship between the input feature channel dimension and the preset embedding dimension E. , , The initial values ​​of each matrix element follow a normal distribution with a mean of 0 and a standard deviation of 0.02. (This is from the previous step.) , , Projection features are generated after a dynamic projection layer. , , ;

[0128] S523 uses the projection features from the previous step as the query vector, key vector, and value vector of the multi-head attention mechanism to calculate multi-head attention;

[0129] S524 generates a spatial attention map based on the multi-head attention calculated in the previous step;

[0130] The specific steps for flattening the feature space include:

[0131] S5211 performs dimensional permutation and flattening on the maximum projected feature to generate the query matrix. ;

[0132] S5211 performs dimensional permutation and flattening on the coronal projection features to generate the key matrix. ;

[0133] S5211 performs dimension permutation and flattening on the sagittal projection features to generate a value matrix. ;

[0134] The multi-head attention formula in step S523 is:

[0135]

[0136] in, , , Indicates from projection features Extract the query matrix corresponding to the i-th head. Indicates the current attention head index. Indicates the feature dimension of a single head.

[0137] E represents the total embedding dimension after projection. Indicates the number of attention heads It is the dimension of the key vector. The key matrix of the i-th head This represents the value matrix of the i-th head. The function is used to convert attention scores into a probability distribution.

[0138] Finally, a spatial attention map is generated, and its formula is:

[0139]

[0140] Here, reshape indicates reshaping into a spatial graph. Represents the average value along dimension D, where Indicates batch, Indicates altitude, The width is indicated. The above description of the embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principle of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A method for microvessel segmentation in OCTA images that integrates multi-view features, characterized in that, Includes the following steps: Step S1: Split the input 3D data and 2D image data into training set, validation set, and test set, respectively; simultaneously obtain the blood vessel label map of the 2D image; Step S2: Preprocess and augment the input data; Step S3: Input the preprocessed 3D data into the 3D feature extraction network to extract the multi-scale spatial features of the 3D image and obtain the 3D features; Step S4: Input the preprocessed 2D data into the 2D feature extraction network to extract the multi-scale features of the 2D image and obtain the 2D features; Step S5: Input the 3D features and 2D features into a cross-dimensional feature fusion network to generate fused features; The cross-dimensional feature fusion module specifically includes the following steps: Step S51: Map the input 3D features to a two-dimensional feature space using a dynamic multi-view projection method to obtain a multi-view feature set; Step S52: The multi-view feature set is input into the multi-view attention module. Global average pooling is performed on each view feature map. Weight coefficients for each view are generated through a fully connected layer and a Sigmoid activation function. The features of the multi-view are weighted and fused according to the weight coefficients to obtain the attention map. Step S53: Multiply the original 2D feature map element-wise with the attention map obtained in step 51 to achieve feature selection enhancement. Then, add the enhanced features to the original features through residual connection. The output features include the attention focus area and low-level detail features. S6. Input the fused features into the decoder to generate a prediction map of the retina.

2. The method for microvessel segmentation in OCTA images by fusing multi-view features according to claim 1, characterized in that, The preprocessing and data augmentation in step S2 specifically include: Step S21: Crop the original 2D and 3D images to a centered size of 256×256 pixels; Step S22: Randomly rotate the original 2D image by an angle of -10 to 10 degrees, and randomly change the brightness and contrast of the image; Step S23: Convert the pixel value range of the input 3D data and 2D data from [0, 255] to [0, 1].

3. The microvessel segmentation method for OCTA images fused with multi-view features according to claim 1, characterized in that, The 3D feature extraction network described in step S3 is implemented based on a pre-trained dynamic 3D UNet, and the extraction of multi-scale 3D spatial features specifically includes: Step S31: Input the preprocessed 3D data into the 3D convolutional neural network DynUNet, which includes an input layer, an encoder, a bottleneck layer, a decoder, and an output layer; Step S32: The encoder consists of five levels of 3D downsampling blocks. Each downsampling block uses a residual convolutional block with a stride of 2×2×2 for feature extraction. Each convolutional layer contains two consecutive 3×3×3 convolutional kernels. Each convolutional layer is followed by an instance normalization layer and a LeakyReLU activation layer. The number of channels in the output feature map increases sequentially for each level, namely 32, 64, 128, 256, and 320. Step S33: The decoder consists of five levels of 3D upsampling blocks. Each upsampling block uses a transposed convolution with a stride of 2×2×2 and a kernel size of 2×2×2 to double the feature map size. After skip connections, multi-scale fusion is formed. The fused features are passed through two consecutive 3×3×3 convolution kernels. Each convolutional layer is followed by an instance normalization layer and a LeakyReLU activation layer. The skip connections described in steps S34 and S33 align the number of channels in the encoder and decoder feature maps using 1×1×1 convolutional kernels.

4. The microvessel segmentation method for OCTA images based on multi-view features according to claim 1, characterized in that, The 2D feature extraction network in step S4 is implemented on a pre-trained UNet++, and the extraction of multi-scale 2D texture features specifically includes: Step S41: Input the preprocessed 2D data into a 2D convolutional neural network, the network comprising an encoder, a decoder and dense skip connections; Step S42: The encoder consists of five 2D downsampling modules. Each downsampling module uses a convolutional module with a stride of 2 and uses double convolutional blocks for feature extraction. Each convolutional layer is followed by batch normalization and ReLU activation function. The feature map space size is halved by a 2×2 max pooling layer. The number of channels of the output feature map at each level doubles sequentially, namely 64, 128, 256, and 320. Step S43: The decoder consists of four nested upsampling modules. Each upsampling block uses a 2×2 transposed convolution to upsample the feature map. The upsampled features are then concatenated with the following three types of features along the channel dimension: skip connection features from the same level encoder, same-scale features from the previous level decoder, and upsampled features from the deep decoder. Two consecutive 3×3 convolution operations are then performed on the concatenated feature map. Step S44: The dense skip connection adjusts the number of channels in the feature maps of each level through a 1×1 convolution kernel, and fuses the same-scale features of different paths by adding elements one by one, establishing cross-level feature reuse channels and forming a dense feature pyramid.

5. The microvessel segmentation method for OCTA images based on multi-view features according to claim 1, characterized in that, The dynamic multi-view projection method in step S51 includes the following steps: Step S511: Project the maximum value along the first dimension; Step S512: Learnable slice selection along the second and third dimensions, which calculates slice saliency scores through dynamically initialized 1D convolutional kernels; Step S513: Select the optimal slice based on the score.

6. The microvessel segmentation method for OCTA images based on multi-view features according to claim 5, characterized in that, The formula for the maximum projection of the first dimension in step S511 is: Where xy represents the input In the xy direction, represents the maximum projection along the depth direction, b represents the block size, h represents the height, w represents the width, c represents the number of channels, and d represents the depth. Represents the maximum value function; In step S512, the selection of learnable slices for the second and third dimensions first involves feature compression and dimensional rearrangement, averaging the 3D data along the height dimension, using the following formula: in, , This represents the data after dimensionality averaging. xz represents slices along the X and Z axes in the 3D volume data, yz represents slices along the Y and Z axes in the 3D volume data, H represents the height dimension, W represents the width dimension, and V(b,c,d,h,w) represents the original 4D volume data after dimensionality rearrangement. Become , Become ; In step S513, the importance of the slice is scored using dynamic convolution, and the formula is as follows: 2) 2) in This represents the importance score of the XZ slice calculated for a given batch index b, width index w, channel index c, and depth index d. in This represents the importance score of the XZ slice calculated for a given batch index b, height index h, channel index c, and depth index d. This represents the one-dimensional convolution kernel weights used for the XZ slice. Let represent the weights of the one-dimensional convolution kernel used for the YZ slice, where c represents the channel index, 1 represents one-dimensional convolution, and k represents the k-th element in the convolution kernel. This represents the XZ slice data after dimensional rearrangement and reshaping. This represents the YZ slice data after dimensional rearrangement and reshaping, where b represents the batch index, w represents the width index, h represents the height index, and c represents the channel index. This represents the depth index plus the convolution kernel offset; Finally, the optimal depth index for each location is selected by averaging along the channel dimension and aggregating along the spatial dimension, using the following formula: in This represents the optimal depth index selected given the batch index b and the width index w. This indicates the optimal depth index selected for a given batch index b and height index h. This indicates selecting the depth value d with the highest importance among all depths. Collect slices and rearrange their dimensions to match the output shape, using the following formula: in This represents the collected XZ slices. This represents the collected YZ slices. The `Permute` function is used to adjust the dimension order of the tensor, and the `Reshape` function is used to change the shape of the tensor. Here, `b` is the batch index, `c` is the channel index, `d` is the depth index, and `w` is the width index. It is the optimal depth index along the height direction. This represents the optimal depth index along the width direction.

7. The method for microvessel segmentation in OCTA images by fusing multi-view features according to claim 1, characterized in that, The multi-view attention mechanism described in step S52 fuses three-view features on top of the multi-head attention mechanism. Its specific steps include: Step S521: Perform feature reshaping on the input three views to generate spatially flattened features. , , ; Step S522: Generate a trainable parameter matrix set based on the mapping relationship between the input feature channel dimension and the preset embedding dimension E. , , The initial values ​​of each matrix element follow a normal distribution with a mean of 0 and a standard deviation of 0.

02. Step S521... , , Projection features are generated after a dynamic projection layer. , , ; Step S523: Use the projection features from step S522 as the query vector, key vector, and value vector of the multi-head attention mechanism to calculate multi-head attention; Step S524: Generate a spatial attention map based on the multi-head attention calculated in step S523.

8. The microvessel segmentation method for OCTA images fusion with multi-view features according to claim 7, characterized in that, The specific steps for flattening the feature space in step S521 include: Step S5211: Perform dimension permutation and flattening on the maximum projected feature to generate the query matrix. ; Step S5212: Perform dimensional permutation and flattening on the coronal projection features to generate the key matrix. ; Step S5213: Perform dimension permutation and flattening on the sagittal projection features to generate a value matrix. .

9. The microvessel segmentation method for OCTA images based on multi-view features according to claim 7, characterized in that... The multi-head attention formula in step S523 is: in, , , Indicates from projection features Extract the query matrix corresponding to the i-th head. Indicates the current attention head index. This indicates the feature dimension of a single head. E represents the total embedding dimension after projection. Indicates the number of attention heads It is the dimension of the key vector. The key matrix of the i-th head This represents the value matrix of the i-th head. The function is used to convert attention scores into a probability distribution; Finally, a spatial attention map is generated, and its formula is: Here, reshape indicates reshaping into a spatial graph. Represents the average value along dimension D, where Indicates batch, Indicates altitude, Indicates the width.

Images