A method for joint segmentation of retinal layer and lesion based on U-Net network

Abstract

This invention relates to the field of image processing technology and discloses a joint segmentation method for retinal layers and lesions based on a U-Net network. A parallel multi-scale feature fusion module is set up between the encoder and decoder. Feature shuffling breaks the inherent channel correlation of the Nth layer encoded features, promoting cross-channel information interaction and avoiding the limitations of single-channel features. Subsequently, feature channels are separated to obtain the first and second channel features, which are then fed into independent multi-scale image feature fusion units for parallel processing. This ensures that different feature branches can specifically capture multi-scale information, generating corresponding fused features that are input into a parallel decoding module to obtain coordinate attention features and spatial attention features for segmentation. This invention, through parallel processing, preserves feature diversity and provides complementary features based on differentiated processing, improving the feature's ability to describe complex medical images and effectively reducing boundary missegmentation and false positives for lesions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a retinal layer and lesion joint segmentation method based on a U-Net network. BACKGROUND

[0002] Age-related Macular Degeneration (AMD) is a degenerative disease of the retina, which can cause serious central visual impairment in the late stage, and is the main cause of irreversible blindness in people over 50 years old in developed countries. The volume of retinal fluid and the thickness of each layer of the retina can be used as important biomarkers to track the development of the disease. In Optical Coherence Tomography (OCT), specific lesion areas of AMD can be detected and visualized, such as intraretinal fluid (IRF), subretinal fluid (SRF), subretinal hyperreflective material (SRHM), and pigment epithelial detachment (PED). In actual cases, IRF, SRF, PED, and SHRM may appear simultaneously in clinical cases.

[0003] Segmentation of retinal layers and lesions based on OCT images has long faced many technical challenges: on the one hand, the boundaries of each sub-layer of the retina have significant soft boundary characteristics, and a transition region often forms between pathological and normal tissues. In addition, OCT images are easily affected by factors such as uneven lighting and low contrast, which further blurs the boundaries between layers. On the other hand, the gray scale features of lesions and surrounding normal tissues are highly similar, and there are significant individual differences in lesion size and shape. In addition, the destruction of retinal layer structure and the formation of lesions often occur simultaneously, and the characteristics of the two further increase the difficulty of segmentation.

[0004] To solve the above problems, automatic segmentation methods based on deep learning have become a research hotspot. Among them, U-Net and its improved models have been widely used due to their excellent performance in medical image segmentation tasks. Early studies mostly used single-task segmentation frameworks, i.e., building separate retinal layer segmentation models and lesion segmentation models. This approach has obvious limitations: first, it ignores the pathological relationship between retinal layer structure and lesions. The formation of lesions often leads to local layer structure damage, while the integrity of the layer structure directly affects the localization and qualification of the lesions. Independent modeling cannot fully utilize the complementary information of the two. Second, the repeated feature extraction process leads to large model parameter quantity and low inference efficiency, which is not conducive to deployment and application on clinical terminal devices.

[0005] In recent years, joint segmentation frameworks gradually emerged, which can complete retinal layer and lesion segmentation simultaneously through a single model, and to some extent, achieve information sharing. However, the existing joint segmentation methods still have many shortcomings: firstly, the feature extraction mechanism has defects. Although the traditional Convolutional Neural Network (CNN) model can capture local spatial details, reduce the number of parameters through weight sharing, and maintain translation invariance, these characteristics make CNN particularly suitable for processing complex structures and subtle differences in medical images, and it is good at capturing local texture features. The hierarchical receptive field increment can model the details of layers and lesions at the same time. However, the intrinsic locality of convolution operation limits the modeling of long-distance dependencies of CNN, and it is difficult to integrate global context information. CNN usually shows limitations in explicitly modeling long-range dependencies, and it is difficult to capture long-distance topological relationships between retinal layers. The segmentation effect of the continuous boundary of large-area lesions is limited, and false positives are easily caused. At the same time, although the Transformer architecture can rely on its self-attention mechanism to perform well in global modeling, and has excellent task transferability in large-scale pre-training, it still faces challenges in promoting to medical imaging datasets. The sample size of these datasets is often limited, and the classes are imbalanced. In addition, the high computational cost of Transformer limits its application in medical OCT image analysis. Secondly, the feature fusion strategy is unreasonable. Most methods use simple feature stacking or concatenation, which cannot effectively decouple the semantic information of retinal layers and lesions, resulting in mutual interference of their features. Especially in the lesion area, the blurred features of the layer structure are easily covered by the lesion features, causing the layer segmentation accuracy to decrease.

[0006] In summary, the existing segmentation methods cannot efficiently model the collaborative relationship between long-range dependencies and local details when segmenting the OCT images of the retina, cannot effectively decouple the layer and lesion semantics, resulting in mutual interference of features, limited segmentation effect of the continuous boundary of lesions, and easy occurrence of boundary fracture, local area missegmentation, and false positives. Moreover, the feature fusion introduced by the existing methods mostly uses fixed-weight feature splicing or stacking, which cannot dynamically adjust the fusion strategy according to the scale characteristics of different regions, cannot focus on small-scale detail features, and the segmentation accuracy of layer structure and lesions of different scales is uneven. SUMMARY

[0007] Therefore, the technical problem to be solved by the present application is to overcome the problem that the existing technology cannot efficiently model the collaborative relationship between long-range dependencies and local details, cannot effectively decouple the layer and lesion semantics, resulting in low segmentation accuracy and poor segmentation accuracy.

[0008] To address the aforementioned technical problems, this invention provides a method for joint segmentation of the retinal layer and lesions based on a U-Net network, comprising:

[0009] The OCT image to be segmented is obtained and input into the encoder of the improved U-Net network. After passing through N layers of encoding modules, the encoding features of the corresponding layers are obtained.

[0010] The Nth layer encoded features are input into the parallel multi-scale feature fusion module of the improved U-Net network. After feature shuffling, hybrid features are generated. Based on the feature channels, the hybrid features are separated to generate first channel features and second channel features. These are then input into two parallel multi-scale map feature fusion units to output the first fused features and the second fused features.

[0011] The first and second fusion features are input into the decoder of the improved U-Net network. After passing through the parallel decoding module with N-1 layers and skipping connections to the corresponding layer's encoding module, the coordinate attention features and spatial attention features of the corresponding layer are obtained.

[0012] The coordinate attention features and spatial attention features output from the first-layer decoding module are input into the segmentation head of the improved U-Net network, and the convolution outputs the joint segmentation results of the retinal layer and lesions.

[0013] Preferably, each layer of the encoder contains two cascaded encoding units, and the output of the second encoding unit is used as the encoding feature output by each layer of the encoding module.

[0014] Preferably, the encoding unit includes:

[0015] The input features are processed through parallel main coding branches and auxiliary coding branches;

[0016] In the main encoding branch, the input features pass through a series of convolutional layers, batch normalization layers, activation function layers, and convolutional and batch normalization layers connected in sequence along the forward propagation direction, and the output is the main encoded features;

[0017] In the auxiliary coding branch, the input features pass through a convolutional layer, a batch normalization layer, and a channel attention unit connected in series along the forward propagation direction, and output auxiliary coding features.

[0018] The main coding feature and the auxiliary coding feature are added element by element, and then activated by an activation function layer to output the coding feature;

[0019] In this process, the channel attention unit sequentially passes the input features through a global average pooling layer, a linear layer, a ReLU activation function layer, and a linear layer and a Sigmoid activation function layer connected in the forward propagation direction. Then, it multiplies the input features of the channel attention unit channel by channel to output auxiliary encoded features.

[0020] Preferably, the channel features are input into the multi-scale map feature fusion unit, and the fused features are output, including:

[0021] The channel features are input into four parallel branches; the first branch directly outputs the channel features; the second branch passes through a graph inference module and outputs the original scale features; the third branch passes through 2×2 max pooling, a graph inference module, and upsampling to output the medium-scale features; the fourth branch passes through 3×3 max pooling, a graph inference module, and upsampling to output the large-scale features; the outputs of the four parallel branches are concatenated and then passed through a CBR unit to output the fused features.

[0022] The graph reasoning module includes:

[0023] The input features are processed through three parallel CBR units with different parameters, and the outputs are dimensionality-reduced features, inverse projection matrix, and dual projection matrix; the dual projection matrix is ​​transposed to obtain the projection matrix.

[0024] After performing matrix multiplication between the dimensionality-reduced features and the projection matrix, the input is given to the graph convolutional network, which outputs the first convolutional features. The first convolutional features are then multiplied with the inverse projection matrix to obtain the first graph convolutional features.

[0025] Multiply the inverse projection matrix with the transpose of the dimensionality-reduced feature, then input the matrix into the graph convolutional network to output the second convolutional feature; multiply the projection matrix with the second convolutional feature and then transpose the matrix to obtain the second graph convolutional feature.

[0026] The convolutional features of the first and second images are concatenated along the channel dimension, then reorganized along the matrix dimension to convert them into a three-dimensional feature map. After passing through a CBR unit, the global image convolutional features are output.

[0027] The global graph convolutional features are residually concatenated with the features of the input graph inference module to output graph inference features.

[0028] Preferably, the graph convolutional network outputs convolutional features, including:

[0029] Obtaining input features from the input graph convolutional network adjacency matrix Perform symmetric normalization on the adjacency matrix to obtain the normalized adjacency matrix. , represented as ;

[0030] The input features are transformed based on the normalized adjacency matrix, and the output convolutional features are generated. , represented as:

[0031] ;

[0032] in, The adjacency matrix represents the connection information matrix, denoted as: , Represents the identity matrix; express The degree matrix; This represents the weights of the first layer of the graph convolutional network. This represents the weights of the second layer of the graph convolutional network. This represents the softmax activation function. This represents the ReLU activation function.

[0033] Preferably, the parallel decoding module acquires coordinate attention features and spatial attention features, including:

[0034] The parallel decoding module acquires two input features and inputs them into two decoding branches respectively; each decoding branch includes:

[0035] The input features are processed by the CBR unit, and the initial features are output.

[0036] The initial features are processed by an attention unit to output enhanced features;

[0037] After the enhanced features are added to the initial features channel by channel, the output is processed by the CBR unit and then added to the corresponding layer's encoded features channel by channel to output the attention features;

[0038] One decoding branch has a coordinate attention unit, which outputs coordinate attention features; the other decoding branch has a spatial attention unit, which outputs spatial attention features.

[0039] Specifically, when the parallel decoding module is at layer N-1, the two input features are the first fusion feature and the second fusion feature; when the parallel decoding module is not at layer N-1, the two input features are the coordinate attention feature and the spatial attention feature output by the parallel decoding module of the previous layer.

[0040] Preferably, the initial feature input coordinate attention unit outputs enhanced features for the coordinates, including:

[0041] Global average pooling is performed on each row and column of the initial features along the coordinate axis to obtain the horizontal and vertical feature vectors;

[0042] After concatenating the horizontal and vertical feature vectors, the vectors are reduced in dimensionality through convolution, then separated and passed through activation functions to obtain the horizontal and vertical attention weight maps.

[0043] The initial features are then multiplied element-wise with the horizontal attention weight map and the vertical attention weight map to obtain the enhanced features for the coordinates.

[0044] Preferably, the initial feature input space attention unit outputs enhanced features in the space, including:

[0045] Max pooling and average pooling are performed on the initial features along the channel dimension to obtain the first spatial feature summary map and the second spatial feature summary map;

[0046] The first spatial feature summary map and the second spatial feature summary map are concatenated along the channel dimension to obtain a dual-channel feature map;

[0047] A 7×7 convolutional layer is used to convolve the dual-channel feature map, and then an activation function is applied to obtain the spatial attention weight map.

[0048] The initial features are multiplied element-wise with the spatial attention weight map to obtain the enhanced features of the space.

[0049] Preferably, after obtaining the coordinate attention features and spatial attention features of the corresponding layer, the method further includes inputting the coordinate attention features and spatial attention features in parallel into the cross-graph convolution module, and outputting optimized coordinate attention features and spatial attention features, including:

[0050] The input coordinate attention features and spatial attention features are passed through a series of 1×1 convolutional and batch normalization layers to perform feature projection, thereby obtaining coordinate projection features and spatial projection features.

[0051] After concatenating the coordinate projection features and the spatial projection features, the global weight matrix is ​​obtained by sequentially passing them through a cascaded 3×3 convolution, a ReLU activation function layer, a 1×1 convolution, and a Sigmoid activation function layer.

[0052] After multiplying the global weight matrix element-wise with the coordinate projection features and spatial projection features respectively, the result is input into the graph convolutional network, which outputs the global coordinate projection features and the global spatial projection features.

[0053] The global coordinate projection features and coordinate attention features are added element by element to obtain the optimized coordinate attention features;

[0054] The global spatial projection features and spatial attention features are added element by element to obtain the optimized spatial attention features.

[0055] Preferably, the first The output of the layer coding module is downsampled and used as the first layer. Input to the layer coding module; the first The output of the layer parallel decoding module is upsampled and used as the first... Input to the layer-parallel decoding module; .

[0056] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0057] The U-Net-based joint segmentation method for retinal layers and lesions described in this invention establishes a parallel multi-scale map feature fusion module between the encoder and decoder. Feature shuffling breaks the inherent channel correlation of the Nth layer encoded features, promoting cross-channel information interaction and avoiding the limitations of single-channel features. Subsequently, feature channels are separated, and the shuffled features are split into first and second channel features, which are then fed into independent multi-scale map feature fusion units, forming a parallel processing structure. This ensures that different feature branches can specifically capture multi-scale information, generating corresponding fused features that are input into the parallel decoding module to obtain coordinate attention features and spatial attention features, improving the localization accuracy of fine structures and strengthening global spatial correlation perception. Through parallel processing, this invention preserves feature diversity and provides complementary features for the final fusion through independent multi-scale map feature fusion units and differentiated processing of decoding branches. This enhances the ability of features to describe complex medical image structures, effectively reducing boundary missegmentation and false positives for lesions, and further improving the joint segmentation accuracy of retinal layers and lesions in OCT images.

[0058] The encoding module of this invention is a dual-branch structure including a main branch and an auxiliary branch. The main encoding branch extracts local details to capture fine spatial details and interlayer local correlations of the retinal sublayers, while the auxiliary encoding branch enhances the feature response of lesions and key retinal layers, suppresses background noise and interference from irrelevant channels, effectively solves the problem of insufficient capture of key structural features in OCT images by traditional encoders, and improves the accuracy of encoding feature extraction.

[0059] The multi-scale graph feature fusion unit in the parallel multi-scale graph feature fusion module of this invention performs graph inference at the original scale, medium scale, and large scale for the input features respectively. It can capture small-scale details of tiny lesions and fine layer boundaries, and can also explore the continuous boundaries of large-area lesions and long-distance topological relationships between retinal layers. This ensures that multi-scale information is effectively integrated in a unified dimension, improves the perception ability of the retinal region of OCT images, enhances the ability of features to describe complex medical image structures, provides high-quality fusion feature support for the decoding stage, and further guarantees the accuracy of segmentation results.

[0060] To match the output of the parallel multi-scale image feature fusion module, this invention designs a parallel decoding module, providing a dual-branch structure for the first and second fused features. The two decoding branches generate coordinate attention features and spatial attention features, respectively. The coordinate attention features can accurately capture details strongly correlated with coordinates, such as lesion boundaries and retinal layer contours, along the coordinate axes, improving the localization accuracy of fine structures. The spatial attention features can focus on the spatial dependence between lesions and normal tissues, strengthening the perception of global spatial correlation. The generation of the two attention features covers the feature requirements of OCT images from different dimensions, effectively decoupling the semantic interference between the retinal layer and lesions, improving the ability to distinguish different types of structures, and thus improving the modeling ability of the retinal layer region and lesion region, ensuring the joint segmentation accuracy of the retinal layer and lesions.

[0061] This invention also designs a cross-graph convolution module to enable information interaction between the two outputs of the parallel decoding module, integrates the advantages of coordinates and spatial attention, and constructs global associations between retinal layers, between layers and lesions, and between lesions. This significantly improves the network's ability to segment layers and lesion regions, lays a solid foundation for the segmentation head to output accurate results, and makes the segmentation more in line with the actual needs for distinguishing the fine structure of retinal OCT images. Attached Figure Description

[0062] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0063] Figure 1 This is a schematic diagram of the image processing process of the U-Net-based joint segmentation method for retinal layer and lesion based on the present invention;

[0064] Figure 2 This is a structural diagram of the encoding module;

[0065] Figure 3 This is a schematic diagram of the structure of the parallel multi-scale graph feature fusion module;

[0066] Figure 4 This is a structural diagram of the graph reasoning module;

[0067] Figure 5 This is a schematic diagram of the parallel decoding module;

[0068] Figure 6 This is a schematic diagram of the cross-graph convolution module;

[0069] Figure 7 In the image (a), the first original OCT image to be segmented is... Figure 7 (b) in the image is the second original OCT image to be segmented. Figure 7(c) in the image is the third original OCT image to be segmented. Figure 7 (d) in the image is the fourth original OCT image to be segmented;

[0070] Figure 8 In the image (a), the manually segmented label map is the first original OCT image to be segmented. Figure 8 (b) in the image is the manually segmented label map of the second original OCT image to be segmented. Figure 8 (c) in the image is the manually segmented label map of the third original OCT image to be segmented. Figure 8 (d) in the image is the manually segmented label map of the fourth original OCT image to be segmented;

[0071] Figure 9 In the image (a), the segmentation label map of the first original OCT image to be segmented is based on the AAU-Net network. Figure 9 (b) in the image is the segmentation label map of the second original OCT image to be segmented based on the AAU-Net network. Figure 9 (c) in the image is the segmentation label map of the third original OCT image to be segmented based on the AAU-Net network. Figure 9 (d) in the image is the segmentation label map of the fourth original OCT image to be segmented based on the AAU-Net network;

[0072] Figure 10 In the image (a), the segmentation label map of the first original OCT image to be segmented is based on the DconnNet network. Figure 10 (b) in the image is the segmentation label map of the second original OCT image to be segmented based on the DconnNet network. Figure 10 (c) in the image is the segmentation label map of the third original OCT image to be segmented based on the DconnNet network. Figure 10 (d) in the image is the segmentation label map of the fourth original OCT image to be segmented based on the DconnNet network;

[0073] Figure 11 In the image (a), the segmentation label map of the first original OCT image to be segmented is based on the UNet++ network. Figure 11 (b) in the image is the segmentation label map of the second original OCT image to be segmented based on the UNet++ network. Figure 11 (c) in the image is the segmentation label map of the third original OCT image to be segmented based on the UNet++ network. Figure 11 (d) in the image is the segmentation label map of the fourth original OCT image to be segmented based on the UNet++ network;

[0074] Figure 12 (a) in the image is the segmentation label map of the first original OCT image to be segmented based on the RollingUNet network. Figure 12(b) in the image is the segmentation label map of the second original OCT image to be segmented based on the RollingUNet network. Figure 12 (c) in the image is the segmentation label map of the third original OCT image to be segmented based on the RollingUNet network. Figure 12 (d) in the image is the segmentation label map of the fourth original OCT image to be segmented based on the RollingUNet network;

[0075] Figure 13 (a) in the image is the segmentation label map of the first original OCT image to be segmented based on the ADSUNet network. Figure 13 (b) in the image is the segmentation label map of the second original OCT image to be segmented based on the ADSUNet network. Figure 13 (c) in the image is the segmentation label map of the third original OCT image to be segmented based on the ADSUNet network. Figure 13 (d) in the image is the segmentation label map of the fourth original OCT image to be segmented based on the ADSUNet network;

[0076] Figure 14 In the image (a), the segmentation label map of the first original OCT image to be segmented is based on the improved U-Net network proposed in this invention. Figure 14 (b) in the figure is the segmentation label map of the second original OCT image to be segmented based on the improved U-Net network proposed in this invention. Figure 14 (c) in the figure is the segmentation label map of the third original OCT image to be segmented based on the improved U-Net network proposed in this invention. Figure 14 In the diagram, (d) is the segmentation label map of the fourth original OCT image to be segmented based on the improved U-Net network proposed in this invention. Detailed Implementation

[0077] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0078] Reference Figure 1 The diagram illustrates the image processing steps of the U-Net-based joint segmentation method for retinal layer and lesion, as shown in the present invention. Specifically, it includes:

[0079] The OCT image to be segmented is obtained and input into the encoder of the improved U-Net network. After passing through N layers of encoding modules, the encoding features of the corresponding layers are obtained.

[0080] The Nth layer encoded features are input into the parallel multi-scale graph feature fusion module of the improved U-Net network. After feature shuffling, hybrid features are generated. Based on the feature channels, the hybrid features are separated to generate first channel features and second channel features. These are then input into two parallel multi-scale graph feature fusion units to output the first fused features and the second fused features.

[0081] The first and second fusion features are input into the decoder of the improved U-Net network. After passing through the parallel decoding module with N-1 layers and skipping connections to the corresponding layer's encoding module, the coordinate attention features and spatial attention features of the corresponding layer are obtained.

[0082] The coordinate attention features and spatial attention features output from the first-layer decoding module are input into the segmentation head of the improved U-Net network, and the convolution outputs the joint segmentation results of the retinal layer and lesions.

[0083] Specifically, no. The output of the layer coding module is downsampled and used as the first layer. Input to the layer coding module; the first The output of the layer parallel decoding module is upsampled and used as the first... Input to the layer-parallel decoding module; .

[0084] Reference Figure 2 The diagram shows the structure of the encoding module. In this embodiment, each layer of the encoder's encoding module contains two cascaded encoding units. The output of the second encoding unit is used as the encoded feature output by each layer of the encoding module. Each encoding unit includes:

[0085] The input features are processed through parallel main coding branches and auxiliary coding branches;

[0086] In the main encoding branch, the input features pass through a series of convolutional layers, batch normalization layers, activation function layers, and convolutional and batch normalization layers connected in sequence along the forward propagation direction, and the output is the main encoded features;

[0087] In the auxiliary coding branch, the input features pass through a convolutional layer, a batch normalization layer, and a channel attention unit connected in series along the forward propagation direction, and output auxiliary coding features.

[0088] The main coding feature and the auxiliary coding feature are added element by element, and then activated by an activation function layer to output the coding feature;

[0089] In this process, the channel attention unit sequentially passes the input features through a global average pooling layer, a linear layer, a ReLU activation function layer, and a linear layer and a Sigmoid activation function layer connected in the forward propagation direction. Then, it multiplies the input features of the channel attention unit channel by channel to output auxiliary encoded features.

[0090] The encoding module of this invention is a dual-branch structure including a main branch and an auxiliary branch. The main encoding branch extracts local details to capture fine spatial details and interlayer local correlations of the retinal sublayers, while the auxiliary encoding branch enhances the feature response of lesions and key retinal layers, suppresses background noise and interference from irrelevant channels, effectively solves the problem of insufficient capture of key structural features in OCT images by traditional encoders, and improves the accuracy of encoding feature extraction.

[0091] Reference Figure 3 The diagram shown is a schematic representation of the parallel multi-scale map feature fusion module; refer to... Figure 4 The diagram shown is a schematic representation of the graph inference module. In this embodiment, based on this structure, channel features are input into the multi-scale graph feature fusion unit, and fused features are output, including:

[0092] The channel features are input into four parallel branches; the first branch directly outputs the channel features; the second branch passes through a graph inference module and outputs the original scale features; the third branch passes through 2×2 max pooling, a graph inference module, and upsampling to output the medium-scale features; the fourth branch passes through 3×3 max pooling, a graph inference module, and upsampling to output the large-scale features; the outputs of the four parallel branches are concatenated and then passed through a CBR unit to output the fused features.

[0093] The graph reasoning module includes:

[0094] The input features are processed through three parallel CBR units with different parameters, and the outputs are dimensionality-reduced features, inverse projection matrix, and dual projection matrix; the dual projection matrix is ​​transposed to obtain the projection matrix.

[0095] After performing matrix multiplication between the dimensionality-reduced features and the projection matrix, the input is given to the graph convolutional network, which outputs the first convolutional features. The first convolutional features are then multiplied with the inverse projection matrix to obtain the first graph convolutional features.

[0096] Multiply the inverse projection matrix with the transpose of the dimensionality-reduced feature, then input the matrix into the graph convolutional network to output the second convolutional feature; multiply the projection matrix with the second convolutional feature and then transpose the matrix to obtain the second graph convolutional feature.

[0097] The convolutional features of the first and second images are concatenated along the channel dimension, then reorganized along the matrix dimension to convert them into a three-dimensional feature map. After passing through a CBR unit, the global image convolutional features are output.

[0098] The global graph convolutional features are residually concatenated with the features of the input graph inference module to output graph inference features.

[0099] The Graph Convolutional Network (GCN) outputs convolutional features, including:

[0100] Obtaining input features from the input graph convolutional network adjacency matrix Perform symmetric normalization on the adjacency matrix to obtain the normalized adjacency matrix. , represented as ;

[0101] The input features are transformed based on the normalized adjacency matrix, and the output convolutional features are generated. , represented as:

[0102] ;

[0103] in, The adjacency matrix represents the connection information matrix, denoted as: , Represents the identity matrix; express The degree matrix; This represents the weights of the first layer of the graph convolutional network. This represents the weights of the second layer of the graph convolutional network. This represents the softmax activation function. This represents the ReLU activation function.

[0104] In this embodiment, the CBR unit includes a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in series along the forward propagation direction. By adjusting the parameters of the 3×3 convolutional layer, the batch normalization layer, and the ReLU activation function layer, the CBR unit can capture different information from the input features and generate corresponding feature outputs.

[0105] The multi-scale graph feature fusion unit in the Parallel Multi-scale Graph Feature Fusion (PMG) module of this invention performs graph inference at the original scale, medium scale, and large scale for the input features respectively. It can capture small-scale details of tiny lesions and fine layer boundaries, and can also explore the continuous boundaries of large-area lesions and long-distance topological relationships between retinal layers. This ensures that multi-scale information is effectively integrated in a unified dimension, improves the perception of the retinal region of OCT images, enhances the ability of features to describe complex medical image structures, provides high-quality fused feature support for the decoding stage, and further guarantees the accuracy of segmentation results.

[0106] Reference Figure 5 The diagram shows the structure of the parallel decoding module; the parallel decoding module acquires coordinate attention features and spatial attention features, including:

[0107] The parallel decoding module acquires two input features and inputs them into two decoding branches respectively; each decoding branch includes:

[0108] The input features are processed by the CBR unit, and the initial features are output.

[0109] The initial features are processed by an attention unit to output enhanced features;

[0110] After the enhanced features are added to the initial features channel by channel, the output is processed by the CBR unit and then added to the corresponding layer's encoded features channel by channel to output the attention features;

[0111] One decoding branch has a coordinate attention unit, which outputs coordinate attention features; the other decoding branch has a spatial attention unit, which outputs spatial attention features.

[0112] Specifically, when the parallel decoding module is at layer N-1, the two input features are the first fusion feature and the second fusion feature; when the parallel decoding module is not at layer N-1, the two input features are the coordinate attention feature and the spatial attention feature output by the parallel decoding module of the previous layer.

[0113] Specifically, the initial features are input to the coordinate attention unit, and the output is the enhanced features of the coordinates, including: performing global average pooling on each row and column of the initial features along the coordinate axis to obtain horizontal and vertical feature vectors; concatenating the horizontal and vertical feature vectors, performing convolutional dimensionality reduction, then splitting them and passing them through activation functions to obtain horizontal and vertical attention weight maps; and performing element-wise multiplication of the initial features with the horizontal and vertical attention weight maps to obtain the enhanced features of the coordinates.

[0114] Specifically, the initial features are input into the spatial attention unit, and the output is the enhanced features of the space, including: performing max pooling and average pooling on the initial features along the channel dimension to obtain a first spatial feature summary map and a second spatial feature summary map; concatenating the first spatial feature summary map and the second spatial feature summary map along the channel dimension to obtain a dual-channel feature map; convolving the dual-channel feature map using a 7×7 convolutional layer, and then passing it through an activation function to obtain a spatial attention weight map; and multiplying the initial features and the spatial attention weight map element-wise to obtain the enhanced features of the space.

[0115] To match the output of the parallel multi-scale image feature fusion module, this invention designs a parallel decoding module, providing a dual-branch structure for the first and second fused features. The two decoding branches generate coordinate attention (CA) features and spatial attention (SA) features, respectively. The coordinate attention feature can accurately capture details strongly correlated with coordinates, such as lesion boundaries and retinal layer contours, along the coordinate axes, improving the localization accuracy of fine structures. The spatial attention feature can focus on the spatial dependence between lesions and normal tissues, strengthening the perception of global spatial correlation. The generation of the two attention features covers the feature requirements of OCT images from different dimensions, effectively decoupling the semantic interference between the retinal layer and lesions, improving the ability to distinguish different types of structures, and thus improving the modeling ability of the retinal layer region and lesion region, ensuring the joint segmentation accuracy of the retinal layer and lesions.

[0116] Based on the above embodiments, this embodiment sets up a Cross Graph Convolution (CGC) module after each parallel decoding module, referring to... Figure 6 The diagram shows the structure of the cross-graph convolution module. After obtaining the coordinate attention features and spatial attention features of the corresponding layer, the module further includes inputting the coordinate attention features and spatial attention features into the cross-graph convolution module in parallel, and outputting optimized coordinate attention features and spatial attention features, including:

[0117] The input coordinate attention features and spatial attention features are passed through a series of 1×1 convolutional and batch normalization layers to perform feature projection, thereby obtaining coordinate projection features and spatial projection features.

[0118] After concatenating the coordinate projection features and the spatial projection features, the global weight matrix is ​​obtained by sequentially passing them through a cascaded 3×3 convolution, a ReLU activation function layer, a 1×1 convolution, and a Sigmoid activation function layer.

[0119] After multiplying the global weight matrix element-wise with the coordinate projection features and spatial projection features respectively, the result is input into the graph convolutional network, which outputs the global coordinate projection features and the global spatial projection features.

[0120] The global coordinate projection features and coordinate attention features are added element by element to obtain the optimized coordinate attention features;

[0121] The global spatial projection features and spatial attention features are added element by element to obtain the optimized spatial attention features.

[0122] This invention designs a cross-graph convolution module to realize information interaction between the two outputs of the parallel decoding module. It integrates the advantages of coordinate and spatial attention to build global associations between retinal layers, between layers and lesions, and between lesions. This significantly improves the network's ability to segment layers and lesion regions, laying a solid foundation for the segmentation head to output accurate results and making the segmentation more in line with the actual needs for distinguishing the fine structure of retinal OCT images.

[0123] Based on the above embodiments, in this embodiment of the invention, the retinal layer and lesion joint segmentation method based on the U-Net network provided by the present invention is used to segment OCT images. The U-Net network structure in this embodiment is a U-shaped encoder-decoder structure. The overall logic is based on parallel feature processing and multi-scale graph structure modeling, and a parallel multi-scale graph feature fusion module is proposed to deeply mine and fuse multi-scale graph structure feature information, thereby improving the model's ability to understand OCT images. This module constructs a dual-branch parallel link through feature separation, providing diverse feature inputs for subsequent multi-scale exploration. In the parallel multi-scale graph feature fusion module, branches with different pooling kernels are used to perform graph inference on the original scale, medium scale, and large scale features respectively, to mine non-local correlations between features, and then feature integration is performed. The parallel decoding module accurately fuses multi-dimensional features and enhances the representation of key information. It first uses a dual-branch parallel structure, relying on convolutional layers to complete basic feature extraction, embedding coordinate attention and spatial attention respectively to accurately capture key features from the spatial dimension. Skip connections achieve feature complementarity between the encoder and decoder. Subsequently, a cross-graph convolution module constructs graph-structured relationships between features, mining semantic dependencies across branches and scales, and then outputs the results through feature fusion. The decoder expands feature diversity through parallel branches, uses an attention mechanism to focus on key information, and strengthens global associations based on cross-graph convolution, achieving deep decoding from local feature optimization to global semantic integration.

[0124] This embodiment uses an encoder-decoder network structure to segment the retinal layer and lesions. The encoder progressively downsamples and compresses features through multiple convolutions and max pooling, reducing spatial dimensions and increasing channel dimensions to extract multi-scale basic features of the image. At the bottleneck layer of the network, multi-scale graph convolution features are extracted through the PMG module. The decoder gradually restores the feature spatial resolution through upsampling and parallel decoding, during which a cross-graph convolution module is added to fuse feature information from different levels and channels, build cross-correlation between features, optimize the fusion accuracy of semantics and details, and improve the consistency of segmentation boundaries and regions. The encoder and decoder pass shallow details through skip connections. Specifically, as shown... Figure 1 As shown, the network employs a U-shaped encoder-decoder architecture. The input image X sequentially enters a multi-layer encoder, where max pooling reduces the feature map size and expands the receptive field, extracting encoded features from different levels. Features of the last layer in the encoding stage The PMG module performs multi-scale feature fusion and inference to mine richer contextual information, providing a higher-quality feature foundation for decoding. The first and second fused features output from the PMG module are... , Input to the decoder, and generate features for each decoder layer step by step. During each decoding process, the features decoded in the previous layer... , Encoder features passed via skip connections After being processed and fused by the parallel decoding module, decoder features are generated through cross-graph convolution. and In the final stage of the network, the decoder's final features are... , The segments are concatenated along the channel dimension, and after passing through a segmentation head (1x1 convolution), the final segmentation result O is output. Based on this, the detailed workflow of each module in this embodiment is described, specifically including:

[0125] ① Encoder:

[0126] Each layer of the encoder consists of two cascaded encoding modules, such as... Figure 2 As shown, the input for each layer is (i=0,1,2,3,4), where The output is The encoder module processes the input in two parallel branches: the main branch first extracts local spatial details through 3×3 convolutions, stabilizes the data distribution through batch normalization (BN), introduces non-linearity through ReLU activation, and then deepens feature abstraction through another 3×3 convolution, followed by BN processing again; the auxiliary branch first compresses the channel dimension with 1×1 convolutions to reduce computation, and then enters the squeeze and excitation (SE) module after BN. The SE module is a simple channel attention processing module. It first aggregates the spatial information of each channel into a single-valued statistic through global average pooling, then performs linear mapping through a linear layer to extract high-order features, uses ReLU activation to filter effective information, performs inverse mapping through a linear layer, and finally uses Sigmoid to generate channel attention weights in the 0-1 interval, which are multiplied channel by channel with the initial input features of the SE module to achieve key channel enhancement and secondary channel suppression. After processing the two paths, the deep detail features of the main branch and the channel enhancement features of the auxiliary branch are added element by element to fuse spatial details and channel attention information. Finally, ReLU activation is used to further enhance the feature expression, outputting encoded features that have both rich spatial information and channel importance awareness.

[0127] ② Parallel multi-scale map feature fusion module:

[0128] In the parallel multi-scale graph feature fusion module, the input features Channel shuffle breaks the inherent correlation between channels, promoting cross-channel information interaction; subsequently, feature separation (i.e., dividing into first-half channel features and second-half channel features) yields two sets of features. and Each feature branch is fed into an independent multi-scale map feature fusion unit, forming a parallel processing structure. This ensures that different feature branches can specifically capture multi-scale information, ultimately yielding the output. , This parallel design preserves feature diversity while providing complementary features for the final fusion through differentiated processing of the two fusion units.

[0129] The multi-scale map feature fusion unit targets single-path input features, using features For example, the input features are divided into four paths. The first path is unprocessed; the second path directly enters the graph inference module to obtain the original scale features. The third approach obtains mesoscale features through 2×2 max pooling and graph reasoning. The fourth approach obtains large-scale features through 3×3 max pooling and graph reasoning. . and After upsampling and restoring the size, it is compared with the first channel. Second Route The features are aggregated through concatenation, and then fused and nonlinearly enhanced using CBR (3×3 convolution + BN + ReLU) to obtain the desired result. This ensures the effective integration of multi-scale information within a unified dimension, enhancing the feature's ability to describe complex medical image structures. Another feature... Obtained through the same process .

[0130] As a core component of the multi-scale graph feature fusion module, the graph reasoning module, such as Figure 4 As shown, input features , , and These represent the number of channels, height, and width of the feature map, respectively; the input features are used to construct projection relationships in three ways: one way is processed by CBR (3×3 convolution + BN + ReLU) to generate dimensionality-reduced features. After passing through CBR (3×3 convolution + BN + ReLU), the inverse projection matrix is... The other path is processed by CBR (3×3 convolution + BN + ReLU) and matrix transpose to generate the projection matrix. ,in This represents the number of channels after dimensionality reduction. The dimensionality-reduced features and the projection matrix are multiplied together, and then fed into the GCN module. Its output is multiplied with the inverse projection matrix to obtain the first graph convolutional feature. , represented as The inverse projection matrix is ​​multiplied by the transpose of the dimensionality-reduced feature, and then fed into the GCN module. The projection matrix is ​​then multiplied by its output and transposed to obtain another graph convolutional feature. , represented as The two convolutional features are concatenated along the channel dimension, and then the matrix dimension is reorganized to transform the convolutional features from a two-dimensional feature map into a larger one. The 3D feature map is then processed by CBR to obtain the global graph convolutional features. , represented as M and input features The output of the graph reasoning module is obtained by adding the residuals together.

[0131] A graph convolutional network can be represented as f(N,A), where N represents the input features and A is the adjacency matrix. First, the adjacency matrix A needs to be symmetrically normalized to obtain the normalized adjacency matrix, represented as... ;in, A matrix representing the connection information between a node and its neighboring nodes. It is the identity matrix. for The degree matrix. for The symmetrically normalized matrix. The computation result of the Graph Convolutional Network (GCN) in this invention can be expressed as:

[0132] ;

[0133] Let represent the weights of the i-th layer (i=1,2), softmax represents the softmax activation function, and ReLU represents the ReLU activation function.

[0134] ③ Parallel decoding module:

[0135] The decoder of this invention is designed with the characteristics of the PMG module in mind, and is based on a dual-branch parallel decoding architecture. In each parallel decoding module, for the high-level decoding features of the input... and The input has two parallel branches, including:

[0136] First, the feature map passes through the CBR module, followed by the Coordinate Attention (CA) module. This module aggregates feature information along the coordinate axes, performing global average pooling on each row and column of the input feature map to obtain feature vectors in the horizontal and vertical directions. These two vectors are then concatenated, convolved for dimensionality reduction, and then separated, with sigmoid functions used to generate attention weight maps representing the importance of horizontal and vertical positions. Finally, the original feature map is element-wise multiplied with these two weight maps, accurately capturing spatial location information, such as lesion boundaries and tissue contours—details strongly correlated with coordinates—while encoding channel relationships, thus enhancing the feature response in key regions. The CA-processed features are added to the CBR output features, allowing for better integration of shallow details and deep semantics. Then, the CBR is used again to deepen the feature encoding, and the results are combined with the corresponding encoder features. The coordinate attention branch output is obtained by adding and merging the components. , represented as ;

[0137] First, the input feature map passes through the CBR module, followed by the Spatial Attention (SA) module. This SA module focuses on the spatial distribution patterns of features, uncovering the spatial dependencies of lesions and tissues in medical images. First, the input feature map undergoes max pooling and average pooling along the channel dimension, generating two two-dimensional spatial feature summary maps. Then, these two summary maps are concatenated along the channel dimension to form a two-channel feature map. Next, a 7x7 convolutional layer is used to convolve this two-channel feature map, followed by a sigmoid activation function to generate a spatial attention weight map. The value of each pixel in this map is between 0 and 1, representing the importance of its corresponding spatial location. Finally, this spatial attention weight map is element-wise multiplied with the original input feature map to obtain the spatially attention-weighted output feature map. The SA-processed features are added to the CBR output features. Then, the CBR is used again to deepen the feature encoding, and the results are combined with the corresponding encoder features. By adding and fusing, we obtain the output of the spatial attention branch. , represented as: ;

[0138] In this embodiment, the two branches in the parallel decoding module complement each other from the perspectives of coordinates and spatial attention to mine features, enabling the decoder to fully cover the feature requirements of different dimensions of medical images.

[0139] and Entering the cross-graph convolution module and enabling cross-branch deep fusion, we first perform feature projection on the features of the two branches (1×1 convolution + batch normalization) to generate projected features. , The projected features are concatenated to integrate information from coordinate and spatial attention mining. Then, global weights are constructed using 3x3 convolution, ReLU, 1x1 convolution, and Sigmoid operations, generating a weight matrix that encodes the correlation strength between the two-branch features. Weight matrix After element-wise multiplication with the projected features, each feature is input into the GCN, allowing the bi-branch features to achieve nonlocal interactions in the graph space and uncover long-range dependencies. Finally, the GCN output is added to the other original feature to obtain the fused cross-branch information. and , represented as , .

[0140] This embodiment establishes a connection between parallel decoders through cross-graph convolution, integrating the advantages of coordinate and spatial attention to improve the recognition of complex structures and subtle lesions in medical images. This lays a solid foundation for the segmentation head to output accurate results, making the segmentation more in line with the clinical requirements for distinguishing fine structures.

[0141] This invention proposes a segmentation network with a parallel feature-based encoder-decoder structure for automatically segmenting the retinal layer and lesions in OCT images. The encoder employs a two-convolutional-layer architecture with a simple channel attention mechanism. The two convolutional layers extract local features to construct details of the layer and lesions. Channel attention extracts information from each channel of the feature map and assigns corresponding weights, enhancing the encoder's ability to model the retinal layer and lesions. A parallel multi-scale graph feature fusion module deeply mines the network's deep feature information and enhances the non-local associations between features through graph convolution, improving the model's understanding of OCT images. The parallel decoder uses a dual-branch parallel structure to decode deep features from two dimensions. Then, through a cross-graph convolution module, a graph structure association between the features of the two decoders is constructed, achieving deep decoding from local feature extraction to global semantic integration.

[0142] Based on the above description, this embodiment verifies the joint segmentation method for retinal layer and lesion based on U-Net network provided by the present invention. The loss function used in this embodiment is a combination of Dice loss and focus loss. Dice loss directly measures the degree of overlap of segmentation results and is a commonly used loss function in image segmentation, but it has certain limitations. When the data classes are extremely imbalanced, its contribution to the gradient of rare classes is small, causing the model to tend to learn common classes and perform poorly on rare classes. At the same time, in the early stage of training, when the predicted results differ greatly from the true labels, the calculation of Dice coefficients is unstable, which can lead to gradient vanishing and slow training convergence. Focus loss can effectively alleviate the class imbalance problem by adjusting the loss weights and focusing on difficult samples, making the model pay more attention to rare classes. The combination of the two can maintain effective supervision of segmentation accuracy while dealing with class imbalance, accelerate the model convergence speed, and improve segmentation performance, especially when dealing with complex scenes and imbalanced data.

[0143] Total loss of this application , represented as: ; This indicates Dice's loss. This indicates a loss at the focal point.

[0144] The dataset used in this embodiment is the publicly available AMD-SD dataset. The AMD-SD dataset was acquired using a Zeiss Cirrus-HD-5000 camera and includes 138 patients and 156 pairs of eyes. Specific information is shown in Table 1, and the image size is 570×380. The gold standard of the original dataset includes five types of lesions: IRF, SRF, PED, SHRM, and IS / OS (Ellipsoid zone).

[0145] Table 1. Detailed information about the AMD-SD dataset

[0146] Number of patients 138 Number of eyes 156 Number of slices 3049 Mean age (standard deviation) 66.7（9.1） Proportion of females (%) 61（44.2）

[0147] The original AMD-SD dataset only contains the gold standard for lesion identification, but not the gold standard for retinal layering. For research purposes, this embodiment supplements the AMD-SD dataset with layer labels, dividing the retinal layer into three layers: Layer 1: Internal limiting membrane (ILM) - inner plexiform layer (IPL); Layer 2: Internal nuclear layer (INL) - retinal pigment epithelium (RPE); Layer 3: retinal pigment epithelium - Bruch's membrane complex (BM). To objectively and fairly evaluate the segmentation algorithm proposed in this embodiment, all experiments were conducted on a patient-by-patient basis, ensuring a relatively balanced distribution of images with lesions. The training set consisted of 2346 images, and the validation set consisted of 703 images.

[0148] The network proposed in this embodiment was trained end-to-end, with a batch size of 4 for both training and validation. Considering GPU memory and training time costs, the OCT images and manually segmented label images were downsampled to 384×256 pixels before input. A poly learning rate descent strategy was applied to optimize the network, using AdamW optimizer, with a learning rate of [missing information]. The calculation is as follows:

[0149] ;

[0150] The basic learning rate Set to 0.0001. Set to 0.9, This indicates the number of iterations during the training process. This represents the total number of iterations in the entire training process, set to 100. The model is based on the public platform PyTorch and a GeForce RTX 3090 GPU with 24GB of VRAM. During network training, after each iteration, the model is validated against the validation set data, and the model parameters with the highest average DSC in the validation set are retained as the network model parameters for the final test.

[0151] To evaluate the segmentation results of the retinal layer and lesions in OCT images, three evaluation metrics were used: Dice similarity coefficient (DSC), intersection-to-union ratio (IoU), and sensitivity (Sen). Their definitions are as follows:

[0152] ; ; ;

[0153] TP, TN, FP, and FN represent the true positive, true negative, false positive, and false negative scores for segmentation, respectively. The metrics for each category are calculated separately, and the overall average is also calculated. Meanwhile, to ensure a fairer calculation of evaluation metrics, if the gold standard image does not contain a label for a particular category, the evaluation metric for that category in the corresponding segmentation prediction image is not calculated and is skipped.

[0154] Table 2 lists the results of the comparative experiments. Layer 1 represents ILM-IPL, layer 2 represents INL-RPE, layer 3 represents RPE-BM, lesion 1 represents PED, lesion 2 represents SHRM, lesion 3 represents SRF, lesion 4 represents IS / OS, and lesion 5 represents IRF.

[0155] Table 2 Comparison Experiment Results of AMD-SD Dataset

[0156]

[0157] As shown in Table 2, this invention demonstrates significant improvements across all metrics. The average Dice, IoU, and Sen scores are the highest, at 77.55, 68.19, and 79.00 respectively. Based on these average metrics, this invention outperforms existing methods in both retinal layer and lesion segmentation performance. Looking at individual retinal layer and lesion segmentation, except for IS / OS lesion segmentation which is inferior to RollingUnet, the segmentation results for other layers and lesions are the highest. The method in this embodiment exhibits good and balanced segmentation performance. The visualization of the segmentation results is shown below. Figures 7 to 14 As shown, Figure 7 In the image (a), the first original OCT image to be segmented is... Figure 7 (b) in the image is the second original OCT image to be segmented. Figure 7 (c) in the image is the third original OCT image to be segmented. Figure 7 (d) in the image is the fourth original OCT image to be segmented; Figure 8 In the image (a), the manually segmented label map is the first original OCT image to be segmented. Figure 8 (b) in the image is the manually segmented label map of the second original OCT image to be segmented. Figure 8 (c) in the image is the manually segmented label map of the third original OCT image to be segmented. Figure 8 (d) in the image is the manually segmented label map of the fourth original OCT image to be segmented; Figure 9 In the image (a), the segmentation label map of the first original OCT image to be segmented is based on the AAU-Net network. Figure 9 (b) in the image is the segmentation label map of the second original OCT image to be segmented based on the AAU-Net network.Figure 9 (c) in the image is the segmentation label map of the third original OCT image to be segmented based on the AAU-Net network. Figure 9 (d) in the image is the segmentation label map of the fourth original OCT image to be segmented based on the AAU-Net network; Figure 10 In the image (a), the segmentation label map of the first original OCT image to be segmented is based on the DconnNet network. Figure 10 (b) in the image is the segmentation label map of the second original OCT image to be segmented based on the DconnNet network. Figure 10 (c) in the image is the segmentation label map of the third original OCT image to be segmented based on the DconnNet network. Figure 10 (d) in the image is the segmentation label map of the fourth original OCT image to be segmented based on the DconnNet network; Figure 11 In the image (a), the segmentation label map of the first original OCT image to be segmented is based on the UNet++ network. Figure 11 (b) in the image is the segmentation label map of the second original OCT image to be segmented based on the UNet++ network. Figure 11 (c) in the image is the segmentation label map of the third original OCT image to be segmented based on the UNet++ network. Figure 11 (d) in the image is the segmentation label map of the fourth original OCT image to be segmented based on the UNet++ network; Figure 12 (a) in the image is the segmentation label map of the first original OCT image to be segmented based on the RollingUNet network. Figure 12 (b) in the image is the segmentation label map of the second original OCT image to be segmented based on the RollingUNet network. Figure 12 (c) in the image is the segmentation label map of the third original OCT image to be segmented based on the RollingUNet network. Figure 12 (d) in the image is the segmentation label map of the fourth original OCT image to be segmented based on the RollingUNet network; Figure 13 (a) in the image is the segmentation label map of the first original OCT image to be segmented based on the ADSUNet network. Figure 13 (b) in the image is the segmentation label map of the second original OCT image to be segmented based on the ADSUNet network. Figure 13 (c) in the image is the segmentation label map of the third original OCT image to be segmented based on the ADSUNet network. Figure 13 (d) in the image is the segmentation label map of the fourth original OCT image to be segmented based on the ADSUNet network; Figure 14 In the image (a), the segmentation label map of the first original OCT image to be segmented is based on the improved U-Net network proposed in this invention. Figure 14(b) in the figure is the segmentation label map of the second original OCT image to be segmented based on the improved U-Net network proposed in this invention. Figure 14 (c) in the figure is the segmentation label map of the third original OCT image to be segmented based on the improved U-Net network proposed in this invention. Figure 14 (d) is the segmentation label map of the fourth original OCT image to be segmented based on the improved U-Net network proposed in this invention; among them, the red ILM-IPL corresponds to the region from the internal limiting membrane (ILM) to the internal plexiform layer (IPL), which is an important superficial structure for retinal nerve signal transmission, and its integrity is closely related to diseases such as glaucoma; the orange INL-RPE corresponds to the region from the inner nuclear layer (INL) to the retinal pigment epithelium (RPE), which includes key sublayers such as the outer plexiform layer and the outer nuclear layer. Destruction of this region often indicates diseases such as macular degeneration; the blue RPE-BM corresponds to the region from the retinal pigment epithelium (RPE) to the Bruch's membrane (BM). The integrity of the RPE layer is crucial for retinal nutrition supply, and abnormalities in this region are a typical manifestation of age-related macular degeneration; Yellow PED corresponds to pigment epithelial detachment (PED), a lesion formed by fluid accumulation between the RPE layer and BM, commonly seen in diseases such as macular degeneration; green SHRM corresponds to subretinal high reflectance material (SHRM), mostly inflammatory exudates or fibrotic tissue, and is an important indicator of retinal disease activity; gray SRF corresponds to subretinal fluid (SRF), i.e., fluid accumulation between retinal layers, often accompanied by lesions such as macular edema; purple IS / OS corresponds to the junction area between the inner segment (IS) and outer segment (OS) of photoreceptor cells, and the breakage or absence of the IS / OS layer directly affects visual function, making it an important marker of abnormal retinal photosensitivity; cyan IRF corresponds to intraretinal fluid (IRF), i.e., fluid accumulation within the retina, a common pathological manifestation of diseases such as diabetic retinopathy. Figures 7 to 14 As can be seen, compared with other segmentation networks, the present invention achieves more complete lesion segmentation, with clearer boundaries between lesions and layers. The network proposed in this invention overcomes the difficulties in segmenting the retinal layer and lesion regions, such as the large horizontal span of OCT images, blurred continuous boundaries between the retinal layer and lesions, large differences in the morphology of different lesions, and imbalanced sample distribution in the dataset. The network proposed in this invention can accurately segment the retinal layer and lesions in retinal OCT images, surpassing many other advanced segmentation networks.

[0158] The U-Net-based joint segmentation method for retinal layers and lesions described in this invention incorporates a parallel multi-scale feature fusion module between the encoder and decoder. This module breaks down the inherent channel correlations of the Nth layer encoded features through feature shuffling, promoting cross-channel information interaction and avoiding the limitations of single-channel features. Subsequently, feature channels are separated, with the shuffled features divided into first and second channel features, each fed into an independent multi-scale map feature fusion unit, forming a parallel processing structure. This ensures that different feature branches can specifically capture multi-scale information, generating corresponding fused features that are input into the parallel decoding module. This acquires coordinate attention features and spatial attention features, improving the accuracy of fine structure localization and strengthening global spatial correlation perception. Through parallel processing, this invention preserves feature diversity while providing complementary features for final fusion via independent multi-scale map feature fusion units and differentiated processing of decoding branches. This enhances the ability of features to describe complex medical image structures, effectively reducing boundary missegmentation and false positives for lesions, and further improving the joint segmentation accuracy of retinal layers and lesions in OCT images. The encoding module of this invention has a dual-branch structure including a main branch and an auxiliary branch. The main encoding branch extracts local details, capturing fine spatial details and interlayer local correlations of the retinal sublayers, while the auxiliary encoding branch enhances the feature responses of lesions and key retinal layers, suppressing background noise and interference from irrelevant channels. This effectively solves the problem of insufficient capture of key structural features in OCT images by traditional encoders, improving the accuracy of encoded feature extraction. The multi-scale graph feature fusion unit in the parallel multi-scale feature fusion module of this invention performs graph inference at the original scale, medium scale, and large scale for the input features. This captures small-scale details of tiny lesions and fine layer boundaries, and also uncovers continuous boundaries of large-area lesions and long-distance topological relationships between retinal layers. This ensures effective integration of multi-scale information in a unified dimension, improving the perception of the retinal region in OCT images and enhancing the descriptive ability of features for complex medical image structures. This provides high-quality fusion feature support for the decoding stage, further ensuring the accuracy of the segmentation results. To match the output of the parallel multi-scale image feature fusion module, this invention designs a parallel decoding module, providing a dual-branch structure for the first and second fused features. The two decoding branches generate coordinate attention features and spatial attention features, respectively. The coordinate attention features can accurately capture details strongly correlated with coordinates, such as lesion boundaries and retinal layer contours, along the coordinate axes, improving the localization accuracy of fine structures. The spatial attention features can focus on the spatial dependence between lesions and normal tissues, strengthening the perception of global spatial correlation. The generation of the two attention features covers the feature requirements of OCT images from different dimensions, effectively decoupling the semantic interference between the retinal layer and lesions, improving the ability to distinguish different types of structures, and thus improving the modeling ability of the retinal layer region and lesion region, ensuring the joint segmentation accuracy of the retinal layer and lesions.This invention also designs a cross-graph convolution module to enable information interaction between the two outputs of the parallel decoding module, integrates the advantages of coordinates and spatial attention, and constructs global associations between retinal layers, between layers and lesions, and between lesions. This significantly improves the network's ability to segment layers and lesion regions, lays a solid foundation for the segmentation head to output accurate results, and makes the segmentation more in line with the actual needs for distinguishing the fine structure of retinal OCT images.

[0159] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for retinal layer and lesion joint segmentation based on a U-Net network, characterized in that, The method comprises the following steps: An OCT image to be segmented is acquired and input into an encoder of an improved U-Net network, and an encoding feature of a corresponding layer is acquired through N-layer encoding modules; The N-layer encoding feature is input into a parallel multi-scale graph feature fusion module of the improved U-Net network, mixed features are generated through feature mixing, the mixed features are separated based on feature channels to generate first channel features and second channel features, and the first channel features and the second channel features are input into two parallel multi-scale graph feature fusion units respectively, and first fusion features and second fusion features are output; The first fusion features and the second fusion features are input into a decoder of the improved U-Net network, and coordinate attention features and spatial attention features of a corresponding layer are acquired through N-1-layer parallel decoding modules which are connected with the corresponding layer encoding modules through jumping; The coordinate attention features and the spatial attention features output by the first decoding module are input into a segmentation head of the improved U-Net network, and a retinal layer and lesion joint segmentation result is output through convolution; The multi-scale graph feature fusion unit comprises four parallel branches; the first branch directly outputs channel features; the second branch outputs original scale features through a graph reasoning module; the third branch outputs medium scale features through 2*2 maximum pooling, a graph reasoning module and up-sampling; and the fourth branch outputs large scale features through 3*3 maximum pooling, a graph reasoning module and up-sampling; the outputs of the four parallel branches are spliced and then input into a CBR unit to output fusion features; The graph reasoning module comprises the following steps: input features are input into three parallel CBR units with different parameters respectively to output dimension-reduced features, an inverse projection matrix and a dual projection matrix; the dual projection matrix is transposed to obtain a projection matrix; the dimension-reduced features and the projection matrix are multiplied to obtain first convolution features; the first convolution features and the inverse projection matrix are multiplied to obtain first graph convolution features; the inverse projection matrix and the transpose of the dimension-reduced features are multiplied to obtain second convolution features; the projection matrix and the second convolution features are multiplied and then transposed to obtain second graph convolution features; the first graph convolution features and the second graph convolution features are spliced according to a channel dimension, reorganized in a matrix dimension, converted into a three-dimensional feature map, input into a CBR unit to output global graph convolution features; and the global graph convolution features and the input features of the graph reasoning module are connected in a residual manner to output graph reasoning features.

2. The U-Net network-based retinal layer and lesion joint segmentation method according to claim 1, characterized in that, Each layer of the encoder comprises two serial encoding units, and the output of the second encoding unit is taken as the encoding feature of each layer.

3. The U-Net network-based retinal layer and lesion joint segmentation method according to claim 2, characterized in that, In the encoding unit, the following steps are included: Input features are input into a main encoding branch and an auxiliary encoding branch in parallel; In the main encoding branch, the input features are input into a convolution layer, a batch normalization layer, an activation function layer, a convolution layer and a batch normalization layer which are connected in series along a positive propagation direction, and main encoding features are output; In the auxiliary encoding branch, the input features are input into a convolution layer, a batch normalization layer and a channel attention unit which are connected in series along the positive propagation direction, and auxiliary encoding features are output; The main encoding features and the auxiliary encoding features are added element by element, activated by an activation function layer, and encoding features are output. The channel attention unit sequentially passes the input features through a global average pooling layer, a linear layer, a ReLU activation function layer, a linear layer, and a Sigmoid activation function layer in series along a positive propagation direction, and then performs channel-by-channel multiplication on the features of the input channel attention unit to output auxiliary encoding features.

4. The U-Net network-based retinal layer and lesion joint segmentation method according to claim 1, characterized in that, The graph convolution network outputs convolutional features, including: Obtaining input features for an input graph convolutional network An adjacency matrix of the input graph convolutional network Symmetrically normalizing the adjacency matrix to obtain a normalized adjacency matrix Is represented as ; Transforming input features based on a normalized adjacency matrix, outputting convolutional features , is expressed as: ； wherein denotes a connection information matrix of the adjacency matrix, denoted as , denotes an identity matrix; denotes a degree matrix of denotes a weight of a first layer of the graph convolution network, denotes a weight of a second layer of the graph convolution network, denotes a softmax activation function, denotes a ReLU activation function.

5. The U-Net network-based retinal layer and lesion joint segmentation method according to claim 1, characterized in that, The parallel decoding module obtains coordinate attention features and spatial attention features, including: The parallel decoding module obtains two input features, which are respectively input into two decoding branches; in each decoding branch, the parallel decoding module includes: The input feature passes through the CBR unit to output an initial feature; The initial feature passes through the attention unit to output a reinforced feature; The reinforced feature and the initial feature are added channel by channel, and then processed by the CBR unit to output, and then added channel by channel with the encoding features of the corresponding layer to output attention features; The attention unit in one decoding branch is a coordinate attention unit, and the decoding branch outputs coordinate attention features; the attention unit in the other decoding branch is a spatial attention unit, and the decoding branch outputs spatial attention features; When the parallel decoding module is the N-1 layer, the two input features are the first fusion feature and the second fusion feature; when the parallel decoding module is not the N-1 layer, the two input features are the coordinate attention features and the spatial attention features output by the last layer parallel decoding module.

6. The U-Net network-based retinal layer and lesion joint segmentation method according to claim 5, characterized in that, The initial feature is input into the coordinate attention unit to output a coordinate reinforced feature, including: Global average pooling is performed on each row and column in the initial feature along the coordinate axis direction to obtain a horizontal feature vector and a vertical feature vector; The horizontal feature vector and the vertical feature vector are spliced, then passed through a convolution dimension reduction, and then separated to pass through an activation function respectively to obtain a horizontal attention weight map and a vertical attention weight map; The initial feature is sequentially multiplied with the horizontal attention weight map and the vertical attention weight map element by element to obtain the coordinate reinforced feature.

7. The U-Net network-based retinal layer and lesion joint segmentation method according to claim 5, characterized in that, The initial feature is input into the spatial attention unit to output a spatial reinforced feature, including: Maximum pooling and average pooling are respectively performed on the initial feature in the channel dimension to obtain a first spatial feature summary map and a second spatial feature summary map; The first spatial feature summary map and the second spatial feature summary map are spliced along the channel dimension to obtain a double-channel feature map; A 7×7 convolution layer is used to convolve the double-channel feature map, and then an activation function is used to obtain a spatial attention weight map; The initial feature and the spatial attention weight map are multiplied element by element to obtain the spatial reinforced feature.

8. The U-Net network-based retinal layer and lesion joint segmentation method according to claim 1, characterized in that, After obtaining the coordinate attention features and the spatial attention features of the corresponding layer, the coordinate attention features and the spatial attention features are also input into the cross-graph convolution module in parallel to output optimized coordinate attention features and spatial attention features, including: The coordinate attention features and the spatial attention features are respectively projected by a 1×1 convolution and a batch normalization layer in series to obtain coordinate projection features and spatial projection features; The coordinate projection features and the spatial projection features are spliced, and then sequentially passed through a 3×3 convolution, a ReLU activation function layer, a 1×1 convolution, and a Sigmoid activation function layer in series to obtain a global weight matrix; The global weight matrix is multiplied element by element with the coordinate projection feature and the space projection feature respectively, and then input into the graph convolution network to output a global coordinate projection feature and a global space projection feature; The global coordinate projection feature is added element by element with the coordinate attention feature to obtain an optimized coordinate attention feature; The global space projection feature is added element by element with the space attention feature to obtain an optimized space attention feature.

9. The U-Net network-based retinal layer and lesion joint segmentation method according to claim 1, characterized in that, The output of the first layer encoding module is inputted into the second layer encoding module after being down-sampled. The output of the second layer encoding module is inputted into the third layer encoding module after being down-sampled. The output of the third layer encoding module is inputted into the fourth layer encoding module after being down-sampled. The output of the fourth layer parallel decoding module is inputted into the third layer parallel decoding module after being up-sampled. The output of the third layer parallel decoding module is inputted into the second layer parallel decoding module after being up-sampled. The output of the second layer parallel decoding module is inputted into the first layer parallel decoding module after being up

Images