Joint segmentation method of multiple lesions in retinal OCT images based on hybrid network
By combining Transformer and CNN in a hybrid network architecture, and utilizing multi-scale gated attention modules and group location embedding modules, the problem of lesion segmentation in retinal OCT images was solved, achieving high-precision segmentation of lesion areas and improving the effectiveness of diagnosis and treatment of ophthalmic diseases.
Patent Information
- Application Number
- CN202411135135.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-19
AI Technical Summary
Existing retinal OCT image segmentation methods struggle to distinguish lesion areas with pixel gray levels similar to surrounding healthy tissue. Furthermore, in the late stages of the disease, retinal structural damage leads to the loss of boundaries between various lesions, exacerbating the difficulty of segmenting lesions with similar morphologies.
A hybrid network architecture is adopted, combining a Transformer encoder and a CNN encoder. Through a multi-scale gating attention module and a group location embedding module, global and local features are integrated, and a joint loss function is used to optimize the network segmentation effect and improve the accuracy of lesion boundary recognition.
It significantly improves the automatic identification and segmentation performance of complex lesion areas, enhancing the effectiveness of ophthalmic disease diagnosis and treatment, especially in terms of identification accuracy and applicability.
Smart Images

Figure CN118657800B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image segmentation, and in particular to a method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network. Background Art
[0002] Fundus lesions may cause multiple lesions in the retina at the same time. For example, common lesions of pathological myopia include internal limiting membrane detachment (ILMD), inner retinal schisis (IRS), outer retinal schisis (ORS), retinal detachment (RD), etc. Common lesions of age-related macular degeneration include subretinal fluid, intraretinal fluid, pigment epithelial detachment, etc. There is a certain correlation between the locations of lesions. Joint segmentation of multiple lesions can make full use of the correlation of lesions, allowing the network to focus on the characteristics of different types of lesions and avoid incorrect segmentation between different types of lesions.
[0003] With the advancement of OCT imaging technology, doctors can quickly obtain cross-sectional images of patients' retinas and clearly observe the hierarchical structure and lesion areas of the retina for detailed analysis. However, the segmentation of lesions in retinal OCT images faces some challenges: 1) In the early stages of the disease, some lesion areas are small and their pixel grayscale is similar to that of the surrounding healthy tissue, making them difficult to distinguish; 2) In the late stages of the disease, the retinal structure may be severely damaged, resulting in the loss of the boundaries between multiple lesions; 3) The morphological similarity, close proximity distribution and consistency of grayscale of the lesions further increase the difficulty of segmentation.
[0004] Medical image segmentation is the process of extracting regions of interest from medical images, accurately locating structures such as lesions and organs, and providing key information for medical processes. It is of great significance for medical diagnosis and treatment planning. With the development of convolutional neural networks (CNNs) and transformer architectures, these technologies have been widely used in the segmentation of lesion areas in OCT images. CNNs are known for their excellent feature extraction capabilities and sensitivity to local contextual information, and are suitable for processing details and structures in images. However, their limited ability to capture global context and process long-distance dependencies may affect the processing effect of complex medical images. At the same time, the transformer architecture can effectively process the global context of the entire image and optimize the processing of long-distance dependency information through its self-attention mechanism. Nevertheless, the high computational cost of transformers and the need for a large amount of training data may limit their application in resource-limited environments. The hybrid network architecture that combines the advantages of CNN and transformers can improve feature extraction efficiency and global context processing capabilities, achieve more accurate image segmentation effects, and is particularly suitable for medical image analysis that requires high precision and high efficiency. Summary of the invention
[0005] In view of the above-mentioned problems, the present invention is proposed.
[0006] Therefore, the technical problem solved by the present invention is: the existing method for segmenting lesions in retinal OCT images has the problem that some lesion areas are small in area, and their pixel grayscale is similar to that of the surrounding healthy tissue, making them difficult to distinguish. In the late stage of the disease, the retinal structure may be severely damaged, resulting in the loss of the dividing lines between various lesions. The problem also arises as to how to segment images with similar morphology, closely adjacent distribution and consistent grayscale of lesions with high precision.
[0007] To solve the above technical problems, the present invention provides the following technical solutions: a method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network, comprising acquiring retinal OCT image data; performing multi-scale global and local feature extraction on the OCT image using a hybrid encoder, extracting and filtering global information using a multi-scale gated attention module, supplementing spatial position information through a group position embedding module, integrating global and local features through a three-way fusion module, and optimizing the segmentation effect of the network on the contour area through a joint loss function; performing model training, and using the trained model to achieve segmentation of lesions in the OCT image.
[0008] As a preferred solution of the method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network described in the present invention, the hybrid encoder is composed of a Transformer encoder and a CNN encoder, and the input image enters the two encoders in two ways. In the Transformer encoder part, the lesion image is downsampled twice through the feature guidance layer, and the output is the feature Then, the features are downsampled twice, serialized, and sequentially enter the four-stage multi-scale gated attention module and group position embedding module to obtain four Transformer features of different scales. , , , .
[0009] As a preferred solution of the method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network described in the present invention, the feature guidance layer includes an image resolution reduction operation, using two convolutional layer operations, each convolutional layer includes a 3×3 convolution with a step size of 2, group normalization and a GELU activation function, and the activation function is expressed as:
[0010] ,
[0011] Among them, x is the input feature value, and the output is twice the feature of the downsampled after passing through the first convolution layer After the second convolutional layer, the output is downsampled four times and the features are input into the point-by-point attention module. In the point-by-point attention module, the deep convolution and Sigmoid activation functions are used in turn to calculate the weight of each position on the entire feature map, and multiplied point by point with the input feature map. The features are enhanced according to the different importance of each position on the feature map. Finally, the output is used as the input of the multi-scale gated attention module through layer normalization.
[0012] As a preferred solution of the method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network described in the present invention, the CNN encoder includes a CNN encoder part, the image undergoes two 3×3 convolutions, a combination of batch normalization and ReLU activation functions and a maximum pooling operation, and then sequentially passes through four stages of ResNet18 basic residual modules to obtain four CNN features of different scales. , , ; The decoder part includes a three-way fusion module with four stages.
[0013] As a preferred solution of the method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network according to the present invention, wherein: the multi-scale gated attention module of the four stages includes feature extraction through the multi-scale gated attention module, setting are input features, where , and Represent the height, width and number of channels respectively. Three parallel paths are input. Path 1 uses two downsampling convolution operations with strides of r1 and r2 to obtain features of different scales. The strides are set to 8 and 4 in the first stage, 4 and 2 in the second stage, 2 and 1 in the third stage, and 1 and 1 in the fourth stage. Layer normalization and GELU activation function are used to standardize the features of each sample and capture the complex features in the data. A linear layer is used to reduce the number of channels, and the corresponding affine transformation is used to obtain and vector, , , is the step size, i=1, 2; Path 2 obtains the corresponding two groups through the linear layer of vector, ; Path three uses a linear layer to obtain the gate vector , used to screen for positive features, ;
[0014] pass and Perform a dot product to get the attention score, and then use The activation function enhances the sparsity of attention, and then Perform point multiplication to obtain attention values of different scales, and finally concatenate the attention values of different scales to obtain Sim features, which can be expressed as:
[0015] ,
[0016] in, Indicates a concatenation operation; then use The vector is multiplied point by point with the Sim feature, and the feature is reorganized into Size, and add it to the input X through the residual structure to get the output feature .
[0017] As a preferred solution of the method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network described in the present invention, wherein: the segmentation effect of the contour area of the network optimized by the joint loss function includes the smoothed Dice loss , expressed as:
[0018] ,
[0019] ,
[0020] in, is the number of categories, The model predicts The pixels belong to The output of the class, The manually segmented label represents Is the pixel the The value of the class, Represents the smoothing factor ;
[0021] Tags The pixels belong to Probability of class It is expressed as:
[0022] ,
[0023] Use the bottom-level gradient information of the CNN encoder to calculate the gradient class activation map, use cross entropy to calculate the loss value of each class in the soft label and class activation map, and then use the square root method to balance the loss contribution of different categories. Represents the value of the jth pixel in the class activation map of the i-th class, and the class activation map loss Calculated as:
[0024] ,
[0025] The joint loss function of the entire network It is expressed as:
[0026] .
[0027] Another object of the present invention is to provide a multi-lesion joint segmentation system in retinal OCT images based on a hybrid network, which can group and calculate features through a group position embedding module (GPE) to enhance the acquisition of spatial position information. This solves the problem that the current segmentation of lesions in retinal OCT images contains some lesion areas with small areas, whose pixel grayscale is similar to that of the surrounding healthy tissues and is difficult to distinguish.
[0028] As a preferred solution of the hybrid network-based multi-lesion joint segmentation system in retinal OCT images described in the present invention, it includes: a data acquisition module, a model building module, and an optimization training module. The data acquisition module is used to obtain retinal image OCT image data. The model building module is used to filter information flow, use images of multiple scales for feature extraction, group and calculate attention features, and supplement spatial position information, and optimize the network's segmentation effect on contours and boundary areas through a loss function. The optimization training module is used to perform model training segmentation evaluation indicators.
[0029] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of a method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network.
[0030] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network.
[0031] Beneficial effects of the present invention: In the Transformer part of the hybrid network-based multi-lesion joint segmentation method in retinal OCT images provided by the present invention, in order to make full use of the global receptive field to capture the long-distance connection between lesions and efficiently filter the information flow, a new multi-scale gated attention module (MSGA) is proposed. In addition, the features are grouped and calculated through the group position embedding module (GPE) to enhance the acquisition of spatial position information. In terms of acquiring short-distance information, ResNet18 is used as another encoder to acquire multi-resolution features. And the dual-branch features are integrated through a newly designed three-way fusion module (TPF), which further improves the recognition accuracy of the lesion boundary. Finally, a cross-entropy loss function based on the class activation map is used to significantly improve the network's segmentation ability of the lesion edge area. It can significantly improve the automatic recognition and segmentation performance of complex lesion areas, which has important clinical value for improving the diagnosis and treatment effects of ophthalmic diseases. The present invention achieves better results in both recognition accuracy and applicability. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0033] Figure 1An overall flow chart of a hybrid network-based method for joint segmentation of multiple lesions in retinal OCT images provided in the first embodiment of the present invention.
[0034] Figure 2 The overall network structure of HybridFormer for a hybrid network-based multi-lesion joint segmentation method in retinal OCT images provided in the first embodiment of the present invention.
[0035] Figure 3 A schematic diagram of a feature-guided layer FG, GELU: GELU activation function, and Sigmoid: Sigmoid activation function of a hybrid network-based multi-lesion joint segmentation method in retinal OCT images provided in the first embodiment of the present invention.
[0036] Figure 4 A schematic diagram of a multi-scale gated attention module for a hybrid network-based multi-lesion joint segmentation method in retinal OCT images provided in the first embodiment of the present invention.
[0037] Figure 5 A schematic diagram of a group position embedding module of a hybrid network-based multi-lesion joint segmentation method in retinal OCT images provided in the first embodiment of the present invention.
[0038] Figure 6 A schematic diagram of a three-way fusion module of a hybrid network-based multi-lesion joint segmentation method in retinal OCT images provided in the first embodiment of the present invention.
[0039] Figure 7 A hybrid network-based multi-lesion joint segmentation method in retinal OCT images provided in the second embodiment of the present invention includes (a) original image, (b) manual segmentation label, (c) UNet++, (d) HRNet, and (e) segmentation visualization diagram of Multi_ResUNet.
[0040] Figure 8 Segmentation visualization diagram of (f) ResUNet++ (g) CSNet (h) Attention UNet (i) TransAttentionUNet (j) CASNet for a hybrid network-based multi-lesion joint segmentation method in retinal OCT images provided in the second embodiment of the present invention.
[0041] Fig. 9A segmentation visualization diagram of (k) nnUNet (l) SegFormer_b1 (m) PVT_tiny (n) Swin Transformer (o) Shunted Transformr_small of a multi-lesion joint segmentation method in retinal OCT images based on a hybrid network provided in the second embodiment of the present invention.
[0042] Fig.10 A segmentation visualization diagram of the HybridFormer proposed by (p) ResTransformerv1_large (q) ResTransformerv2_base (r) TransUNet (s) UTNet (t) H2Former (u) for a hybrid network-based multi-lesion joint segmentation method in retinal OCT images provided in the second embodiment of the present invention. DETAILED DESCRIPTION
[0043] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0044] Example 1, reference Figure 1-Figure 6 , as an embodiment of the present invention, provides a method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network, comprising:
[0045] S1: Acquire retinal image OCT image data.
[0046] Furthermore, the acquisition of retinal OCT image data includes collecting 33-line, 64-line and 96-line three-dimensional scans through a micro-OCT scanner, all covering a retinal range of 12mm×8mm, with an image resolution of 2048×1382, forming a data set. The data set is divided into a training set, a validation set, and a test set in a ratio of 6:2:2. Before entering the network, the image and the manual segmentation label are downsampled and processed into a 512×256 size image.
[0047] S2: Use a hybrid encoder to extract multi-scale global and local features from the OCT image, use a multi-scale gated attention module to extract and filter global information, supplement spatial position information through a group position embedding module, integrate global and local features through a three-way fusion module, and optimize the network's segmentation effect on the contour area through a joint loss function.
[0048] Further, refer to Figure 2 The information flow for the overall network structure of HybridFormer is selected. The encoder consists of a Transformer encoder and a CNN encoder. The input image enters the two encoders in two ways. In the Transformer encoder part, the lesion image is downsampled twice through the feature guidance layer, and the output is the feature Then, the features are downsampled twice, serialized, and sequentially enter the four-stage multi-scale gated attention module and group position embedding module to obtain four Transformer features of different scales. , , , .
[0049] The feature-guided layer performs image resolution reduction operations and uses two convolutional layers. Each convolutional layer includes a 3×3 convolution with a step size of 2, group normalization, and a Gaussian error linear unit activation function. The GELU activation function is expressed as:
[0050] ,
[0051] Among them, x is the input feature value, and the output is twice the feature of the downsampled after passing through the first convolution layer After the second convolutional layer, the output is downsampled four times and the features are input into the point-by-point attention module. In the point-by-point attention module, the deep convolution and Sigmoid activation functions are used in turn to calculate the weight of each position on the entire feature map, and multiplied point by point with the input feature map. The features are enhanced according to the different importance of each position on the feature map. Finally, the output is used as the input of the multi-scale gated attention module through layer normalization.
[0052] In the CNN encoder part, the image undergoes two 3×3 convolutions, batch normalization, a combination of ReLU activation functions, and a maximum pooling operation, and then passes through four stages of ResNet18 basic residual modules to obtain CNN features of four different scales. , , .
[0053] The decoder part consists of a three-way fusion module with four stages.
[0054] It should be noted that the four-stage multi-scale gated attention module includes feature extraction through the multi-scale gated attention module. are input features, where , and Represent the height, width and number of channels respectively. Three parallel paths are input. Path 1 uses two downsampling convolution operations with strides of r1 and r2 to obtain features of different scales. The stride is set to:
[0055] The first stage is 8 and 4, the second stage is 4 and 2, the third stage is 2 and 1, and the fourth stage is 1 and 1. Layer normalization and GELU activation function are used to standardize the features of each sample and capture the complex features in the data. A linear layer is used to reduce the number of channels, and the corresponding affine transformation is used to obtain and vector, , , is the step size, i=1, 2.
[0056] Path 2 uses a linear layer to obtain the corresponding two groups of vector, .
[0057] Path three uses a linear layer to obtain the gate vector , used to screen for positive features, .
[0058] pass and Perform a dot product to get the attention score, and then use The activation function enhances the sparsity of attention, and then Perform point multiplication to obtain attention values of different scales, and finally concatenate the attention values of different scales to obtain Sim features, which can be expressed as:
[0059] ,
[0060] in, Represents a concatenation operation.
[0061] Reuse The vector is multiplied point by point with the Sim feature, and the feature is reorganized into Size, and add it to the input X through the residual structure to get the output feature .
[0062] Through the above process, the features acquired by the module not only obtain efficient and robust global modeling capabilities, but also filter the captured information through the gating mechanism.
[0063] It should also be noted that although the attention mechanism can effectively capture the dynamic correlation in the data, when processing image data, serializing the image into a vector sequence will result in the loss of some spatial position information. Therefore, in order to compensate for the position information when calculating attention, a group position embedding module (GPE) is designed. The structure of the group position embedding module is as follows Figure 5 As shown in the figure. The module first uses group normalization to process the features, that is, it performs independent normalization operations on the features within each group, reducing the internal offset of the data and improving the generalization of new data. Then, the position attention module is used. This module first obtains the attention weight of each position of the image through group convolution and depth convolution, and then obtains the attention weight of each position of the image through the activation function. The attention weight is multiplied point by point with the feature map after group convolution, so as to dynamically adjust the model's attention to different positions of the image. Finally, effective feature fusion is performed through layer normalization and linear layer operations in sequence. The group position embedding module helps the model understand the spatial structure in the data by adding additional position information to each input element.
[0064] Furthermore, since the encoder part uses a parallel structure of Transformer and CNN, each extracts features in different ways. Using simple splicing or convolution operations to fuse features may cause interference between features or make the network biased towards a certain type of feature, thereby ignoring other information. Therefore, a three-way fusion module (TPF) based on pixel rearrangement is designed. The structure of the three-way fusion module is as follows: Figure 6 As shown in the figure, the module receives three inputs, namely, high-level features from the previous decoder, low-level features from the Transformer encoder, and low-level features from the CNN encoder. The module first uses deep convolution to perform channel conversion operations on high-level features, then splices them with two low-level features through a splicing operation, and then performs a pixel rearrangement operation, which arranges the feature values of every 4 channels at the same position into 2×2 pixels in space, thereby achieving upsampling by 2 times in both spatial directions, while reducing the channel dimension to one-fourth. The features are then weighted and combined in the direction of the number of channels through deep convolution, and finally the output features are used as high-level input features for the three-way fusion module in the next stage.
[0065] It should be noted that the Dice loss function can alleviate the problem of severe class imbalance and small proportion of objects of interest, ensuring that the model does not favor larger background classes. However, since the Dice loss function depends on the ratio of predicted and true labels, it is piecewise continuous and nonlinear in the parameter space, which means that the loss function has multiple local minima and is non-convex, making it difficult to find the global optimal point.
[0066] The high-level features and low-level features are fused and up-sampled through the three-way fusion module. After transposed convolution, the decoded features are obtained The input of the three-way fusion module of the Nth layer is the decoding features obtained in the previous stage And the Transformer features of the corresponding stage and CNN features , the output is the decoded feature , N=4,3,2,1, the final decoding features It is the segmentation result map with K+1 channels, and each channel represents the probability that the pixel belongs to K types of lesions and background areas.
[0067] The loss function is used to optimize the network's segmentation effect on contours and boundary areas, and the smoothed Dice loss , expressed as:
[0068] ,
[0069] ,
[0070] in, is the number of categories =5, The model predicts The pixels belong to The output of the class, The manually segmented label represents Is the pixel the The value of the class, Represents the smoothing factor .
[0071] Since the lesion areas usually present unique morphological and structural characteristics in the retinal tissue, the model needs to be able to finely identify the main areas and detailed contour information of these areas. Therefore, a class activation map loss function is proposed, which uses the bottom-level gradient information of the CNN branch in the encoder as the gradient input to calculate the gradient class activation map. And because in the class activation map, in the lesion area that the network pays attention to, the attention gradually decreases from the center to the edge, directly calculating the loss with the manual segmentation label will result in too large a loss value, making it difficult for the network to converge during training. Therefore, soft labels are used instead of the original manual segmentation labels. The first label in the label is The pixels belong to Probability of class It is expressed as:
[0072] ,
[0073] Use cross entropy to calculate the loss value of each class in the soft label and class activation map, and then use the square root method to balance the loss contribution of different categories. Represents the value of the jth pixel in the class activation map of the i-th class, and the class activation map loss Calculated as:
[0074] ,
[0075] The joint loss function of the entire network It is expressed as:
[0076] .
[0077] S3: Perform model training and use the trained model to segment lesions in OCT images.
[0078] Furthermore, the model training and segmentation evaluation indicators include setting the batch size to 4 during training, validation, and testing, downsampling the OCT images and manually segmented label images to 256×512 pixels before input, and applying the cosine annealing learning rate algorithm to optimize the network. The learning rate lr is expressed as:
[0079] ,
[0080] The basic learning rate Set to 0.15, represents the number of iterations in the training process, Indicates the total number of iterations of the entire training process. During the network training process, after each round of iteration, the model is used to verify the validation set data, and the model parameters with the highest average DSC in the validation set are retained as the network model parameters for the final test.
[0081] Example 2, reference Figure 7-10 , which is an embodiment of the present invention, provides a method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.
[0082] First, the experimental data is clinical images obtained in a hospital, collected by a micro-OCT scanner. It includes 33-line, 64-line and 96-line three-dimensional scans, all covering a retinal range of 12mm×8mm. The image resolution is 2048×1382. The experimental dataset includes 3D scans of 43 patients, including 2162 two-dimensional OCT images of internal limiting membrane separation (ILMD), inner layer schisis (IRS), outer layer schisis (ORS), and retinal detachment (RD) lesions. The manual segmentation labels of each image are manually annotated under the guidance of professional ophthalmologists. According to the ratio of 6:2:2, the dataset is divided into training set, validation set, and test set, with the data volume of 1301, 433, and 428 respectively. Before entering the network, the images and manual segmentation labels are downsampled and processed into 512×256 size images.
[0083] In order to evaluate the segmentation results of ILMD, IRS, ORS and RD in OCT images, three evaluation indicators are used: Dice Similarity Coefficient (DSC), Intersection of Union (IoU) and Sensitivity (Sen). They are defined as follows:
[0084] ,
[0085] Where TP, TN, FP and FN are the true positive, true negative, false positive and false negative of the segmentation respectively. The indicators for each category are calculated separately and the overall average is calculated.
[0086] Compare the ablation results and set up scale ablation experiments on the multi-scale gated attention module.
[0087] Table 1 Scale ablation experiment data table of multi-scale gated attention module (%)
[0088] ,
[0089] The ablation experiment results of the multi-scale gated attention module are shown in Table 1. As shown in Table 1, the multi-scale version of the gated attention has achieved better results in single-class DSC, average DSC and average intersection-over-union ratio of ILMD, IRS and ORS, which are 0.31%, 1.60%, 2.1%, 0.80% and 0.91% higher than the single-scale version, respectively. This shows that the multi-scale operation is more superior in the overall segmentation effect. The single-scale version is better in single-class DSC and average sensitivity of RD, which are 0.81% and 0.90% higher than the multi-scale version, respectively. This shows that at a single scale, the network will focus more on RD, which is also because the lesion area of RD is larger, thereby reducing the network's ability to recognize lesions in other small areas.
[0090] Setting up an ablation experiment on the number of channels per encoder layer.
[0091] Table 2 Ablation experiment data table on the number of channels per layer of the encoder (%)
[0092] ,
[0093] The ablation experiment results on the number of channels per layer of the encoder are shown in Table 2. In the hybrid network designed by the present invention, the number of channels per layer of the network is set to [32, 64, 128, 256], which is also considered to balance the network depth and the size of the data set. It can be seen that the network with the channel dimension number we used achieved the best results on the single-class DSC, average DSC, and average IoU of ILMD and IRS, and also achieved the second best results on the single-class DSC of ORS and RD, and the gap is not large. In contrast, the model with the channel dimension number of [16, 32, 64, 128] achieved better results on the single-class DSC of ORS and RD, but the effect was much lower on the single-class DSC of ILMD and IRS, which means that the parameters of the network are more focused on the larger ORS and RD lesion areas. When the number of channel dimensions increases, the effect decreases instead, which shows that the increase in the number of parameters will generate redundant information, affecting the feature extraction of the network.
[0094] Set up an ablation experiment on the loss function.
[0095] Table 3 Ablation experiment data table on loss function (%)
[0096] ,
[0097] The ablation experiment results of the loss function are shown in Table 3. Compared with the ordinary Dice loss function, the network using the smooth Dice loss function improves the overall performance, with an increase of 0.53% and 0.15% in average DSC and average IoU respectively. Although the single-class DSC and average sensitivity of IRS have decreased to 0.13% and 0.07% respectively, other indicators have been improved. After using the mixed loss function, it can be seen that compared with the network using the smooth Dice loss function, the network's attention to each class is more balanced, and the network's attention to IRS and ORS lesions with higher segmentation difficulty is improved. The average DSC and average IoU are also increased by 0.49% and 0.83% respectively. This shows that the designed class activation map cross entropy loss function can improve the network's ability to extract inter-class information and better identify the boundary and contour information of each class area.
[0098] Comparing segmentation results
[0099] Table 4 Segmentation performance data of different segmentation networks (%)
[0100] ,
[0101] The segmentation performance comparison between the hybrid network of the present invention and different segmentation networks is shown in Table 4. The hybrid network proposed in the present invention achieves the best indicators in single-class DSC, average DSC and average intersection-over-union ratio of ILMD and IRS, and the average DSC, average IoU and average Sen indicators are 77.30%, 72.17% and 79.43% respectively, which shows that the network can accurately capture smaller targets. The second best result was also achieved on the single-class DSC of ORS, reaching 80.06%, and the good result was also achieved on the single-class DSC of RD, reaching 85.13%, showing a good and balanced segmentation performance.
[0102] like Figure 7-10 The grayscale values shown in the figure represent the segmentation diagram of internal limiting membrane separation, inner layer splitting, outer layer splitting and retinal detachment from low to high. The network proposed in the present invention can overcome the difficulties in segmenting multiple types of lesions, such as large horizontal span, large differences in the morphology of different lesions, large or small areas, damage or even loss of tissue structure leading to disappearance of boundaries between lesions, etc. The hybrid network proposed in the present invention can accurately segment multiple types of lesions in retinal OCT images, surpassing many other advanced segmentation networks.
[0103] Example 3, an embodiment of the present invention, provides a system for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network, including a data acquisition module, a model building module, and an optimization training module.
[0104] The data acquisition module is used to obtain retinal OCT image data. The model building module is used to filter information flow, use images of multiple scales for feature extraction, calculate attention features in groups, supplement spatial position information, and optimize the network's segmentation effect on contours and boundary areas through loss functions. The optimization training module is used to perform model training segmentation evaluation indicators.
[0105] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0106] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0107] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk case (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0108] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logical function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.
[0109] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A hybrid network-based joint segmentation method for multiple lesions in retinal OCT images. It is characterized in that include: Acquire retinal OCT image data; A hybrid encoder is used to extract multi-scale global and local features from the OCT image, a multi-scale gated attention module is used to extract and filter global information, a group position embedding module is used to supplement spatial position information, a three-way fusion module is used to integrate global and local features, and a joint loss function is used to optimize the segmentation effect of the network on the contour area; Perform model training and use the trained model to segment lesions in OCT images; The hybrid encoder consists of a Transformer encoder and a CNN encoder. The input image enters the two encoders in two ways. In the Transformer encoder part, the lesion image is downsampled twice through the feature guidance layer, and the output is the feature Then the features are downsampled twice, serialized, and sequentially enter the four-stage multi-scale gated attention module and group position embedding module to obtain four Transformer features of different scales. The four-stage multi-scale gated attention module includes feature extraction through the multi-scale gated attention module, let X∈R H×W×C is the input feature, where H, W and C represent the height, width and number of channels respectively. X is input into three parallel paths. Path 1 uses two steps of r 1 and r 2 Downsampling convolution operations are performed to obtain features of different scales. The step size is set to 8 and 4 in the first stage, 4 and 2 in the second stage, 2 and 1 in the third stage, and 1 and 1 in the fourth stage. Layer normalization and GELU activation functions are used to standardize the features of each sample and capture the complex features in the data. Linear layers are used to reduce the number of channels, and the corresponding Q and K vectors are obtained through affine transformation. N=H×W,r i is the step length, i=1,2; Path 2 uses a linear layer to obtain the V vector corresponding to the two sets of Q and K, V i ∈R N×C / 2 ,i=1,2; Path three uses the linear layer to obtain the gate vector Gate, which is used to filter the features of positive effects, Gate∈R N×C ; By V i and Q i Perform dot product to get the attention score, and then use relu 2 The activation function enhances the sparsity of attention, and then i Perform point multiplication to obtain attention values of different scales, and finally concatenate the attention values of different scales to obtain Sim features, which can be expressed as: Among them, Concat represents the concatenation operation; Then use the Gate vector to perform point-by-point product with the Sim feature, reorganize the feature into H×W×C size, and add it to the input X through the residual structure to obtain the output feature O∈R H×W×C ; The decoder includes the fusion of high-level features and low-level features and upsampling operations through a three-way fusion module. After transposed convolution, the decoded features are obtained The input of the three-way fusion module of the Nth layer is the decoding features obtained in the previous stage And the Transformer features of the corresponding stage and CNN features The output is the decoded features N = 4, 3, 2, 1, the final decoded features is the segmentation result map with K+1 channels, where each channel represents the probability that the pixel belongs to K types of lesions and background areas; The three-way fusion module of layer N first uses a depthwise convolution to Perform channel conversion and then concatenate them with and The splicing is then performed, and then the pixel rearrangement operation is performed to arrange the feature values of every 4 channels at the same position into 2×2 pixels in space, achieving a 2-fold upsampling in both spatial directions, while the channel dimension is reduced to a quarter. The output features are then obtained through deep convolution as the input of the three-way fusion module in the N-1 layer. The segmentation effect of the contour area by optimizing the network through the joint loss function includes the smoothed Dice loss It is expressed as: Among them, K is the number of categories, p ij The output result of the model predicting that the jth pixel belongs to the i-th category, y ij is the value in the manual segmentation label indicating whether the j-th pixel belongs to the i-th category, ε represents the smoothing factor ε = 1e-6; The probability S that the jth pixel in the label belongs to the i-th class ij It is expressed as: Use the bottom-level gradient information of the CNN encoder to calculate the gradient class activation map, use cross entropy to calculate the loss value of each class in the soft label and class activation map, and then use the square root method to balance the loss contribution of different categories. Use CAM ij Represents the value of the jth pixel in the class activation map of the i-th class, and the class activation map loss loss CAM Calculated as: The joint loss function of the entire network is loss total It is expressed as:
2. The method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network as claimed in claim 1, Features: The feature guidance layer includes an image resolution reduction operation, using two convolutional layer operations, each convolutional layer includes a 3×3 convolution with a step size of 2, group normalization and a GELU activation function, and the activation function is expressed as: Among them, x is the input feature value, and the output is twice the feature of the downsampled after passing through the first convolution layer The second convolutional layer outputs four times the features that are downsampled and input into the point-by-point attention module. In the point-by-point attention module, deep convolution and Sigmoid activation functions are used in turn to calculate the weight of each position on the entire feature map, and multiplied point by point with the input feature map. Feature enhancement is performed according to the different importance of each position on the feature map. Finally, the output is used as the input of the multi-scale gated attention module through layer normalization.
3. The method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network as claimed in claim 2, Features: The CNN encoder includes a CNN encoder part, where the image undergoes two 3×3 convolutions, batch normalization, a combination of ReLU activation functions, and a maximum pooling operation, and then sequentially passes through four stages of ResNet18 basic residual modules to obtain CNN features of four different scales.
4. A system using the hybrid network-based multi-lesion joint segmentation method in retinal OCT images as claimed in any one of claims 1 to 3, Features: Including data collection module, model building module, and optimization training module; The data acquisition module is used to acquire retinal image OCT image data; The model building module is used to filter information flow, use images of multiple scales for feature extraction, calculate attention features in groups, supplement spatial position information, and optimize the network's segmentation effect on contours and boundary areas through a loss function; The optimization training module is used to perform model training segmentation evaluation indicators.
5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, It is characterized in that When the processor executes the computer program, the steps of the method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network according to any one of claims 1 to 3 are implemented.
6. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method for joint segmentation of multiple lesions in retinal OCT images based on a hybrid network according to any one of claims 1 to 3 are implemented.
