A method, system, device and medium for automatic segmentation of vitreoretinal interface abnormalities based on OCT images

Abstract

The present invention discloses a method, system, device, and medium for automatic segmentation of vitreoretinal interface abnormalities based on OCT images, relating to the field of image recognition technology. The method comprises inputting the OCT image to be segmented into a high-resolution segmentation network, extracting features from the OCT image using a four-stage feature encoding module to obtain multi-scale semantic features; fusing the multi-scale semantic features based on feature splicing and dynamic attention selection; inputting a portion of the encoder output into a length regression auxiliary network, and directly inputting the remaining portion into a decoder module to obtain a segmentation result image; processing the segmentation result image and the corresponding gold standard image through an edge feature smoothing module to calculate a smoothing loss function; and introducing a Hausdorff distance loss function to calculate the distance between the segmentation result and the gold standard. The present invention incorporates novel feature splicing and dynamic attention selection into the encoder portion of the segmentation network, achieving fully automatic segmentation of PVD and ERM in OCT images.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, system, device and medium for automatically segmenting vitreoretinal interface abnormalities based on OCT images. Background Art

[0002] Abnormal vitreoretinal interface (VRI) is a common symptom in patients with high myopia, mainly including posterior vitreous detachment (PVD) and epiretinal membranes (ERM). The formation of PVD and ERM may cause serious damage to the visual system. The occurrence of PVD causes abnormal traction on the retina, thereby increasing the risk of retinal detachment. ERM can exert traction on the peripheral retina and blood vessels, and together with the elongated axial length, lead to the formation of paravascular and intraretinal defects in high myopia. Early diagnosis and treatment of PVD and ERM can effectively prevent the retinal lesions caused by them and reduce vision loss.

[0003] Optical Coherence Tomography (OCT) is a non-invasive imaging technique that allows for the visualization of cross-sectional images of the retina at high resolution. By automatically segmenting abnormalities in the VRI, the morphology and extent of the lesions can be accurately analyzed, helping doctors to more clearly see the outline and boundary of the lesions, and helping to determine the severity of the lesions and the progression of the disease. However, the segmentation of PVD and ERM in OCT images faces some challenges: (1) The distribution of categories is uneven, because not all categories appear in a specific scan. (2) The lesions are small and have elongated curved features, and the background is complex and occupies a large proportion of the image. (3) In some special cases, when the PVD is close to the internal limiting membrane, the PVD and ERM will have similar features to each other, which can easily lead to false detection and segmentation. So far, no one has achieved fully automatic segmentation of posterior vitreous detachment and epiretinal membrane in OCT images.

[0004] With the rise of convolutional neural networks (CNNs), they have been increasingly used for lesion segmentation tasks in OCT images. Their excellent feature extraction capabilities lead to superior segmentation performance. However, many CNN models first extract features through downsampling and then upsample to restore the feature map to the input image resolution. The size of PVD and ERM will be further reduced after multiple downsampling through the network. In particular, some PVDs have low contrast with the surrounding background, making them difficult to identify after downsampling. In addition, many methods do not fully utilize the curved features presented by PVD and ERM to design networks, making their edge segmentation of curved targets inaccurate. Most current research focuses on the segmentation of a single category of ERM. The relevant models lack the ability to distinguish different categories in the image and are not suitable for multi-category segmentation tasks of PVD and ERM. Summary of the Invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the problem to be solved by the present invention is that most of the current research focuses on the segmentation of a single category of ERM, and the related models lack the ability to distinguish different categories in an image and are not suitable for multi-category segmentation tasks of PVD and ERM.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: a method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images, comprising: inputting the OCT image to be segmented into a high-resolution segmentation network, the encoder part of the high-resolution segmentation network using a four-stage feature encoding module to extract features from the OCT image to be detected to obtain multi-scale semantic features; the multi-scale semantic features are feature fused based on feature splicing and dynamic attention selection, and the details of slender targets and contextual information are captured; the output of the encoder is transmitted to a length regression auxiliary network and a decoder module respectively; after receiving the transmission information, the length regression auxiliary network outputs a one-dimensional vector representing the target length for calculating the length loss; after receiving the transmission information, the decoder module outputs a segmentation result image; the segmentation result image and the corresponding gold standard obtained by manual segmentation are processed by an edge feature smoothing module to calculate the smoothing loss function; the Hausdorff distance loss function is introduced to calculate the distance between the segmentation result and the gold standard obtained by manual segmentation for network training.

[0008] As a preferred solution of the method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images described in the present invention, the encoder includes a bottleneck module, a residual module, an upsampling module and a downsampling module; in the bottleneck module, the input features pass through a 1×1 convolution layer to reduce the number of channels, and then pass through a 3×3 convolution layer for feature extraction, and finally pass through a 1×1 convolution layer to increase the number of channels, and are added to the input through a residual connection, and are output after passing through a ReLU activation function; in the residual module, the input features pass through two 3×3 convolution layers, are added to the input through a residual connection, and are output after passing through a ReLU activation function; the upsampling module includes a 1×1 convolution layer with a stride of 1 and a batch normalization layer to perform upsampling operations on the feature data; the downsampling module contains three 3×3 convolution layers with a stride of 2 and a batch normalization layer to downsample the feature data by 2 times, 4 times, and 8 times.

[0009] As a preferred embodiment of the method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images of the present invention, before extracting features from the OCT image to be detected using the four-stage feature encoding module, the input image is subjected to two 3×3 convolutional layers with a stride of 2 to reduce the scale to 1 / 4; the four-stage feature encoding module includes: ij (j≤i) is the output feature of different branches in the four stages, where i=1,2,3,4 represents the stage number from left to right, and j=1,2,3,4 represents the branch number from top to bottom.

[0010] As a preferred embodiment of the method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images of the present invention, the four-stage feature encoding module further comprises: in stage 1, the output feature F of stage 1 is obtained through four bottleneck modules. 11 ; In stage 2, the output feature F of stage 1 is 11 Through a 3×3 convolution module with a step size of 1, and then through 4 residual modules, the feature F' is obtained 21 , in another channel, F 11 Through a 3×3 convolution module with a stride of 2 and 4 residual modules, the feature F' is obtained 22 , the feature F' 21 and F' 22 Fusion is performed to obtain the two-way output feature F of stage 2 21 and F 22 , the fusion calculation is expressed as: F 21 =F' 21 +up(F' 22 ,2),F 22 =down(F' 21 ,2)+F' 22, where up(F,n) means upsampling feature F by n times, and down(F,n) means downsampling feature F by n times;

[0011] In stage 3, the output features F of stage 2 are 22 Through a 3×3 convolution module with a stride of 2, and then through 4 residual modules, the feature F' is obtained 33 ; In the other two channels, the output features F of stage 2 are 21 、F 22 Through 4 residual modules respectively, the feature F' is obtained 31 and F' 32 , the feature F' 31 、F' 32 、F' 33 Fusion is performed to obtain the three-way output feature F of stage 3 31 、F 32 and F 33 , the fusion calculation is expressed as,

[0012] F 31 =F′ 31 +up(F′ 32 ,2)·+up(F′ 33 ,4)

[0013] F 32 =down(F′ 31 ,2)+F′ 32 +up(F′ 33 ,2)

[0014] F 33 =down(F′ 31 ,4)+down(F′ 32 ,2)+F′ 33

[0015] In stage 4, the output features F of stage 3 are 33 Through a 3×3 convolution module with a stride of 2, and then through 4 residual modules, the feature F' is obtained 44 , the output features F of stage 3 are converted into 31 、F 32 、F 33 Through 4 residual modules respectively, the feature F' is obtained 41 、F' 42 、F' 43 , the feature F' 41 、F' 42 、F' 43 、F' 44 Fusion is performed to obtain the four-way output feature F of stage 4 41 、F 42、F 43 and F 44 , the fusion calculation is expressed as,

[0016] F 41 =F′ 41 +up(F′ 42 ,2)+up(F′ 43 ,4)+up(F′ 44 ,8)

[0017] F 42 =down(F′ 41 ,2)+F′ 42 +up(F′ 43 ,2)+up(F′ 44 ,4)

[0018] F 43 =down(F′ 41 ,4)+down(F′ 42 ,2)+F′ 43 +up(F′ 44 ,2)

[0019] F 44 =down(F′ 41 ,8)+down(F′ 42 ,4)+down(F′ 43 ,2)+F′ 44 .

[0020] As a preferred embodiment of the automatic segmentation method for vitreoretinal interface abnormalities based on OCT images described in the present invention, the feature stitching includes stitching the features of the same scale obtained in the four stages to compensate for the spatial position and detail information that may be lost in each stage. The output after feature stitching is expressed as:

[0021] F concat1 =Concat(F 11 ,F 21 ,F 31 ,F 41 )

[0022] F concat2 =Concat(F 22 ,F 32 ,F 42 )

[0023] F concat3 =Concat(F 33 ,F 43 )

[0024] Among them, F concat1It represents the output feature map after fusing the resolution of the same scale under 1 branch number at different stages, F concat2 It represents the output feature map after fusing the resolution of the same scale under the 2 branch numbers at different stages, F concat3 Indicates the output feature map after fusing the resolution of the same scale under the three branch numbers at different stages; Concat means splicing the feature map in the channel dimension; F concat1 、F concat2 and F concat3 Input the dynamic attention selection module and convolution layer respectively to obtain F1, F2, and F3. 44 Input the dynamic attention selection module to obtain F4, F1, F2, F3, and F4 as the four outputs of the encoder; the dynamic attention selection module includes three steps: separation, feature fusion, and feature selection; in the separation part, let C, H, and W be the number of channels, height, and width of the input feature X respectively, and R be a real number set, and the input feature X∈R C×H×W After 3×3 and 5×5 convolutions, feature maps U1 and U2 are obtained respectively, and then U1 and U2 are added to obtain feature map U; in the feature fusion part, U is globally average pooled and compressed into a vector S, S∈R C×1 , the dimension of S is reduced through the fully connected layer to obtain the feature vector Z∈R d×1 , d is the dimension after dimensionality reduction; in the feature selection part, the feature vector Z is transformed by two matrices A∈R C×d , B∈R C×d , use the Softmax function to calculate the attention between channels to obtain the weight vector a∈R C×1 , b∈R C×1 ;a c is the cth element in a, b c is the cth element in b, put a c , b c The feature map U corresponding to the cth channel in U1 and U2 respectively 1c and U 2c After multiplication, add them together to get the feature map V on the cth channel c , stack the c feature maps of all channels to obtain the final feature map V that integrates multi-scale information.

[0025] As a preferred solution of the automatic segmentation method of vitreoretinal interface abnormality based on OCT images of the present invention, the features of four different scales of the encoder are input into the length regression auxiliary network, and the highest scale feature F1∈R H×W×C After downsampling with a 3×3 convolution kernel with a step size of 2, and extracting features through a 3×3 convolution with a step size of 1, the scale is halved and the number of channels is doubled. The features obtained are Combine F'1 with the high-scale features output by the encoder Perform the addition operation, then perform downsampling and feature extraction through two layers of convolution, and the obtained features are Combine F'2 with the high-scale features output by the encoder Perform the addition operation, then perform downsampling and feature extraction through two layers of convolution, and the obtained features are Combine F'3 with the high-scale features output by the encoder After the addition operation, downsampling and feature extraction are performed through two layers of convolution, and the obtained features are The obtained feature F'4 undergoes a global average pooling operation, a fully connected layer, and a Sigmoid activation function to obtain the final regression output, which is a one-dimensional vector with M channels, representing the length prediction of M types of lesion areas. The result obtained by manual segmentation is used as the gold standard, and the number of pixels in each type of lesion area is calculated as the true value of the length regression task. The mean square error between the regressed predicted length and the true value is calculated as the length loss function to constrain the main segmentation network.

[0026] As a preferred solution of the method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images described in the present invention, wherein: after the decoder module receives the transmission information, it outputs a segmentation result map, including: the decoder module upsampling the input feature F2 by 2 times, upsampling the feature F3 by 4 times, and upsampling the feature F4 by 8 times, and then splicing them with F1, and passing through a 1×1 convolution layer to obtain a segmentation result map; the edge feature smoothing module processes the output image of the network and the gold standard obtained by manual segmentation through a Gaussian blur filter with a size of 15×15 and a standard deviation of 8, and half of the mean square error between the blurred results is used as the smoothing loss function.

[0027] To solve the above technical problems, the present invention provides the following technical solution: a system for automatically segmenting vitreoretinal interface abnormalities based on OCT images, comprising: a feature extraction unit, a feature fusion unit, an output unit, and a model evaluation and optimization unit; the feature extraction unit inputs the OCT image to be segmented into a high-resolution segmentation network. The encoder portion of the high-resolution segmentation network uses a four-stage feature encoding module to extract features from the OCT image to be detected, thereby obtaining multi-scale semantic features; the feature fusion unit fuses the multi-scale semantic features based on feature splicing and dynamic attention selection, and captures slender object details and contextual information; the output unit transmits the encoder output to a length regression auxiliary network and a decoder module respectively; after receiving the transmitted information, the length regression auxiliary network outputs a one-dimensional vector representing the object length for calculating the length loss; after receiving the transmitted information, the decoder module outputs a segmentation result image; the model evaluation and optimization unit processes the segmentation result image and the corresponding gold standard obtained by manual segmentation through an edge feature smoothing module, calculates a smoothing loss function, and introduces a Hausdorff distance loss function to calculate the distance between the segmentation result and the gold standard obtained by manual segmentation for network training.

[0028] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the above-mentioned method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images.

[0029] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images.

[0030] The beneficial effects of the present invention are as follows: the present invention proposes a high-resolution network ALR-HRNet including an ALR module, which realizes the fully automatic segmentation of PVD and ERM in OCT images. Novel feature splicing and dynamic attention selection are added to the encoder part of the segmentation network. After fusing feature maps with the same resolution, the dynamic attention selection module is used to process feature maps with different resolutions, fully integrating deep semantic features and shallow detail information, and improving the generalization ability of the model. An edge feature smoothing module is used to smooth curved and slender targets, thereby improving the recognition ability of the edges of slender targets. A length regression auxiliary module is used to focus on the length information of the lesion features, providing additional geometric constraints to the main network, so that the main segmentation network pays more attention to the slender shape of the target. The Hausdorff distance loss function is used to make the network pay more attention to edge information. The present invention is verified and evaluated in a retinal OCT image dataset. The experimental results show that the method of the present invention can effectively realize the segmentation of PVD and ERM in OCT images, and its performance is optimal compared with other advanced networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 This is a structural diagram of the high-resolution network ALR-HRNet including a length regression auxiliary module for an automatic segmentation method of vitreoretinal interface abnormalities based on OCT images in Example 1.

[0033] Figure 2 This is a structural diagram of the bottleneck module and residual module of an automatic segmentation method for vitreoretinal interface abnormalities based on OCT images in Example 1.

[0034] Figure 3 This is a structural diagram of upsampling and downsampling of an automatic segmentation method for vitreoretinal interface abnormalities based on OCT images in Example 1.

[0035] Figure 4 This is a structural diagram of the dynamic attention selection (SK) module of an automatic segmentation method for vitreoretinal interface abnormalities based on OCT images in Example 1.

[0036] Figure 5 This is a structural diagram of the length regression auxiliary module of an automatic segmentation method for vitreoretinal interface abnormalities based on OCT images in Example 1.

[0037] Figure 6 This is a visualization of the experimental results comparing ALR-HRNet with other segmentation networks for an automatic segmentation method of vitreoretinal interface abnormalities based on OCT images in Example 2.

[0038] in, Figure 6 Middle (a), retinal OCT image; Figure 6 middle (b), gold standard; Figure 6 (c) The high-resolution network ALR-HRNet proposed in this invention, which includes a length regression auxiliary module; Figure 6 Middle (d), global transformer based on deep and shallow level feature fusion and dual local attention network GT-DLA-dsFF; Figure 6 Middle (e), U-shaped codec network U-Net; Figure 6 Middle (f), residual U-shaped network ResUNet; Figure 6 Middle (g), improved residual U-shaped network ResUNet++; Figure 6 Middle (h), Transformer Encoding U-shaped Network TransUNet; Figure 6 Middle (i), linear structure segmentation network CS2-Net; Figure 6 Middle (j), Contextual Pyramid Fusion Network CPFNet; Figure 6 Middle (k), Multi-scale feature fusion network MFI-Net. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0041] Example 1, with reference to Figure 1-Figure 5 , which is the first embodiment of the present invention, provides a method for automatically segmenting vitreoretinal interface abnormalities based on OCT images, including: Figure 1Figure 2 shows the overall architecture of a high-resolution segmentation network (ALR-HRNet) with an auxiliary length regression module for automatic segmentation of PVD and ERM images based on OCT images. The ALR-HRNet is based on an encoder-decoder architecture. The OCT image to be segmented is input into the ALR-HRNet network. In the encoder, a four-stage feature encoding module extracts features from the OCT image to generate multi-scale semantic features. These features are then subjected to feature concatenation and dynamic attention selection (FCDAS) to achieve feature fusion and enhance the ability to capture details and contextual information of slender objects. Part of the encoder output is directly input into the decoder module to obtain the segmentation result; part is input into the auxiliary length regression network (ALR), which outputs a one-dimensional vector representing the object length for calculating the length loss. The output of the segmentation network and the corresponding ground truth are processed by the edge feature smoothing module (EFS) to calculate the smoothing loss function. Furthermore, the Hausdorff distance (HD) loss, Dice loss, and cross entropy (CE) loss are directly calculated between the segmentation result and the ground truth. The sum of these five loss functions is used as the total loss function for network training.

[0042] 1. The encoder consists of a bottleneck module, a residual module, an upsampling module, and a downsampling module.

[0043] like Figure 2 As shown in the figure, in the bottleneck module, the input features pass through a 1×1 convolution layer to reduce the number of channels, then pass through a 3×3 convolution layer for feature extraction, and finally pass through a 1×1 convolution layer to increase the number of channels. They are added to the input through a residual connection and output after a ReLU activation function.

[0044] In the residual module, the input features pass through two 3×3 convolutional layers, are added to the input through the residual connection, and are output after the ReLU activation function.

[0045] like Figure 3 As shown in Figure 2, the upsampling module includes a 1×1 convolutional layer with a stride of 1 and a batch normalization layer to perform upsampling operations on the feature data.

[0046] The downsampling module contains three 3×3 convolutional layers with a stride of 2 and a batch normalization layer, which downsamples the feature data by 2 times, 4 times, and 8 times.

[0047] In the encoder, the input image first passes through two 3×3 convolutional layers with a stride of 2, and the scale becomes 1 / 4, and then passes through four stages of feature encoding modules in sequence. Assuming F ij (j≤i) is the output feature of different branches in the four stages, where i=1,2,3,4 represents the stage number from left to right, and j=1,2,3,4 represents the branch number from top to bottom.

[0048] In stage 1, the output feature F of stage 1 is obtained after 4 bottleneck modules 11 .

[0049] In stage 2, the output features F of stage 1 are 11 Through a 3×3 convolution module with a step size of 1, and then through 4 residual modules, the feature F' is obtained 21 , in another channel, F 11 Through a 3×3 convolution module with a stride of 2 and 4 residual modules, the feature F' is obtained 22 , the feature F' 21 and F' 22 Fusion is performed to obtain the two-way output feature F of stage 2 21 and F 22 , the fusion calculation is expressed as: F 21 =F' 21 +up(F' 22 ,2),F 22 =down(F' 21 ,2)+F' 22 , where up(F,n) means upsampling feature F by a factor of n, and down(F,n) means downsampling feature F by a factor of n.

[0050] In stage 3, the output features F of stage 2 are 22 Through a 3×3 convolution module with a stride of 2, and then through 4 residual modules, the feature F' is obtained 33 ; In the other two channels, the output features F of stage 2 are 21 、F 22 Through 4 residual modules respectively, the feature F' is obtained 31 and F' 32 , the feature F' 31 、F' 32 、F' 33 Fusion is performed to obtain the three-way output feature F of stage 3 31 、F 32 and F 33 , the fusion calculation is expressed as,

[0051] F 31 =F′ 31 +up(F′ 32 ,2)·+up(F′ 33 ,4)

[0052] F 32 =down(F′ 31 ,2)+F′ 32 +up(F′ 33 ,2)

[0053] F 33=down(F′ 31 ,4)+down(F′ 32 ,2)+F′ 33

[0054] In stage 4, the output features F of stage 3 are 33 Through a 3×3 convolution module with a stride of 2, and then through 4 residual modules, the feature F' is obtained 44 , the output features F of stage 3 are converted into 31 、F 32 、F 33 Through 4 residual modules respectively, the feature F' is obtained 41 、F' 42 、F' 43 , the feature F' 41 、F' 42 、F' 43 、F' 44 Fusion is performed to obtain the four-way output feature F of stage 4 41 、F 42 、F 43 and F 44 , the fusion calculation is expressed as,

[0055] F 41 =F′ 41 +up(F′ 42 ,2)+up(F′ 43 ,4)+up(F′ 44 ,8)

[0056] F 42 =down(F′ 41 ,2)+F′ 42 +up(F′ 43 ,2)+up(F′ 44 ,4)

[0057] F 43 =down(F′ 41 ,4)+down(F′ 42 ,2)+F′ 43 +up(F′ 44 ,2)

[0058] F 44 =down(F′ 41 ,8)+down(F′ 42 ,4)+down(F′ 43 ,2)+F 44

[0059] Second, the features of the same scale obtained in the four stages are spliced ​​together to compensate for the spatial position and detail information that may be lost in each stage. The output after feature splicing is expressed as,

[0060] F concat1 =Concat(F 11 , F 21 , F 31 , F 41 )

[0061] F concat2 =Concat(F 22 , F 32 , F 42 )

[0062] F concat3 =Concat(F 33 , F 43 )

[0063] Among them, F concat1 It represents the output feature map after fusing the resolution of the same scale under 1 branch number at different stages, F concat2 It represents the output feature map after fusing the resolution of the same scale under the 2 branch numbers at different stages, F concat3 It represents the output feature map after fusing the resolution of the same scale under the three branch numbers at different stages.

[0064] F concat1 、F concat2 and F concat3 Input the dynamic attention selection module and convolution layer respectively to obtain F1, F2, and F3. 44 Input the dynamic attention selection module to obtain F4, F1, F2, F3, and F4 as the four outputs of the encoder.

[0065] like Figure 4 As shown in Figure 3, the dynamic attention selection module consists of three steps: separation, feature fusion, and feature selection.

[0066] In the separation part, let C, H, W be the number of channels, height and width of the input feature X, R be a set of real numbers, and let the input feature X∈R C×H×W After 3×3 and 5×5 convolutions, feature maps U1 and U2 are obtained respectively, and then U1 and U2 are added together to obtain feature map U.

[0067] In the feature fusion part, U is globally average pooled and compressed into a vector S, S∈R C×1 , the dimension of S is reduced through the fully connected layer to obtain the feature vector Z∈R d×1 , d is the dimension after dimensionality reduction.

[0068] In the feature selection part, the feature vector Z is transformed by two matrices A∈R C×d , B∈R C×d , use the Softmax function to calculate the attention between channels to obtain the weight vector a∈R C×1 , b∈R C×1 .

[0069] Assume a c is the cth element in a, b c is the cth element in b, and its calculation method is expressed as,

[0070]

[0071] Among them, A c ∈R 1×d represents the element in row c of matrix A, B c ∈R 1×d Represents the elements of the cth row in the matrix B. In order to fuse multi-scale features, a c , b c The feature map U corresponding to the cth channel in U1 and U2 respectively 1c and U 2c After multiplication, fusion is performed to obtain the feature map V on the cth channel c , V c It can be expressed as,

[0072] V c =a c U 1c +b c U 2c

[0073] The c feature maps of all channels are stacked to obtain the final feature map V that integrates multi-scale information.

[0074] Third, for curved and slender targets, their length information is one of the most intuitive features, which can provide additional spatial dimension constraints for the model. Based on this, this paper proposes a length regression auxiliary network module (ALR) to further optimize the segmentation accuracy of slender and curved lesion features, such as Figure 5 As shown:

[0075] The four different scale features of the encoder are input into the length regression auxiliary network, and the highest scale feature F1∈R H×W×C After downsampling with a 3×3 convolution kernel with a step size of 2, and extracting features through a 3×3 convolution with a step size of 1, the scale is halved and the number of channels is doubled. The features obtained are Combine F'1 with the high-scale features output by the encoder Perform the addition operation, then perform downsampling and feature extraction through two layers of convolution, and the obtained features are Combine F'2 with the high-scale features output by the encoder Perform the addition operation, then perform downsampling and feature extraction through two layers of convolution, and the obtained features are Combine F'3 with the high-scale features output by the encoder After the addition operation, downsampling and feature extraction are performed through two layers of convolution, and the obtained features are

[0076] The obtained feature F'4 is subjected to global average pooling operation, fully connected layer and Sigmoid activation function to obtain the final regression output, which is a one-dimensional vector with 2 channels, representing the length prediction of the two types of lesion areas.

[0077] The results obtained by manual segmentation are used as the gold standard. The number of pixels in each type of lesion area is calculated as the true value of the length regression task. The mean square error between the length predicted by regression and the true value is calculated as the length loss function to constrain the main segmentation network.

[0078] Fourth, the decoder module upsamples the input feature F2 by 2 times, the feature F3 by 4 times, and the feature F4 by 8 times, and then concatenates them with F1, passes through a 1×1 convolution layer, and obtains the segmentation result image.

[0079] 5. Traditional segmentation models usually use loss functions such as Dice loss function and Cross-Entropy (CE) loss function. However, these pixel-level loss functions are very sensitive to the boundaries of linear structures. When processing linear structures, even a small spatial deviation may lead to a large loss, which limits the performance of traditional models in linear structure segmentation. To solve this problem, the present invention proposes an edge feature smoothing module (EFS) module, as shown in the attached figure. Figure 4 The EFS module applies a Gaussian blur filter with a size of 15×15 and a standard deviation of 8 to the network output image and the reference image. The blurred result is then fed into a smoothing loss function. This method smoothes the distribution of prediction errors with a milder penalty, helping the model more accurately handle subtle differences in edge positions.

[0080] 6. To further improve the performance of the model in segmenting curved and slender objects in OCT images, this paper introduces a loss function based on the Hausdorff distance. The Hausdorff distance is a method for measuring the distance between two point sets and is often used to evaluate the quality of image segmentation. The segmentation result image is binarized with a threshold of 0.5 to obtain the predicted label point set P. Let the true label point set be T. The Hausdorff distance hausdorff(P,T) from P to T can be expressed as,

[0081]

[0082] Where p and t are points in the point sets P and T respectively, and ||pt|| represents the Euclidean distance from p to t.

[0083] The Hausdorff distance hausdorff(T,P) from T to P can be expressed as,

[0084]

[0085] The sum of the squares of these two distances is used as the distance weight W HD , the formula is,

[0086] W HD =hausdorff 2 (P,T)+hausdorff 2 (T,P)

[0087] Final Hausdorff distance (HD) loss function L HD It can be expressed as,

[0088]

[0089] Among them, p d is the predicted value of the d-th pixel, t d is the true value of the d-th pixel, and N is the total number of pixels.

[0090] In the EFS module, the gold standard and the output result graph are Gaussian blurred and the smoothing loss function L is calculated. Smooth , expressed as,

[0091]

[0092] in, is the value of the dth pixel in the output image after Gaussian blur, is the value of the d-th pixel in the gold standard after Gaussian blur.

[0093] The length prediction value output by the ALR module is used to calculate the length loss function L MSE , expressed as,

[0094]

[0095] in, is the predicted value of the length of the mth category, l m is the true value of the length of the mth category, and M is the number of categories.

[0096] The loss function of the segmentation main network is the Dice loss function L Dice and the cross entropy (CE) loss function L CE , where the Dice loss function can be expressed as,

[0097]

[0098] Here, ε is a smoothing factor that prevents either the numerator or the denominator from reaching zero.

[0099] The cross entropy loss function can be expressed as,

[0100]

[0101] The total loss function is,

[0102] L seg =L HD +L Smooth +L MSE +L Dice +L CE .

[0103] Example 2, reference Figure 6 , which is the second embodiment of the present invention, and which differs from the first embodiment in that: a method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images also includes, in order to verify the effectiveness of the method of the present invention, the method is verified by using the task of segmenting vitreoretinal interface abnormalities in retinal optical coherence tomography (OCT) images.

[0104] (1) Dataset: The experimental data used in the present invention are clinical images obtained in a hospital. They are acquired by a micro-OCT scanner and include 33-line, 64-line and 96-line three-dimensional scans, all covering a retinal range of 12mm×8mm. The image resolution is 2048×1382. The dataset comes from 1973 B-scan slices of 46 patients, of which 914 images contain posterior vitreous detachment and 1236 images contain epiretinal membrane. The annotation of the dataset was carried out under the guidance of professional ophthalmologists. The experiment of the present invention adopts a five-fold cross-validation method to ensure the reliability and generalization ability of the results. The data is divided according to the patient, that is, all the data of each patient will only appear in one of the five folds. Before entering the network, the image and the gold standard are downsampled and processed into 512×256 size images.

[0105] (2) Model training and testing. The experimental configuration is as follows: GPU is GeForce RTX 3090, memory is 24GB. Deep learning framework is Pytorch 1.10.0, language environment is Python 3.8. Integrated development environment is Visual Studio Code. The initial learning rate is 0.005, momentum is 0.9, and weight decay coefficient is 1e -4 The poly learning strategy is used to adjust the learning rate. The formula is as follows:

[0106]

[0107] Among them lter and lter t Represents the current number of iterations and the total number of iterations, the basic learning rate lr b The total number of iterations for network model training is 100 and the batch size is 4.

[0108] During the network training phase, online data augmentation methods were introduced, including random rotation within ±30°, random upside-down flipping, and left-right flipping, to expand the data volume and accelerate network convergence.

[0109] (3) Evaluation metrics: In order to objectively evaluate the pixel-level segmentation performance of ALR-HRNet in the segmentation tasks of PVD and ERM in OCT images, this paper adopts four image segmentation pixel-level evaluation metrics, including Dice Similarity Coefficient (DSC), Intersection Over Union (IoU), Sensitivity (Sen), and Specificity (Spe), as shown in the following formulas:

[0110]

[0111]

[0112]

[0113]

[0114] Among them, TP represents true positives, that is, the number of samples that are correctly segmented from positive samples; FP represents false positives, that is, the number of samples that are incorrectly segmented from positive samples; TN represents true negatives, that is, the number of samples that are correctly segmented from negative samples; FN represents false negatives, that is, the number of samples that are incorrectly segmented from negative samples.

[0115] In addition, in order to evaluate the topological level segmentation performance, this paper uses the centerline Dice coefficient (Centerline DSC, clDice) as the topological level evaluation index. First, extract the real lesion area V T and predicted lesion area V P The skeletons are S T and S P Then, calculate S T The topological accuracy and S P The topological sensitivity of is as follows:

[0116]

[0117]

[0118] Finally, clDice is defined as the harmonic mean of topological accuracy and topological sensitivity, which is expressed as,

[0119]

[0120] (4) Experimental results: (a) Ablation experiments include:

[0121] In order to verify the effectiveness of the feature concatenation and dynamic attention selection (FCDAS), edge feature smoothing module (EFS), length regression auxiliary module (ALR) and Hausdorff distance loss function (HDLoss) added in the invention, corresponding ablation experiments were performed, and the results are shown in Table 1.

[0122] Table 1 Ablation test results of each module of the method of the present invention

[0123]

[0124]

[0125] As shown in Table 1, compared to the baseline network, adding feature concatenation from FCDAS alone improved average clDice by 0.71%, average DSC by 0.81%, and average IoU by 0.83%. Adding SK-Attention from FCDAS alone improved average clDice by 0.44%, average DSC by 0.49%, and average IoU by 0.52%. Adding FCDAS as a whole improved average clDice by 1.23%, average DSC by 1.18%, and average IoU by 1.15% compared to the baseline network, achieving better segmentation performance than either feature concatenation or SK-Attention alone. This demonstrates that combining feature concatenation with dynamic attention enhances the model's ability to discern target structure, ensuring that both high-resolution detail and low-resolution global information are fully utilized. When HDLoss is added to FCDAS, the average clDice improves by 0.54%, the average DSC improves by 0.47%, and the average IoU improves by 0.33% compared to the FCDAS module alone. This demonstrates that the Hausdorff distance loss effectively enhances the model's capture of edge details, thereby improving segmentation accuracy. Adding the EFS module to the FCDAS and HDLoss modules improves the average clDice by 0.51%, the average DSC by 0.54%, and the average IoU by 0.57%. This improvement demonstrates that the EFS module, by smoothing edge features, further optimizes segmentation boundaries and improves segmentation accuracy. Finally, adding the ALR module to the FCDAS, EFS, and HDLoss modules improves the average clDice by 0.99%, the average DSC by 0.70%, and the average IoU by 1.06%. This demonstrates that the ALR module accurately captures object length information through regression analysis, making it particularly suitable for segmenting curved objects. Finally, after integrating all the proposed modules, the average clDice reaches 82.24%, the average DSC reaches 77.92%, and the average IoU is improved to 69.94%.

[0126] (b) Comparative Experiments: In this comparative experiment, our proposed method, ALR-HRNet, was compared with eight other advanced image segmentation networks. The results are shown in Appendix 2. The comparison networks include U-Net, ResUNet, ResUNet++, CPFNet, CS2-Net, MFI-Net, TransUNet, and GT-DLA-dsHFF. CS2-Net, MFI-Net, and GT-DLA-dsHFF are specifically designed for linear structures. To ensure fair comparison, the implementation details of all other networks remain consistent with ALR-HRNet. As shown in Appendix 2, our proposed method, ALR-HRNet, achieves state-of-the-art results in most metrics, outperforming other segmentation networks. Among them, average clDice, average DSC, average IoU, and average Sen achieved optimal results of 82.24%, 77.92%, 69.94%, and 83.60%, respectively. Average Spe reached 99.91%, achieving high performance.

[0127] Table 2 Comparative experimental results of the ALR-HRNet proposed in this invention and other segmentation networks

[0128]

[0129]

[0130] like Figure 6 As shown in the figure, a visual comparison of the segmentation results of the ALR-HRNet proposed in the present invention and other advanced segmentation networks is shown. Through the comparison, we can intuitively find that ALR-HRNet has obvious advantages in segmentation effect. In the first column of images, the PVD area with low contrast in the upper left corner can be accurately identified and segmented by ALR-HRNet, while other methods show varying degrees of omission or misclassification. In the second and third columns of images, only ALR-HRNet predicts the correct length of the lesion, while other methods will more or less predict it to be too long or too short. TransUNet even predicts PVD as ERM.

[0131] Example 3 is the third embodiment of the present invention, which is different from the previous two embodiments in that: an automatic segmentation system for vitreoretinal interface abnormalities based on OCT images, including a feature extraction unit, a feature fusion unit, an output unit and a model evaluation and optimization unit; the feature extraction unit inputs the OCT image to be segmented into a high-resolution segmentation network, and the encoder part of the high-resolution segmentation network uses a four-stage feature encoding module to extract features from the OCT image to be detected to obtain multi-scale semantic features; the feature fusion unit fuses the multi-scale semantic features based on feature splicing and dynamic attention selection, and captures the details and contextual information of slender targets; the output unit inputs a part of the encoder output into a length regression auxiliary network, outputs a one-dimensional vector representing the target length, which is used to calculate the length loss, and the other part is directly input into the decoder module to obtain the segmentation result; the model evaluation and optimization unit processes the output of the high-resolution segmentation network and the corresponding gold standard through an edge feature smoothing module, calculates the smoothing loss function, and introduces the Hausdorff distance loss function to calculate the distance between the segmentation result and the gold standard for network training.

[0132] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0133] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0134] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0135] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, it can be implemented using a combination of any of the following technologies known in the art: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0136] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images, characterized by: include, The OCT image to be segmented is input into a high-resolution segmentation network. The encoder part of the high-resolution segmentation network uses a four-stage feature encoding module to extract features from the OCT image to be detected and obtain multi-scale semantic features. Multi-scale semantic features are fused based on feature splicing and dynamic attention selection, capturing the details of slender targets and contextual information; The output of the encoder is transmitted to the length regression auxiliary network and the decoder module respectively; After receiving the transmission information, the length regression auxiliary network outputs a one-dimensional vector representing the target length for calculating the length loss; After receiving the transmission information, the decoder module outputs the segmentation result map; The segmentation result image and the gold standard obtained by the corresponding manual segmentation are processed by the edge feature smoothing module to calculate the smoothing loss function; The Hausdorff distance loss function is introduced to calculate the distance between the segmentation result and the gold standard obtained by manual segmentation for network training; The encoder includes a bottleneck module, a residual module, an upsampling module and a downsampling module; In the bottleneck module, the input features pass through a 1×1 convolution layer to reduce the number of channels, then pass through a 3×3 convolution layer for feature extraction, and finally pass through a 1×1 convolution layer to increase the number of channels. The features are added to the input through a residual connection and then output after a ReLU activation function. In the residual module, the input features pass through two 3×3 convolutional layers, are added to the input through the residual connection, and are output after the ReLU activation function; The upsampling module includes a 1×1 convolutional layer with a stride of 1 and a batch normalization layer to perform upsampling operations on feature data; The downsampling module contains three 3×3 convolutional layers with a stride of 2 and a batch normalization layer, which downsample the feature data by 2, 4, and 8 times; Before extracting features from the OCT image to be detected using the four-stage feature encoding module, the input image is subjected to two 3×3 convolutional layers with a stride of 2 to reduce its scale to 1 / 4; The four-stage feature encoding module includes: are the output features of different branches in the four stages, where Indicates the stage number from left to right, Indicates the branch number from top to bottom; The four-stage feature encoding module also includes, in stage 1, obtaining the output features of stage 1 through 4 bottleneck modules ; In stage 2, the output features of stage 1 are Through a 3×3 convolution module with a step size of 1, and then through 4 residual modules, the feature , in another channel Through a 3×3 convolution module with a stride of 2 and 4 residual modules, the feature , the features and Fusion is performed to obtain the two-way output features of stage 2 and , the fusion calculation is expressed as: , ,in Represents the feature Upsampling times, Represents the feature Downsampling times; In stage 3, the output features of stage 2 are Through a 3×3 convolution module with a stride of 2, and then through 4 residual modules, the feature ; In the other two channels, the output features of stage 2 、 Through 4 residual modules respectively, the features are obtained and , the features 、 、 Fusion is performed to obtain the three-way output features of stage 3 、 and , the fusion calculation is expressed as, In stage 4, the output features of stage 3 are Through a 3×3 convolution module with a stride of 2, and then through 4 residual modules, the feature , the output features of stage 3 are converted into 、 、 Through 4 residual modules respectively, the features are obtained 、 、 , the features 、 、 、 Fusion is performed to obtain the four-way output features of stage 4 、 、 and , the fusion calculation is expressed as, + + 。 2. The method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images according to claim 1, characterized in that: The feature splicing includes splicing the features of the same scale obtained in the four stages to compensate for the spatial position and detail information that may be lost in each stage. The output after feature splicing is expressed as: in, It represents the output feature map after fusing the resolution of the same scale under 1 branch number at different stages. It represents the output feature map after fusing the resolution of the same scale under the 2 branch numbers at different stages. It represents the output feature map after fusing the resolution of the same scale under the three branch numbers at different stages; Concat means splicing the feature map in the channel dimension; Will 、 and Input the dynamic attention selection module and convolution layer respectively, and get 、 、 ,Will Input dynamic attention selection module, get , 、 、 、 As four outputs of the encoder; The dynamic attention selection includes three steps: separation, feature fusion and feature selection; In the separation part, let C, H, and W be the number of channels, height, and width of the input feature X, respectively. is a set of real numbers, and the input features The feature maps are obtained by 3×3 and 5×5 convolution respectively. and , and then and Add to get the feature map ; In the feature fusion part, Perform global average pooling and compress into a vector , , after the fully connected layer Perform dimensionality reduction to obtain the feature vector , is the dimension after dimensionality reduction; In the feature selection part, the feature vector After two matrix transformations , , use the Softmax function to calculate the attention between channels to obtain the weight vector , ; yes Middle elements, yes Middle elements, , Respectively and The corresponding feature maps of channels and After multiplication, add and get Feature maps on channels ,Bundle The feature maps of all channels are stacked to obtain the final feature map that integrates multi-scale information .

3. The method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images according to claim 2, characterized in that: The four different scale features of the encoder are input into the length regression auxiliary network, and the highest scale feature After downsampling with a 3×3 convolution kernel with a step size of 2, and extracting features through a 3×3 convolution with a step size of 1, the scale is halved and the number of channels is doubled. The features obtained are ,Will High-scale features output by the encoder Perform the addition operation, then perform downsampling and feature extraction through two layers of convolution, and the obtained features are ,Will High-scale features output by the encoder Perform the addition operation, then perform downsampling and feature extraction through two layers of convolution, and the obtained features are ,Will High-scale features output by the encoder After the addition operation, downsampling and feature extraction are performed through two layers of convolution, and the obtained features are ; The obtained features After the global average pooling operation, the fully connected layer and the Sigmoid activation function, the final regression output is obtained, which is a one-dimensional vector with M channels, representing the length prediction of M types of lesion areas; The results obtained by manual segmentation are used as the gold standard. The number of pixels in each type of lesion area is calculated as the true value of the length regression task. The mean square error between the length predicted by regression and the true value is calculated as the length loss function to constrain the main segmentation network.

4. The method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images according to claim 3, characterized in that: After the decoder module receives the transmission information, the output segmentation result map includes: Upsampling 2 times, features Upsampling 4 times, features After upsampling 8 times After splicing and passing through a 1×1 convolutional layer, the segmentation result map is obtained; The edge feature smoothing module processes the network output image and the gold standard obtained by manual segmentation through a Gaussian blur filter with a size of 15×15 and a standard deviation of 8, and half of the mean square error between the blurred results is used as the smoothing loss function.

5. A system for automatically segmenting vitreoretinal interface abnormalities based on OCT images, using the method for automatically segmenting vitreoretinal interface abnormalities based on OCT images according to any one of claims 1 to 4, characterized in that: It includes feature extraction unit, feature fusion unit, output unit and model evaluation and optimization unit; The feature extraction unit inputs the OCT image to be segmented into the high-resolution segmentation network. The encoder part of the high-resolution segmentation network uses a four-stage feature encoding module to extract features from the OCT image to be detected and obtain multi-scale semantic features. The feature fusion unit fuses multi-scale semantic features based on feature splicing and dynamic attention selection, and captures slender target details and context information; The output unit transmits the output of the encoder to the length regression auxiliary network and the decoder module respectively; After receiving the transmission information, the length regression auxiliary network outputs a one-dimensional vector representing the target length for calculating the length loss; after receiving the transmission information, the decoder module outputs a segmentation result image; The model evaluation and optimization unit processes the segmentation result image and the corresponding gold standard obtained by manual segmentation through an edge feature smoothing module, calculates a smoothing loss function, and introduces a Hausdorff distance loss function to calculate the distance between the segmentation result and the gold standard obtained by manual segmentation for network training.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for automatically segmenting vitreoretinal interface abnormalities based on OCT images according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for automatic segmentation of vitreoretinal interface abnormalities based on OCT images according to any one of claims 1 to 4 are implemented.

Images