Choroidal neovascularization segmentation method and system in optical coherence tomography images

Abstract

The present invention discloses a method and system for segmenting choroidal neovascularization in optical coherence tomography images. By constructing a segmentation network based on UNet, two paths are provided during downsampling to respectively extract multi-scale global features and multi-scale global features with local semantic information and long-distance dependency information. The features extracted on the two paths are fused to output fused features of each scale, which can fully extract feature information. During upsampling, the fused features and upsampled features of the same scale are enhanced and then fused, which can further focus on the generation of lesion features during the upsampling process. In the upsampling process, lesion boundary information is used as supervisory information to enhance the expression of boundary information, thereby enabling refined segmentation and improving the segmentation effect.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a method and system for segmenting choroidal neovascularization in optical coherence tomography images. Background Art

[0002] Myopic choroidal neovascularization (mCNV) is a common complication of high myopia. Its lesions involve the macula of the retina and have a serious impact on vision. Currently, the main treatment for mCNV is intraocular injection of anti-vascular endothelial growth factor (VEGF), which requires multiple injections to convert the lesions from active to inactive.

[0003] The best tool for observing the transition from active to inactive lesions is optical coherence tomography (OCT). This is a non-invasive test that can observe the fine structures of the different layers of the retina and choroid. It can be used to observe the size of lesions, helping doctors determine the effectiveness of each injection treatment or the possibility of recurrence.

[0004] With the development of deep learning, although deep learning-based image segmentation methods have been used for OCT image segmentation, due to the small size of the choroidal neovascularization lesion area, low contrast, blurred boundaries, and the presence of a large amount of speckle noise in OCT images, these problems lead to insufficient feature information extracted by existing image segmentation methods when segmenting choroidal neovascularization, resulting in inaccurate segmentation and poor segmentation effect.

[0005] Therefore, the existing technology needs to be improved and enhanced. Summary of the Invention

[0006] The main purpose of the present invention is to provide a method, system, intelligent terminal and storage medium for segmenting choroidal neovascularization in optical coherence tomography images, which can solve the current problems of insufficient feature information extracted during choroidal neovascularization segmentation, inaccurate segmentation and poor segmentation effect.

[0007] To achieve the above objectives, the present invention provides, in a first aspect, a system for segmenting choroidal neovascularization in optical coherence tomography images, comprising a segmentation network constructed based on UNet, for segmenting choroidal neovascularization in optical coherence tomography images, wherein the segmentation network is provided with an encoder for downsampling and a decoder for upsampling;

[0008] The encoder is provided with a dual-path network, wherein a first path of the dual-path network is provided with a plurality of Transformer blocks, and the first path is used to extract first global features at multiple scales; a second path of the dual-path network is provided with a plurality of hybrid units for extracting local semantic information and long-distance dependency information, and the second path is used to extract second global features at multiple scales, wherein the scales of the first global features correspond one-to-one to the scales of the second global features;

[0009] The encoder is further provided with a plurality of dual-path fusion units, and the dual-path fusion units are used to fuse the first global feature and the second global feature at each scale;

[0010] The decoder is provided with a plurality of bidirectional attention gating units and cascade units, the cascade unit is used to cascade the features output by the dual-path fusion unit and the features output by the bidirectional attention gating unit, and the bidirectional attention gating unit is used to fuse the upsampled features and the features output by the dual-path fusion unit at adjacent levels, and the upsampled features are features obtained by upsampling the features output by the cascade unit or the dual-path fusion unit;

[0011] The output module is used to obtain and output the segmentation result based on the features output by the segmentation network.

[0012] Optionally, the hybrid unit includes a connected residual convolutional network and a Transformer network.

[0013] Optionally, a channel attention encoding module is provided on one path of the dual-path fusion unit to enhance the interdependence of the channel attention map extracted by the mixing unit, and a position attention encoding module is provided on the other path to enhance the spatial information of the features extracted by the Transformer block.

[0014] Optionally, the bidirectional attention gating unit includes two bidirectional attention gating modules and a summation module, the two bidirectional attention gating modules are used to enhance the features output by the dual-path fusion unit and the up-sampled features, respectively, and the summation module is used to perform matrix addition on the features output by the two bidirectional attention gating modules.

[0015] Optionally, the bidirectional attention gating module is provided with a first pooling layer, a dot product layer and a second pooling layer, and the first pooling layer and the second pooling layer are both provided with a maximum pooling branch and an average pooling branch. The first pooling layer is used to perform pooling operations and full connection operations on the features input into the bidirectional attention gating module, the dot product layer is used to perform matrix dot product of the features output by the first pooling layer and the features input into the bidirectional attention gating module, and the second pooling layer is used to perform pooling operations on the features output by the dot product layer.

[0016] Optionally, the loss functions used to train the segmentation network include: lesion classification loss, boundary classification loss and Dice loss.

[0017] A second aspect of the present invention provides a method for segmenting choroidal neovascularization in an optical coherence tomography image, the method comprising:

[0018] Pre-training a segmentation network, wherein the segmentation network is the segmentation network in any of the above-mentioned systems for segmenting choroidal neovascularization in optical coherence tomography images;

[0019] Input the optical coherence tomography image into the trained segmentation network;

[0020] Get the features of the trained segmentation network output;

[0021] Based on the features, a segmentation result is obtained and output.

[0022] Optionally, the loss function for training the segmentation network is:

[0023]

[0024]

[0025]

[0026]

[0027] in, is the lesion classification loss, is the boundary classification loss, L Dice Dice loss, GT i and Respectively, the i-th pixel is the true annotation of the lesion and the lesion boundary, PO i and PB i They represent the predicted values ​​of the i-th pixel belonging to the lesion and the lesion boundary, respectively. α, β, and γ are the weights of the lesion classification loss, boundary classification loss, and Dice loss, respectively.

[0028] A third aspect of the present invention provides an intelligent terminal, which includes a memory, a processor, and a choroidal neovascularization segmentation program in optical coherence tomography images stored in the memory and runnable on the processor. When the choroidal neovascularization segmentation program in optical coherence tomography images is executed by the processor, any step of the choroidal neovascularization segmentation method in optical coherence tomography images is implemented.

[0029] A fourth aspect of the present invention provides a computer-readable storage medium, on which is stored a choroidal neovascularization segmentation program in an optical coherence tomography image. When the choroidal neovascularization segmentation program in an optical coherence tomography image is executed by a processor, any one of the steps of the choroidal neovascularization segmentation method in an optical coherence tomography image is implemented.

[0030] It can be seen from the above that compared with the existing technology, by constructing a segmentation network based on UNet, two paths are set up during downsampling to respectively extract multi-scale global features and multi-scale global features with local semantic information and long-distance dependency information, and the features extracted on the two paths are fused to output the fused features of each scale, which can fully extract feature information; during upsampling, the fused features and upsampled features of the same scale are enhanced and then fused, which can focus more on the generation of lesion features during the upsampling process, so that the lesion boundary information is used as supervisory information to enhance the expression of boundary information during the upsampling process, and can perform refined segmentation and improve the segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 1 is a network architecture diagram of a segmentation network in a choroidal neovascularization segmentation system in optical coherence tomography images provided by an embodiment of the present invention;

[0033] Figure 2 yes Figure 1 Network architecture diagram of the hybrid unit in the embodiment;

[0034] Figure 3 yes Figure 1 Network architecture diagram of the dual-path fusion unit in the embodiment;

[0035] Figure 4 yes Figure 1A schematic diagram of the network architecture of a bidirectional attention gating unit in an embodiment;

[0036] Figure 5 It is a qualitative analysis effect diagram of the segmentation results of each network model;

[0037] Figure 6 1 is a schematic diagram of a specific process of a method for segmenting choroidal neovascularization in an optical coherence tomography image provided by an embodiment of the present invention;

[0038] Figure 7 This is a block diagram of the internal structure principle of a smart terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0040] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0041] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0042] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0043] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0044] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0046] Accurate segmentation of myopic choroidal neovascularization (OCT) in OCT is crucial for assessing the progression of MCNE. Currently, medical image segmentation primarily utilizes convolutional neural network-based UNet networks or various UNet variants, such as ResUNet, UNet++, and nnUNet, or visual Transformer networks. Although these networks have been widely used in medical image segmentation, the small size, low contrast, and blurred boundaries of MCNE lesions in OCT images require network models capable of extracting and fusing both global and local feature information to fully extract feature information. Furthermore, the network model must be sensitive to the boundary information of the lesion region to accurately segment the MCNE lesion region and achieve good segmentation results. Currently, single UNet networks, UNet variants, and visual Transformer networks have limitations in extracting feature information, failing to fully extract feature information and resulting in poor segmentation results. Although some hybrid networks such as the UTNet network (a hybrid network of a convolutional neural network and a Transformer network) have emerged, tests have shown that the recognition of boundary information is still not sensitive enough when segmenting the lesion area of ​​myopic choroidal neovascularization, and the segmentation results are not precise enough.

[0047] In order to improve the segmentation accuracy of the lesion area of ​​myopic choroidal neovascularization in OCT images, the present invention provides a choroidal neovascularization segmentation system and method in optical coherence tomography images. By constructing a segmentation network based on UNet, two paths are provided during downsampling to respectively extract multi-scale global features and multi-scale global features with local semantic information and long-distance dependency information, and the features extracted on the two paths are fused to output fused features of each scale, which can fully extract feature information; during upsampling, the fused features and upsampled features of the same scale are enhanced and then fused, which can focus more on the generation of lesion features during the upsampling process, so that the lesion boundary information is used as supervisory information to enhance the expression of boundary information during the upsampling process, and can refine the segmentation results and improve the segmentation effect.

[0048] Exemplary Systems

[0049] An embodiment of the present invention provides a choroidal neovascularization segmentation system in optical coherence tomography images, which runs on electronic devices such as smart terminals, PCs, and hospital backend servers, and is used to segment myopic choroidal neovascularization in optical coherence tomography images. It should be noted that the system is not limited to segmenting myopic choroidal neovascularization, but can also segment various choroidal neovascularizations in optical coherence tomography images. The segmentation system mainly includes two parts: a segmentation network constructed based on UNet and an output module. After the optical coherence tomography image is input into the segmentation network, the segmentation network outputs feature information such as the classification of each pixel in the optical coherence tomography image as belonging to the background area or the lesion area, and the classification of pixels belonging to the lesion interior or the lesion boundary. The output module then determines the lesion area based on this feature information, crops the optical coherence tomography image, and segments and outputs an image of the lesion area of ​​myopic choroidal neovascularization.

[0050] Figure 1 The architecture of the segmentation network is shown, which includes an encoder for downsampling and a decoder for upsampling. Figure 1 The left part is the downsampling stage, and the middle part is the upsampling stage. The encoder is a dual-path network. Figure 1 The multiple Transformer blocks at the top are the first path. Each Transformer block outputs the first global feature of a scale. For example, the first Transformer block includes 3 Transformers, and the scale of the output feature is The second Transformer block includes 4 Transformers, and the scale of the output features is Etc. In this embodiment, four Transformer blocks are set on the first path, which can downsample the first global features of four scales. Figure 1 The hybrid module in the second path is used to extract local semantic information and long-distance dependency information. The feature output by the hybrid unit is the second global feature, and the scale of the second global feature output by the hybrid unit corresponds one-to-one with the scale of the first global feature output by the Transformer block. In this embodiment, there are four hybrid units, each of which corresponds to a Transformer block. In other words, the second path can downsample the second global features at four scales, and the sizes of the four scales of the second path correspond one-to-one with the sizes of the four scales of the first path.

[0051] The multiple first global features extracted by the first path and the multiple second global features extracted by the second path can fully extract the feature information of the optical coherence tomography image from multiple angles of local semantic information and global texture information.

[0052] The network architecture of the hybrid unit is shown in the figure below: Figure 2 As shown in the figure, it includes a connected residual convolutional network and a Transformer network. The input of the hybrid unit is: features extracted from the OCT image or features extracted by the hybrid unit of the previous layer. These input features first pass through the residual convolutional network and then pass through the Transformer network to output the second global feature. Among them, the residual convolutional network includes two layers of 3*3 convolutional layers, which can extract local semantic information. The local semantic features output by the residual convolutional network are then cut into blocks, and the embedded block sequence is obtained and input into the Transformer network. The long-distance dependency information is extracted through the multi-head attention and multi-layer perceptron in the Transformer network.

[0053] In order to fuse the first global features and the second global features extracted by the two paths respectively, the encoder is also provided with a plurality of two-path fusion units (Two-path fusion, TPFusion), and the number of two-path fusion units is the same as the total number of feature scales. For example, in this embodiment, four two-path fusion units are provided accordingly, and one two-path fusion unit is used to fuse the first global features and the second global features of the same scale, output the fused features and send them to the decoder of the same level. Among them, the level refers to the upsampling stage of the decoder or the downsampling stage of the encoder, and each level will output features of one scale. For example: Figure 1 The encoder and decoder in each consist of four layers, and the scale of the first layer features is: The scale of the second-level features is: Each level of the encoder includes a dual-path fusion unit and a mixing unit. The decoder, from the first to the third level, includes a bidirectional attention gating unit (DAG in the figure) and a cascade unit (cascade C in the figure).

[0054] The network architecture of the dual-path fusion unit is shown in the figure below: Figure 3 As shown in , the dual-path fusion unit includes two parallel paths. Assume that the features extracted by the Transformer block are The features extracted by the hybrid unit are Since the local features extracted by the convolutional neural network focus more on the semantic features of the image, Figure 3 The channel attention encoding module is set on the left path in the figure, which uses the channel attention encoding to enhance the features extracted by the hybrid unit, that is, to enhance the mutual dependence of the channel attention map output by the hybrid unit. Since the features extracted by the Transformer have long-range context feature dependencies and pay less attention to spatial information, Figure 3 There is a position attention encoding module on the path on the right side, which uses position attention encoding to enhance the spatial information of the features extracted by the Transformer block.

[0055] According to the definition of channel attention coding and position attention coding, the fused feature F obtained by the dual-path fusion unit TP It can be expressed as:

[0056]

[0057] Among them, Point_CNN(·) represents point convolution with a kernel size of 1×1, Sum(·) represents matrix addition, M(·) represents matrix multiplication, and Resh(·) represents feature dimension reshaping (i.e., rearrangement).

[0058] The specific process is: the feature F extracted by the hybrid unit M Input the path with channel attention encoding module, and transform the feature F M With the transposed features Matrix multiplication, the result of multiplication and feature F M The rearranged result is matrix multiplied, and then the multiplied result is rearranged again, and the obtained result is combined with the feature F M Add together to obtain the features after channel attention encoding.

[0059] Features F extracted by the Transformer block T Input the path with position attention encoding module, and transform the feature F T With the transposed features Matrix multiplication, the result of multiplication and feature F TThe rearranged result is matrix multiplied, and then the multiplied result is rearranged again, and the obtained result is combined with the feature F T Add together to obtain the features after position attention encoding;

[0060] Then the features after channel attention encoding and position attention encoding are matrix-added, and then a point convolution with a kernel size of 1×1 is performed to obtain the fused feature F output by the dual-path fusion unit. TP .

[0061] After the dual-path fusion unit, the features extracted by the four Transformer blocks are complemented in terms of spatial information, and the features extracted by the hybrid unit are complemented by mutual dependencies in the channel attention map, so that the features extracted by the two paths can be effectively fused and complemented in different feature dimensions, thereby improving the feature information extraction effect of the segmentation network.

[0062] Although the encoder can extract features with rich semantics and global texture, it still needs to be further fused by the decoder to express features related to the lesions of myopic choroidal neovascularization. Figure 1 As shown in the figure, the decoder is equipped with several bidirectional attention gating units (DAGs) and cascade units. The cascade unit is used to cascade the fused features output by the dual path fusion unit (TPFusion) and the features output by the bidirectional attention gating unit (DAG). The upsampled features obtained by upsampling the features output by the cascade unit at each level are the output features of each level in the decoder. For example: Figure 1 The upsampled features obtained by upsampling the features output by the cascade units of the first layer of the decoder are the output features of the first layer in the decoder. Since the fourth layer of the decoder is the lowest layer, the upsampled features of this layer are the features obtained by directly upsampling the features output by the dual-path fusion unit. The bidirectional attention gating unit is used to fuse the upsampled features and the features output by the dual-path fusion unit at adjacent levels. The bidirectional attention gating unit of the first level is used to fuse the features output by the dual-path fusion unit of the first level with the upsampled features of the second level, and the bidirectional attention gating unit of the second level is used to fuse the features output by the dual-path fusion unit of the second level with the upsampled features of the third level. The features output by the bidirectional attention gating unit are used to input into the cascade units of the same level.

[0063] Specifically, in order to make the segmentation network pay more attention to the generation of lesion features during the upsampling process, the present invention designs a bidirectional attention gating unit using the channel attention mechanism. Figure 4 As shown, the bidirectional attention gating unit includes two bidirectional attention gating modules and a summation module. A bidirectional attention gating module is used to enhance the fused feature F output by the dual-path fusion unit. TP, a bidirectional attention gating module is used to enhance the upsampled features F up The summation module is used to perform matrix addition on the features output by the two bidirectional attention gating modules to obtain the bidirectional attention gating feature map F DAG And output.

[0064] The specific structure of the bidirectional attention gating module includes a first pooling layer and a second pooling layer arranged one above the other, with a dot product layer provided between them. Parallel maximum pooling branches and average pooling branches are provided on both the first pooling layer and the second pooling layer. The first pooling layer is used to perform pooling and full connection operations on the features input to the bidirectional attention gating module, and then perform matrix addition before inputting the dot product layer. The dot product layer is used to perform matrix dot product on the features output by the first pooling layer and the features input to the bidirectional attention gating module, and then input them into the second pooling layer. After performing maximum pooling and average pooling operations, they are cascaded, and the cascaded results are matrix multiplied with the output features of the dot product layer to obtain the output features of the bidirectional attention gating module.

[0065] The specific process is: First, the fusion feature F TP The maximum pooling layer and the average pooling layer are input respectively, and the pooled results are activated by the ReLu activation function respectively, and the number of channels of the features is changed to The features obtained by the maximum pooling branch and the average pooling branch are weighted and summed, and then combined with the fusion feature F TP Perform matrix multiplication to obtain the attention map F′ TP In order to maintain the integrity of the features, the obtained attention map is passed through the maximum pooling layer and the average pooling layer again, and the features obtained by the two pooling processes are cascaded and then matrix multiplied with the obtained attention map to obtain the fusion feature F TP The corresponding single attention gate feature F″ TP .

[0066] Upsample feature F up The same operation as above is performed to obtain the up-sampled feature F up The corresponding single attention gate feature.

[0067] Finally, the fusion feature F TP The corresponding single attention gate feature F″ TP and upsampled features F up The corresponding single attention gate feature F″ up Perform weighted summation to obtain the final dual attention gate feature map F DAG The specific expression is:

[0068] F′ TP =Sum(FC(ReLU(PoolMax (F TP ))),FC(ReLU(Pool Avg (F TP )))),

[0069] F″ TP =M(F′ TP ,Concat(Pool Max (M(F TP ,F′ TP )),Pool Avg (M(F TP ,F′ TP )))),

[0070] F′ up =Sum(FC(ReLU(Pool Max (F up ))),FC(ReLU(Pool Avg (F up )))),

[0071] F″ up =M(F′ up ,Concat(Pool Max (M(F up ,F′ up )),Pool Avg (M(F up ,F′ up )))),

[0072] F DAG =Sum(F″) TP ,F″ up )

[0073] Among them, Pool Max (·) and Pool Avg (·) denotes max pooling and average pooling operations, respectively. FC(·) denotes a fully connected layer, Sum(·) denotes matrix addition, and M(·) denotes matrix multiplication. Concat(·) denotes a cascade operation.

[0074] Finally, after linearly projecting the upsampled features of the first layer of the decoder, the classification information of each pixel as belonging to the background area or the lesion area, as well as the classification information of the pixel inside the lesion or the lesion boundary are output. The lesion area is determined based on this classification information, and the optical coherence tomography image is cropped to segment the lesion area image of myopic choroidal neovascularization and output it.

[0075] Through the bidirectional attention gating module, the upsampling process of the decoder can pay more attention to the generation of feature information of the lesion area, which has a certain gain effect on the prediction results of the classification information.

[0076] Since the segmentation task of choroidal neovascularization in optical coherence tomography images can be regarded as a pixel-level binary classification problem, that is, there are only two categories: background and lesions. Therefore, binary cross entropy is selected as the lesion classification loss. In order to use the lesion boundary information to supervise the segmentation task for refined segmentation, the boundary information of the lesion is also used as weak supervision information to constrain the segmentation network to generate a predicted segmentation mask with rich boundary information. The boundary information can also be considered as a pixel-level binary classification problem, so the weak supervision boundary loss function also selects the binary cross entropy function as the boundary classification loss. Lesion classification loss and boundary classification loss The specific expression is:

[0077]

[0078]

[0079] Among them, GT i and Respectively, the i-th pixel is the true annotation of the lesion and the lesion boundary, PO i and PB i They represent the predicted values ​​of whether the i-th pixel belongs to the lesion and the lesion boundary, respectively.

[0080] In order to prevent overfitting during network training and ensure that the segmentation network can accurately predict the lesion area, Dice loss is also used. Dice loss has been proven to have good performance and stability in multiple medical image segmentation tasks. Dice The specific expression is:

[0081]

[0082] Among them, GT i Indicates that the i-th pixel is a lesion, PO i Indicates the predicted value that the i-th pixel belongs to the lesion boundary.

[0083] Finally, the loss function of the segmentation network includes: lesion classification loss, boundary classification loss and Dice loss, which are specifically expressed as:

[0084]

[0085] Among them, α, β, and γ are the weights of lesion classification loss, boundary classification loss, and Dice loss, respectively. In this embodiment, they are set to 0.4, 0.2, and 0.4, respectively.

[0086] It should be noted that the framework of the segmentation network 600 can be a basic UNet network or various deformation networks of the UNet network.

[0087] To demonstrate the effectiveness of the segmentation network presented in this paper, several networks with the best performance in medical image segmentation in recent years were selected for comparative experiments, including U-Net, Atten U-Net, Swin-UNet, UTNet, TMUNet, and Res_UTNet. The quantitative comparison results of each network for the mCNV OCT lesion segmentation task presented in this paper are shown in Table 1:

[0088] Table 1 Quantitative comparison results of different networks on OCT lesion segmentation tasks (%)

[0089]

[0090] As can be seen from Table 1, the comparison networks selected by the present invention are all network structures designed with the U-Net network as the baseline network model, among which Swin-UNet, UTNet, TMUNet and Res_UTNet are all network models that combine the CNN structure with the Transformer structure. These four networks are also classic networks that deeply fuse CNN and Transformer. Compared with the comparison networks, the segmentation network of the present invention has achieved the best results in all the segmentation indicators involved in the comparison. In particular, the Dice coefficient and IoU, the two most commonly used and most objective indicators for evaluating segmentation, are compared with the baseline U-Net network. The Dice coefficient of the segmentation network of the present invention has increased by nearly 3 percentage points, and the IoU score has increased by more than 4.5 percentage points. By comparing the two algorithms of U-Net and UTNet, it can be seen that after the introduction of the Transformer framework, the segmentation performance of the network has been greatly improved, wherein the Dice coefficient and IoU score have both increased by more than 1 percentage point, which shows that the introduction of the Transformer framework can strengthen the network's extraction of global features, thereby improving the network's expression of lesion features. Compared with the second-ranked model in Table 1, the segmentation network of the present invention has improved by 1.14% and 2.65% in the Dice coefficient and IoU index respectively. This result shows that the segmentation network of the present invention can fully integrate the local semantic features extracted by the CNN network and the global features of the long-distance feature dependency extracted by the Transformer network, which has a certain effect on the overall segmentation performance of the network. In addition, the segmentation results of each network model were also qualitatively analyzed, and the results are as follows: Figure 5 As shown. Figure 5It can be seen that the segmentation results of the U-Net and Atten U-Net models are poor. The segmentation of the lesion area in the image is not complete, and some are even undetectable. This shows that when only the CNN model is used for image segmentation, the network focuses more on the semantic feature information in the image and has limited learning ability for spatial and global feature information. In addition, the OCT images of mCNV are noisy and the boundary information is fuzzy, which can easily lead to the CNN network learning a lot of interference information when learning local semantic features, causing the network to learn some erroneous information and fail to fully express the lesion characteristics, such as Figure 5 In the segmentation results of the second row, Atten U-Net and Swin-UNet methods hardly detect the lesion area, resulting in the segmentation result having only the background area. By observing the first and third rows, when the network introduces the Transformer framework (such as UTNet, TMUNet, Res_UTNet, etc.), the global features of long-distance dependence can be well learned, so that the lesion feature information can be fully expressed. Compared with these network models, the segmentation network of the present invention can not only better learn the global features of local semantic features and long-distance dependence, but also learn the boundary information of the lesion well, which also makes up for the shortcoming of the blurred boundary information of the lesion in the OCT image.

[0091] In summary, since the encoding stage extracts and fuses multi-scale features on two paths, and the decoding stage uses a bidirectional attention gating unit to fuse the features obtained in the encoding stage, the model focuses more on the generation of lesion features during the upsampling process, so that the lesion boundary information is used as supervisory information to enhance the expression of boundary information and improve the refined segmentation effect.

[0092] Exemplary Methods

[0093] The embodiment of the present invention also provides a method for segmenting choroidal neovascularization in optical coherence tomography images. The above segmentation network is used to segment choroidal neovascularization, and the segmentation network is pre-trained. Specifically, Figure 6 As shown, this embodiment includes the following steps:

[0094] Step S100: Inputting the optical coherence tomography image into the trained segmentation network.

[0095] Step S200: Obtain features output by the trained segmentation network.

[0096] Step S300: Based on the above features, obtain the segmentation result and output it.

[0097] Specifically, after the optical coherence tomography image is input into the trained segmentation network, the features output by the segmentation network are the classification information of each pixel belonging to the background area or the lesion area, as well as the classification information of the pixels inside the lesion or the lesion boundary. Based on this classification information, the lesion area can be determined, and then the optical coherence tomography image is cropped to segment the lesion area image of myopic choroidal neovascularization and output it.

[0098] Specifically, in this embodiment, the specific functions of each step of the above-mentioned choroidal neovascularization segmentation method in the optical coherence tomography image can refer to the corresponding description in the above-mentioned choroidal neovascularization segmentation system in the optical coherence tomography image, and will not be repeated here.

[0099] Based on the above embodiment, the present invention also provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 7 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a choroidal neovascularization segmentation program in optical coherence tomography images. The internal memory provides an environment for the operation of the operating system and the choroidal neovascularization segmentation program in the optical coherence tomography images in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the choroidal neovascularization segmentation program in the optical coherence tomography image is executed by the processor, the steps of any one of the above-mentioned methods for segmenting choroidal neovascularization in optical coherence tomography images are implemented. The display screen of the intelligent terminal can be a liquid crystal display or an electronic ink display.

[0100] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention and does not constitute a limitation on the smart terminal to which the solution of the present invention is applied. The specific smart terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0101] An embodiment of the present invention also provides a computer-readable storage medium, on which a choroidal neovascularization segmentation program in an optical coherence tomography image is stored. When the choroidal neovascularization segmentation program in an optical coherence tomography image is executed by a processor, the steps of any one of the methods for segmenting choroidal neovascularization in an optical coherence tomography image provided by an embodiment of the present invention are implemented.

[0102] It should be understood that the sequence numbers of the steps in the above embodiments do not imply a specific order of execution; the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0103] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0104] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0105] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0106] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For example, the division of the modules or units described above is merely a logical functional division. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented.

[0107] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0108] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A choroidal neovascularization segmentation system in optical coherence tomography images, characterized in that: include: A segmentation network based on UNet is used to segment choroidal neovascularization in optical coherence tomography images, wherein the segmentation network is provided with an encoder for downsampling and a decoder for upsampling; The encoder is provided with a dual-path network, wherein a first path of the dual-path network is provided with a plurality of Transformer blocks, and the first path is used to extract first global features at multiple scales; The second path of the dual-path network is provided with a plurality of hybrid units for extracting local semantic information and long-range dependency information, and the second path is used to extract second global features at multiple scales, and the scales of the first global features correspond one-to-one to the scales of the second global features; The encoder is further provided with a plurality of dual-path fusion units, and the dual-path fusion units are used to fuse the first global feature and the second global feature at each scale; The decoder is provided with a plurality of bidirectional attention gating units and cascade units, the cascade unit is used to cascade the features output by the dual-path fusion unit and the features output by the bidirectional attention gating unit, and the bidirectional attention gating unit is used to fuse the upsampled features and the features output by the dual-path fusion unit at adjacent levels, and the upsampled features are features obtained by upsampling the features output by the cascade unit or the dual-path fusion unit; An output module, configured to obtain and output a segmentation result based on the features output by the segmentation network; A channel attention encoding module is provided on one path of the dual-path fusion unit to enhance the interdependence of the channel attention map extracted by the mixing unit, and a position attention encoding module is provided on the other path to enhance the spatial information of the features extracted by the Transformer block; The bidirectional attention gating unit includes two bidirectional attention gating modules and a summing module. The two bidirectional attention gating modules are respectively used to enhance the features output by the dual-path fusion unit and the up-sampled features. The summing module is used to perform matrix addition on the features output by the two bidirectional attention gating modules.

2. The choroidal neovascularization segmentation system in optical coherence tomography images according to claim 1, wherein: The hybrid unit includes a connected residual convolutional network and a Transformer network.

3. The choroidal neovascularization segmentation system in optical coherence tomography images according to claim 1, wherein: The bidirectional attention gating module is provided with a first pooling layer, a dot product layer and a second pooling layer. The first pooling layer and the second pooling layer are both provided with a maximum pooling branch and an average pooling branch. The first pooling layer is used to perform pooling operations and full connection operations on the features input into the bidirectional attention gating module. The dot product layer is used to perform matrix dot product of the features output by the first pooling layer and the features input into the bidirectional attention gating module. The second pooling layer is used to perform pooling operations on the features output by the dot product layer.

4. The choroidal neovascularization segmentation system in optical coherence tomography images according to claim 1, wherein: The loss functions used in training the segmentation network include: lesion classification loss, boundary classification loss and Dice loss.

5. A method for segmenting choroidal neovascularization in optical coherence tomography images, characterized in that: A pre-trained segmentation network, wherein the segmentation network is the segmentation network in the choroidal neovascularization segmentation system in optical coherence tomography images according to any one of claims 1 to 4; Input the optical coherence tomography image into the trained segmentation network; Get the features of the trained segmentation network output; Based on the features, a segmentation result is obtained and output.

6. The method for segmenting choroidal neovascularization in optical coherence tomography images according to claim 5, wherein: The loss function for training the segmentation network is: ， ， ， ， in is the lesion classification loss, is the boundary classification loss, is the Dice loss, and Respectively represent pixels are the true annotations of the lesion and the lesion boundary, and Respectively represent The predicted value of pixels belonging to the lesion and the lesion boundary, 、 are the weights of lesion classification loss, boundary classification loss, and Dice loss, respectively.

7. An intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a choroidal neovascularization segmentation program in an optical coherence tomography image stored in the memory and runnable on the processor. When the choroidal neovascularization segmentation program in an optical coherence tomography image is executed by the processor, the steps of the choroidal neovascularization segmentation method in an optical coherence tomography image as described in claim 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a choroidal neovascularization segmentation program in an optical coherence tomography image. When the choroidal neovascularization segmentation program in an optical coherence tomography image is executed by a processor, the steps of the choroidal neovascularization segmentation method in an optical coherence tomography image as described in claim 5 are implemented.

Images