Meibomian gland area and meibomian gland atrophy area segmentation device and method based on CNN
By introducing information fusion module and parallel path connection module in the CNN model, the problem of missing details and noise in the U-shaped codec network is solved, and the segmentation performance of meibomian gland area and meibomian gland atrophy area is improved, and more Good feature recovery quality.
Patent Information
- Application Number
- CN202310106210.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-13
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-02-13
AI Technical Summary
The U-shaped codec network leads to the loss of detailed information during the downsampling process, and the traditional U-Net network introduces noise through jump connections, affecting the quality of the decoder's recovery of features, especially in the meibomian gland area and meibomian gland atrophy area. There are challenges in segmentation tasks.
A meibomian gland region and meibomian gland atrophy region segmentation model is designed based on CNN, using feature encoding module, information fusion module and parallel path connection module, to enhance the high-level semantic information of shallow features and fill in deep features through the information fusion module. The spatial position information of the system is replaced by a parallel path connection module to suppress background noise and improve the quality of the decoder output characteristics.
Effectively solve the problem of missing details and introducing noise in the U-shaped codec network, and improve the segmentation performance of meibomian gland area and meibomian gland atrophy area. Through the combination of information fusion module and parallel path connection module, improve the The network segmentation performance of the target and the decoder feature recovery quality are obtained.
Smart Images

Figure CN116205934B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image segmentation, and in particular to a CNN-based meibomian gland region and meibomian gland atrophy region segmentation device and method. Background Art
[0002] Image segmentation is the process of assigning labels to pixels in an image, and is the basis of image analysis and understanding. Traditional image segmentation methods mainly include: threshold-based segmentation methods, edge-based segmentation methods, region-based segmentation methods, and graph-based methods. In 2014, the Fully Convolutional Network (FCN) was born, pioneering the application of convolutional neural networks in semantic segmentation, and image segmentation officially entered the era of deep learning. In 2015, the equally milestone U-Net was born, and its special U-shaped structure became widely popular in the field of medical image segmentation. Most of the subsequent networks used in medical image segmentation borrowed the encoder-decoder structure of this model, using jump connections to fuse shallow features while upsampling feature maps.
[0003] Meibomian Gland Dysfunction (MGD) is the main cause of dry eye disease. The degree of meibomian gland atrophy is of great significance for the clinical diagnosis of MGD. In the segmentation tasks of Meibomian Gland Area (MGA) and Meibomian Gland Atrophy Area (MGAA), MGA and MGAA are challenging due to their blurred boundaries and diverse shapes. In the medical image segmentation tasks, networks represented by U-Net perform outstandingly and can achieve good results in many segmentation tasks. Although U-Net has achieved good segmentation results with its special U-type encoder-decoder structure and novel jump connection method, it still has the following shortcomings:
[0004] (1) Although U-Net’s downsampling of feature maps enables the convolutional network to have a larger receptive field, as the network depth increases, continuous downsampling will lead to the loss of detailed information.
[0005] (2) The skip connection operation is essentially to better compensate for the information loss caused by the downsampling operation during the encoding process, and also to better assist the decoder in recovering feature information. However, simply using a skip connection to fuse the features extracted by the encoder with the feature information output by the decoder will introduce background noise or useless information in the shallow features, which will affect the quality of the final decoder output features.
[0006] Therefore, there is an urgent need for a model and method that can correctly segment the meibomian gland area and the meibomian gland atrophy area. Summary of the invention
[0007] To this end, an embodiment of the present invention provides a CNN-based meibomian gland region and meibomian gland atrophy region segmentation model and method, which is used to solve the problem of loss of detail information caused by downsampling in the U-type codec network in the prior art and the problem that the traditional U-Net network introduces noise in the jump connection, resulting in the deterioration of the decoder's feature recovery quality.
[0008] In order to solve the above problems, an embodiment of the present invention provides a CNN-based meibomian gland region and meibomian gland atrophy region segmentation model, which includes:
[0009] Feature encoding module, used to extract feature maps from original images;
[0010] An information fusion module (IFM), connected to the feature encoding module, is used to optimize the channel and spatial information of the shallow features and deep features of the input feature map, enhance the high-level semantic information of the shallow features and fill in the spatial position information of the deep features;
[0011] A parallel path connection module (PPC), connected to the feature encoding module, for refining the features of the feature graph extracted by the feature encoding module;
[0012] The feature decoding module is connected to the information fusion module IFM and the parallel path connection module PPC respectively, and is used to restore the extracted features to obtain the prediction results.
[0013] Preferably, the feature encoding module includes a total of four layers, and performs two 3x3 convolutions, batch normalization and Relu activation processing after each downsampling; after each downsampling, the number of channels is doubled and the resolution is reduced by half, and the number of input channels is 32, 64, 128, 256, and 512 respectively.
[0014] Preferably, the shallow feature map of the second-to-last layer and the deep feature map of the first-to-last layer of the feature encoding module are used as inputs of the information fusion module IFM.
[0015] Preferably, optimizing the channel and spatial information of the shallow features and deep features of the input feature map, enhancing the high-level semantic information of the shallow features and filling the spatial position information of the deep features specifically includes:
[0016] First, the deep feature map and the shallow feature map are respectively reduced in dimension by 1x1 convolution, and the number of channels of the deep feature map and the shallow feature map are uniformly set to 128 channels; then the deep feature map is restored to the same resolution as the shallow feature map by bilinear interpolation upsampling, and the two feature information are fused and added to obtain richer semantic feature information F. M ;
[0017] In the channel dimension, a channel attention mechanism is introduced to combine the obtained channel information with the semantic feature information F M Multiply them together to get the channel fusion result c y ; The channel attention mechanism includes global average pooling, one-dimensional convolution and Sigmoid activation function;
[0018] In the spatial dimension, we first use global average pooling (GAP) and global maximum pooling (GMP) to pool the deep features x L Perform global and local feature refinement to obtain global features and local features; then concatenate the global features and local features to obtain the spatial information fusion feature F A ; Then embed a multi-scale module with different dilated convolutions to extract multi-scale features, and finally obtain the probability map through Softmax, which is multiplied by the input deep feature map to obtain the spatial fusion result s y ;
[0019] Finally, the input features are fused with the enhanced features to obtain the final output result O y .
[0020] Preferably, the output result c of the information fusion module IFM in the channel fusion mechanism y for:
[0021] c y =Sigmoid(Conv 1x1 (Avg(Conv 1x1 (x H )+Up(Conv 1x1 (x L )))))
[0022] Among them, x L is the deep feature of the penultimate layer of the feature encoding module, x H Represents the shallow feature map of the penultimate layer of the feature encoding module, c y It represents the output result of the information fusion module IFM in the channel fusion mechanism. represents the bilinear interpolation upsampling operation, represents global average pooling, Represents a 1x1 convolution operation.
[0023] Preferably, the final output of the information fusion module IFM can be expressed as:
[0024] O y =x L +s y *c y
[0025] Among them, O y It is the output feature result after the shallow features and deep features of the feature encoding module are optimized by the information fusion module IFM. y is the optimized output result of the information fusion module IFM in the spatial dimension, c y is the optimized output structure of the information fusion module IFM in the channel dimension, x L It is the deep feature of the penultimate layer of the feature encoding module.
[0026] Preferably, the parallel path connection module PPC specifically includes:
[0027] First, use 1x1 convolution to reduce the input feature x in The channel dimension is ; then two residual combinations consisting of 3x3 convolution and 1x1 convolution are used in parallel; finally, the number of channels is restored through 1x1 convolution to obtain the output feature x out .
[0028] Preferably, the feature decoding module includes four layers, and each layer between the feature encoding module and the feature decoding module is connected by a parallel path connection module PPC across the connection layer.
[0029] Preferably, the model adopts a joint segmentation loss function based on cross entropy loss and Dice loss, which is expressed as follows:
[0030] L seg =L CE +αL Dice
[0031] Among them, L CE represents the cross entropy loss, L Dice represents the Dice loss, L seg It represents the joint segmentation loss of cross entropy loss and Dice loss, and α represents the trade-off coefficient between cross entropy loss and Dice loss, which is set to 0.5.
[0032] The embodiment of the present invention further provides a CNN-based method for segmenting a meibomian gland region and a meibomian gland atrophy region, the method comprising:
[0033] S1: Collect different meibomian gland infrared images to form a data set, and randomly divide the data set into training set, validation set and test set;
[0034] S2: constructing a CNN-based segmentation model of the meibomian gland region and meibomian gland atrophy region as described in any of the above;
[0035] S3: Use the training set and validation set to train the model and save the model weights when the loss function converges;
[0036] S4: Load the saved model weights, and use the trained model to predict the meibomian gland area and meibomian gland atrophy area in the image to achieve segmentation.
[0037] It can be seen from the above technical solutions that the present invention has the following advantages:
[0038] The embodiment of the present invention provides a CNN-based meibomian gland area and meibomian gland atrophy area segmentation model and method. The present invention designs an information fusion module IFM that can combine the advantages of deep features and shallow features, which can effectively enhance the high-level semantic information of shallow features and fill in the spatial position information of deep features. The overall fusion improves the network's segmentation performance for the target; in addition, the present invention adopts parallel path connection instead of skip connection, and refines the features extracted by the encoder by stacking several 1x1 convolutions for residual connection, which can effectively suppress background noise; finally, an evaluation was carried out on 505 infrared meibomian gland images to verify the performance of the designed model. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:
[0040] Figure 1 A schematic diagram of a CNN-based segmentation model for meibomian gland area and meibomian gland atrophy area provided in an embodiment;
[0041] Figure 2 It is the overall structure diagram of the information fusion module IFM;
[0042] Figure 3 It is a schematic diagram of the overall structure of the parallel path connection module PPC;
[0043] Figure 4is a schematic diagram of a CNN-based method for segmenting a meibomian gland region and a meibomian gland atrophy region according to an embodiment;
[0044] Figure 5 (a) is the infrared image of the meibomian gland. Figure 5 (b) Gold standard image: meibomian gland area MGA (white), atrophic area MGAA (gray);
[0045] Figure 6 Comparison of the segmentation visualization results of the network model IF-Net designed for the present invention with other networks, meibomian gland area MGA (white), atrophy area MGAA (gray), where Figure 6 (a) is the original image;
[0046] Figure 6 (b) is the gold standard image; Figure 6 (c) is the visualization result of U-Net; Figure 6 (d) is the visualization result of CS2-Net; (e) is the visualization result of GDCSeg-Net; (f) is the visualization result of CE-Net; (g) is the visualization result of IF-Net. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] like Figure 1 As shown, the embodiment of the present invention proposes a CNN-based meibomian gland region and meibomian gland atrophy region segmentation model, the model comprising:
[0049] A feature encoding module (10), used for extracting a feature map from an original image;
[0050] An information fusion module IFM (20), connected to the feature encoding module, is used to optimize the channel and spatial information of the shallow features and deep features of the input feature map, enhance the high-level semantic information of the shallow features and fill in the spatial position information of the deep features;
[0051] A parallel path connection module PPC (30), connected to the feature encoding module, for refining the features of the feature graph extracted by the feature encoding module;
[0052] The feature decoding module (40) is connected to the information fusion module IFM and the parallel path connection module PPC respectively, and is used to restore the extracted features to obtain prediction results.
[0053] The embodiment of the present invention provides a CNN-based meibomian gland region and meibomian gland atrophy region segmentation model, which uses the U-Net structure as the reference network. The model consists of four parts: a feature encoding module, an information fusion module IFM, a parallel path connection module PPC, and a feature decoding module. First, the feature encoding module is used to extract a feature map with rich semantic information from the original image; then the information fusion module IFM is used to optimize the channel and spatial information of the input shallow features and deep features, enhance the high-level semantic information of the shallow features and fill in the spatial position information of the deep features, and the overall fusion improves the network's segmentation performance for the target; secondly, the parallel path connection module PPC is added, and a 1x1 residual convolution is added to the feature information of the encoding stage after each convolution through the residual structure. On the one hand, the semantic information loss can be reduced to a certain extent, and on the other hand, the noise of the feature can be suppressed by stacking multiple 1x1 convolutions, thereby improving the quality of the output features of the decoder. The network model designed by the present invention has achieved good segmentation performance by comparing with other excellent networks, and has effectively solved the problems in the prior art of the loss of detail information caused by downsampling in the U-type codec network and the introduction of noise in the jump connection of the traditional U-Net network, which leads to the deterioration of the quality of feature recovery of the decoder.
[0054] The present invention adopts the U-Net network structure as the reference network and designs a CNN-based segmentation model for the meibomian gland region and the meibomian gland atrophy region, which is called the Information Fusion Network (IF-Net), for realizing the automatic segmentation of the meibomian gland region MGA and the atrophy region MGAA. The network model designed by the present invention is specifically described below.
[0055] The feature encoding module, i.e. the process of extracting feature information by the network, mainly compresses the original image size and gradually extracts feature maps with rich semantic information. In the encoding stage of the present invention, a total of four layers are included. In order to reduce the amount of model parameters, the number of channels in each layer is reduced to 1 / 2 of the corresponding original U-Net channel number, and two 3x3 convolutions, batch normalization and Relu activation processes are performed after each downsampling. After each downsampling, the number of channels is doubled and the resolution is reduced by half. The number of input channels is 32, 64, 128, 256, and 512 respectively. Therefore, the output feature map of the entire feature encoding module is 1 / 8 the size of the original image.
[0056] The overall structure of the information fusion module IFM is as follows: Figure 2As shown. Generally speaking, shallow features focus on detail information such as edges, shapes, textures, etc., but lack high-level semantic features, while deep features have relatively rich high-level semantic information, but lack spatial detail information. In order to enhance the high-level semantic feature information of shallow features while providing richer spatial information for deep features, the present invention designs an information fusion module IFM. The present invention selects the shallow feature map of the second-to-last layer and the deep feature map of the first-to-last layer of the feature encoding module as the input of the information fusion module IFM.
[0057] The channel and spatial information of the shallow and deep features of the input feature map are optimized to enhance the high-level semantic information of the shallow features and fill in the spatial position information of the deep features. Specifically, it includes:
[0058] In order to reduce the computational cost of shallow and deep features, before optimizing the channel and spatial information, the present invention first reduces the number of channels of the deep feature map and the shallow feature map by 1x1 convolution, and reduces the number of channels of the deep feature map and the shallow feature map to 1 / 4 of the original, and uniformly sets them to 128 channels; then, the deep feature map is restored to the same resolution as the shallow feature map by bilinear interpolation upsampling, and the two feature information are fused and added to obtain richer semantic feature information F. M ; However, the feature information obtained in this way is mixed with a lot of irrelevant information. In order to improve the network's ability to accurately identify information and obtain the global dependency of the feature map itself, and establish the correlation between channels, the present invention introduces a channel attention mechanism, which includes global average pooling, one-dimensional convolution and Sigmoid activation function; the present invention uses a one-dimensional convolution to achieve interaction between local channels, extract dependencies, and avoid the destruction of the dependency between learning channels caused by channel compression. Finally, the Sigmoid activation function is used to obtain the probability map of the channel, and the correlation of the channel is judged accordingly. The higher the probability, the higher the correlation of the channel, so as to obtain highly correlated channel information and compare it with the semantic feature information F M Multiply them together to get the channel fusion result c y Therefore, the output result of the information fusion module IFM in the channel fusion mechanism is c y for:
[0059] c y =Sigmold(Conv 1x1 (Avg(Conv 1x1 (x H )+Up(Conv 1x1 (x L )))))
[0060] Among them, x Lis the deep feature of the penultimate layer of the feature encoding module, x H Represents the shallow feature map of the penultimate layer of the feature encoding module, c y It represents the output result of the information fusion module IFM in the channel fusion mechanism. represents the bilinear interpolation upsampling operation, represents global average pooling, Represents a 1x1 convolution operation.
[0061] For the edge features of the meibomian gland area, it is necessary to extract edge, shape, texture and other detail information at different scales from the shallow feature map with rich detail information. In order to extract useful spatial information features of the feature map, global average pooling and global maximum pooling are first used to pool the deep feature x L Perform global and local feature refinement to obtain global features and local features; then concatenate the global features and local features to obtain the spatial information fusion feature F A ; Then embed a multi-scale module composed of three dilated convolutions with different dilation rates (1, 3 and 5) to extract multi-scale features, obtain multi-scale feature information with different receptive fields, and use the Softmax activation function to normalize the spatial dimension to obtain the importance of each pixel relative to the region in the space, and then multiply it with the input deep feature map, weight the importance of each pixel, and aggregate to obtain the prediction result of the object area. The purpose of this is to filter the information of the feature map, give priority to pixels with high importance, that is, useful spatial features, and weaken pixels with low importance, that is, useless feature information. Finally, the spatial fusion result s is obtained according to the features. y .
[0062] Finally, the input features are fused with the enhanced features to obtain the final output result O y , which is expressed as:
[0063] O y =x L +s y *c y
[0064] Among them, O y It is the output feature result after the shallow features and deep features of the feature encoding module are optimized by the information fusion module IFM. y is the optimized output result of the information fusion module IFM in the spatial dimension, c y is the optimized output structure of the information fusion module IFM in the channel dimension, x L It is the deep feature of the penultimate layer of the feature encoding module.
[0065] The overall structure of the parallel path connection module PPC is as follows Figure 3 The parallel path connection module PPC specifically includes: firstly, using 1x1 convolution to reduce the input feature x in The channel dimension is reduced to 1 / 4 of the original number of channels; then two residual combinations consisting of 3x3 convolution and 1x1 convolution are used in parallel to extract more detailed feature information, which can make up for the loss of semantic information in the jump connection process to a certain extent; finally, the number of channels is restored through 1x1 convolution to obtain the output feature x out The features extracted by the encoder are refined by stacking multiple 1x1 convolutions, which can effectively suppress background noise. The present invention introduces the idea of residual structure, and after residual optimization processing of 1x1 convolution, the time cost is greatly reduced, which speeds up the learning speed of the network.
[0066] The feature decoding stage, that is, the process of recovering the features extracted by the encoding by the network, is achieved by upsampling the features extracted by the encoder to a fixed resolution size and completing pixel-level prediction and classification. The feature decoding module of the present invention corresponds to the feature encoding module, and also includes four layers. Each layer of the cross-connection layer between the feature encoding module and the feature decoding module is connected using a parallel path connection module PPC. For the decoder feature recovery stage of each layer, on the one hand, the features extracted by the encoder are respectively passed through the parallel path connection module PPC to suppress noise and retain more semantic features. On the other hand, the features extracted by the encoder are restored to the same resolution as the features of the encoder through a bilinear interpolation upsampling operation, and then the two feature maps are spliced together, and the information at different levels is fused as the input of the next layer of decoding stage. Finally, a prediction result with the same resolution as the original image is obtained to update the model parameters.
[0067] In addition, the present invention adopts a joint segmentation loss function based on cross entropy loss and Dice loss, which is expressed as follows:
[0068] L seg =L CE +αL Dice
[0069] Among them, L CE represents the cross entropy loss, L Dice represents the Dice loss, L seg It represents the joint segmentation loss of cross entropy loss and Dice loss, and α represents the trade-off coefficient between cross entropy loss and Dice loss, which is set to 0.5.
[0070] like Figure 4 As shown, the embodiment of the present invention further provides a method for segmenting a meibomian gland region and a meibomian gland atrophy region based on CNN, the method comprising:
[0071] S1: Collect different meibomian gland infrared images to form a data set, and randomly divide the data set into training set, validation set and test set;
[0072] S2: constructing a CNN-based segmentation model of the meibomian gland region and meibomian gland atrophy region as described in any of the above;
[0073] S3: Use the training set and validation set to train the model and save the model weights when the loss function converges;
[0074] S4: Load the saved model weights, and use the trained model to predict the meibomian gland area and meibomian gland atrophy area in the image to achieve segmentation.
[0075] The method is based on the above-designed CNN-based meibomian gland area and meibomian gland atrophy area segmentation model to achieve automatic segmentation of the meibomian gland area MGA and the atrophy area MGAA, which will not be described here to avoid redundancy.
[0076] The advantages of the method of the present invention will be described below through specific experiments.
[0077] 1. Experimental Dataset
[0078] The dataset comes from the First Affiliated Hospital of Soochow University, with a total of 505 infrared images of meibomian glands, and the image size is 1024×1280. All images come from different eyes of 300 patients with different degrees of atrophy. The present invention randomly divides the 505 images into a training set (303), a validation set (101) and a test set (101). In order to reduce computational overhead, the image size is adjusted to 512×512 and input into the network. In order to increase the diversity of the data, data enhancement strategies such as horizontal and vertical flipping and rotation (-45 degrees to 45 degrees) are adopted.
[0079] 2. Experimental Evaluation Indicators
[0080] The degree of meibomian gland atrophy is one of the important indicators for evaluating meibomian gland dysfunction. The atrophy rate of meibomian glands (Glands Drop, GD) can be calculated by the ratio of the area of meibomian gland atrophy to the sum of the area of meibomian gland and atrophy (i.e., eyelid area). Figure 5 In (b), assuming that the area of the meibomian gland area MGA is S1 (white area) and the area of the meibomian gland atrophy area MGAA is S2 (gray area), the atrophy rate of the meibomian gland GD is:
[0081]
[0082] The grading standard for the degree of meibomian gland atrophy was set according to the above formula, as shown in Table 1.
[0083] Table 1
[0084] MG shrinkage area ratio (%) Meiboscore 0 0 1-33 1 34-67 2 68-100 3
[0085] In order to quantitatively evaluate the segmentation performance of the network model designed by the present invention for MGA and MGAA, the present invention uses two commonly used segmentation evaluation indicators, Dice Similarity Coefficient (DSC) and Intersection over Union (IoU), as well as the error (Root Mean Square Deviation, RMSD) generated when predicting the percentage of image shrinkage. The formula is as follows:
[0086]
[0087] Among them, TP is the number of true positive pixels (i.e., the pixels in the meibomian gland area and the atrophic area are correctly segmented), TN is the number of false positive pixels (i.e., the background pixels are correctly segmented), FP is the number of false positive pixels (i.e., the background pixels are incorrectly segmented into pixels in the meibomian gland area and the atrophic area), FN is the number of false negative pixels (i.e., the pixels in the meibomian gland area and the atrophic area are incorrectly segmented into background pixels), and A i represents the shrinkage rate of the i-th image predicted by the neural network, It represents the actual shrinkage rate of the i-th image calculated by the formula, and N is the total number of images in the test set.
[0088] 3. Experimental results and analysis
[0089] Table 2 shows the comparison of the segmentation evaluation indicators of the network model IF-Net designed by the present invention and other network models in the meibomian gland area (MGA) and atrophy area (MGAA) as well as the ablation experiment results. As shown in Table 2, CE-Net uses multiple convolution branches with different receptive fields to improve the model's ability to obtain multi-scale information. Although the addition of multi-scale information can improve the model's attention to targets of different sizes, multiple convolutions introduce a large number of parameters, which increases the computational cost of the model. Although CS2-Net introduces a strip pooling kernel, which can effectively help the network recognize strip targets, it performs poorly for regional segmentation targets in the meibomian gland area and atrophy area. GDCSeg-Net solves the problem of semantic information and spatial information imbalance in feature fusion from a global and local perspective, respectively, strengthens the network's ability to extract global features, long-distance feature dependency modeling, and feature spatial information recovery, and improves the network's performance to a certain extent. The average Dice and average IoU are 0.89% and 1.17% higher than the baseline network, respectively, but too many parameters are introduced in the feature fusion stage.
[0090] Table 2
[0091]
[0092] The network model IF-Net designed by the present invention outperforms other methods in all evaluation indicators in the segmentation of the meibomian gland region and the atrophic region. In order to verify the effectiveness of the information fusion module IFM and the parallel path connection module PPC designed by the present invention in the segmentation of the meibomian gland region and the atrophic region, the present invention conducted an ablation experiment as shown in Table 3. For fairness and rationality, the same hyperparameters were used for training and testing in the ablation experiment.
[0093] Table 3
[0094]
[0095] From the experimental results of adding the information fusion module IFM (baseline network + IFM) alone in Table 3, it can be seen that all indicators are better than the baseline network. In the performance of Dice and IoU, the segmentation of the meibomian gland area MGA is improved by 0.81% and 1.10% respectively, and the segmentation of the atrophic area MGAA is improved by 0.93% and 2.04% respectively, and the average improvement is 0.86% and 1.03% respectively. This is mainly due to the introduction of the attention mechanism by the information fusion module IFM, which integrates the richer semantic features in the high-level features and the richer spatial information in the low-level features, enhances the semantic feature information of the low-level features and filters the feature channel information, so that the network can focus more on the lesion area and improve the recognition ability of the detail features. It can be seen from the experiment that the performance improvement is more obvious in the segmentation of the atrophic area. This is because the network will pay more attention to the detail features after the information fusion module IFM is inserted. In addition, the area of the meibomian gland area is larger and the difficulty in segmentation is less, so the improvement may not be very obvious.
[0096] From the experimental results of adding the parallel path connection module PPC (baseline network + PPC) alone, it can be seen that in the performance of Dice and IoU, the segmentation of the meibomian gland area MGA increased by 0.31% and 0.16% respectively, and in the atrophic area by 0.82% and 1.39% respectively, and on average by 0.55% and 0.73% respectively. The parallel path connection module PPC with a residual structure can retain more detailed feature information during the cross-connection process, make up for the lack of semantic information, and effectively deal with the shortcomings of the decoder feature recovery quality caused by the introduction of background noise in simple jump connections.
[0097] Finally, the best results were achieved in the experiment combining the information fusion module IFM and the parallel path connection module PPC (baseline network + IFM + PPC). In the segmentation of the meibomian gland area, Dice and IoU increased by 1.21% and 1.40% respectively, and in the segmentation of the atrophic area, they increased by 1.90% and 2.80% respectively, and on average increased by 1.53% and 2.04% respectively. This shows that both the information fusion module IFM and the parallel path connection module improve the segmentation performance of the network, while introducing fewer parameters and increasing the learning speed of the network.
[0098] When you get Figure 6 After the segmentation results are obtained, the meibomian gland atrophy area ratio is calculated according to the above formula, and the classification is performed according to the classification standard provided in Table 1. Since the images in the data set used by the present invention all have different degrees of meibomian gland atrophy, that is, there is no sample of level 0, the final classification results are shown in Table 3. From the accuracy (Acc) in Table 3, it can be seen that the method designed by the present invention can obtain the highest average accuracy of 88.10%, and performs best in both atrophy degree levels 1 and 2.
[0099] In summary, the present invention analyzes the segmentation of the meibomian gland region and the atrophic region and the grading of the atrophy degree, and verifies the designed method. The present invention designs a CNN-based meibomian gland region and meibomian gland atrophy region segmentation model IF-Net. The present invention designs an information fusion module IFM that can combine the advantages of deep features and shallow features, which can effectively enhance the high-level semantic information of shallow features and fill in the spatial position information of deep features, and the overall fusion improves the network's segmentation performance of the target. In addition, ordinary jump connections only directly combine the features extracted by the encoder with the decoder features, which will inevitably mix the background noise of the shallow features, which will affect the quality of feature recovery by the decoder to a certain extent. Therefore, the present invention adopts parallel path connection instead of jump connection, and refines the features extracted by the encoder by stacking several 1x1 convolutions for residual connection, which can effectively suppress the background noise. The network model IF-Net designed by the present invention is compared with other superior networks, and better segmentation results are achieved. The experimental results verify the effectiveness of the method designed by the present invention.
[0100] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
[0101] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0104] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.
Claims
1. A CNN-based segmentation device for meibomian gland area and meibomian gland atrophy area, characterized in that: include: Feature encoding module, used to extract feature maps from original images; The information fusion module IFM is connected to the feature encoding module and is used to optimize the channel and spatial information of the shallow features and deep features of the input feature map, enhance the high-level semantic information of the shallow features and fill in the spatial position information of the deep features, specifically including: First, the deep feature map and the shallow feature map are respectively reduced in dimension by 1x1 convolution, and the number of channels of the deep feature map and the shallow feature map are uniformly set to 128 channels; then the deep feature map is restored to the same resolution as the shallow feature map by bilinear interpolation upsampling, and the two feature information are fused and added to obtain the semantic feature information F M ; In the channel dimension, a channel attention mechanism is introduced to combine the obtained channel information with the semantic feature information F M Multiply them together to get the channel fusion result c y ; The channel attention mechanism includes global average pooling, one-dimensional convolution and Sigmoid activation function; In the spatial dimension, global average pooling and global maximum pooling are first used to pool the deep features x L Perform global and local feature refinement to obtain global features and local features; then concatenate the global features and local features to obtain the spatial information fusion feature F A ; Then embed a multi-scale module with different dilated convolutions to extract multi-scale features, and finally obtain the probability map through Softmax, which is multiplied by the input deep feature map to obtain the spatial fusion result s y ; Finally, the input features are fused with the enhanced features to obtain the final output result O y ; The output result of the information fusion module IFM in the channel fusion mechanism is c y for: c y =Sigmoid(Conv 1x1 (Avg(Conv 1x1 (x H )+Up(Conv 1x1 (x L ))))) Among them, x L is the deep feature of the penultimate layer of the feature encoding module, x H Represents the shallow feature map of the penultimate layer of the feature encoding module, c y It represents the output result of the information fusion module IFM in the channel fusion mechanism, Up() represents the bilinear interpolation upsampling operation, Avg() represents the global average pooling, Conv 1x1 () represents a 1x1 convolution operation; The final output of the information fusion module IFM is expressed as: O y =x L +s y *c y Among them, O y It is the output feature result after the shallow features and deep features of the feature encoding module are optimized by the information fusion module IFM. y is the optimized output result of the information fusion module IFM in the spatial dimension, c y is the optimized output structure of the information fusion module IFM in the channel dimension, x L It is the deep feature of the penultimate layer of the feature encoding module; A parallel path connection module PPC, connected to the feature encoding module, is used to refine the features of the feature graph extracted by the feature encoding module; The feature decoding module is connected to the information fusion module IFM and the parallel path connection module PPC respectively, and is used to restore the extracted features to obtain the prediction results.
2. The CNN-based meibomian gland area and meibomian gland atrophy area segmentation device according to claim 1, characterized in that: The feature encoding module includes four layers in total, and performs two 3x3 convolutions, batch normalization and ReLU activation processes after each downsampling; after each downsampling, the number of channels doubles and the resolution decreases by half, and the number of input channels is 32, 64, 128, 256, and 512 respectively.
3. The CNN-based meibomian gland area and meibomian gland atrophy area segmentation device according to claim 1, characterized in that: The shallow feature map of the second-to-last layer and the deep feature map of the first-to-last layer of the feature encoding module are used as inputs of the information fusion module IFM.
4. The CNN-based meibomian gland area and meibomian gland atrophy area segmentation device according to claim 1, characterized in that: The parallel path connection module PPC specifically includes: First, use 1x1 convolution to reduce the input feature x in The channel dimension is ; then two residual combinations consisting of 3x3 convolution and 1x1 convolution are used in parallel; finally, the number of channels is restored through 1x1 convolution to obtain the output feature x out .
5. The CNN-based meibomian gland area and meibomian gland atrophy area segmentation device according to claim 1, characterized in that: The feature decoding module includes four layers, and each layer between the feature encoding module and the feature decoding module is connected by a parallel path connection module PPC across the connection layer.
6. The CNN-based meibomian gland area and meibomian gland atrophy area segmentation device according to claim 1, characterized in that: The device adopts a joint segmentation loss function based on cross entropy loss and Dice loss, which is expressed as follows: L seg =L CE +αL Dice Among them, L CE represents the cross entropy loss, L Dice represents the Dice loss, L seg It represents the joint segmentation loss of cross entropy loss and Dice loss, and α represents the trade-off coefficient between cross entropy loss and Dice loss, which is set to 0.
5.
7. A CNN-based method for segmenting meibomian gland regions and meibomian gland atrophy regions, characterized in that: include: S1: Collect different meibomian gland infrared images to form a data set, and randomly divide the data set into training set, validation set and test set; S2: constructing a CNN-based meibomian gland area and meibomian gland atrophy area segmentation device as described in any one of claims 1 to 6; S3: Train the device using the training set and the validation set, and save the device weight when the loss function converges; S4: Load the saved device weights, and use the trained device to predict the meibomian gland area and meibomian gland atrophy area in the image, thereby achieving segmentation.
