Visual cup optic disk region generalization segmentation method and device and electronic equipment
In the visual cup disc segmentation task, using technical means such as YUV color space, homomorphic filtering, curvature filtering, feature mixing and multi-scale feature extraction, the model's generalization ability in the target domain data with large distribution differences is solved, and more efficient visual cup disc segmentation performance is achieved.
Patent Information
- Application Number
- CN202510077888.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-17
- Publication Date
- 2025-05-16
AI Technical Summary
In the prior art, in the segmentation task of view cup discs, when the model is applied to target domain data with large distribution differences, the generalization ability is poor, resulting in poor performance.
Low-level features are extracted by converting the fundus image to the YUV color space, and homomorphic filtering and curvature filtering are performed to extract high-frequency features. Then, through feature mixing and multi-scale feature extraction, advanced features with multi-scale information are generated, and feature adjustments are performed using the domain knowledge base to finally generate output features for view cup view disc segmentation.
The model's perception of the detailed characteristics of the visual cup disc is improved, the diversity and robustness of the features are enhanced, and the generalization ability and segmentation performance of the model are significantly improved.
Smart Images
Figure CN120014265A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image segmentation, and in particular to a generalized segmentation method, device, electronic device and computer-readable storage medium for optic cup and optic disc domain. Background Art
[0002] With the development of image recognition technology, it has also begun to be widely used in the medical field. For example, the study of optic cups and discs in fundus images plays a key role in the diagnosis of diseases. The changes in the morphology and structure of fundus images can directly reflect lesions such as glaucoma. It is of great research value to use deep learning methods to automatically segment fundus images. However, traditional convolutional neural networks rely on large-scale data sets and professional annotations, and only perform well on specific data sets, and have poor generalization on some other data sets. At the same time, due to different acquisition conditions, equipment and environments, different data sets have significant differences in image appearance, contrast, quality, field of view, etc. The trained model overfits the characteristics of the source domain data, so when the model is applied to target domain data with large distribution differences, the performance is poor. Summary of the invention
[0003] In view of the above problems, embodiments of the present application provide a method, device, electronic device and computer-readable storage medium for generalized segmentation of optic cup and optic disc domain, so as to improve the generalization ability of segmenting optic cup and optic disc model.
[0004] On the one hand, the present application discloses a generalized segmentation method for optic cup and optic disc domain, comprising:
[0005] Converting the color space of the fundus images in different domains into a YUV color space, and performing feature extraction on the fundus images in different domains based on the YUV color space to obtain first low-level features of the fundus images in different domains;
[0006] Performing homomorphic filtering on the first low-level feature and extracting high-frequency components thereof to obtain high-frequency features;
[0007] Performing curvature filtering on the high-frequency features to obtain second low-level features;
[0008] Performing internal mixing of the mean and the standard deviation on the first low-level feature to generate a third low-level feature in a different domain from the fundus image;
[0009] Extract features from the third low-level features to obtain first high-level features;
[0010] Extract features at different scales from the first high-level features, and then merge the features at different scales to obtain a second high-level feature with multi-scale information;
[0011] encoding the second high-level feature into a code specific to the domain in which the second high-level feature resides;
[0012] Using the multi-domain features of the domain knowledge base and the code, adjusting the second high-level feature to obtain an adjusted third high-level feature;
[0013] The third high-level feature is combined with the second high-level feature and the second low-level feature to generate a final output feature.
[0014] Optionally, the process of performing homomorphic filtering on the first low-level feature and extracting its high-frequency component includes:
[0015] Performing a logarithmic transformation on the first low-level feature to obtain a transformed first feature;
[0016] Performing Fourier transform on the first feature and centering it to obtain a second feature;
[0017] converting the second feature from the spatial domain to the frequency domain;
[0018] Performing filtering processing on the second feature in the frequency domain to obtain a third feature with high frequency enhancement;
[0019] Performing an inverse Fourier transform on the third feature to convert the third feature into a spatial domain;
[0020] An exponential transformation is performed on the third feature in the spatial domain, and a high-frequency component in the third feature is extracted to obtain the high-frequency feature.
[0021] Optionally, the process of performing curvature filtering on the high-frequency features to obtain the second low-level features further includes:
[0022] Filtering out a first channel with the largest curvature from each channel of the high-frequency feature;
[0023] The first channels of all the high-frequency features are concatenated to obtain the second low-level features.
[0024] Optionally, the process of selecting a first channel with the largest curvature from each channel of the high-frequency feature further includes:
[0025] Rearranging the high-frequency features, splitting the multiple channels into single channels, and obtaining channel features of each single channel of the high-frequency features;
[0026] Convolving each channel feature of the high-frequency feature to obtain a convolved feature map;
[0027] Summing the feature map in the width dimension, calculating the horizontal cumulative curvature of all pixels in each channel;
[0028] Summing the feature map in the height dimension, calculating the vertical cumulative curvature of all pixels in each channel;
[0029] The vertical curvature is further accumulated on the horizontal cumulative curvature of all pixels in each channel to obtain the total curvature intensity of each channel;
[0030] According to the total curvature intensity of each channel, the first channel with the largest curvature is screened out.
[0031] Optionally, the process of performing internal mixing of the mean and the standard deviation of the first low-level feature to generate a third low-level feature in a domain different from the fundus image includes:
[0032] Calculate the mean and standard deviation corresponding to each of the first low-level features according to the channel dimension of each of the first low-level features;
[0033] The mean of each of the first low-level features is merged with the mean of another different feature through a proportional coefficient to obtain a fused mean;
[0034] The standard deviation of each of the first low-level features is merged with the standard deviation of another different picture through a proportional coefficient to obtain a fused standard deviation;
[0035] The third low-level feature corresponding to each first low-level feature is obtained according to the fusion mean and the fusion standard deviation.
[0036] Optionally, after generating the final output features, it also includes:
[0037] The final output features are subjected to upsampling and convolution operations to generate a segmentation result of the optic cup and optic disc.
[0038] On the other hand, the present application also discloses a generalized segmentation device for optic cup and optic disc domain, comprising:
[0039] A low-level feature extraction module, used for converting the color space of the fundus images of different domains into a YUV color space, and performing feature extraction on the fundus images of different domains based on the YUV color space to obtain first low-level features of the fundus images of different domains;
[0040] A high-frequency feature extraction module, used for performing homomorphic filtering on the first low-level feature and extracting its high-frequency component to obtain a high-frequency feature;
[0041] A curvature filtering module, used for performing curvature filtering on the high-frequency features to obtain second low-level features;
[0042] A feature mixing module, used for performing internal mixing of the mean and the standard deviation of the first low-level feature to generate a third low-level feature in a different domain from the fundus image;
[0043] The high-level feature extraction module is used to extract features from the third low-level features to obtain first high-level features.
[0044] A multi-scale feature extraction module, used to extract features at different scales from the first high-level features, and then merge the features at different scales to obtain a second high-level feature with multi-scale information;
[0045] An encoding module, used for encoding the second high-level feature into a code specific to the domain to which it belongs;
[0046] A feature adjustment module, used to adjust the second high-level feature by using the multi-domain features of the domain knowledge base and the code to obtain an adjusted third high-level feature;
[0047] A feature output module is used to combine the third high-level feature with the second high-level feature and the second low-level feature to generate a final output feature.
[0048] Optionally, also include:
[0049] The segmentation result output module is used to perform upsampling and convolution operations on the final output features to generate a segmentation result of the optic cup and optic disc.
[0050] On the other hand, the present application also discloses an electronic device, which includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the aforementioned generalized segmentation method for optic cup and optic disc domain.
[0051] On the other hand, the present application further discloses a computer-readable storage medium, including a program code. When the storage medium is run on an electronic device, the program code is used to enable the electronic device to execute the above-mentioned generalized segmentation method for optic cup and optic disc domain.
[0052] The present application not only enhances the diversity of features but also makes the model focus on the detailed features of the optic cup and optic disc, thereby improving the generalization ability of the model by extracting features in the YUV color space, mixing the low-level features generated in the backbone network, and strengthening the low-level features separately outside the backbone network. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0054] Figure 1This is a flow chart of a generalized segmentation method for the optic cup and optic disc domain in an embodiment of the present application;
[0055] Figure 2 A schematic diagram of a network structure of a generalized segmentation method for the optic cup and optic disc domain in an embodiment of the present application;
[0056] Figure 3 This is a schematic diagram of a MobilenetV2 backbone network design in an embodiment of the present application;
[0057] Figure 4 This is a schematic diagram of a homomorphic filtering and curvature combination process in an embodiment of the present application;
[0058] Figure 5 This is a schematic diagram of the optic cup and optic disc segmentation results on four data sets in an embodiment of the present application;
[0059] Figure 6 This is a schematic diagram of the structure of a generalized segmentation device for the optic cup and optic disc domain in an embodiment of the present application;
[0060] Figure 7 A schematic diagram of the hardware structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.
[0062] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.
[0063] At present, in order to overcome the problem of poor cross-domain segmentation performance of medical images, various domain generalization segmentation schemes have been proposed. According to different processing schemes, there are many methods such as meta-learning, data enhancement, domain alignment, and regularization strategies. Data enhancement method is one of the most widely used domain generalization methods. By expanding the source domain data to simulate the distribution of the target domain data, the domain offset between the source domain and the target domain is reduced. For example, some scholars use Fourier transform to capture frequency domain information from the source domain image, combine different frequency domain information to generate new composite images as additional enhanced images of the source domain data, and jointly train the segmentation model to improve the generalization and robustness of the model. Other scholars have proposed a new deep stacking transformation method, which simulates domain shift by performing data enhancement on the source domain data distribution. The segmentation model trained with the enhanced data can effectively summarize the data distribution of the unknown domain. This method allows the model to obtain better results on the unknown domain by expanding the data without further training.
[0064] The optic cup and optic disc segmentation method based on convolutional neural network has made great progress, but there are significant style differences in the acquisition conditions, equipment and environment of images in different domains. The model is easy to overfit the style features of the source domain, resulting in poor generalization ability. To solve the above problems, some novel domain generalization methods have been developed in the field of deep learning images, but there is still a problem of insufficient accuracy, and there is still a lot of room for improvement. For example, in cross-domain images, low-frequency noise (such as uneven illumination) will mask the details, and these details (such as the edge of the optic cup and optic disc) are crucial in the segmentation task. Secondly, the existing model does not consider that conventional deep learning feature extraction is insensitive to local geometric capture. However, local geometric information, such as the boundary information change points of the optic cup and optic disc and lesions, is of great significance for distinguishing normal and abnormal areas. Therefore, how to overcome the interference of multi-domain fundus images in background, noise, style, etc., and further improve the pertinence and effectiveness of multi-domain feature extraction and utilization, is still the difficulty and focus of current research.
[0065] In order to solve the problem that the trained model has poor performance when applied to target domain data with large distribution differences, the present application embodiment discloses a generalized segmentation method for optic cup and optic disc domain, see Figure 1 and then Figure 2 As shown, the method can be applied to the identification of optic cup and optic disc in the medical field, including:
[0066] S101: converting the color space of the fundus images in different domains into the YUV color space, and performing feature extraction on the fundus images in different domains based on the YUV color space to obtain the first low-level features of the fundus images in different domains.
[0067] Since there is a strong correlation between channels in the existing RGB color space, this may cause the model to learn redundant information, while in the YUV color space, brightness and chromaticity are separated, and the correlation between channels is significantly reduced, making model training more efficient. Therefore, the RGB color space of the samples is uniformly converted to the YUV color space.
[0068] After converting the samples into YUV color space, they are fed into the model for feature extraction, so that the model can focus on edge feature extraction and feature extraction of color change areas. This is because brightness information helps capture structural features, while chromaticity information is more sensitive to color differences in different areas. At the same time, it also reduces the interference of illumination changes on feature extraction and improves the model's adaptability to multi-domain images.
[0069] When extracting features of the optic cup and optic disc, the model can focus on extracting the shape and edge features of the optic cup and optic disc and extracting features of color changes in the lesion area.
[0070] In order to perform better deep feature extraction on images in the future, we can first extract low-level features of images from a batch of data sets. Low-level features can refer to the basic information directly extracted from the original data of the image. Low-level features are related to the physical properties of the image and contain the basic structure and texture information of the image. Common low-level features include: 1. Edge features: describe the areas where the grayscale value changes sharply in the image; 2. Corner features: describe the points where the edges cross or the curvature changes greatly in the image; 3. Texture features: describe the repeated patterns or local structures in the image; 4. Color features: describe the distribution of colors in the image.
[0071] For example, a batch of fundus images are input into a convolutional neural network, the convolutional neural network extracts features from the input images, and outputs low-level features of the images. The first low-level features can be expressed in the form of a feature map (hereinafter referred to as the first low-level feature map), such as the convolutional neural network outputs the first low-level feature map x(24,24,64,64), where: the first 24 is the batch size, the second 24 is the number of channels, and 64 and 64 are the width and height of the feature map.
[0072] The sub-network composed of the first four convolutional layers of the MobilenetV2 backbone network can be used to extract low-level features of fundus images in different domains based on the YUV color space. The MobilenetV2 backbone network and feature hybrid are as follows: Figure 3 shown.
[0073] S102: Perform homomorphic filtering on the first low-level feature and extract its high-frequency component to obtain a high-frequency feature;
[0074] S103: Perform curvature filtering on the high-frequency features to obtain the second low-level features.
[0075] The first low-level features can be enhanced by combining homomorphic filtering and curvature. The main function of homomorphic filtering is to enhance the details in the image and improve the contrast, especially by separating the brightness and reflection components to highlight the local information. Curvature calculation is mainly used to measure the intensity of local geometric changes and highlight significant high-frequency features and important structures. Direct curvature calculation of the original image may contain a large number of meaningless flat area features. First extracting the high-frequency part through homomorphic filtering and then calculating the curvature can significantly reduce invalid calculations and improve efficiency. At the same time, the high-frequency components obtained by homomorphic filtering have already paid attention to the details, and the curvature calculation can further focus on significant areas, such as edges and densely textured areas, and improve the model's perception of these important features. More importantly, the high-frequency components may contain a lot of noise or invalid information. Through curvature calculation, truly meaningful high-frequency features can be screened out to further reduce data redundancy.
[0076] S102 and S103 correspondence Figure 2 Medium HFAC module.
[0077] S104: Perform internal mixing of the mean and the standard deviation on the first low-level feature to generate a third low-level feature in a different domain from the fundus image.
[0078] In order to increase the diversity of features, the first low-level features of each image are merged with the first low-level features of other different images to obtain new third low-level features.
[0079] Further, the process of S104 in the embodiment of the present disclosure may specifically include S1041 to S1044, see Figure 2 As shown, it corresponds to the Fmix part.
[0080] S1041: Calculate the mean and standard deviation corresponding to each first low-level feature according to the channel dimension of each first low-level feature.
[0081] For each low-level feature map (24, 64, 64) of each image, the mean μ and standard deviation σ are calculated according to the channel dimension. Then the feature map is normalized to zero mean and unit variance according to the channel. At the same time, the 24 batch samples are randomly shuffled to obtain the rearranged index p of each sample and the corresponding shuffled mean μ p and standard deviation σ p .
[0082] S1042: fusing the mean of each first low-level feature with the mean of another different feature through a proportional coefficient to obtain a fused mean;
[0083] S1043: fusing the standard deviation of each first low-level feature with the standard deviation of another different picture through a proportional coefficient to obtain a fused standard deviation;
[0084] S1044: Obtain a third low-level feature corresponding to each first low-level feature according to the fusion mean and the fusion standard deviation.
[0085] like Figure 2 As shown in Figure 1, the mean and standard deviation of each sample feature map after mixing are calculated using the mixing ratio λ, where λ is a mixing coefficient sampled from the Beta distribution. Finally, the normalized feature map is restored using the mixed mean and standard deviation. Therefore, the output feature map x after feature mixing enhancement is mix It can be expressed as: Where x represents the first low-level feature map.
[0086] It is understandable that feature fusion does not reduce the number of features, but only generates a new feature that is different from the original feature.
[0087] By randomly mixing the mean and standard deviation of multi-domain sample features, features with unknown domain distribution are randomly generated to enhance feature diversity and force the model to learn features of different styles during training, thereby improving the model's robustness to unseen domain data and reducing dependence on a single domain distribution.
[0088] S105: Extract features from the third low-level features to obtain first high-level features.
[0089] High-level features are abstract information further extracted from low-level features. High-level features are related to the semantic content of the sample. High-level features may include: 1. Shape features: describing the outline and geometric shape of objects in the image. 2. Object detection and recognition: identifying and locating specific objects from the image. 3. Semantic segmentation: dividing the image into regions with semantic meaning, each region representing a specific object or scene. 4. Image captioning: generating natural language descriptions based on images.
[0090] Low-level features are the most basic information units in image processing and provide the basis for the extraction of high-level features. By combining, aggregating and abstracting low-level features, high-level features that can reflect the high-level semantic information of the sample are obtained.
[0091] The first high-level feature can be obtained by inputting the third low-level feature into the remaining convolutional layers of the MobilenetV2 backbone network for high-level feature extraction.
[0092] S106: extracting features at different scales from the first high-level features, and then merging the features at different scales to obtain second high-level features with multi-scale information.
[0093] Atrous Spatial Pyramid Pooling (ASPP) can be used to perform multi-scale aggregation on each first high-level feature to improve feature utilization and reduce feature loss of details. Convolution operations are performed on the first high-level features using dilated convolutions with different dilation rates (such as 1, 6, 12, 18) to capture features of different scales and obtain the final second high-level features with multi-scale information.
[0094] The atrous spatial pyramid pooling method captures multi-scale contextual information simultaneously by using multiple atrous convolutions with different dilation rates. Each atrous convolution is sampled at a different dilation rate to obtain feature maps of different receptive field sizes in the same layer. Specifically: 1. Atrous convolution: Unlike standard convolution, atrous convolution inserts "holes" between weights, that is, skips certain input values, which effectively increases the receptive field of the convolution kernel without increasing the number of parameters or the amount of computation. A larger dilation rate means a larger jump interval, so a wider range of spatial information can be captured. 2. Spatial pyramid structure: The atrous spatial pyramid pooling method adopts a pyramid-like structure with multiple parallel atrous convolution branches, each using a different dilation rate. This can process information of different scales at the same time, ensuring that the model can adapt to target objects of various sizes. 3. Global Average Pooling (GAP): In addition to multiple atrous convolution branches, ASPP also usually includes a global average pooling branch. This branch performs a pooling operation on the entire feature map to generate a fixed-size feature vector, which is then upsampled back to the original resolution and combined with the results of other branches. This can introduce image-level context information and help improve segmentation results. 4. Fusion mechanism: The outputs of all these branches are concatenated together and fused through an additional convolutional layer to obtain features with multi-scale information.
[0095] S107: Encode the second high-level feature into a code specific to the domain to which it belongs.
[0096] S108: Using the multi-domain features and codes of the domain knowledge base, adjust the second high-level features to obtain the adjusted third high-level features. Figure 2 The Mpool module in .
[0097] S109: Combine the third high-level feature with the second high-level feature and the second low-level feature to generate a final output feature. Figure 2 In the DoFE module.
[0098] It can be seen that the embodiment of the present application not only enhances the diversity of features, but also makes the model pay attention to the detailed features of the optic cup and optic disc, thereby improving the generalization ability of the model by extracting features in the YUV color space, mixing the low-level features generated in the backbone network, and strengthening the low-level features separately outside the backbone network.
[0099] In some embodiments, the process of performing homomorphic filtering on the first low-level feature and extracting its high-frequency component to obtain the high-frequency feature in S102 includes S1021 to S1026, see Figure 4 shown.
[0100] S1021: Perform logarithmic transformation on the first low-level feature to obtain a transformed first feature.
[0101] The first low-level feature (24, 64, 64) is logarithmically transformed to obtain the transformed first feature, so that the brightness and contrast characteristics of the image can be more easily processed.
[0102] S1022: Perform Fourier transform on the first feature and center it to obtain a second feature;
[0103] S1023: Convert the second feature from the spatial domain to the frequency domain;
[0104] S1024: Filter the second feature in the frequency domain to obtain a third feature with high frequency enhancement.
[0105] By filtering the transformed second feature, the high-frequency components, such as the details and edges of the optic cup and optic disc, are enhanced, and the low-frequency components, such as the smooth areas with less changes in the image, are suppressed. The filter formula is:
[0106]
[0107] In the formula, r h and r l are the gain factors of high frequency and low frequency, and their values are 2.0 and 0.5 respectively. c is a control parameter that determines the filter bandwidth of the filter, and its value is 4. D(u,v) is the frequency distance, D0 is the cutoff frequency, and its value can be 10. u corresponds to the frequency variable in the horizontal direction in the frequency domain, and v corresponds to the frequency variable in the vertical direction.
[0108] S1025: Perform inverse Fourier transform on the third feature to convert the third feature into a spatial domain;
[0109] S1026: Perform an exponential transformation on the third feature in the spatial domain, extract the high-frequency component in the third feature, and obtain a high-frequency feature.
[0110] In some embodiments, the process of performing curvature filtering on the high-frequency features to obtain the second low-level features in S103 may include S1031 and S1032.
[0111] S1031: Filter out the first channel with the largest curvature from each channel of the high-frequency feature;
[0112] Furthermore, the process of S1031 selecting the first channel with the largest curvature from each channel of the high-frequency feature may specifically include S10311 to S10316.
[0113] S10311: rearrange the high-frequency features, split the multiple channels into single channels, and obtain channel features of each single channel of the high-frequency features;
[0114] S10312: Convolve each channel feature of the high-frequency feature to obtain a convolved feature map.
[0115] A 3x3 convolution can be used to convolve the channel features of the high-frequency features.
[0116] S10313: Sum the feature map in the width dimension and calculate the horizontal cumulative curvature of all pixels in each channel;
[0117] S10314: sum the feature map in the height dimension and calculate the vertical cumulative curvature of all pixels in each channel;
[0118] S10315: further accumulating the vertical curvature on the horizontal cumulative curvature of all pixels in each channel to obtain the total curvature intensity of each channel;
[0119] S10316: Filter out the first channel with the largest curvature according to the total curvature strength of each channel.
[0120] In order to further highlight the key features in the image, the high-frequency features corresponding to each extracted image are converted back to the spatial domain and rearranged so that each channel is regarded as an independent single-channel image. After rearrangement, the high-frequency feature shape of a single channel can be (576, 1, 64, 64), that is, the batch size is 576, the number of channels is 1, the width is 64, and the height is 64. Then a 3x3 convolution operation is applied to each channel. Subsequently, the sum is taken over the width and height dimensions of the feature map, and the cumulative curvature in the horizontal and vertical directions of all pixels in each channel is calculated as the total curvature intensity of each channel. Finally, each feature map is traversed to filter out the channel with the largest curvature in the current feature map.
[0121] S1032: Concatenate the first channels of all high-frequency features to obtain the second low-level features.
[0122] On the basis of the above-mentioned embodiment, after S109 generates the final output features, it may further include: performing upsampling and convolution operations on the final output features to generate a segmentation result of the optic cup and optic disc.
[0123] In order to demonstrate the superiority of the proposed method in fundus data segmentation, the disclosed embodiments of the present application are compared with other recent domain generalization segmentation models. In each experiment, training is performed on three fundus datasets to obtain the optimal segmentation model, and the performance of the segmentation model is tested and evaluated on the remaining domains. In order to ensure the fairness of the experiment, the model proposed in the present invention was experimented three times. The final result is the average of the three experimental results. As can be seen from the results in Table 1, the Dice averages of the optic disc and optic cup on the Domain 2 dataset surpass all existing mainstream methods.
[0124] As can be seen from Table 1, the method proposed in the embodiment of the present application achieves 85.50% / 93.63%, 82.66% / 92.33%, 85.61% / 91.98%, and 87.38% / 94.04% Dice of the optic cup and optic disc in Domain 1, Domain 2, Domain 3, and Domain 4, respectively. The method has obvious performance advantages over most fundus domain generalization models in Domain 1, Domain 3, and Domain 4, especially in Domain 2, where the method can segment the optimal experimental results, which are significantly higher than other mainstream algorithms. In general, the model proposed in the present invention has strong generalization ability and robustness when processing the task of optic disc and optic cup segmentation in fundus images.
[0125] Table 1 Comparison of four datasets on different methods
[0126]
[0127] Among them, DOFE (Domain-Oriented Feature Embedding), FedDG (Federated Domain Generalization), CDDSA (Contrastive Domain Disentanglement and Style Augmentation), AADG (Automatic Augmentation for Domain Generalization), BEFDG (Blockchain Enabled Federal Domain Generalization), DCAC (Domain and Content Adaptive Convolution), FACT (Fourier-based Augmentation with Applications), and DCAM-NET (multi-region and multi-scale convolution attention mechanism).
[0128] Figure 5 The segmentation result diagrams of the baseline model and the present application on 4 data sets are shown, and each row represents a segmentation result extracted from a domain. The green and blue in the segmentation diagram represent the boundaries of the optic disc and optic cup, respectively, and the red represents the actual data. It can be seen that the segmentation results produced by the method proposed in the present application are more accurate than the baseline model. Specifically, for the sample images of Domain 2, since the proportion of glaucoma patients in this dataset is relatively large, the fundus images of glaucoma patients have differences in scale and shape of the optic cup and optic disc, as well as interference factors such as fundus exudates, which makes it more challenging for the model to segment the optic cup, while the method of the present application can still segment the OC and OD with accurate boundaries. In addition, for the sample images of Domain 1, other segmentation models are instable, resulting in poor segmentation results for some fundus images, among which the results of the method of the present application are still better than other methods.
[0129] In order to verify the effectiveness of each module, the embodiment of the present application conducted an ablation experiment on the RIM-ONE-r3 dataset. The performance comparison results after adding each module are shown in Table 2.
[0130] As can be seen from Table 2, when the feature mixing enhancement module is used, the segmented optic disc and optic cup Dice of the model are improved by 2.2% and 4.68% respectively compared with the previous model. After introducing the low-level feature enhancement module combining homomorphic filtering and curvature, the optic disc and optic cup Dice are slightly improved compared with the case without adding it. Finally, the three RGB channels of the image are converted to YUV color space and then sent to the model training. The segmented optic disc and optic cup Dice are improved by 0.23% and 0.72% respectively. In general, compared with the original baseline model, the method of this application improves the segmented optic disc and optic cup Dice by 2.95% and 5.15% respectively, which verifies the effectiveness of this application.
[0131] Table 2 Ablation experiments on the RIM-ONE-r3 dataset
[0132]
[0133] The present disclosure also discloses a generalized segmentation device for the optic cup and optic disc domain, see Figure 6 As shown, including:
[0134] A low-level feature extraction module 21, used to convert the color space of the fundus images of different domains into a YUV color space, and perform feature extraction on the fundus images of different domains based on the YUV color space to obtain first low-level features of the fundus images of different domains;
[0135] A high-frequency feature extraction module 22, configured to perform homomorphic filtering on the first low-level feature and extract its high-frequency component therefrom to obtain a high-frequency feature;
[0136] A curvature filtering module 23 is used to perform curvature filtering on the high-frequency features to obtain the second low-level features;
[0137] A feature mixing module 24, configured to perform internal mixing of the mean and the standard deviation of the first low-level feature to generate a third low-level feature in a different domain from the fundus image;
[0138] The high-level feature extraction module 25 is used to extract features from the third low-level features to obtain the first high-level features.
[0139] A multi-scale feature extraction module 26, used to extract features at different scales from the first high-level features, and then merge the features at different scales to obtain a second high-level feature with multi-scale information;
[0140] An encoding module 27, for encoding the second high-level feature into a code specific to the domain to which it belongs;
[0141] A feature adjustment module 28, used to adjust the second high-level feature by using the multi-domain features and codes of the domain knowledge base to obtain an adjusted third high-level feature;
[0142] The feature output module 29 is used to combine the third high-level feature with the second high-level feature and the second low-level feature to generate a final output feature.
[0143] The embodiments of the present application not only enhance the diversity of features, but also make the model focus on the detailed features of the optic cup and optic disc, thereby improving the generalization ability of the model by extracting features in the YUV color space, mixing the low-level features generated in the backbone network, and strengthening the low-level features separately outside the backbone network.
[0144] Among them, the process of performing homomorphic filtering on the first low-level feature and extracting its high-frequency component therefrom includes: performing logarithmic transformation on the first low-level feature to obtain the transformed first feature; performing Fourier transformation on the first feature and centering it to obtain the second feature; converting the second feature from the spatial domain to the frequency domain; filtering the second feature in the frequency domain to obtain a high-frequency enhanced third feature; performing inverse Fourier transformation on the third feature and converting the third feature to the spatial domain; performing exponential transformation on the third feature in the spatial domain to extract the high-frequency component in the third feature and obtain a high-frequency feature.
[0145] The process of performing curvature filtering on the high-frequency features to obtain the second low-level features also includes: selecting the first channel with the largest curvature from each channel of the high-frequency features; and concatenating the first channels of all high-frequency features to obtain the second low-level features.
[0146] Among them, the process of selecting the first channel with the largest curvature from each channel of the high-frequency feature also includes: rearranging the high-frequency features, splitting multiple channels into single channels, and obtaining channel features of each single channel of the high-frequency feature; convolving each channel feature of the high-frequency feature to obtain a convolved feature map; summing the feature map in the width dimension to calculate the horizontal cumulative curvature of all pixels in each channel; summing the feature map in the height dimension to calculate the vertical cumulative curvature of all pixels in each channel; further accumulating the curvature in the vertical direction on the horizontal cumulative curvature of all pixels in each channel to obtain the total curvature intensity of each channel; and selecting the first channel with the largest curvature according to the total curvature intensity of each channel.
[0147] Among them, the process of internally mixing the mean and standard deviation of the first low-level features to generate a third low-level feature in a different domain from the fundus image includes: calculating the mean and standard deviation corresponding to each first low-level feature according to the channel dimension of each first low-level feature; fusing the mean of each first low-level feature with the mean of another different feature through a proportional coefficient to obtain a fused fusion mean; fusing the standard deviation of each first low-level feature with the standard deviation of another different image through a proportional coefficient to obtain a fused fusion standard deviation; and obtaining the third low-level feature corresponding to each first low-level feature according to the fused mean and the fused standard deviation.
[0148] It may also include: a segmentation result output module, which is used to perform upsampling and convolution operations on the final output features to generate a segmentation result of the optic cup and optic disc. In some possible implementations, the optic cup and optic disc recognition device according to the present application may include at least a processor and a memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the optic cup and optic disc domain generalization segmentation method according to various exemplary embodiments of the present application described in this specification. For example, the processor may execute the following steps: Figure 1 Follow the steps shown in .
[0149] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application. The electronic device can implement the functions of the generalized segmentation method and device for the optic cup and optic disc domain. Figure 7 , electronic equipment includes:
[0150] At least one processor 801, and a memory 802 connected to the at least one processor 801. The specific connection medium between the processor 801 and the memory 802 is not limited in the embodiment of the present application. Figure 7 In the example, the processor 801 and the memory 802 are connected via a bus 800. Figure 7 The connections between other components are shown in bold lines, and are not intended to be limiting. The bus 800 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 7 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 801 can also be called a controller, and there is no limitation on the name.
[0151] In the embodiment of the present application, the memory 802 stores instructions that can be executed by at least one processor 801. The at least one processor 801 can execute the generalized segmentation method for the optic cup and optic disc domain discussed above by executing the instructions stored in the memory 802. The processor 801 can implement Figure 7 The functions of each module in the device shown.
[0152] Among them, the processor 801 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 802 and calling the data stored in the memory 802, the various functions of the device and process data, the device can be monitored as a whole.
[0153] In one possible design, the processor 801 may include one or more processing units, and the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 801. In some embodiments, the processor 801 and the memory 802 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.
[0154] The processor 801 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the generalized segmentation method for the optic cup and optic disc domain disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0155] The memory 802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 802 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 802 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 802 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0156] By programming the processor 801, the code corresponding to the generalized segmentation method for the optic cup and optic disc domain described in the above embodiment can be fixed into the chip, so that the chip can execute the generalized segmentation method when running. Figure 1The steps of the generalized segmentation method for the optic cup and optic disc domain in the embodiment shown are as follows: How to design and program the processor 801 is a technique known to those skilled in the art and will not be described in detail here.
[0157] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the generalized segmentation method for the optic cup and optic disc domain discussed above.
[0158] In some possible implementations, various aspects of the generalized segmentation method for the optic cup and optic disc domain provided in the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the generalized segmentation method for the optic cup and optic disc domain according to various exemplary embodiments of the present application described above in this specification.
[0159] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0160] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0163] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A generalized segmentation method for optic cup and optic disc domain, characterized in that: include: Converting the color space of the fundus images in different domains into a YUV color space, and performing feature extraction on the fundus images in different domains based on the YUV color space to obtain first low-level features of the fundus images in different domains; Performing homomorphic filtering on the first low-level feature and extracting high-frequency components thereof to obtain high-frequency features; Performing curvature filtering on the high-frequency features to obtain second low-level features; Performing internal mixing of the mean and the standard deviation on the first low-level feature to generate a third low-level feature in a different domain from the fundus image; Extract features from the third low-level features to obtain first high-level features; Extract features at different scales from the first high-level features, and then merge the features at different scales to obtain a second high-level feature with multi-scale information; encoding the second high-level feature into a code specific to the domain in which the second high-level feature resides; Using the multi-domain features of the domain knowledge base and the code, adjusting the second high-level feature to obtain an adjusted third high-level feature; The third high-level feature is combined with the second high-level feature and the second low-level feature to generate a final output feature.
2. The generalized segmentation method for optic cup and optic disc domain according to claim 1, characterized in that: The process of performing homomorphic filtering on the first low-level feature and extracting its high-frequency component therefrom includes: Performing a logarithmic transformation on the first low-level feature to obtain a transformed first feature; Performing Fourier transform on the first feature and centering it to obtain a second feature; converting the second feature from the spatial domain to the frequency domain; Performing filtering processing on the second feature in the frequency domain to obtain a third feature with high frequency enhancement; Performing an inverse Fourier transform on the third feature to convert the third feature into a spatial domain; An exponential transformation is performed on the third feature in the spatial domain, and a high-frequency component in the third feature is extracted to obtain the high-frequency feature.
3. The generalized segmentation method for optic cup and optic disc domain according to claim 1, characterized in that: The process of performing curvature filtering on the high-frequency features to obtain the second low-level features also includes: Filtering out a first channel with the largest curvature from each channel of the high-frequency feature; The first channels of all the high-frequency features are concatenated to obtain the second low-level features.
4. The generalized segmentation method for optic cup and optic disc domain according to claim 3, characterized in that: The process of selecting the first channel with the largest curvature from each channel of the high-frequency feature includes: Rearranging the high-frequency features, splitting the multiple channels into single channels, and obtaining channel features of each single channel of the high-frequency features; Convolving each channel feature of the high-frequency feature to obtain a convolved feature map; Summing the feature map in the width dimension, calculating the horizontal cumulative curvature of all pixels in each channel; Summing the feature map in the height dimension, calculating the vertical cumulative curvature of all pixels in each channel; The vertical curvature is further accumulated on the horizontal cumulative curvature of all pixels in each channel to obtain the total curvature intensity of each channel; According to the total curvature intensity of each channel, the first channel with the largest curvature is screened out.
5. The generalized segmentation method for optic cup and optic disc domain according to claim 1, characterized in that: The process of performing internal mixing of the mean and the standard deviation on the first low-level feature to generate a third low-level feature in a different domain from the fundus image includes: Calculate the mean and standard deviation corresponding to each of the first low-level features according to the channel dimension of each of the first low-level features; The mean of each of the first low-level features is merged with the mean of another different feature through a proportional coefficient to obtain a fused mean; The standard deviation of each of the first low-level features is merged with the standard deviation of another different picture through a proportional coefficient to obtain a fused standard deviation; The third low-level feature corresponding to each first low-level feature is obtained according to the fusion mean and the fusion standard deviation.
6. The generalized segmentation method for optic cup and optic disc domain according to claim 1, characterized in that: After generating the final output features, it also includes: The final output features are subjected to upsampling and convolution operations to generate a segmentation result of the optic cup and optic disc.
7. A generalized segmentation device for optic cup and optic disc domain, characterized in that: include: A low-level feature extraction module, used for converting the color space of the fundus images of different domains into a YUV color space, and performing feature extraction on the fundus images of different domains based on the YUV color space to obtain first low-level features of the fundus images of different domains; A high-frequency feature extraction module, used for performing homomorphic filtering on the first low-level feature and extracting its high-frequency component to obtain a high-frequency feature; A curvature filtering module, used for performing curvature filtering on the high-frequency features to obtain second low-level features; A feature mixing module, used for performing internal mixing of the mean and the standard deviation of the first low-level feature to generate a third low-level feature in a different domain from the fundus image; The high-level feature extraction module is used to extract features from the third low-level features to obtain first high-level features. A multi-scale feature extraction module, used to extract features at different scales from the first high-level features, and then merge the features at different scales to obtain a second high-level feature with multi-scale information; An encoding module, used for encoding the second high-level feature into a code specific to the domain to which it belongs; A feature adjustment module, used to adjust the second high-level feature by using the multi-domain features of the domain knowledge base and the code to obtain an adjusted third high-level feature; A feature output module is used to combine the third high-level feature with the second high-level feature and the second low-level feature to generate a final output feature.
8. The generalized segmentation device for optic cup and optic disc domain according to claim 7, characterized in that: Also includes: The segmentation result output module is used to perform upsampling and convolution operations on the final output features to generate a segmentation result of the optic cup and optic disc.
9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the generalized segmentation method for optic cup and optic disc domain according to any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that: The storage medium comprises a program code, and when the storage medium is run on an electronic device, the program code is used to enable the electronic device to execute the generalized segmentation method for the optic cup and optic disc domain as claimed in any one of claims 1 to 6.