Cross-domain optic cup and optic disc automatic segmentation method and device based on double-path self-supervision

Abstract

The application discloses a kind of based on double-path self-supervision's cross-domain view cup view disc automatic segmentation method and device, method includes: using source domain-target domain and target domain-source domain's two-way fundus image domain transformation, respectively generate target domain style source domain image and source domain style target domain image, the source domain image has segmentation label, target domain image is no segmentation label;Using DeepLabV3+ image segmentation network as framework, MobileNetV2 feature extractor is matched, constructs double-path fundus image segmentation network;The prediction segmentation result of the unsupervised image output by double-path network is fused, and the fusion pseudo label is obtained, and the further training of no label image is guided using fusion pseudo label;Using a cross-domain contrast constraint mechanism, the similarity of image features before and after domain transformation is optimized, so that the information related to image structure is retained in features.The device comprises a memory and a processor.The application improves the accuracy of target domain image prediction segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, image semantic segmentation, cross-domain learning, self-supervised learning and computer vision, and in particular to a cross-domain optic cup and optic disc automatic segmentation method and device based on a dual-path self-supervision. BACKGROUND

[0002] With the continuous innovation of information technology, massive medical image data is available for analysis. Medical images are important information carriers and diagnostic criteria in today's medical clinical practice, and have very high research value. Medical image processing technology has wide application prospects and is an important application field of computer vision technology. With the proposal of the concept of "big health" industry, the development of medical image processing technology is attracting more and more attention.

[0003] Fundus images are important medical image materials and are the main diagnostic basis for glaucoma disease. In fundus images, the optic disc is a light red disc structure about 3mm from the macular position to the nasal side of the retina, and the optic cup is a physiological depression in the center of the optic disc. Segmenting the optic cup and optic disc regions, the ratio of their vertical diameters (referred to as the cup-disc ratio) is an important medical indicator, i.e., the main reference standard for diagnosing glaucoma disease. Generally, the higher the cup-disc ratio, the higher the risk of glaucoma [1] . To address the problem of large consumption of medical resources for manual optic cup and optic disc segmentation, many existing studies focus on the automatic segmentation function of fundus image optic cup and optic disc. Most of the research methods [2,3,4] are based on the principle of deep learning, construct a deep neural network, and output the optic cup and optic disc segmentation results by fine-tuning and improving advanced mainstream image semantic segmentation networks such as FCN [5] , U-Net [6] , DeepLab [7] , etc.

[0004] However, due to domain shift between different fundus image datasets, most of the trained models cannot obtain good segmentation results in new unlabeled datasets, limiting the performance and generalization ability of the method. The domain shift phenomenon of fundus image datasets mainly comes from the following points: 1) different hospitals use different devices to shoot fundus images; 2) the conditions such as light intensity of the shooting site are inconsistent; 3) the parameter settings of the camera equipment, such as image resolution, are inconsistent. The existence of domain shift phenomenon makes the application range of the optic cup and optic disc segmentation model limited to a single dataset, and it does not have good generalization ability and lacks clinical applicability.

[0005] To solve this problem, domain adaptation methods can be used to overcome domain shift and adapt the trained model to the target domain image. Domain adaptation [8]Domain adaptation is an important research direction in computer vision field, which mainly aims to perform the same task using data from different domains, and to overcome the domain differences between data to make the model obtain good generalization performance. After domain adaptation training, the model can only use the label information of the source domain data for supervision during training, and can show good performance on the target domain data, so that the model has good generalization ability.

[0006] In recent years, some research works focus on the automatic segmentation of optic disc and cup in fundus images in cross-domain scenes, and have achieved some results. Wang et al. [9] proposed a pOSAL network, which is based on patch extraction for segmentation, and uses an adversarial domain discriminator to confuse the source domain and target domain features, so as to retain spatial and structural information. Kadambi et al.

[10] proposed a network named WGAN, which realizes domain adaptation based on Wasserstein distance. Wang et al.

[11] proposed a network named BEAL, which designs two prediction branches for the boundary and entropy map of the segmentation result, and applies adversarial networks on the two branches respectively, so as to learn the domain invariance of the predicted boundary and the predicted entropy. Chen et al.

[12] proposed an unsupervised domain adaptation network for input space and output space alignment, which uses adversarial training and combines input feature alignment and output space alignment to alleviate domain shift.

[0007] The above existing researches are mostly based on adversarial domain adaptation, which alleviates the distribution difference between source domain data and target domain data in feature space. These achievements provide sufficient ideas and reference methods, but there are still some deficiencies. At present, the main problems in this field are as follows:

[0008] 1. How to fully eliminate the interference of domain style related features, so that the extracted image features are more suitable for optic cup and disc segmentation task;

[0009] 2. How to excavate additional supervision information besides limited labeled images, and effectively utilize unlabeled data to improve the potential of the model;

[0010] 3. How to design an image segmentation network based on domain adaptation, so that the trained model can output accurate optic cup and disc segmentation results on target domain images, and can be generalized to different target domain data sets. SUMMARY

[0011] The application provides a cross-domain optic cup and optic disc automatic segmentation method and device based on a dual-path self-supervision, which combines a bidirectional domain transformation, a dual-path segmentation network, pseudo-label fusion and a cross-domain comparison module, establishes an optic cup and optic disc segmentation model in a cross-domain scene, and improves the accuracy of a target domain image prediction segmentation result, as described below:

[0012] In a first aspect, a cross-domain optic cup and optic disc automatic segmentation method based on a dual-path self-supervision is provided, which includes the following steps:

[0013] A source domain-target domain and target domain-source domain bidirectional fundus image domain transformation is used to generate a source domain image in a target domain style and a target domain image in a source domain style, respectively, wherein the source domain image has a segmentation label, and the target domain image has no segmentation label;

[0014] A DeepLabV3+ image segmentation network is used as a framework, and a MobileNetV2 feature extractor is used to build a dual-path fundus image segmentation network;

[0015] The prediction segmentation results of the unsupervised images output by the dual-path network are fused to obtain fused pseudo-labels, and the fused pseudo-labels are used to guide the further training of the unlabeled images;

[0016] A cross-domain comparison constraint mechanism is used to optimize the similarity of image features before and after the domain transformation, so that the features retain information related to the image structure.

[0017] The dual-path is two segmentation networks that use the same image segmentation framework but do not share network parameters.

[0018] The source domain-target domain and target domain-source domain bidirectional fundus image domain transformation is as follows:

[0019] The source domain image X s and the target domain image X t are subjected to two-dimensional image Fourier transform to obtain their respective frequency spectra A s and A t .

[0020] A central rectangular block in the amplitude graph of the source domain frequency spectrum A s is intercepted and replaced in the corresponding position of the target domain frequency spectrum A t to generate a new frequency spectrum A t→s , and then A t→s is subjected to inverse Fourier transform to generate a target domain image X t→s in a source domain style, which has the same shape, position, texture and other information as the original target domain image X t .

[0021] The target domain frequency spectrum A tlow-frequency information in the source domain spectrum graph A is replaced into the target domain spectrum graph A s Then, the replaced spectrum graph A is subjected to inverse Fourier transform to obtain a source domain image X with the target domain style s→t s→t The shape, position, texture and other information of the source domain image X are consistent with those of the original source domain image X s .

[0022] The fusion of the prediction segmentation results of the unsupervised image output by the dual-path network is as follows:

[0023] The unsupervised image X t and X t→s are input into the dual-path image segmentation network, and the output prediction segmentation results are Y′ t and Y′ t→s , respectively.

[0024] Y′ t→s and Y′ are fused to generate a fused pseudo label Y′ mix , and the fusion method is as follows:

[0025] For the cth channel graph Y′ t (c) in Y′ t , the ith pixel point prediction value is Y′ t (c,i), and the confidence value S t (i,c) is calculated by the following formula:

[0026] S t (i,c) = 1-exp(1-p max / p min )

[0027] p max = max(Y′ t (c,i), 1-Y′ t (c,i))

[0028] p min = min(Y′ t (c,i), 1-Y′ t (c,i))

[0029] Where exp(·) is the exponential function with natural constant e as the base, and the confidence values of all prediction points form a confidence map S t (c).

[0030] Y′ t→s is processed in the same way to obtain a confidence map S t→s (c), and S t (c) and S t→s (c) are connected to obtain a relative confidence map S′​t (c) and S' t→s (c), reorganize the relative confidence map of each channel into a confidence matrix S' t and S' t→s ;

[0031] Using S' t and S' t→s Weighted fusion is performed on Y' t and Y' t→s to obtain the fused pseudo label Y' mix , as follows:

[0032] Y' mix = S' t ·Y' t +S' t→s ·Y' t→s

[0033] In further training, the pseudo label Y' mix is used as supervision information for the unlabeled image X t and X t→s .

[0034] Further, the cross-domain contrast constraint mechanism is:

[0035] The original image X s and the copy image X s→t are regarded as positive samples, and negative samples are randomly selected, X t and X t→s are the same;

[0036] Using the contrast loss function L c , the distance between the features of the positive samples is narrowed, and the form of L c is:

[0037]

[0038] Where, log(·) is the logarithmic function with base 2, F(·) represents the encoding feature of the image, d(·) represents the Euclidean distance, exp(d(·) / 2σ 2 ) represents the Gaussian kernel function, and n represents the number of images input in a training batch.

[0039] The contrast loss function L c is applied to the positive sample images X s and X s→t , and X t and X t→s , respectively, to strengthen the semantic information related to the segmentation task in the encoding feature.

[0040] In a second aspect, a device for cross-domain cup and disc automatic segmentation based on dual-path self-supervision is provided, and the device comprises a processor and a memory, wherein the memory stores program instructions, and the processor invokes the program instructions stored in the memory to enable the device to perform the method steps of any one of the first aspect.

[0041] In a third aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program, wherein the computer program comprises program instructions, and the program instructions are executed by a processor to enable the processor to perform the method steps of any one of the first aspect.

[0042] The technical scheme provided by the application has the following beneficial effects:

[0043] 1. The application adopts image bidirectional domain transformation to generate domain conversion copies for all images, thereby expanding the number of images and available data information; and a dual-path image segmentation network is used to input source domain style and target domain style images for training, thereby realizing domain decoupling, effectively eliminating the interference of domain-related features on the prediction segmentation result of the input sample image in the same image segmentation network, and improving the feature effectiveness.

[0044] 2. The application designs a pseudo-label generation method, fuses the prediction segmentation results of homologous unlabeled images in the dual-path based on confidence weighting to generate fused pseudo-labels, and further supervises the unlabeled images using the fused pseudo-labels, thereby effectively mining additional supervision information, enhancing the interactivity of the dual-path segmentation network, and improving the segmentation performance of the model through a self-supervised manner.

[0045] 3. The application designs a cross-domain contrast loss function to strengthen the information related to the segmentation task in the image features, further improve the feature effectiveness, guarantee the similarity of the dual-path positive sample image segmentation results and the reliability of the fused pseudo-labels, and thereby improve the segmentation performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 It is a flowchart of a cross-domain cup and disc automatic segmentation method based on dual-path self-supervision.

[0047] Figure 2 It is a schematic diagram of a cross-domain cup and disc automatic segmentation network based on dual-path self-supervision.

[0048] Figure 3 It is an experimental effect diagram.

[0049] Figure 4 It is a structural schematic diagram of a device for cross-domain cup and disc automatic segmentation based on dual-path self-supervision. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application are described in further detail below.

[0051] Embodiment 1

[0052] A cross-domain cup-disc automatic segmentation method based on double-path self-supervision, see Figure 1 and Figure 2 The method comprises the following steps:

[0053] 101: using the source domain-target domain and target domain-source domain bidirectional fundus image domain transformation, respectively generating the source domain image (with segmentation label) of the target domain style and the target domain image (without segmentation label) of the source domain style;

[0054] 102: using DeepLabV3+ image segmentation network as a framework, combining MobileNetV2 (lightweight mobile neural network) feature extractor to construct a double-path fundus image segmentation network.

[0055] Among them, the double-path is that two segmentation networks use the same image segmentation framework (DeepLabV3+), but do not share network parameters. All original images and generated copy images are divided by domain style and input into two paths for training.

[0056] 103: fuse the prediction segmentation results of the unsupervised images output by the double-path network to obtain the fusion pseudo label, and use the fusion pseudo label to guide the further training of the unlabeled images, so that the supervision information is more abundant;

[0057] 104: using a cross-domain contrast constraint mechanism to optimize the similarity of image features before and after domain transformation, so that the features retain more information related to image structure, and improve the effectiveness of image features in the segmentation task.

[0058] In summary, the embodiments of the present application combine bidirectional domain transformation, double-path segmentation network, pseudo label fusion and cross-domain contrast modules, establish a cup-disc segmentation model in a cross-domain scene, and improve the accuracy of the prediction segmentation result of the target domain image.

[0059] Embodiment 2

[0060] The scheme in embodiment 1 is further introduced in combination with specific examples and calculation formulas, and details are described below:

[0061] 201: using the source domain-target domain and target domain-source domain bidirectional fundus image domain transformation, respectively generating the source domain image (with segmentation label) of the target domain style and the target domain image (without segmentation label) of the source domain style;

[0062] Among them, the above step 201 mainly includes:

[0063] 1) Two-dimensional image Fourier transform is performed on the original image:

[0064] In the embodiment of the present application, the source domain image X s and the target domain image X t are converted into frequency spectrum graphs by using Fast Fourier transform (FFT), as shown below:

[0065] A s = F ou (X s ) (1)

[0066] A t = F ou (X t ) (2)

[0067] wherein F ou (·) represents a Fast Fourier transform function, A s and A t are frequency spectrum graphs obtained by frequency domain transformation of X s and X t respectively, and the frequency spectrum graph is mainly composed of an amplitude graph and a phase graph. The same frequency domain transformation is performed on all source domain and target domain images in the training data set.

[0068] 2) Bidirectional domain transformation is realized by low-frequency information exchange:

[0069] In the embodiment of the present application, bidirectional domain transformation is performed on the source domain and target domain images to generate their respective copy images, and the specific mode is shown as follows:

[0070] In the frequency spectrum graphs A s and A t , the high and low values of the frequency are mainly embodied in the amplitude graph. A rectangle α s is intercepted at the central position of the amplitude graph of A s , and the ratio of the side length of the rectangle α s to the total side length of the amplitude graph is 0.01. The low-frequency information contained in the image X s is inside the rectangle, including illumination, overall chroma and other information. In the image X s , the low-frequency information is mainly related to the overall style (domain style) of the image; and the high-frequency information is mainly related to the information of texture, details and the like.

[0071] The same processing is performed on the frequency spectrum graph A t to intercept the rectangle α t . The rectangle α s is replaced at the low-frequency position of the amplitude graph of A t , and the rectangle αt Replace the amplitude graph of the spectrum A s Low frequency position, that is, the low frequency information exchange of the image is realized, and the transformed spectrum A t→s And A s→t .

[0072] The spectrum A t→s And A s→t Inverse Fourier transform is carried out:

[0073]

[0074]

[0075] Wherein, Indicates the inverse Fourier transform function. The obtained image X t→s Converts into the source domain style, and the image structure, details, texture and the like are completely consistent with the image X t As a copy image of X t X s→t Also the same.

[0076] 202: Use DeepLabV3+ image segmentation network as a framework, match MobileNetV2 (lightweight mobile neural network) feature extractor, construct a double-path fundus image segmentation network, divide all original images and generated copy images according to the domain style, and input into two paths for training;

[0077] Wherein, the above step 202 mainly includes:

[0078] 1) Construct an image segmentation network:

[0079] In order to realize the image segmentation function, the embodiment of the application utilizes DeepLabV3+ image segmentation network matched with MobileNetV2 feature extractor to construct optic disc and cup segmentation network. First, the image is passed through the MobileNetV2 network, and the "empty" convolution is used, that is, the convolution kernel with different sparseness is used to control the resolution of the output feature map, so that image features of different levels and different scales are obtained, and the fusion features containing the whole image information are fused to realize multi-scale image coding. The coded image features are passed through the image decoder, and the decoder of DeepLabV3+ uses the deconvolution method for up-sampling to restore the resolution and spatial dimension of the feature map. In order to optimize the detail effect of the output image, DeepLabV3+ uses the feature map of the intermediate layer to enlarge the output image. The output content of the decoder is the optic cup and disc segmentation result, and the size of the original image is the same, and the channel number is 2, which corresponds to the segmentation result of the optic cup and disc.

[0080] 2) Input the image into the double-path image segmentation network for training:

[0081] According to the above framework, two structure-identical image segmentation networks are established, but the parameters are not shared during training. The double-path segmentation network is represented as M s and M t , respectively. M s is trained on source domain style images, and M t is trained on target domain style images. Since only single domain data is input in a single segmentation network, the decoupling of source domain and target domain data is achieved, and the influence of domain style-related features on cross-domain image segmentation results is weakened.

[0082] During training, labeled images X s and X s→t are input into M s and M t , respectively, and their segmentation labels are both Y s . To train the image segmentation target, pixel-level cross-entropy loss is used to train the optic cup and disc segmentation target, and its specific form is:

[0083]

[0084] where h and w are the height and width of the image, c is the number of channels, i is each pixel point, Y′ s (i) and Y′ s→t (i) are the predicted segmentation results of the labeled images output by the segmentation networks M s and M t . This loss function can narrow the similarity between the real segmentation label and the predicted segmentation result, so that the image segmentation network can output the optic cup and disc segmentation result.

[0085] 203: Fuse the predicted segmentation results of the unlabeled images output by the double-path network to obtain fused pseudo labels, and use the fused pseudo labels to guide the further training of the unlabeled images to make the supervision information more abundant;

[0086] Wherein, the above step 203 mainly includes:

[0087] 1) Generate a confidence matrix for the unlabeled image prediction result:

[0088] For unlabeled images X t and X t→s , the double-path segmentation network M t and M s outputs the optic cup and disc prediction segmentation result, represented as Y′ t and Y′ t→s . Ideally, Y′ t and Y′ t→sShould be completely consistent, the true segmentation results of the corresponding image. The embodiment of the application designs a fusion mechanism, in Y' t and Y' t→s Each predicted pixel point, respectively calculate the relative confidence value, confidence matrix S' t and S' t→s . The specific steps are as follows:

[0089] For Y' t The c-th channel image Y' t (c), the i-th pixel point prediction value is Y' t (c, i), the confidence value S t (i, c) of the prediction point is calculated by using the following formula:

[0090] S t (i, c) = 1-exp(1-p max / p min ) (6)

[0091] p max = max(Y' t (c, i), 1-Y' t (c, i)) (7)

[0092] p min = min(Y' t (c, i), 1-Y' t (c, i)) (8)

[0093] Where, exp(·) is the exponential function with natural constant e as the base, max represents the larger value, and min represents the minimum value. The confidence values of all prediction points form the confidence map S t (c).

[0094] Y' t→s Do the same processing, get the confidence map S t→s (c). Connect S t (c) and S t→s (c), pass through the softmax layer, get the relative confidence map S' t (c) and S' t→s (c), and then split it. After splitting, the relative confidence maps of each channel are reorganized into confidence matrices S' t and S' t→s .

[0095] 2) Generate fused pseudo labels

[0096] The confidence matrix S' t and S' t→s obtained by the above steps, the value of each position corresponds to Y't and Y' t→s corresponding prediction point, and S' t and S' →s The sum of the corresponding values of S' t and Y' t→s are all 1. The confidence matrix is used to weight the fusion of the prediction segmentation results Y' mix and Y' mix , that is, the fused pseudo label Y' mix is obtained, as shown in the following formula:

[0097] Y' t = S' t · Y' t→s + S' t→s · Y' t→s (9)

[0098] In further training, the pseudo label Y' mix is used as supervision information for the unlabeled images X t and X t→s . The self-supervision of X t and X t→s is performed using a pixel-level cross-entropy loss function, denoted as L t , as shown in the following formula:

[0099]

[0100] The pseudo label generation method included in the embodiment of the present application gives higher weight to the prediction segmentation result with higher confidence on each pixel point of the prediction segmentation graph output by the dual-path segmentation network. In this way, the generated pseudo label not only fuses the prediction segmentation results of the dual paths, but also tends to the segmentation result with higher prediction certainty at the pixel point level, so that the fused pseudo label has higher confidence.

[0101] 3) Maximum square loss constraint

[0102] In the fundus image, the optic disc region contains the optic cup region, and the outline of the optic disc region is more distinct, so the prediction of the optic disc region is easier than the prediction of the optic cup region. Therefore, most segmentation networks produce higher prediction values for the optic disc class and relatively lower prediction values for the optic cup class, thereby causing the model training to be biased towards the prediction performance improvement of the optic disc. To prevent the above situation, the embodiment of the present application uses the maximum square loss function L max to constrain the pseudo label Y' mix (i) to adapt to the situation of inter-class prediction probability imbalance, weaken the dominant role of the high-confidence class in training, and improve the confidence of the prediction results of each class. The specific form is as follows:

[0103]

[0104] where Y' is the pseudo label mix (i) represents the pseudo label Y' of the i-th pixel value mix The maximum square loss function can guide each probability value of the fused pseudo label to be more biased towards 0 or 1, so as to be closer to the characteristics of the true label. In addition, the gradient of the maximum square loss is linear, and when the model is updated, even if the gradient value obtained by the high-confidence class is higher, other classes can also obtain sufficient gradient, so as to weaken the dominant role of the high-confidence class in training, and the confidence of the fused pseudo label in each class is improved.

[0105] 204: using a cross-domain contrast constraint mechanism to optimize the similarity of image features before and after domain transformation, so that the features retain more information related to the structure of the image, and improve the effectiveness of the image features in the segmentation task;

[0106] The step 204 mainly includes:

[0107] The semantic information contained in the encoded features of the image should be maximally related to the optic disc and optic cup segmentation task, and irrelevant semantic information should be excluded, so that the output segmentation result is more accurate.

[0108] In summary, the embodiment of the application combines bidirectional domain transformation, double-path segmentation network, pseudo label fusion and cross-domain contrast modules, establishes an optic cup and optic disc segmentation model in a cross-domain scene, and improves the accuracy of the predicted segmentation result of the target domain image.

[0109] Embodiment 3

[0110] The following specific examples are used to verify the feasibility of the schemes in embodiments 1 and 2, and the calculation formula is described in detail as follows:

[0111] During training, the original image X s and the copy image X s→t are regarded as positive samples, and negative samples are randomly selected. X t and X t→s The output segmentation results of the positive sample images should be consistent, and therefore the features thereof should also have structural similarity. Based on the above consideration, in the embodiment of the application, the contrast loss function L c is used to narrow the distance between the features of the positive samples, so as to strengthen the information related to the segmentation task in the features, and make the output segmentation results of the positive samples close to each other. The specific form of L c is as follows:

[0112]

[0113] where log(·) is a logarithmic function with base 2, F(·) represents the encoded features of the image, d(·) represents the Euclidean distance, and exp(d(·) / 2σ2 ) represents a Gaussian kernel function measuring the similarity between two features, and n represents the number of images input in a training batch. For positive sample images X s and X s→t , and X t and X t→s , the contrastive loss function L c is applied respectively, which strengthens the semantic information (mainly structural information) related to the segmentation task in the encoded features, thus improving the accuracy of the predicted segmentation results and the consistency of the predicted results of positive sample images.

[0114] The evaluation index used is the Dice coefficients (DI), which is calculated as follows:

[0115]

[0116] where N tp , N fp and N fn represent the number of true positive, false positive and false negative pixels in the predicted segmentation results respectively. A higher DI index implies better segmentation performance. DI disc is used to evaluate the segmentation results of the optic disc, and DI cup is used to evaluate the segmentation results of the optic cup.

[0117] For glaucoma diagnosis, the absolute error δ is the error between the predicted cup-disc ratio CDR p and the true cup-disc ratio CDR g , i.e. δ = |CDR p - CDR g |, where VD cup represents the vertical cup diameter of the segmentation map, and VD disc represents the vertical disc diameter of the segmentation map. A smaller δ value implies better segmentation performance.

[0118] Tables 1, 2 and 3 respectively show the experimental results of the proposed algorithm on the Drishti-GS

[13] , RIM-ONE-r3

[14] and REFUGE Validation / Test

[15] three data sets. Compared with the experimental results of previous work, the proposed method has certain superiority. Some segmented visual results are shown in Figure 3 , and the segmentation results of the proposed method have high consistency with the true results.

[0119] Table 1 Experimental results on the Drishti-GS

[13] data set

[0120]

[0121] Table 2 RIM-ONE-r3

[14] Comparison of experimental results on datasets

[0122]

[0123]

[0124] Table 3 REFUGE Validation Test

[15] Comparison of experimental results on datasets

[0125]

[0126] Example 4

[0127] A cross-domain cup-disc automatic segmentation based on dual-path self-supervision, see Figure 4 The device comprises a processor and a memory, and the memory stores program instructions; the processor invokes the program instructions stored in the memory to enable the device to perform the following method steps in Example 1:

[0128] The dual paths are two segmentation networks that adopt the same image segmentation framework but do not share network parameters.

[0129] The source domain-target domain and target domain-source domain bidirectional fundus image domain transformation is:

[0130] The source domain image X s and the target domain image X t are respectively subjected to two-dimensional image Fourier transform to obtain their respective frequency spectrums A s and A t .

[0131] A central rectangular block in the amplitude graph of the source domain frequency spectrum A s is intercepted and replaced in the corresponding position of the target domain frequency spectrum A t to generate a new frequency spectrum A t→s ; then A t→s is subjected to inverse Fourier transform to generate a target domain image X t→s in the source domain style, which is consistent with the original target domain image X t in shape, position, texture, etc.

[0132] Low-frequency information in the target domain frequency spectrum A t is replaced in the source domain frequency spectrum A s ; then the replaced frequency spectrum A s→t is subjected to inverse Fourier transform to obtain a source domain image X s→t in the target domain style.shape, position, texture, etc. information of the original source domain image X s are consistent.

[0133] The fusion of the unsupervised image prediction segmentation results output by the double-path network is:

[0134] The unsupervised image X t and X t→s are input into the double-path image segmentation network, and the output prediction segmentation results are Y' t and Y' t→s , respectively.

[0135] Y' t→s and Y' t are fused to generate a fused pseudo label Y' mix , and the fusion method is:

[0136] For the cth channel graph Y' t (c) in Y' t , the ith pixel point prediction value is Y' t (c, i), and the confidence value S t (i, c) is calculated using the following formula:

[0137] S t (i, c) = 1 - exp(1 - p max / p min )

[0138] p max = max(Y' t (c, i), 1 - Y' t (c, i))

[0139] p min = min(Y' t (c, i), 1 - Y' t (c, i))

[0140] Where exp(·) is the exponential function with natural constant e as the base, and the confidence values of all prediction points form a confidence map S t (c).

[0141] Y' t→s is processed in the same way to obtain a confidence map S t→s (c), and S t (c) and S t→s (c) are connected to obtain relative confidence maps S' t (c) and S' t→s (c) through a softmax layer, and the relative confidence maps of each channel are reorganized into a confidence matrix S' t and S' t→s;

[0142] S′ t and S′ t→s are weighted fusion, to obtain the fusion pseudo label Y′ t and Y′ t→s , as follows: mix

[0143] Y′ mix =S′ t ·Y′ t +S′ t→s ·Y′ t→s

[0144] In further training, the pseudo label Y′ mix is used as supervision information of the unlabeled image X t and X t→s .

[0145] Further, the cross-domain contrast constraint mechanism is:

[0146] The original image X s and the copy image X s→t are regarded as positive samples, and negative samples are randomly selected, X t and X t→s are the same;

[0147] The contrast loss function L c is used to narrow the distance between the features of the positive samples, and the form of L c is:

[0148]

[0149] Wherein, log(·) is the logarithmic function with base 2, F(·) represents the encoding feature of the image, d(·) represents the Euclidean distance, exp(d(·) / 2σ 2 ) represents the Gaussian kernel function, and n represents the number of images input in a training batch.

[0150] The contrast loss function L c is applied to the positive sample images X s and X s→t , and X t and X t→s , respectively, to strengthen the semantic information related to the segmentation task in the encoding feature.

[0151] It should be pointed out here that the device description in the above embodiments corresponds to the method description in the embodiments, and the embodiments of the present application will not be repeated here.

[0152] ​The execution subject of the processor 1 and the memory 2 described above can be a computer, a single-chip microcomputer, a microcontroller, or the like having a computing function, and in a specific implementation, embodiments of the present application do not limit the execution subject, and selection is made according to the needs in actual application.

[0153] The memory 2 and the processor 1 transmit data signals through the bus 3, and embodiments of the present application do not repeat the same.

[0154] Based on the same inventive concept, embodiments of the present application also provide a computer-readable storage medium, which includes a stored program, and when the program is running, the device where the storage medium is located executes the method steps in the above embodiments.

[0155] The computer-readable storage medium includes, but is not limited to, a flash memory, a hard disk, a solid-state disk, and the like.

[0156] It should be noted here that the readable storage medium description in the above embodiments corresponds to the method description in the embodiments, and embodiments of the present application do not repeat the same.

[0157] In the above embodiments, all or part can be realized by software, hardware, firmware, or any combination thereof. When realized by software, all or part can be realized in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present application is generated.

[0158] The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in or transmitted by a computer-readable storage medium. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available medium can be a magnetic medium or a semiconductor medium, etc.

[0159] Embodiments of the present application do not limit the model of each device unless otherwise specified, and any device that can complete the above functions is acceptable.

[0160] Reference:

[0161] [1] Zhang N, Wang J, Li Y, et al. Prevalence of primary open angle glaucoma in the last 20 years: a meta-analysis and systematic review [J]. Scientific Reports, 2021, 11(1): 1-12.

[0162] [2] Fu Y, Chen J, Li J, et al. Optic disc segmentation by U-net and probability bubble in abnormal fundus images [J]. Pattern Recognition, 2021, 117: 107971.

[0163] [3] Xiong H, Liu S, Sharan R V, et al. Weak label based Bayesian U-Net for optic disc segmentation in fundus images [J]. Artificial Intelligence in Medicine, 2022, 126: 102261.

[0164] [4] Guo F, Li W, Kuang Z, et al. MES-Net: a new network for retinal image segmentation [J]. Multimedia Tools and Applications, 2021, 80(10): 14767-14788.

[0165] [5] Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation [C]. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015: 3431-3440.

[0166] [6] Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation [C]. In International Conference on Medical image computing and computer-assisted intervention, 2015: 234-241.

[0167] [7] Chen L-C, Papandreou G, Kokkinos I, et al. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs [J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(4): 834-848.

[0168] [8] Liu X, Yoo C, Xing F, et al. Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectives [J]. APSIPA Transactions on Signal and Information Processing, 2022.

[0169] [9] Wang S, Yu L, Yang X, et al. Patch-based output space adversarial learning for joint optic disc and cup segmentation [J]. IEEE transactions on medical imaging, 2019, 38(11): 2485-2495.

[0170]

[10] Kadambi S, Wang Z, Xing E. WGAN domain adaptation for the joint optic disc-and-cup segmentation in fundus images [J]. International Journal of Computer Assisted Radiology and Surgery, 2020, 15(7): 1205-1213.

[0171]

[11] Wang S, Yu L, Li K, et al. Boundary and entropy-driven adversarial learning for fundus image segmentation [C]. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2019: 102-110.

[0172]

[12] Chen C, Wang G. IOSUDA: an unsupervised domain adaptation with input and output space alignment for joint optic disc and cup segmentation [J]. Applied Intelligence, 2021, 51(6): 3880-3898.

[0173]

[13] Sivaswamy J, Krishnadas S, Chakravarty A, et al. A comprehensive retinal image dataset for the assessment of glaucoma from the optic nerve head analysis [J]. JSM Biomedical Imaging Data Papers, 2015, 2(1): 1004.

[0174]

[14] Fumero F, Alayón S, Sanchez J L, et al. RIM-ONE: An open retinal image database for optic nerve evaluation [C]. In 2011 24th international symposium on computer-based medical systems (CBMS), 2011: 1-6.

[0175]

[15] Orlando J I, Fu H, Breda J B, et al. Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs [J]. Medical image analysis, 2020, 59: 101570.

[0176]

[16] Zhang Y, Miao S, Mansi T, et al. Task driven generative modeling for unsupervised domain adaptation: Application to x-ray image segmentation [C]. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2018: 599-607.

[0177]

[17] Hoffman J, Wang D, Yu F, et al. Fcns in the wild: Pixel-level adversarial and constraint-based adaptation [J]. arXiv preprint arXiv:1612.02649, 2016.

[0178]

[18] Javanmardi M, Tasdizen T. Domain adaptation for biomedical image segmentation using adversarial training [C]. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018: 554-558.

[0179]

[19] Liu L, Zhang Z, Li S, et al. S-CUDA: Self-cleansing unsupervised domain adaptation for medical image segmentation [J]. Medical Image Analysis, 2021, 74: 102214.

[0180]

[20] Liu P, Kong B, Li Z, et al. CFEA: collaborative feature ensembling adaptation for domain adaptation in unsupervised optic disc and cup segmentation [C]. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2019: 521-529.

[0181] The types of the devices are not limited as long as the devices can complete the above functions, except for the types of the devices which are specially described in the embodiments of the present application.

[0182] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0183] The above-mentioned is only the preferred embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A dual-path self-supervised based cross-domain automatic segmentation method for cup and disc, characterized in that, The method comprises: Utilizing bidirectional fundus image domain transformation of source domain-target domain and target domain-source domain, respectively generating source domain image of target domain style and target domain image of source domain style, the source domain image has segmentation label, and the target domain image is without segmentation label; Utilizing DeepLabV3+ image segmentation network as a framework, and MobileNetV2 feature extractor, a double-path fundus image segmentation network is constructed; Fusion is performed on the predicted segmentation results of the unsupervised images output by the double-path network to obtain fusion pseudo labels, and the fusion pseudo labels are used to guide further training of the unlabeled images; A cross-domain contrast constraint mechanism is utilized to optimize the similarity of image features before and after domain transformation, so that the features retain information related to image structure; The fusion pseudo labels are obtained by fusing the predicted segmentation results of the unsupervised images output by the double-path network. unsupervised images and are input into a dual-path image segmentation network, and the output prediction segmentation results are and ; Will and fusion, generate fusion pseudo label , the fusion mode is: For the first channel graph , the first pixel point prediction value is , the confidence value is calculated by using the following formula : ； ； ； wherein, is an exponential function with base is a natural constant confidence values of all prediction points form a confidence map Do the same processing, get confidence map Do the same processing, get confidence map , get relative confidence map and Connect, through the softmax layer, get relative confidence map and , reorganize the relative confidence map of each channel into a confidence matrix and ; use and right and Perform weighted fusion to obtain fusion pseudo-labels. As shown in the following formula: ； In further training, the pseudo labels are used as supervision information for the no-label images and .

2. The cross-domain optic cup and disk automatic segmentation method based on dual-path self-supervision according to claim 1, wherein, The double path is that two segmentation networks adopt the same image segmentation framework but do not share network parameters.

3. The dual-path self-supervised based cross-domain optic cup and disc automatic segmentation method according to claim 1, wherein, The bidirectional fundus image domain transformation of source domain-target domain and target domain-source domain is: performing two-dimensional Fourier transform on the source domain image and the target domain image to obtain their respective frequency spectrums and ; Extracting the source domain spectrum Replace a central rectangular block in the amplitude plot with the target domain spectrum. At the corresponding position, a new spectrum is generated. Then Perform inverse Fourier transform to generate a target domain image in the style of the source domain. Its shape, position, and texture information are consistent with the original target domain image. Consistent; Replacing low frequency information in target domain spectrum with source domain spectrum , and then performing inverse Fourier transform on the replaced spectrum to obtain a source domain image with target domain style , whose shape, position, and texture information are consistent with the original source domain image .

4. The dual-path self-supervised based cross-domain optic cup and disc automatic segmentation method according to claim 1, wherein, The cross-domain contrast constraint mechanism is: The original image and the copy image are regarded as positive samples, and the negative samples are randomly selected, and the same reason; Using a contrastive loss function , pulling the distance between features of positive samples, is of the form: ； wherein, is a logarithm function with base 2, denotes the Euclidean distance, denotes a Gaussian kernel function, denotes the number of images inputted in a training batch; for positive sample images and and and respectively apply a contrastive loss function to reinforce semantic information in the encoded features that is relevant to the segmentation task.

5. A dual-path self-supervised based cross-domain optic cup and disc automatic segmentation device, characterized in that, The device comprises a processor and a memory, the memory stores program instructions, and the processor invokes the program instructions stored in the memory to enable the device to perform the method of any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program comprises program instructions, and the program instructions are executed by the processor to enable the processor to perform the method of any one of claims 1-4.

Images