Eye ground color photo optic cup and optic disc segmentation method and device, equipment, medium and product

Abstract

The invention relates to the field of eye fundus color photo optic cup and optic disc segmentation, and provides an eye fundus color photo optic cup and optic disc segmentation method and device, equipment, a medium and a product, and the method comprises the steps: obtaining a to-be-segmented eye fundus color photo; inputting the fundus photo to be segmented into the trained fundus photo optic cup and optic disc segmentation model to obtain a corresponding optic cup and optic disc segmentation result; the fundus color photo optic cup and optic disc segmentation model is constructed by replacing an encoder in a DPT architecture with a DINOv2 encoder. The training process comprises a process of diffusion training of a DINOv2 encoder, a consistency training process combining image-level spatial transformation and feature-level hidden space disturbance, and a process of adaptive optimization of pseudo labels in the consistency training process based on category prototype guidance. According to the method, the utilization rate of limited annotation data is improved, the medical image annotation cost is remarkably reduced, meanwhile, the image segmentation precision is ensured, and reliable technical support can be provided for eye disease diagnosis.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of fundus color photograph optic cup optic disc segmentation, and particularly relates to a fundus color photograph optic cup optic disc segmentation method and device, equipment, medium and product. BACKGROUND

[0002] With the rapid development of deep learning technology, especially the breakthrough of Convolutional Neural Network (CNN), the image field has made remarkable progress in the past decade. Deep learning methods, through end-to-end training, have shown their powerful ability in image classification, object detection, image generation and segmentation, etc. Especially in the fields of medical image processing, automatic driving, security monitoring, etc., the application of deep learning models has greatly promoted the process of automation and intelligentization.

[0003] In the image segmentation task, deep learning models can automatically identify different regions or targets in the image, such as tumors, blood vessels or organs, etc. This usually requires a large amount of labeled data to train the model, so as to obtain accurate segmentation results by using the trained model. However, in many practical applications, such as the segmentation of optic cup and disc and blood vessels in fundus color photographs, the segmentation of each layer structure of the eye in Optical Coherence Tomography (OCT) images, the segmentation of blood vessels in Optical Coherence Tomography Angiography (OCTA), the segmentation of nerve fibers in corneal confocal microscopy, etc., it is difficult to obtain a large amount of high-quality labeled data. Moreover, the process of labeling data with labels is not only time-consuming and laborious, but also often limited by the availability of expert resources due to the need for professional knowledge. This makes the scarcity of labeled data a major bottleneck for the application of deep learning in the field of ophthalmic images.

[0004] To solve this problem, semi-supervised learning has gradually become an important research direction in image segmentation tasks. Semi-supervised learning methods use a large amount of unlabeled data and a small amount of labeled data to improve the learning ability of the model. This method not only alleviates the problem of insufficient labeled data, but also improves the generalization ability of the model, especially in the field of medical images where data labeling is difficult or expensive, and has great potential.

[0005] Semi-supervised learning methods can achieve segmentation using a small amount of labels, and have some applications in medical field in recent years. For example, Wang et al. proposed three new methods, MI-SelfL, MI-SemiL and MI-SESEL, which exploit the intrinsic relationship and complementarity of multi-sequence medical images using self-supervised and semi-supervised learning methods, and verified the effectiveness of these methods in improving model performance in liver cell carcinoma grading and prostate cancer classification tasks. Tang et al. proposed a novel semi-supervised medical image segmentation method, which combines an adversarial training mechanism and a collaborative consistency learning strategy, and introduces it into the average teacher model. Through the adversarial training mechanism, the discriminator can generate a confidence map for the unlabeled data, providing more reliable supervision information for the student network. In the adversarial training process, the semi-supervised medical image segmentation method further proposes a collaborative consistency learning strategy to help the main discriminator obtain higher quality supervision information. Gao et al. proposed a new image segmentation framework SOUSA, which effectively improves the performance of medical image segmentation tasks by combining weak supervision and semi-supervised learning with sparse labeled data and unlabeled data, reducing the dependence on large amounts of high-quality labeled data, and has strong robustness and wide application potential. Yang et al. proposed a semi-supervised 3D medical image segmentation model based on collaborative training, which improves the segmentation accuracy and generalization ability of computed tomography images through multi-view and multi-modal data augmentation, confidence weighted pseudo label generation, and three-stage training strategy, which helps to improve the efficiency and accuracy of clinical diagnosis. Wu et al. proposed a novel mutual consistency network MC-Net+, which effectively utilizes unlabeled data through multi-decoder structure and mutual consistency constraints to improve the performance of medical image semi-supervised segmentation, especially in the segmentation accuracy of fuzzy regions, and outperforms existing methods on multiple public medical datasets, establishing a new benchmark for semi-supervised medical image segmentation. Luo et al. proposed a semi-supervised medical image segmentation method based on reliable pseudo label learning, which evaluates the reliability of pseudo labels by comparing the prediction confidence and intra-class similarity of twin networks, reduces noise introduction, and thus improves segmentation performance, and achieves the latest optimal performance on multiple medical datasets. Yang et al. proposed a unified dual-flow perturbation framework UniMatch, which significantly expands the perturbation space in semi-supervised segmentation by introducing auxiliary feature perturbation flow and dual-flow image perturbation technology, and achieves good performance in segmentation tasks on multiple benchmark datasets.

[0006] However, the above prior art still has some deficiencies. For example, many semi-supervised learning models lack excellent pre-training steps for adapting to target fields. Almost all semi-supervised semantic segmentation works still insist on using the outdated Resnet encoder and use the small-scale dataset Imagenet-1K for pre-training. Currently, the mainstream consistency learning paradigm in the field of semi-supervised medical image segmentation generally focuses on image-level perturbation strategies (such as spatial transformation / data augmentation), which has double limitations in constraint dimension: on the one hand, the feature space perturbation robustness is missing: existing methods rarely construct a perturbation-invariance constraint mechanism at the feature level, resulting in network sensitivity to hidden space noise; on the other hand, the cross-modal semantic consistency modeling is insufficient: in the multi-modal image analysis scene such as retinal blood vessels, there is a lack of topological preservation strategies for semantic correlation between feature channels. In addition, some prior art also lacks a method for improving the quality of pseudo-labels during training. SUMMARY

[0007] The present disclosure aims to at least solve one of the problems in the prior art, and provide an eye fundus color photograph optic cup and disc segmentation method and device, equipment, medium, and product.

[0008] In one aspect of the present disclosure, an eye fundus color photograph optic cup and disc segmentation method is provided, which comprises:

[0009] Obtaining an eye fundus color photograph to be segmented;

[0010] Inputting the eye fundus color photograph to be segmented into a trained eye fundus color photograph optic cup and disc segmentation model to obtain a corresponding optic cup and disc segmentation result;

[0011] The eye fundus color photograph optic cup and disc segmentation model is constructed by replacing the encoder in the DPT architecture with a DINOv2 encoder, and the training process of the eye fundus color photograph optic cup and disc segmentation model comprises a diffusion training process of the DINOv2 encoder, a consistency training process combining image-level spatial transformation and feature-level hidden space perturbation, and a process of adaptively optimizing the pseudo-label in the consistency training process based on class prototypes.

[0012] Optionally, the diffusion training process of the DINOv2 encoder comprises:

[0013] Obtaining a model dataset comprising a plurality of eye fundus color photographs containing optic cup and disc segmentation annotations;

[0014] Segmentation preprocessing is performed on the eye fundus color photographs in the model dataset to obtain cropped images containing optic cup and disc regions and satisfying a preset pixel requirement;

[0015] extracting features of the cropped image by using the DINOv2 encoder to generate a feature map;

[0016] adding Gaussian noise to the feature map through a forward diffusion process to generate a noisy image, and uniformly and randomly sampling noise intensity during the entire forward diffusion process;

[0017] removing noise in the noisy image through a reverse denoising process to obtain a corresponding denoised image;

[0018] In the forward diffusion process and the reverse denoising process, the parameters of the DINOv2 encoder are updated using a pre-trained objective function of diffusion to obtain the fundus color photograph optic disc and cup segmentation model after diffusion training.

[0019] Optionally, the forward diffusion process and the reverse denoising process are both implemented through a DDPM model.

[0020] Optionally, the consistency training process combining image-level spatial transformation and feature-level latent space disturbance includes:

[0021] obtaining the fundus color photograph in the model data set;

[0022] generating a weakly enhanced copy corresponding to the fundus color photograph through weak data enhancement processing, and generating a first strongly enhanced copy and a second strongly enhanced copy corresponding to the fundus color photograph through strong data enhancement processing;

[0023] consistency loop iteration process: inputting the weakly enhanced copy into an exponential moving average teacher model to generate pseudo labels; wherein the exponential moving average teacher model is obtained by applying exponential moving average to the fundus color photograph optic disc and cup segmentation model after diffusion training; inputting the first strongly enhanced copy and the second strongly enhanced copy into the fundus color photograph optic disc and cup segmentation model after diffusion training to obtain corresponding first prediction segmentation results and second prediction segmentation results; calculating consistency loss according to the distribution difference between the pseudo labels and the first prediction segmentation results and the distribution difference between the pseudo labels and the second prediction segmentation results; updating the parameters of the fundus color photograph optic disc and cup segmentation model after diffusion training according to the consistency loss and the gradient, and updating the exponential moving average teacher model according to the updated parameters of the fundus color photograph optic disc and cup segmentation model;

[0024] repeating the consistency loop iteration process until a preset consistency loop iteration number is reached.

[0025] Optionally, the weak data enhancement processing includes any one or more of random scaling processing, random cropping processing, and horizontal flipping processing.

[0026] The strong data enhancement processing includes any one or more of color disturbance processing, grayscale transformation processing, random blur processing, and CutMix operation.

[0027] Optionally, the process of guiding adaptive optimization of the pseudo label in the consistency training process based on the category prototype includes:

[0028] Obtaining the fundus color photograph in the model dataset;

[0029] A pseudo label updating iteration process: inputting the fundus color photograph into a first EMA teacher model to obtain a first pseudo label, and calculating a category prototype corresponding to each category according to the first pseudo label; inputting the fundus color photograph into a second EMA teacher model to obtain a second pseudo label; calculating a distance between the category prototype and the second pseudo label, eliminating the second pseudo label far away from the corresponding category prototype, and updating the pseudo label in the consistency cycle iteration process by using the remaining second pseudo label; inputting the fundus color photograph into the fundus color photograph optic cup and optic disc segmentation model after diffusion training to obtain a corresponding prediction result; determining a pseudo label training loss according to the remaining second pseudo label and the prediction result, updating the parameters of the fundus color photograph optic cup and optic disc segmentation model after diffusion training according to the pseudo label training loss, and updating the first EMA teacher model and the second EMA teacher model according to the updated parameters of the fundus color photograph optic cup and optic disc segmentation model; wherein the first EMA teacher model and the second EMA teacher model are obtained by applying exponential moving average to the fundus color photograph optic cup and optic disc segmentation model after diffusion training.

[0030] Repeating the pseudo label updating iteration process until a preset pseudo label updating iteration number is reached.

[0031] Another aspect of the present disclosure provides a fundus color photograph optic cup and optic disc segmentation device, which comprises:

[0032] An acquisition module is configured to acquire a fundus color photograph to be segmented.

[0033] A segmentation module is configured to input the fundus color photograph to be segmented into a trained fundus color photograph optic cup and optic disc segmentation model to obtain a corresponding optic cup and optic disc segmentation result.

[0034] The fundus color photograph optic cup and optic disc segmentation model is constructed by replacing an encoder in a DPT architecture with a DINOv2 encoder, and a training process of the fundus color photograph optic cup and optic disc segmentation model includes a diffusion training process of the DINOv2 encoder, a consistency training process combining image-level spatial transformation and feature-level latent space disturbance, and a process of guiding adaptive optimization of a pseudo label in the consistency training process based on a category prototype.

[0035] In another aspect of the present disclosure, an electronic device is provided, comprising:

[0036] at least one processor; and

[0037] a memory communicatively connected to the at least one processor; wherein

[0038] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the fundus color photograph optic disc cup segmentation method described above.

[0039] In another aspect of the present disclosure, a computer-readable storage medium is provided, storing a computer program, and the computer program is executed by a processor to implement the fundus color photograph optic disc cup segmentation method described above.

[0040] In another aspect of the present disclosure, a computer program product is provided, comprising a computer program, and the computer program is executed by a processor to implement the fundus color photograph optic disc cup segmentation method described above.

[0041] Compared with the prior art, the present disclosure constructs a fundus color photograph optic disc cup segmentation model by replacing the encoder in the DPT architecture with a DINOv2 encoder, adds a diffusion training process for the DINOv2 encoder in the training process of the fundus color photograph optic disc cup segmentation model, combines an image-level spatial transformation and a feature-level latent space disturbance consistency training process, and guides the self-adaptive optimization of pseudo-labels in the consistency training process based on class prototypes, and uses the trained fundus color photograph optic disc cup segmentation model for fundus color photograph optic disc cup segmentation. The trained fundus color photograph optic disc cup segmentation model only needs a small amount of labeled data to achieve a segmentation performance comparable to that of a fully supervised model, improves the utilization of limited labeled data, significantly reduces the cost of medical image labeling, while ensuring the image segmentation accuracy, effectively solves the problem of high cost and low efficiency of obtaining high-quality labeled data in the fundus color photograph optic disc cup segmentation task, and provides reliable technical support for eye disease diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0042] One or more embodiments are illustrated by way of example in the drawings, which are not intended to be limiting of the embodiments, and like reference numerals refer to like elements in the drawings and are used herein to illustrate exemplary embodiments by way of example. The drawings in which:

[0043] Figure 1 A flowchart of a fundus color photograph optic disc cup segmentation method according to an embodiment of the present disclosure is provided.

[0044] Figure 2 A schematic diagram of an ocular fundus photograph optic cup optic disc segmentation model and its related training process according to another embodiment of the present disclosure;

[0045] Figure 3 A schematic diagram of a consistency learning training process according to another embodiment of the present disclosure;

[0046] Figure 4 A schematic diagram of a prototype online updating pseudo-label process according to another embodiment of the present disclosure;

[0047] Figure 5 A schematic diagram of an ocular fundus photograph and its corresponding optic cup optic disc segmentation result according to another embodiment of the present disclosure;

[0048] Figure 6 A schematic diagram of an ocular fundus photograph optic cup optic disc segmentation device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure clearer, the embodiments of the present disclosure will be described in detail below with reference to the drawings. However, those skilled in the art can understand that in the embodiments of the present disclosure, many technical details are proposed in order to make the readers better understand the present disclosure. However, the technical solutions claimed by the present disclosure can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific embodiments of the present disclosure. The embodiments can be combined and referenced with each other without contradiction.

[0050] One embodiment of the present disclosure relates to an ocular fundus photograph optic cup optic disc segmentation method, the flow of which is shown in Figure 1 and includes steps S110 and S120.

[0051] Step S110, obtaining an ocular fundus photograph to be segmented.

[0052] Specifically, the ocular fundus photograph to be segmented here refers to an ocular fundus photograph that needs to be segmented into an optic cup and an optic disc.

[0053] In step S120, the fundus photograph to be segmented is input into the trained fundus photograph optic cup and disc segmentation model to obtain a corresponding optic cup and disc segmentation result. The fundus photograph optic cup and disc segmentation model is constructed by replacing the encoder in the DPT architecture with a DINOv2 encoder. The training process of the fundus photograph optic cup and disc segmentation model includes a diffusion training process for the DINOv2 encoder, a consistency training process combining image-level spatial transformation and feature-level latent space disturbance, and a process of adaptively optimizing pseudo labels in the consistency training process based on class prototype guidance.

[0054] Specifically, in combination with the above Figure 2 The fundus photograph optic cup and disc segmentation model provided by the embodiment is constructed by replacing the encoder in the DPT architecture with a DINOv2 encoder, i.e., includes a DINOv2 encoder and a decoder, which is the decoder in the DPT architecture. By using this semi-supervised learning framework, a good segmentation model can be trained using a small amount of optic cup and disc segmentation labeled data, thereby solving the problem of medical data labeling difficulty.

[0055] After the fundus photograph optic cup and disc segmentation model is subjected to diffusion training for the DINOv2 encoder, consistency learning training, i.e., consistency training combining image-level spatial transformation and feature-level latent space disturbance, and prototype online update pseudo labels, i.e., adaptive optimization of pseudo labels in the consistency training process based on class prototype guidance, it can receive an input fundus photograph to be segmented. The encoder extracts features from the input fundus photograph, the decoder performs segmentation prediction based on the extracted features, and finally outputs a corresponding optic cup and disc segmentation result.

[0056] The DPT architecture is a dense prediction architecture based on an encoder-decoder design, which uses a transformer as the basic computational building block of the encoder and gradually combines feature representations into the final prediction result through a convolutional decoder, which can provide better fine-grained and more globally consistent predictions. However, the encoder in the DPT architecture may lose some information when processing data, which may make it difficult to recover this information in the decoder, thereby affecting the final prediction accuracy. Therefore, when constructing the fundus photograph optic cup and disc segmentation model, the encoder in the DPT architecture is replaced with a DINOv2 encoder, and the DINOv2 encoder is used as a feature extraction backbone network. Through a domain-specific fine-tuning strategy, the general visual representation capability obtained by pre-training is transferred to the medical image segmentation task, thereby significantly improving the feature expression capability of the model.

[0057] DINOv2 is a visual large model based on the Transformer architecture, which has very strong representation ability and generalization ability. Unlike traditional visual models, DINOv2 is based on the Vision Transformer (ViT) architecture and is trained in an unsupervised training framework. It can learn useful features and representations from a large amount of unlabeled data through self-supervised learning.

[0058] The process of diffusion training the DINOv2 encoder is used to make the DINOv2 encoder learn effective representations for the fundus photograph optic disc and cup segmentation task through diffusion training. After diffusion training, the DINOv2 encoder can improve the accuracy of early pseudo-labels in the consistency training process, thereby preventing the accumulation of errors.

[0059] A dual-channel perturbation consistency learning mechanism is designed by combining image-level spatial transformation and feature-level latent space perturbation. Based on traditional image-level spatial transformation (such as affine transformation and elastic deformation), feature-level latent space perturbation is introduced. By applying random noise perturbation and channel attention mask in the feature space, more comprehensive feature robustness constraints are achieved.

[0060] The process of adaptively optimizing pseudo-labels in the consistency training process based on class prototypes guides the calculation of data classification prototypes. According to the distance between prototype features and samples, low-quality pseudo-labels are removed, thereby achieving online updating of pseudo-labels based on prototypes, improving the quality of pseudo-labels, dynamically optimizing the accuracy and reliability of pseudo-labels, and thus improving the training effect of the model.

[0061] The fundus photograph optic disc and cup segmentation method provided by the embodiments of the present disclosure replaces the encoder in the DPT architecture with a DINOv2 encoder to construct a fundus photograph optic disc and cup segmentation model. In the training process of the fundus photograph optic disc and cup segmentation model, the process of diffusion training the DINOv2 encoder, the consistency training process combining image-level spatial transformation and feature-level latent space perturbation, and the process of adaptively optimizing pseudo-labels in the consistency training process based on class prototypes are added. The trained fundus photograph optic disc and cup segmentation model is used for fundus photograph optic disc and cup segmentation, so that the trained fundus photograph optic disc and cup segmentation model can achieve segmentation performance comparable to that of a fully supervised model with only a small amount of labeled data, improving the utilization of limited labeled data, significantly reducing the cost of medical image labeling, while ensuring image segmentation accuracy, effectively solving the problem of high cost and low efficiency of obtaining high-quality labeled data in the fundus photograph optic disc and cup segmentation task, and providing reliable technical support for eye disease diagnosis.

[0062] An exemplary process of diffusion training the DINOv2 encoder includes: obtaining a model dataset, the model dataset including a plurality of fundus color images containing optic cup and disc segmentation annotations; performing segmentation preprocessing on the fundus color images in the model dataset to obtain cropped images containing optic cup regions and disc regions and satisfying a preset pixel requirement; performing feature extraction on the cropped images using the DINOv2 encoder to generate feature maps; adding Gaussian noise to the feature maps through a forward diffusion process to generate noise images, and uniformly and randomly sampling noise intensity during the entire forward diffusion process; removing the noise in the noise images through a backward denoising process to obtain corresponding denoised images; and updating parameters of the DINOv2 encoder using a diffusion pre-training objective function during the forward diffusion process and the backward denoising process to obtain a fundus color image optic cup and disc segmentation model after diffusion training.

[0063] For example, the model dataset can use the Refuge2 dataset. The Refuge2 dataset is a publicly available dataset dedicated to glaucoma, which contains segmentation annotations about the optic disc and cup, and it contains 2000 images (1200 training images, 400 validation images, and 400 test images). By training using the Refuge2 dataset, the trained fundus color image optic cup and disc segmentation model can be used to segment the disc region and cup region in the fundus color image.

[0064] During segmentation preprocessing, to balance the foreground and background classes, the U-NET network can be used to perform coarse segmentation of the disc region and cup region in the images in the Refuge2 dataset to generate cropped images satisfying a preset pixel requirement, such as 500x500 pixels.

[0065] To improve training efficiency, the DINOv2 encoder trained under the ViT-based unsupervised training framework pre-trained on 1.42 billion images can be used to perform feature extraction on the cropped images to generate corresponding feature maps.

[0066] During the forward diffusion process, for the input feature map x, Gaussian noise is added to the feature map x based on the scalar noise level, i.e., the noise intensity γ, to generate a noise image which can be represented as:

[0067]

[0068] where x is weakened and is attenuated to ensure that the difference between x and ∈ is 1. ∈ represents Gaussian noise, which is subject to a Gaussian distribution I is an identity matrix.

[0069] In particular, the above forward diffusion process and reverse denoising process can be implemented by a DDPM model. The DDPM model, i.e., a denoising diffusion probability model, is a generative deep learning model based on a diffusion process, which generates high-quality images by gradually adding and removing noise. The DDPM model constructs a forward diffusion process from a clear image to pure noise and a reverse denoising process, and uniformly randomly samples the noise intensity γ in the entire diffusion process to guide the model training.

[0070] Based on the above, the pre-training objective function of diffusion can be used to make the DINOv2 encoder learn high-quality representations, thereby updating the parameters of the DINOv2 encoder. The pre-training objective function of diffusion can be represented as: wherein, represents the expectation of the feature map x, represents the expectation of the Gaussian noise ∈, represents the expectation of the noise intensity γ. θ represents an image-to-image translation architecture, which can refer to the DINOv2 encoder here. p(γ) defines the noise schedule of the DDPM model. The pre-training objective function of diffusion represents a single iteration of the diffusion process modeled by the DDPM. During training, the parameters of the DINOv2 encoder can be updated by minimizing the objective function. After the above diffusion training, the DINOv2 encoder can improve the accuracy of early pseudo-labels in the subsequent consistency training process, thereby preventing the accumulation of errors.

[0071] For example, in combination with Figure 3 , the consistency training process, i.e., the consistency learning training process, which combines image-level spatial transformation and feature-level latent space perturbation, includes steps 3-1 to 3-4.

[0072] Step 3-1, obtaining the fundus color photograph in the model dataset.

[0073] Step 3-2, generating a weakly enhanced copy of the fundus color photograph corresponding to the fundus color photograph (weak enhancement in Figure 3 ) through weak data augmentation processing, and generating a first strongly enhanced copy of the fundus color photograph corresponding to the fundus color photograph (strong enhancement 1 in Figure 3 ) and a second strongly enhanced copy of the fundus color photograph (strong enhancement 2 in Figure 3 ) through strong data augmentation processing.

[0074] Step 3-3, Consistency Iterative Process: The weakly enhanced copy is input into the exponential moving average teacher model to generate pseudo-labels; the exponential moving average teacher model is obtained by applying the exponential moving average to the diffusion-trained fundus photography optic disc segmentation model; the first and second strongly enhanced copies are input into the diffusion-trained fundus photography optic disc segmentation model, respectively. The encoder in this model, namely the DINOv2 encoder, extracts features from the first and second strongly enhanced copies, respectively. The extracted features are randomly discarded and then enter the decoder, which obtains the corresponding first predicted segmentation result. Figure 3 Prediction 1) and the second prediction segmentation result ( Figure 3 2) Prediction in the middle; calculate the consistency loss based on the distribution difference between the pseudo-label and the first predicted segmentation result and the distribution difference between the pseudo-label and the second predicted segmentation result; update the parameters of the diffusion-trained fundus color photograph optic cup and optic disc segmentation model based on the consistency loss and gradient, and update the exponential moving average teacher model based on the updated parameters of the fundus color photograph optic cup and optic disc segmentation model.

[0075] Steps 3-4: Repeat the consistency loop iteration process, i.e., step 3-3, until the preset number of consistency loop iterations is reached.

[0076] Specifically, the loss from pure feature augmentation is very small compared to the loss at the input level. The gap widens when the encoder in the weaker DPT architecture is replaced with the more powerful Dinov2 encoder. Therefore, this implementation unifies input-level and feature-level augmentations into a single stream. This approach has three advantages: first, the model can still pursue strong representativeness across a wide augmentation space; second, it eliminates the need to maintain separate streams of different augmentation types, thus improving training efficiency; and third, Dinov2 is more capable than previous resets. Therefore, stacking various augmentations is beneficial.

[0077] The single-stream loss can be expressed as:

[0078]

[0079] Where, p sf This represents the prediction result of the strongly enhanced copy, where h represents the decoder. Indicates feature-level enhancement, g represents the DINOv2 encoder, x s This indicates a strongly enhanced copy. B represents the unlabeled training loss. u This indicates the number of unlabeled images. This indicates an indicator function, which is 1 if the condition is met and 0 otherwise. This represents a weakly enhanced copy of the i-th unlabeled image. confidence of the most likely class that the weakly augmented copy of the ith unlabelled image belongs to, τ denotes the confidence, denotes the cross-entropy, denotes the pseudo label of the ith unlabelled image, denotes the prediction result of the strongly augmented copy of the ith unlabelled image.

[0080] together with Figure 3 , the present embodiment further proposes a complementary channel dropout to obtain two disjoint and complementary function sets of the double stream. Specifically, given the feature map e S1 ∈R B×C×H×W , where e S1 denotes the strongly data augmented image whose features are extracted by the encoder, i.e., DINOv2 encoder, R denotes the real space to which the tensor e S1 belongs, B denotes the batch size, C denotes the number of channels, H denotes the height, and W denotes the width. From x s1 input stream, a down mask M of the same shape as e S1 is randomly generated. M is a binary mask sampled from a binary distribution (probability of 0.5), half of the channels (C / 2) are all set to 1, and the other half are set to 0. Using M, complementary channel dropout can be performed on the features and features : s1 ) and g(x s2 ):

[0081]

[0082] where g(x s1 ) denotes the first strongly augmented copy processed by the encoder, i.e., g(x s2 ) denotes the second strongly augmented copy processed by the encoder, i.e., M and 1-M are two complementary masks. The last scaling factor "2" is to ensure that the output expectation is the same as the normal function. It should be noted that, and are extracted by the shared encoder. They each have the same features in each channel. Therefore, by using the complementary masks M and (1-M), two disjoint function sets will be obtained, which have the same meaning as double stream learning. In addition, although the random dropout is performed at the intersection of the encoder and the decoder, the gradient will go back to the encoder, making the entire model stronger. Given the double complementary stream of input-level and function-level enhancement, the final unlabelled loss is expressed as:

[0083]

[0084] where, a prediction result of the first strong augmented copy, a prediction result of the second strong augmented copy, a loss of the unlabeled image, a prediction result of the first strong augmented copy of the i-th unlabeled image, a prediction result of the second strong augmented copy of the i-th unlabeled image, H represents cross-entropy.

[0085] The embodiment maintains an exponentially moving average teacher model, i.e., an exponentially moving average is applied to the exponentially moving average teacher model obtained by applying the diffusion training to the fundus photograph optic disc and cup segmentation model, instead of using the student model, i.e., the fundus photograph optic disc and cup segmentation model after diffusion training, to generate stable and better pseudo labels. The parameters θ t in the teacher model are updated together with the parameters θ s in the student model: θ t ← γ × θ t + (1-γ) × θ s .

[0086] It should be noted that the weak data augmentation process usually includes some mild image transformations, such as color jittering, slight cropping, scaling, rotation, etc., which can enhance the robustness of the model to small changes in the image without changing the main features of the image. For example, the weak data augmentation process includes any one or more of random scaling processing, random cropping processing, and horizontal flipping processing. The scaling ratio of the random scaling processing can be between 0.5 and 2.0. The probability of horizontal flipping can be set to 0.5.

[0087] The strong data augmentation process uses more significant transformations, such as larger cropping, rotation, color shift, random blur, etc., which can force the model to learn more challenging and more diverse feature representations. For example, the strong data augmentation process includes any one or more of color perturbation processing, grayscale transformation processing, random blur processing, and CutMix operation. The random blur processing can be Gaussian blur processing.

[0088] During the training process, the embodiment uses weak augmentation to generate pseudo labels, and uses these pseudo labels as supervision signals for the training of strong augmented images.

[0089] For example, in combination with Figure 4 , the process of adaptively optimizing the pseudo labels in the consistency training process based on the category prototype, i.e., the prototype online updating pseudo label process, includes steps 4-1 to 4-3.

[0090] Step 4-1, obtaining the fundus photograph in the model dataset.

[0091] Step 4-2, pseudo-label update iteration process: Input the fundus color photo into the first EMA teacher model ( Figure 4 The EMA teacher 1 is used to obtain the first pseudo-label, and the category prototype corresponding to each category is calculated based on the first pseudo-label; the fundus color image is input into the second EMA teacher model ( Figure 4 In the EMA teacher 2), the second pseudo-label is obtained; the distance between the category prototype and the second pseudo-label is calculated, and the second pseudo-label that is far away from the corresponding category prototype is removed. The remaining second pseudo-label is used to update the pseudo-label in the consistency loop iteration process; the fundus color photo is input into the fundus color photo optic cup-optic disc segmentation model after diffusion training. Figure 4 The training model in the image is used to obtain the corresponding prediction results; the pseudo-label training loss is determined based on the remaining second pseudo-label and the prediction results; the parameters of the diffuse-trained fundus color photograph optic cup and optic disc segmentation model are updated based on the pseudo-label training loss; and the first EMA teacher model and the second EMA teacher model are updated based on the updated parameters of the fundus color photograph optic cup and optic disc segmentation model. The first EMA teacher model and the second EMA teacher model are both obtained by applying exponential moving average to the diffuse-trained fundus color photograph optic cup and optic disc segmentation model, which includes a Dinov2 encoder and a decoder.

[0092] Step 4-3: Repeat the pseudo-label update iteration process, i.e., step 4-2, until the preset number of pseudo-label update iterations is reached.

[0093] Specifically, the key to the adaptive optimization process of pseudo-labels, i.e., the online updating of pseudo-labels, lies in fixing the soft pseudo-labels and gradually weighting them according to category probability, updating the pseudo-labels as new knowledge is learned. Therefore, this implementation proposes using weighted pseudo-labels. Perform self-training:

[0094]

[0095] in, To adjust the probability, The weights proposed are called modulation weights, which change as training progresses. It is initialized by the source model and remains fixed throughout the learning process, thus serving as a template for subsequent refinement.

[0096] This implementation method gradually corrects false labels by utilizing the distance from the feature to the prototype. Let f(x) t ) (i) Indicates feature x t The function at index i, if it is far from the category prototype η (k)If the feature vector of the i-th sample (or spatial location) of the input data x t is far away from the feature prototype of the k-th class, then it is more likely to be an outlier, and thus the likelihood of being classified as the k-th class decreases. Specifically, the modulation weight softmax over the feature distance to the class prototype can be defined as:

[0097]

[0098] where f denotes the momentum encoder of the feature extractor f, which here is desired to make a reliable feature estimate of x t , and τ is the empirical soft magnet temperature set to τ = 1. denotes the feature vector of the i-th sample (or spatial location) of the input data x t after feature extraction by f, and η (k′) denotes the feature prototype of the k'-th class. In the above definition of η , is actually approximated by the i-th sample (or spatial location) x t of the input data x t (i) after feature extraction by f. The confidence score of the i-th sample (or spatial location) of the input data x t to belong to the k-th class. Here, instead of relying on the class prototype to classify new samples, we try to correct the errors in the pre-generated pseudo labels according to this class prototype context.

[0099] The pseudo label updating iteration process proposed in step 4-2 requires the class prototype to be calculated on the fly.

[0100] First, the class prototype η (k) corresponding to the k-th class is initialized according to the predicted pseudo labels of the target domain images:

[0101]

[0102] where is the indicator function. x t denotes a single sample in the target domain . However, this class prototype calculation is computationally intensive during the training process. To solve this problem, the present embodiment estimates the class prototype as the moving average of the group prototypes in the mini-batch, so that we can track the slowly moving prototypes. Specifically, in each iteration, the class prototype η (k) is estimated as:

[0103] η (k) ←λη (k) +(1-λ)η′ (k)

[0104] where η′ (k)This is the average feature of the k-th class calculated by the momentum encoder in the current training batch. λ is the momentum coefficient, which can be set to 0.9999.

[0105] By updating pseudo-labels online, the network can be trained to segment the target domain. This implementation does not use standard cross-entropy, but instead employs a more robust symmetric cross-entropy (SCE) loss to further enhance the stability of the noise and thus stabilize the early training phase. Specifically, this implementation performs the following symmetric cross-entropy loss...

[0106]

[0107] Here, α and β are set to 0.1 and 1, respectively. Let p represent the standard cross-entropy and the predicted probability. t and pseudo-tags The differences between them. Represents the inverse cross-entropy, and is a pseudo-label. and prediction probability p t The differences between them. Later, we will discuss pseudo-tags. Brace to [1e-4, 1] to avoid numerical issues with log 0.

[0108] This implementation method calculates category prototypes to roughly estimate the distribution of data in the target domain, thus providing a basis for domain adaptation of the pre-trained base model DINOv2 in natural scenes. This effectively reduces the distribution difference between the source and target domains, improving the model's generalization ability in the target domain. By combining the distribution information of the target domain, DINOv2 can better adapt to different data features, thereby improving its performance in the target domain, reducing the large amount of labeled data required in traditional domain adaptation methods, and ultimately achieving more efficient cross-domain transfer learning.

[0109] For example, regarding Figure 5 The fundus images shown in the first column are segmented into optic cups and optic discs through manual annotation, as shown below. Figure 5 As shown in the second column of the images, the trained fundus photography cup-optic disc segmentation model provided by the above implementation method is used to... Figure 5 The segmentation results obtained by segmenting the optic cup and optic disc from the fundus color images shown in the first column are as follows: Figure 5 The corresponding images in the third column are shown below. Figure 5 In the images shown in the second and third columns, the white areas represent the optic cup region, the gray areas represent the optic disc region, and the black areas represent the background. From Figure 5As can be seen, the optic cup and disc segmentation result obtained by using the trained fundus photograph optic cup and disc segmentation model provided in the above embodiments has high similarity with the optic cup and disc segmentation result obtained by manual annotation, which indicates that the trained fundus photograph optic cup and disc segmentation model provided in the above embodiments has high segmentation accuracy and segmentation precision, and can meet the accuracy requirement and segmentation precision requirement of the fundus photograph optic cup and disc segmentation task.

[0110] Another embodiment of the present disclosure relates to a fundus photograph optic cup and disc segmentation device, as shown in the accompanying drawings, comprising an acquisition module 610 and a segmentation module 620. Figure 6 The segmentation module 620 is configured to input the fundus photograph to be segmented into the trained fundus photograph optic cup and disc segmentation model to obtain the corresponding optic cup and disc segmentation result.

[0111] The acquisition module 610 is configured to acquire a fundus photograph to be segmented.

[0112] The segmentation module 620 is configured to input the fundus photograph to be segmented into the trained fundus photograph optic cup and disc segmentation model to obtain the corresponding optic cup and disc segmentation result. The fundus photograph optic cup and disc segmentation model is constructed by replacing the encoder in the DPT architecture with the DINOv2 encoder, and the training process of the fundus photograph optic cup and disc segmentation model comprises the diffusion training process of the DINOv2 encoder, the consistency training process combining image-level spatial transformation and feature-level latent space disturbance, and the process of adaptively optimizing the pseudo label in the consistency training process based on the category prototype guidance.

[0113] The specific implementation method of the fundus photograph optic cup and disc segmentation device provided in the embodiments of the present disclosure can be referred to the fundus photograph optic cup and disc segmentation method provided in the embodiments of the present disclosure, which will not be described here.

[0114] The fundus photograph optic cup and disc segmentation device provided in the embodiments of the present disclosure, compared with the prior art, constructs the fundus photograph optic cup and disc segmentation model by replacing the encoder in the DPT architecture with the DINOv2 encoder, adds the diffusion training process of the DINOv2 encoder, the consistency training process combining image-level spatial transformation and feature-level latent space disturbance, and the process of adaptively optimizing the pseudo label in the consistency training process based on the category prototype guidance in the training process of the fundus photograph optic cup and disc segmentation model, and uses the trained fundus photograph optic cup and disc segmentation model for fundus photograph optic cup and disc segmentation, so that the trained fundus photograph optic cup and disc segmentation model can achieve segmentation performance comparable to full-supervised models with only a small amount of labeled data, improves the utilization rate of limited labeled data, significantly reduces the cost of medical image annotation, while ensuring the image segmentation precision, effectively solves the problem of high cost and low efficiency of obtaining high-quality labeled data in the fundus photograph optic cup and disc segmentation task, and provides reliable technical support for eye disease diagnosis.

[0115] Another embodiment of the present disclosure relates to an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the fundus color cup disc segmentation method according to the above-mentioned embodiments.

[0116] The memory and the processor are connected in a bus mode, the bus can include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage stabilizers, and power management circuits, which are well known in the art, and thus will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, which provide a unit for communicating with various other devices on the transmission medium. Data processed by the processor is transmitted on the wireless medium through the antenna, and further, the antenna also receives data and transmits the data to the processor.

[0117] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store data used by the processor in performing operations.

[0118] Another embodiment of the present disclosure relates to a computer readable storage medium storing a computer program, which, when executed by a processor, implements the fundus color cup disc segmentation method according to the above-mentioned embodiments.

[0119] That is, those skilled in the art can understand that all or part of the steps of the method according to the above-mentioned embodiments can be completed by a program instructing related hardware, the program is stored in a storage medium, and includes a plurality of instructions for causing a device (which can be a single chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method according to the embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage medium capable of storing program codes.

[0120] Another embodiment of the present disclosure relates to a computer program product, comprising a computer program, which, when executed by a processor, implements the fundus color cup disc segmentation method according to the above-mentioned embodiments.

[0121] It is understood by those of ordinary skill in the art that the above embodiments are specific embodiments for implementing the present disclosure, and various changes can be made in form and details in practical applications without departing from the spirit and scope of the present disclosure.

Claims

1. A method for segmenting the optic cup and optic disc in fundus photography, characterized in that, The fundus photography method for optic cup and optic disc segmentation includes: Obtain a color image of the fundus to be segmented; The fundus color image to be segmented is input into the trained fundus color image cup-optic disc segmentation model to obtain the corresponding cup-optic disc segmentation result. The fundus photography optic cup-optic disc segmentation model is constructed by replacing the encoder in the DPT architecture with the DINOv2 encoder. The training process of the fundus photography optic cup-optic disc segmentation model includes a diffusion training process for the DINOv2 encoder, a consistency training process combining image-level spatial transformation and feature-level latent space perturbation, and an adaptive optimization process for pseudo-labels in the consistency training process based on category prototype guidance.

2. The method for segmenting the optic cup and optic disc in fundus photography according to claim 1, characterized in that, The process of performing diffusion training on the DINOv2 encoder includes: Obtain the model dataset, which includes several fundus color images containing visual cup and visual disc segmentation annotations; The fundus images in the model dataset are segmented and preprocessed to obtain a cropped image that includes the optic cup region and the optic disc region and meets the preset pixel requirements; The DINOv2 encoder is used to extract features from the cropped image to generate a feature map; Gaussian noise is added to the feature map through a forward diffusion process to generate a noisy image, and the noise intensity is uniformly and randomly sampled throughout the forward diffusion process. The noise in the noisy image is removed by a reverse denoising process to obtain the corresponding denoised image; In the forward diffusion process and the reverse denoising process, the parameters of the DINOv2 encoder are updated using the pre-trained objective function of diffusion to obtain the diffusion-trained fundus color photograph optic cup-optic disc segmentation model.

3. The method for segmenting the optic cup and optic disc in fundus photography according to claim 2, characterized in that, Both the forward diffusion process and the reverse denoising process are implemented using the DDPM model.

4. The method for segmenting the optic cup and optic disc in fundus photography according to claim 2, characterized in that, The consistency training process, which combines image-level spatial transformation and feature-level latent space perturbation, includes: Obtain the fundus color images from the model dataset; Weakly enhanced copies of the fundus images are generated by weak data augmentation, and first and second strongly enhanced copies of the fundus images are generated by strong data augmentation. Consistency Iterative Process: The weakly enhanced copy is input into the exponential moving average teacher model to generate pseudo-labels; wherein, the exponential moving average teacher model is obtained by applying the exponential moving average to the diffusion-trained fundus chromatic acupuncture cup-optic disc segmentation model; the first strongly enhanced copy and the second strongly enhanced copy are respectively input into the diffusion-trained fundus chromatic acupuncture cup-optic disc segmentation model to obtain corresponding first and second predicted segmentation results; the consistency loss is calculated based on the distribution differences between the pseudo-labels and the first predicted segmentation results and the distribution differences between the pseudo-labels and the second predicted segmentation results; the parameters of the diffusion-trained fundus chromatic acupuncture cup-optic disc segmentation model are updated based on the consistency loss and gradient, and the exponential moving average teacher model is updated based on the updated parameters of the fundus chromatic acupuncture cup-optic disc segmentation model; Repeat the consistency loop iteration process until the preset number of consistency loop iterations is reached.

5. The method for segmenting the optic cup and optic disc in fundus photography according to claim 4, characterized in that, The weak data augmentation process includes any one or more of random scaling, random cropping, and horizontal flipping. The strong data enhancement processing includes any one or more of the following: color perturbation processing, grayscale transformation processing, random blurring processing, and CutMix operation.

6. The method for segmenting the optic disc in fundus photography according to claim 4, characterized in that, The process of adaptively optimizing pseudo-labels during the consistency training process based on category prototype guidance includes: Obtain the fundus color images from the model dataset; The pseudo-label update iteration process is as follows: The fundus photograph is input into the first EMA teacher model to obtain the first pseudo-label; the category prototype corresponding to each category is calculated based on the first pseudo-label; the fundus photograph is input into the second EMA teacher model to obtain the second pseudo-label; the distance between the category prototype and the second pseudo-label is calculated, and second pseudo-labels that are far from the corresponding category prototype are removed; the remaining second pseudo-labels are used to update the pseudo-labels in the consistency loop iteration process; the fundus photograph is input into the diffusion-trained fundus photograph optic cup / optic disc segmentation model to obtain the corresponding prediction result; the pseudo-label training loss is determined based on the remaining second pseudo-labels and the prediction result; the parameters of the diffusion-trained fundus photograph optic cup / optic disc segmentation model are updated based on the pseudo-label training loss; and the first EMA teacher model and the second EMA teacher model are updated based on the updated parameters of the fundus photograph optic cup / optic disc segmentation model; wherein, both the first EMA teacher model and the second EMA teacher model are obtained by applying exponential moving average to the diffusion-trained fundus photograph optic cup / optic disc segmentation model; Repeat the pseudo-label update iteration process until the preset number of pseudo-label update iterations is reached.

7. A fundus photography viewing cup optic disc segmentation device, characterized in that, The fundus photography viewing cup optic disc segmentation device includes: The acquisition module is used to acquire color images of the fundus to be segmented; The segmentation module is used to input the fundus color image to be segmented into a trained fundus color image optic cup-optic disc segmentation model to obtain the corresponding optic cup-optic disc segmentation result. The fundus photography optic cup-optic disc segmentation model is constructed by replacing the encoder in the DPT architecture with the DINOv2 encoder. The training process of the fundus photography optic cup-optic disc segmentation model includes a diffusion training process for the DINOv2 encoder, a consistency training process combining image-level spatial transformation and feature-level latent space perturbation, and an adaptive optimization process for pseudo-labels in the consistency training process based on category prototype guidance.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the fundus photography optic disc segmentation method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the fundus photography optic cup and optic disc segmentation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the fundus photography optic cup and optic disc segmentation method as described in any one of claims 1 to 6.

Images