Cross-domain fundus color photograph arteriovenous classification model, training method and system thereof and application

Abstract

The application discloses a cross-domain fundus color photograph arteriovenous classification model and a training method, system and application thereof. The training method comprises the following steps: obtaining an enhanced image by preprocessing a target domain image; performing arteriovenous segmentation by using an initial segmentation model to obtain a segmentation result; performing uncertainty estimation to obtain an uncertainty degree; screening the uncertainty degrees above a dynamic threshold as high-quality labels, and obtaining pseudo labels based on the high-quality labels, wherein the dynamic threshold is increased at least with the training; and performing training by using the pseudo labels to obtain an arteriovenous classification model. The arteriovenous classification model training method provided by the application trains cross-domain fundus images by using a semi-supervised algorithm, and applies a dynamic threshold adjustment method to a segmentation task, thereby solving the problem that other pseudo label methods cannot improve difficult parts in fundus color photographs, completing the training without manual labeling of target domain data, and the training result is obviously improved compared with using only source domain data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a cross-domain fundus color photograph arteriole and vein classification model and a training method, system and application thereof. BACKGROUND

[0002] According to the statistics of the National Health Commission of China, more than 600 million people in China suffer from myopia, and the number of patients with cataract, glaucoma and diabetic retinopathy is increasing year by year. Fundus examination, as an effective examination method for checking various eye diseases and potential systemic diseases, is the preferred solution due to its non-invasive and safe characteristics. The blood vessel information of the retinal fundus image has been proven to be closely related to a variety of blood vessel-related diseases in clinical research, such as diabetes, hypertension, cardiovascular disease, etc. Therefore, the extraction of blood vessel information in the retinal color photograph image is of great help to the screening and diagnosis of related diseases.

[0003] For example, according to clinical research, the ratio of the area of the arteriole and the vein in the fundus, i.e., the arteriolar-to-velar diameter ratio (AVR), will change with diabetes, hypertension and some cardiovascular diseases, and has become a basis for clinical diagnosis of many eye diseases and systemic diseases.

[0004] However, the current fundus screening in clinical application is still realized in the form of manual film reading. However, due to the huge base number of screening people and the requirement of certain clinical experience for ophthalmologists, the human consumption is great, which often leads to insufficient medical resources. With the help of computer-aided retinal blood vessel analysis, some early retinal vascular diseases can be automatically screened and diagnosed to provide assistance for interventional treatment, which has very important research significance and application value. The classification of retinal arteriole and vein, as an important part of retinal blood vessel analysis, has very important research value.

[0005] At present, the method of deep learning has achieved great success in the task of fundus color photograph retinal arteriole and vein classification, and can effectively classify the arteriole and vein of the fundus color photograph taken by various devices. However, in actual application, it is found that due to the differences in different imaging devices, fields of view, resolutions, light intensities, etc., there are style differences in images taken in different environments or by different devices. This leads to the fact that the model trained under a certain specific environment and device cannot well classify the arteriole and vein of images taken by other devices, which will lead to the decline of the classification effect of the model. It is necessary to re-label the new images for retraining the model under other devices, which consumes a lot of manpower. With the development of semi-supervised methods, some algorithms can be trained with unlabeled data, overcoming the defect of re-labeling, but the effect of these semi-supervised methods applied to the classification of fundus color photograph arteriole and vein is still not good. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a semi-supervised cross-domain retinal blood vessel arteriovenous classification algorithm to solve the problem of repeated labeling.

[0007] To achieve the above-mentioned purposes, the technical scheme adopted by the present application comprises:

[0008] In a first aspect, the present application provides a cross-domain fundus color photograph arteriovenous classification model training method, comprising:

[0009] 1) performing image preprocessing on the target domain image to obtain an enhanced image;

[0010] 2) performing arteriovenous segmentation on the enhanced image using an initial segmentation model to obtain a segmentation result;

[0011] 3) performing uncertainty estimation on the segmentation result to obtain an uncertainty degree, screening the segmentation result with an uncertainty degree above a dynamic threshold as a high-quality label, and obtaining a pseudo label based on the high-quality label, wherein the dynamic threshold is at least increased as the training progresses;

[0012] 4) training the initial segmentation model using the pseudo label to obtain an arteriovenous classification model.

[0013] In a second aspect, the present application further provides a cross-domain fundus color photograph arteriovenous classification model trained by the above-mentioned model training method.

[0014] In a third aspect, the present application further provides a cross-domain fundus color photograph arteriovenous classification method based on the above-mentioned arteriovenous classification model.

[0015] In a fourth aspect, the present application further provides a cross-domain fundus color photograph arteriovenous classification model training system, comprising:

[0016] An image preprocessing module for performing image preprocessing on the target domain image to obtain an enhanced image;

[0017] A pseudo label generation module for performing arteriovenous segmentation on the enhanced image using an initial segmentation model to obtain a segmentation result, performing uncertainty estimation on the segmentation result to obtain an uncertainty degree, screening the segmentation result with an uncertainty degree above a dynamic threshold as a high-quality label, and obtaining a pseudo label based on the high-quality label, wherein the dynamic threshold is at least increased as the training progresses;

[0018] A dynamic curriculum learning module for training the initial segmentation model using the pseudo label to obtain an arteriovenous classification model.

[0019] Based on the above technical scheme, compared with the prior art, the beneficial effects of the present application at least include:

[0020] The training method of the cross-domain fundus color photograph arteriovenous classification model provided by the present application trains the cross-domain fundus images through a semi-supervised algorithm, and applies a dynamic threshold adjustment method to the segmentation task, solving the problem that other pseudo-label methods cannot improve the difficult part in the fundus color photograph, and enabling the training to be completed without manual labeling of the target domain data, and the training result is greatly improved compared with using only the source domain data.

[0021] The above description is only a summary of the technical scheme of the present application, in order to enable those skilled in the art to more clearly understand the technical means of the present application, and can be implemented according to the content of the description, as follows. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is the algorithm flowchart of the training method provided by a typical embodiment of the present application;

[0023] Figure 2 is a result schematic diagram of the existing source domain classification model when segmenting the target domain image provided by a typical embodiment of the present application;

[0024] Figure 3 is a pseudo-label generation flowchart in the training method provided by a typical embodiment of the present application;

[0025] Figure 4 is a comparison diagram of experimental results of the arteriovenous classification model provided by a typical embodiment of the present application. DETAILED DESCRIPTION

[0026] In view of the deficiencies in the prior art, the present inventors have long studied and practiced to propose the technical scheme of the present application. The technical scheme, its implementation process and principles will be further explained as follows.

[0027] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, however, the present application can also be implemented in other ways different from those described herein, therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0028] Moreover, the relationship terms such as "first" and "second" are only used to distinguish one component or method step from another component or method step with the same name, and do not necessarily require or imply any such actual relationship or order between the components or method steps.

[0029] AsFigure 2 As shown in the figure, directly applying the existing source domain classification model to the target domain can cause the problem of decreased arteriovenous segmentation effect, and from left to right in the figure are the original device photographed image (source domain), the original device photographed image segmentation effect, the new device photographed image (target domain) and the new device photographed image segmentation effect.

[0030] Referring to Figure 1 To solve the problem of repeated labeling, the semi-supervised cross-domain retinal blood vessel arteriovenous classification algorithm can be used for fundus blood vessel arteriovenous analysis and intelligent medical diagnosis. However, it should be noted that the training method and the model obtained by training provided by the present application are applied to the arteriovenous classification of fundus color photographs, and only intermediate information or reference information that can be used for disease diagnosis is obtained. In clinical practice, the above-mentioned intermediate information or reference information must be combined with the judgment of doctors to finally diagnose the disease. The segmentation result of the model only reflects the feature information of the image to be segmented through computer program calculation and recognition, and does not involve any data or information obtained by invading the biological body. Therefore, the present application does not belong to the disease diagnosis and treatment method, but only belongs to the method for obtaining or processing medical intermediate information.

[0031] The embodiment of the present application provides a kind of cross-domain fundus color photograph arteriovenous classification model training method, comprising the following steps:

[0032] 1) image preprocessing is carried out on target domain image, and enhanced image is obtained.

[0033] 2) arteriovenous segmentation is carried out on the enhanced image using the initial segmentation model, and segmentation result is obtained.

[0034] 3) uncertainty estimation is carried out on the segmentation result to obtain uncertainty, and the segmentation result with uncertainty above dynamic threshold is selected as high-quality label, and pseudo label is obtained based on the high-quality label, wherein the dynamic threshold is at least improved with the training.

[0035] 4) the initial segmentation model is trained using the pseudo label, and an arteriovenous classification model is obtained.

[0036] As some specific application cases, the training method summarized above can be implemented using the following implementation process:

[0037] 1) image quality is improved by preprocessing fundus color photograph, and feature information in the image is highlighted; 2) a segmentation model is established to segment arteriovenous of fundus color photograph; 3) uncertainty analysis is carried out on the pre-result of unlabeled data, and high-quality prediction result is selected to generate pseudo label; 4) the current learning situation of the model is evaluated, the dynamic threshold is adjusted, and the model is trained using the pseudo label.

[0038] In some embodiments, the initial segmentation model can be obtained by training a source domain target image and its corresponding source domain label. Of course, those skilled in the art can also directly obtain an initial segmentation model trained from elsewhere, and the corresponding training method can belong to existing technology or be specifically set by those skilled in the art. How to obtain the initial segmentation model is not a key step, and will not be repeated here.

[0039] In some embodiments, in step 1), the purpose of image preprocessing can be to highlight the blood vessel features of the source domain image and the target domain image.

[0040] In some embodiments, the image preprocessing can include any one or a combination of more than two of resolution adjustment, image enhancement, and random enhancement.

[0041] As some specific application examples, the source domain and target domain images can be enhanced using an image enhancement algorithm to highlight the blood vessel features of the images, and the flowchart is as shown in Figure 2 The specific operation process is as follows:

[0042] 1) Resolution adjustment: The resolution of the image has a great influence on the segmentation of the blood vessels. Resolution adjustment and cutting of the image can effectively improve the operation efficiency and segmentation effect of the model. In the present application, the bilinear interpolation method is preferably used for the fundus image, and the nearest neighbor interpolation method is used for the label to ensure the consistency of the segmentation result.

[0043] 2) Image enhancement: During shooting, images of different qualities can be taken due to environmental lighting, field of view, shooting equipment, patient lesion influence, operator experience, etc. Low-quality photos have low contrast and similar blood vessel colors, so it is difficult for even a human to classify the fundus blood vessels of such images. Therefore, the fundus color photos need to be uniformly enhanced. In the present application, the brightness is corrected using Gamma lighting correction. The color is enhanced using image weighting combined with a Gaussian filter. The purpose of contrast adjustment is achieved by processing the image histogram using contrast-limited adaptive histogram equalization. The image is masked and processed using a morphological method to remove the artifacts caused by the edge step response of the image during enhancement.

[0044] 3) Random enhancement: Due to the influence of the shooting environment and equipment, some noise such as halos and Gaussian noise can be generated during shooting. The noise in the image can overlap with the fundus blood vessels, affecting the classification of arteriovenous, and leading to a decrease in the segmentation effect. In the present application, Gaussian noise, halo noise, color adjustment, random rotation, scaling, and flipping are added in random enhancement to perform data enhancement processing on the image.

[0045] In some embodiments, in step 2), the arteriovenous segmentation can specifically include:

[0046] A probability estimation is performed on each pixel in the enhanced image by using the initial segmentation model, and based on a result of the probability estimation, the segmentation result is obtained, wherein the probability estimation represents a probability size of the pixel belonging to an artery, a vein or a background. For example, it can be determined that the pixel belongs to an artery or a background when the probability is greater than a threshold, or a maximum probability is selected as the attribution of the pixel.

[0047] As some typical application examples, a deep learning-based network can be used for arteriovenous segmentation, and an enhanced fundus image is input into a U-net network to output a probability estimation of an artery, a vein and a background for each pixel of the fundus image.

[0048] In some embodiments, in step 2), the initial segmentation model can include a plurality of segmentation networks, and in step 3), the uncertainty estimation can specifically include the following steps:

[0049] The arteriovenous segmentation of the enhanced image is performed at different times by using the plurality of segmentation networks to obtain a plurality of segmentation results.

[0050] The information entropy and the variance of each corresponding pixel of the plurality of segmentation results are calculated.

[0051] Based on the information entropy and the variance, the uncertainty of the segmentation result is obtained.

[0052] The above process is shown in Figure 3 As shown in the figure, from left to right are a fundus color photograph original image, an enhanced fundus color photograph image, a prediction result of a single network, an uncertainty heat map and a generated pseudo label.

[0053] The reliability evaluation of the label is an important part of the pseudo label method, that is, how good is the label and how bad is the label. The prediction result of the unlabeled data cannot be directly used as a pseudo label, because the neural network has different segmentation effects for different data, and contains a lot of incorrect supervision information. If the poor prediction result is directly used as a pseudo label, it will cause negative optimization to the model due to the incorrect guidance. Therefore, the prediction result needs to be screened.

[0054] Specifically, the application evaluates the label quality by means of uncertainty estimation. For example, the segmentation model can adopt a double-network mode to output prediction results P m1，t , P m2，tThe information entropy and variance of each pixel are calculated using the prediction results from these two different networks as the uncertainty of the current pixel, in order to complete the evaluation of the prediction performance. t .

[0055] In some implementations, the uncertainty is calculated as follows:

[0056]

[0057] E t =-∑μ t logμ t

[0058]

[0059]

[0060] Where M represents the number of segmentation networks with different initializations, and P i，t μ represents the segmentation result of the i-th segmentation network at time t. t E represents the average confidence level of the segmentation results of all segmentation networks at time t. t Represents the information entropy, U represents the variance. t The uncertainty is represented by λ1 and λ2, both of which are constants.

[0061] After obtaining the uncertain estimation results, it is necessary to select high-quality prediction results to generate pseudo-labels. The traditional approach is to use a fixed preset threshold to divide the prediction results into two parts. However, due to the large variation in the difficulty of the segmentation task, high-certainty results are easily obtained for the simpler parts, while high-certainty results are difficult to obtain for the difficult parts, even if positive results are obtained. Using a fixed threshold for screening excludes the supervision information of the difficult parts, resulting in the difficult parts not being learned. Repeatedly emphasizing the supervision information of the simpler parts does not help the network segment the difficult parts. Therefore, this embodiment of the invention uses a course-based pseudo-label learning method to dynamically adjust the threshold based on the learning progress. A lower threshold is set in the initial stage of learning to ensure that the prediction results of each part have a chance to be learned by the network. As the network's capabilities improve, the threshold is gradually increased to filter out supervision information with low certainty and prevent the generation of erroneous pseudo-labels.

[0062] In some implementations, high-quality labels can be obtained by dynamically enhancing and sharpening the pseudo-labels and their corresponding masks.

[0063] After obtaining high-quality supervised information, it needs to be sharpened to generate pseudo-labels. Unsharpened labels cannot generate gradients and cannot be used for training. The common practice is to use the one-hot encoding method, taking the class with the highest predicted probability in each class as the prediction result to generate pseudo-labels. This method results in excessively large gradients in the early stages of training and may cause some negative optimization. This invention uses a dynamically enhanced sharpening method to sharpen the supervised information to generate pseudo-labels, and simultaneously generates corresponding masks.

[0064] In some implementations, the calculation method for the dynamic enhancement sharpening can be as follows:

[0065] Sharpen(μ)=μ+(onehot(μ)-μ)*α|α∈[0,1]

[0066] Wherein, Sharpen(μ) represents the pseudo-label, μ represents the high-quality label, onehot(μ) represents the result obtained by processing the high-quality label using the onehot sharpening method, and α represents the learning rate.

[0067] In some implementations, the learning rate of the dynamic enhancement sharpening can be gradually increased as training progresses.

[0068] In some implementations, step 4) may specifically include the following steps:

[0069] The initial segmentation model training process is statistically fitted to obtain the learning progress.

[0070] Based on the learning progress, the values ​​of the dynamic threshold and the learning rate are dynamically adjusted.

[0071] In some implementations, the statistical fitting can be performed in each iteration.

[0072] In some implementations, the current segmentation model generated during training can be used to segment the validation set data, and the segmentation effect can be evaluated. The normalized quantization index of the segmentation effect is used as the current learning progress, and the validation set data is the target domain image that is additionally annotated.

[0073] In some implementations, the convex function of the normalized quantization index can be used as the currently adjusted dynamic threshold.

[0074] The dynamic thresholding method is the core algorithm of this invention, which requires dynamically adjusting parameters such as the threshold and learning rate based on the model's learning progress at each stage. After each iteration, this invention performs statistical analysis on the model's segmentation results, fits the learning progress, and adjusts the parameters according to the current learning situation. Therefore, in this embodiment, some additional manually labeled target domain data is used as validation set data. The obtained model is used to segment the validation set data, and the segmentation effect is evaluated. The normalized quantization index of the segmentation effect is used as the current learning progress, and the convex function output of the learning progress is used as the adjusted threshold.

[0075] In some implementations, in step 4), the initial segmentation model can be trained based on the model loss value.

[0076] In some implementations, the model loss value can be calculated based on the mask.

[0077] In some implementations, the model loss value can be calculated as follows:

[0078] R P =||P(x|mask)-P′(x|mask)|| 2

[0079] Among them, R P P(x|mask) represents the pixel in the mask, and P′(x|mask) represents the label pixel in the verification set data.

[0080] The present invention also provides a cross-domain fundus color imaging arteriovenous classification model trained by the model training method in the above embodiments.

[0081] This invention also provides a cross-domain fundus color imaging arteriovenous classification method, which classifies arteries and veins based on the arteriovenous classification model in the above embodiments.

[0082] This invention also provides a cross-domain fundus color imaging arteriovenous classification model training system, comprising: an image preprocessing module for preprocessing a target domain image to obtain an enhanced image; a pseudo-label generation module for segmenting the enhanced image into arteries and veins using an initial segmentation model to obtain segmentation results, performing uncertainty estimation on the segmentation results to obtain uncertainty, selecting segmentation results with uncertainty above a dynamic threshold as high-quality labels, and obtaining pseudo-labels based on the high-quality labels, wherein the dynamic threshold increases at least as training progresses; and a dynamic course learning module for training the initial segmentation model using the pseudo-labels to obtain an arteriovenous classification model.

[0083] Based on the above implementation examples, it is clear that the embodiments of the present invention have at least the following advantages compared with the prior art:

[0084] 1. This invention proposes for the first time a technical solution that applies the dynamic threshold adjustment method of course pseudo-label learning to the arteriovenous segmentation task, solving the problem that other pseudo-label methods have little effect on difficult parts.

[0085] 2. The embodiments of the present invention use information entropy and variance to select high-quality pseudo-labels by leveraging uncertainty, thus solving the problems of low label quality and difficulty in quantifying label quality in other methods.

[0086] 3. The embodiments of the present invention use a dynamic label sharpening method to sharpen pseudo-labels, which solves the negative optimization problem caused by overly hard labels generated in the early stage of training.

[0087] 4. The embodiments of the present invention train cross-domain fundus images using a semi-supervised algorithm, which can complete the training without manually annotating the target domain data, and the training results are significantly improved compared to using only the source domain data.

[0088] Based on the above technical solution, the inventors conducted a comparative experiment using the cross-domain fundus color imaging arteriovenous classification model provided in the above embodiments. The experimental results are as follows: Figure 4 As shown, from left to right, the segmentation results are: the segmentation results experimented on the source domain only, the segmentation results of the mean teacher method, the segmentation results of the pseudo-label method, and the segmentation results trained on the target domain. The results demonstrate that the segmentation performance, especially accuracy, of the algorithm used in this embodiment of the invention achieves a significant improvement over supervised learning algorithms. For example, according to experimental results, compared to existing technologies, the segmentation method provided in this embodiment of the invention improves Acc (accuracy) from 82.72% to 88.98%, Se (sensitivity) from 80.36 to 86.54%, F1 (F1 score) from 80.66 to 87.57%, G-mean (geometric mean) from 82.47 to 88.72%, Kappa (Kappa coefficient) from 78.41% to 86.29, and FDR (false detection rate) from 19.02 to 11.37%.

[0089] It should be understood that the above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for training a cross-domain fundus color imaging arteriovenous classification model, characterized in that, include: 1) Perform image preprocessing on the target domain image to obtain an enhanced image; 2) Perform arteriovenous segmentation on the enhanced image using the initial segmentation model to obtain the segmentation result. The initial segmentation model includes multiple segmentation networks. 3) Perform arteriovenous segmentation on the enhanced image using multiple segmentation networks at different times to obtain multiple segmentation results; Calculate the information entropy and variance of each corresponding pixel in the multiple segmentation results; Based on the information entropy and variance, the uncertainty of the segmentation result is obtained, and the method for calculating the uncertainty is as follows: ； Where M represents the number of segmentation networks with different initializations, and P i，t μ represents the segmentation result of the i-th segmentation network at time t. t E represents the average confidence level of the segmentation results of all segmentation networks at time t. t Represents the information entropy, U represents the variance. t Represents the uncertainty, and All represent constants; Segmentation results with uncertainty above the dynamic threshold are selected as high-quality labels. Based on these high-quality labels, pseudo-labels and their corresponding masks are obtained after dynamic enhancement and sharpening. The calculation method for the dynamic enhancement and sharpening is as follows: ； Wherein, Sharpen(μ) represents the pseudo-label, μ represents the high-quality label, onehot(μ) represents the result obtained by processing the high-quality label using the onehot sharpening method, and α represents the learning rate, which gradually increases as training progresses; 4) In each iteration, the initial segmentation model training process is statistically fitted, the current segmentation model generated during the training process is used to segment the validation set data, and the segmentation effect is evaluated. The normalized quantitative index of the segmentation effect is used as the current learning progress. The validation set data is the target domain image with additional annotation. Based on the learning progress, the convex function of the normalized quantification index is used as the adjusted dynamic threshold, and the learning rate increases as the learning progress increases. The initial segmentation model is trained using the pseudo-labels to obtain an arteriovenous classification model.

2. The model training method according to claim 1, characterized in that, The initial segmentation model is obtained by training the source domain target image and its corresponding source domain label.

3. The model training method according to claim 2, characterized in that, In step 1), the image preprocessing highlights the vascular features of the source domain image and the target domain image.

4. The model training method according to claim 1, characterized in that, The image preprocessing includes any one or a combination of two or more of the following: resolution adjustment, image enhancement, and random enhancement.

5. The model training method according to claim 1, characterized in that, In step 2), the arteriovenous segmentation specifically includes: The probability of each pixel in the enhanced image is estimated using an initial segmentation model, and the segmentation result is obtained based on the result of the probability estimation, wherein the probability estimation represents the probability that the pixel belongs to an artery, vein or background.

6. The model training method according to claim 1, characterized in that, In step 4), the initial segmentation model is trained based on the model loss value.

7. The model training method according to claim 6, characterized in that, The model loss value is calculated based on the mask, and the calculation method for the model loss value is as follows: ； Among them, R P P represents the model loss value, P(x|mask) represents the pixel in the mask, and P'(x|mask) represents the label pixel in the validation set data.

8. A cross-domain fundus color imaging method for classifying arteries and veins, characterized in that, The arteriovenous classification method is based on the arteriovenous classification model obtained by the model training method described in any one of claims 1-7.

9. A cross-domain fundus color imaging arteriovenous classification model training system, used to perform the steps of the model training method according to any one of claims 1-7, characterized in that, include: The image preprocessing module is used to preprocess the target domain image to obtain an enhanced image; The pseudo-label generation module is used to perform arteriovenous segmentation on the enhanced image using an initial segmentation model to obtain segmentation results, perform uncertainty estimation on the segmentation results to obtain uncertainty, select segmentation results with uncertainty above a dynamic threshold as high-quality labels, and obtain pseudo-labels based on the high-quality labels, wherein the dynamic threshold increases at least as training progresses; The dynamic course learning module is used to train the initial segmentation model using the pseudo-labels to obtain an arteriovenous classification model.

Images