Eye fundus image analysis method and device, electronic equipment and storage medium
By extracting the domain-specific features and domain-independent features of fundus images, and optimizing the loss value of independent components constraints, the problem of insufficient adaptability of fundus image analysis models in the prior art to images acquired by rare fundus cameras is solved, and the robustness and accuracy of the analysis results are improved.
Patent Information
- Application Number
- CN202311753820.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-19
- Publication Date
- 2025-06-20
AI Technical Summary
Existing fundus image analysis models cannot adapt to fundus images collected by other rare fundus cameras, resulting in poor robustness and accuracy.
By obtaining the sample fundus image and its annotation information, the domain-specific features and domain-independent features are extracted using the unique feature extraction model and the irrelevant feature extraction model, the independent component constraint loss value is calculated, and the model parameters are adjusted through backpropagation until the loss value converges, and the irrelevant feature extraction model is obtained as the target feature extraction model.
The fundus image analysis model is improved to improve the robustness and accuracy of the analysis results of fundus images taken by different models, and can better adapt to the images collected by different fundus cameras.
Smart Images
Figure CN120182166A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of data processing, and particularly to a fundus image analysis method, apparatus, electronic device, and storage medium. Background Art
[0002] A fundus camera can collect fundus images to facilitate obtaining the fundus conditions of patients. In the prior art, a deep neural network can be trained using pre-annotated sample fundus images to obtain a fundus image analysis model. Then, the trained fundus image analysis model is used to analyze the fundus images to determine information such as the age, gender, and symptoms of the patient, helping to achieve the diagnosis of the condition. Among them, the sample fundus images are usually fundus images collected by several common fundus cameras.
[0003] However, with the emergence of new types of fundus cameras, the number of fundus camera models is increasing. The lens compositions and parameter settings of different fundus cameras vary, so the collected fundus images are also increasingly diverse in style. In this case, the fundus image analysis model only has a good analysis effect on the fundus images collected by several common fundus cameras corresponding to the sample fundus images, and cannot adapt to the fundus images collected by other relatively rare fundus cameras, with poor robustness and accuracy. Summary of the Invention
[0004] The present disclosure provides a fundus image analysis system, method, apparatus, electronic device, and storage medium to at least solve the problem that the fundus image analysis model in the related art cannot adapt to the fundus images collected by other relatively rare fundus cameras, with poor robustness and accuracy. The technical solution of the present disclosure is as follows:
[0005] According to the first aspect of the embodiments of the present disclosure, a fundus image analysis method is provided, including:
[0006] Obtain a sample fundus image and the annotation information of the sample fundus image;
[0007] Input the sample fundus image into a specific feature extraction model for feature extraction to obtain the domain-specific features of the sample fundus image;
[0008] Input the sample fundus image into an irrelevant feature extraction model for feature extraction to obtain the domain-irrelevant features of the sample fundus image;
[0009] Calculate the independent component constraint loss value between the domain-specific features and the domain-irrelevant features;
[0010] Based on the domain-irrelevant features, a predictor determines the predicted value of the sample fundus image; calculate the target loss value between the predicted value and the annotation information;
[0011] According to the independent component constraint loss value and the target loss value, perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, and the predictor until the independent component constraint loss value and the target loss value converge, and obtain the irrelevant feature extraction model as the target feature extraction model and the predictor as the target task model;
[0012] After obtaining the fundus image to be processed, input the fundus image to be processed into the target feature extraction model and the target task model to obtain the analysis result of the fundus image to be processed.
[0013] Optionally, the performing backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, and the predictor according to the independent component constraint loss value and the target loss value until the independent component constraint loss value and the target loss value converge includes:
[0014] Input the domain-specific features and the domain-irrelevant features into a decoder to obtain a reconstructed image;
[0015] Calculate the reconstruction error between the sample fundus image and the reconstructed image;
[0016] According to the independent component constraint loss value, the target loss value, and the reconstruction error, perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, and the decoder until the independent component constraint loss value, the target loss value, and the reconstruction error converge.
[0017] Optionally, the obtaining the sample fundus image and the annotation information of the sample fundus image further includes:
[0018] Obtain the classification information of the domain to which the sample fundus image belongs;
[0019] The performing backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, and the decoder according to the independent component constraint loss value, the target loss value, and the reconstruction error until the independent component constraint loss value, the target loss value, and the reconstruction error converge includes:
[0020] Input the domain-specific features into a classifier to obtain the classification result of the domain to which the sample fundus image belongs; the classification result is used to indicate whether the sample fundus image is an image taken by a common model or an image taken by a rare model;
[0021] Calculate the classification loss value between the classification result and the classification information;
[0022] Based on the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value, perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, the decoder, and the classifier until the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value converge.
[0023] Optionally, the performing backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, the decoder, and the classifier based on the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value until the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value converge includes:
[0024] Perform a weighted sum of the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value to obtain an overall loss value; the weights of the independent component constraint loss value, the reconstruction error, and the classification loss value are less than the weight of the target loss value;
[0025] Perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, the decoder, and the classifier according to the overall loss value until the overall loss value converges.
[0026] Optionally, the calculating the independent component constraint loss value between the domain-specific features and the domain-irrelevant features includes:
[0027] Calculate the minimum KL divergence, the minimum JS divergence, and / or the orthogonal loss value between the domain-specific features and the domain-irrelevant features as the independent component constraint loss value.
[0028] Optionally, the predictor is a multi-task learning model, and according to the domain-irrelevant features, the predictor determines the predicted value of the sample fundus image; calculating the target loss value between the predicted value and the annotation information includes:
[0029] According to the domain-irrelevant features, the multi-task learning model determines the predicted values of the sample fundus image under multiple tasks;
[0030] Calculate the sub-loss values between the predicted value and the annotation information for each task respectively;
[0031] According to the sub-loss values, determine the target loss value between the predicted value and the annotation information.
[0032] According to a second aspect of the embodiments of the present disclosure, there is provided a fundus image analysis device, including:
[0033] An acquisition module, configured to acquire a sample fundus image and annotation information of the sample fundus image;
[0034] A specific feature extraction module, configured to input the sample fundus image into a specific feature extraction model for feature extraction, so as to obtain the domain-specific features of the sample fundus image;
[0035] An irrelevant feature extraction module, configured to input the sample fundus image into an irrelevant feature extraction model for feature extraction, so as to obtain the domain-irrelevant features of the sample fundus image;
[0036] An independent component constraint loss value calculation module, configured to calculate an independent component constraint loss value between the domain-specific features and the domain-irrelevant features;
[0037] A target loss value calculation module, configured to determine a predicted value of the sample fundus image according to the domain-irrelevant features by a predictor; and calculate a target loss value between the predicted value and the annotation information;
[0038] An iteration module, configured to perform backpropagation on model parameters of the specific feature extraction model, the irrelevant feature extraction model, and the predictor according to the independent component constraint loss value and the target loss value until the independent component constraint loss value and the target loss value converge, so as to obtain the irrelevant feature extraction model as a target feature extraction model and the predictor as a target task model;
[0039] An analysis module, configured to, after acquiring a fundus image to be processed, input the fundus image to be processed into the target feature extraction model and the target task model, so as to obtain an analysis result of the fundus image to be processed.
[0040] According to a third aspect of the embodiments of the present disclosure, there is provided a fundus image analysis electronic device, including:
[0041] A processor;
[0042] A memory for storing executable instructions of the processor;
[0043] Wherein, the processor is configured to execute the instructions to implement the fundus image analysis method according to any one of the above.
[0044] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of a fundus image analysis electronic device, enabling the fundus image analysis electronic device to execute the fundus image analysis method according to any one of the above.
[0045] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product including computer programs / instructions which, when executed by a processor, implement the fundus image analysis method according to any one of the above.
[0046] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:
[0047] In the solution provided in the present application, first, a sample fundus image and annotation information of the sample fundus image are obtained; the sample fundus image is input into a domain-specific feature extraction model for feature extraction to obtain the domain-specific features of the sample fundus image; the sample fundus image is input into an irrelevant feature extraction model for feature extraction to obtain the domain-irrelevant features of the sample fundus image; an independent component constraint loss value between the domain-specific features and the domain-irrelevant features is calculated; based on the domain-irrelevant features, a predictor determines the predicted value of the sample fundus image; a target loss value between the predicted value and the annotation information is calculated; according to the independent component constraint loss value and the target loss value, backpropagation is performed on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, and the predictor until the independent component constraint loss value and the target loss value converge, obtaining the irrelevant feature extraction model as the target feature extraction model and the predictor as the target task model; after obtaining the fundus image to be processed, the fundus image to be processed is input into the target feature extraction model and the target task model to obtain the analysis result of the fundus image to be processed.
[0048] In this way, the present application proposes a multi-task learning method based on domain adaptation in the field of deep learning. According to the independent component constraint value between the domain-specific features and the domain-irrelevant features, backpropagation is performed on the model parameters of the specific feature extraction model and the irrelevant feature extraction model to decouple the extracted domain-related features from the domain-irrelevant features. Among them, the domain-irrelevant features are irrelevant to the domain to which the sample fundus image belongs but are related to the target task. Then, the trained irrelevant feature extraction model is used as the target feature extraction model, which can extract information related to the target task from the fundus image to be processed, and then be used to analyze the fundus image to be processed, thereby improving the robustness and accuracy of the analysis results of the target feature extraction model and the target task model for the fundus images to be processed captured by different models.
[0049] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure and do not constitute an improper limitation to the present disclosure.
[0051] Figure 1It is a flowchart of a fundus image analysis method shown according to an exemplary embodiment.
[0052] Figure 2 It is a schematic diagram of a model of a fundus image analysis method shown according to an exemplary embodiment.
[0053] Figure 3 It is a block diagram of a fundus image analysis device shown according to an exemplary embodiment.
[0054] Figure 4 It is a block diagram of an electronic device for fundus image analysis shown according to an exemplary embodiment.
[0055] Figure 5 It is a block diagram of a device for fundus image analysis shown according to an exemplary embodiment. Detailed implementation manners
[0056] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the accompanying drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0058] Figure 1 It is a flowchart of a fundus image analysis method shown according to an exemplary embodiment. As Figure 1 shown, the fundus image analysis method includes:
[0059] In step S11, a sample fundus image and annotation information of the sample fundus image are obtained.
[0060] With the emergence of new types of fundus cameras, the number of fundus camera models is increasing, and the lens compositions and parameter settings of different fundus cameras vary. Therefore, the collected fundus images are also more and more diverse in style. In this case, the fundus image analysis model only has a good analysis effect on the fundus images collected by several common fundus cameras corresponding to the sample fundus images, and cannot adapt to the fundus images collected by other relatively rare fundus cameras, with poor robustness and accuracy.
[0061] Through the fundus image analysis method provided by this application, the robustness and accuracy of the fundus image analysis model can be improved, so that the fundus image analysis model can extract features from fundus images taken by different models and obtain relatively accurate analysis results.
[0062] In this step, sample fundus images and annotation information of the sample fundus images are obtained. Among them, the sample fundus images are the fundus images used for model training, including images taken by rare models and images taken by common models. The annotation information of the sample fundus images is used to annotate the patient information corresponding to the sample fundus images to distinguish the age, gender, symptoms, etc. of the patients corresponding to each sample fundus image.
[0063] It can be understood that the images taken by rare models and the images taken by common models may have different image styles. For example, the flash color temperature, exposure intensity, white balance, etc. of the images taken by rare models may be different from those of the images taken by common models. Different image styles may affect the feature recognition of the sample fundus images, that is, sample fundus images with different image styles taken of the same object may be recognized as having different image features, thereby resulting in different target task analysis results.
[0064] Therefore, in this application, it is necessary to overcome the influence of different image styles on the target task analysis results to improve the accuracy of the target task analysis results of the trained model for fundus images with different image styles.
[0065] In step S12, the sample fundus images are input into a specific feature extraction model for feature extraction to obtain the domain-specific features of the sample fundus images.
[0066] In this step, the sample fundus images are input into a specific feature extraction model for feature extraction to obtain the domain-specific features of the sample fundus images. Among them, the domain-specific features refer to the features related to the domain to which the sample fundus images belong, such as the flash color temperature, exposure intensity, white balance, etc. mentioned above. That is to say, the domain-specific features of the sample fundus images are related to the model used during shooting.
[0067] Among them, any common deep learning network model can be used as the specific feature extraction model, such as Resnet (Deep residual network), InceptionNet convolutional neural network, ViT (Vision Transformer) model, etc. The domain-specific features can be expressed as h = H(X), where X represents the sample fundus image.
[0068] For the same object, different domain-specific features may be identified in sample fundus images with different image styles taken by different models, but these domain-specific features should not affect the analysis results of the target task, especially sample fundus images taken by rare models. Since their image styles are relatively rare, they are more likely to mislead the analysis results.
[0069] Therefore, in this step, the domain-specific features of the sample fundus images are identified so as to decouple and distinguish them from the domain-independent features, thereby facilitating the improvement of the robustness and accuracy of fundus image analysis.
[0070] In step S13, the sample fundus image is input into an irrelevant feature extraction model to extract features, thereby obtaining domain-independent features of the sample fundus image.
[0071] In this step, the sample fundus image is input into an irrelevant feature extraction model for feature extraction to obtain domain-independent features of the sample fundus image, wherein the domain-independent features refer to features that are irrelevant to the domain to which the sample fundus image belongs, including information such as the direction of blood vessels and lesions of the human eye. In other words, the domain-independent features of the sample fundus image are irrelevant to the model used when shooting.
[0072] Among them, similar to the unique feature extraction model, any common deep learning network model can be used as an irrelevant feature extraction model, and the domain-independent feature can be expressed as g=G(X).
[0073] For the same object, sample fundus images of different image styles taken by different models should have the same domain-independent features. Therefore, in this step, the domain-independent features of the sample fundus images will be identified so as to decouple and distinguish them from the domain-specific features, which will help improve the robustness and accuracy of fundus image analysis.
[0074] In step S14, the independent component constraint loss value between the domain-specific features and the domain-independent features is calculated.
[0075] In this step, in order to make the domain-specific features and the domain-independent features independent of each other, an independent component constraint function between the domain-specific features and the domain-independent features can be constructed, where the value of the independent component constraint loss function is used to indicate the correlation between the domain-specific features and the domain-independent features. Therefore, when the value of the independent component constraint loss function is the minimum, its optimal solution is obtained as the independent component constraint loss value, expressed as L indept .
[0076] It can be understood that the larger the independent component constraint loss value is, the greater the correlation between the domain-specific features and the domain-independent features is, and the effect of mutual independence between the domain-specific features and the domain-independent features cannot be achieved. Therefore, it is necessary to further train the specific feature extraction model and the irrelevant feature extraction model.
[0077] Conversely, if the independent component constraint loss value is small enough, it indicates that the correlation between the domain-specific features and the domain-independent features is low, achieving the effect of mutual independence between the domain-specific features and the domain-independent features.
[0078] In one implementation, calculating the independent component constraint loss value between the domain-specific features and the domain-independent features includes:
[0079] Calculating the minimum KL divergence value, the minimum JS divergence value, and / or the orthogonal loss value between the domain-specific features and the domain-independent features as the independent component constraint loss value.
[0080] That is to say, multiple loss functions can be used as the independent component constraint loss function between the domain-specific features and the domain-independent features, including KL divergence, JS divergence, and orthogonal loss function, etc.
[0081] It can be understood that when the KL divergence or JS divergence is minimized, the mutual information between the domain-specific features and the domain-independent features is minimized and the correlation is the weakest; alternatively, an orthogonal loss function can be introduced to force the direction vectors of the domain-specific features and the domain-independent features to be perpendicular to each other, that is, the inner product of the domain-specific features and the domain-independent features is 0, and at this time the correlation between the domain-specific features and the domain-independent features is the weakest.
[0082] In step S15, based on the domain-independent features, the predictor determines the predicted value of the sample fundus image; calculates the objective loss value between the predicted value and the annotation information.
[0083] In this step, the predictor analyzes the domain-independent features of the sample fundus image to obtain the predicted value of the sample fundus image. The predicted value is the information such as the age, gender, and symptoms of the patient corresponding to the sample fundus image predicted by the predictor. Ideally, the predicted value matches the annotation information of the sample fundus image. Therefore, the objective loss value between the predicted value and the annotation information can be further calculated to facilitate the judgment of the prediction accuracy of the predictor.
[0084] In one implementation, the predictor is a multi-task learning model, that is, the predictor can simultaneously analyze the analysis results of the sample fundus image under multiple target tasks based on the domain-independent features. Then, based on the domain-independent features, the predictor determines the predicted value of the sample fundus image; calculates the objective loss value between the predicted value and the annotation information, including:
[0085] Based on domain - independent features, the multi - task learning model determines the predicted values of the sample fundus image under multiple tasks; calculates the sub - loss values between the predicted values and the annotation information for each task respectively; and determines the target loss value between the predicted values and the annotation information according to the sub - loss values.
[0086] That is to say, the target loss value can be the synthesis of the supervised loss functions corresponding to each task in multi - task learning. For example, if the multi - tasks include an age prediction task and a gender prediction task, then, for the age prediction task, the mean absolute error (MAE, Mean Absolute Error) can be used to construct the loss function, and for the gender prediction task, the cross - entropy can be used to construct the loss function. In this way, the sub - loss values of the two different tasks are obtained. Then, the sub - loss values of the loss functions of the above - mentioned multiple tasks can be used as the supervised loss function of multi - task learning, that is, the target loss value, denoted as L. Task 。
[0087] In step S16, according to the independent component constraint loss value and the target loss value, backpropagation is performed on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, and the predictor until the independent component constraint loss value and the target loss value converge, and the irrelevant feature extraction model is obtained as the target feature extraction model, and the predictor is obtained as the target task model.
[0088] In this step, according to the independent component constraint loss value and the target loss value calculated above, backpropagation can be performed on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, and the predictor, continuously adjusting the values of the model parameters until the independent component constraint loss value and the target loss value obtained in the new round of training converge. At this time, it can be considered that the training of the specific feature extraction model, the irrelevant feature extraction model, and the predictor has obtained good results.
[0089] It can be understood that in this application, it is necessary to extract the domain - independent features of the fundus image and analyze the fundus image based on the domain - independent features to obtain the corresponding analysis results. Therefore, the training of the specific feature extraction model is only to assist the training of the irrelevant feature extraction model so that it can extract the domain - independent features independent of the domain - specific features of the fundus image.
[0090] In other words, simply obtaining the irrelevant feature extraction model as the target feature extraction model and the predictor as the target task model can be used for subsequent analysis of the fundus image, without the need to extract the domain - specific features of the fundus image anymore.
[0091] In one implementation, according to the independent component constraint loss value and the target loss value, backpropagation is performed on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, and the predictor until the independent component constraint loss value and the target loss value converge, including:
[0092] Input the domain-specific features and domain-irrelevant features into the decoder to obtain a reconstructed image; calculate the reconstruction error between the sample fundus image and the reconstructed image; according to the independent component constraint loss value, the target loss value, and the reconstruction error, perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, and the decoder until the independent component constraint loss value, the target loss value, and the reconstruction error converge.
[0093] Specifically, the extracted domain-specific features and domain-irrelevant features can be added together to obtain the overall features of the sample fundus image, denoted as f = g + h, and then the overall features are input into the decoder to convert the overall features into the reconstructed image X pred = Decoder(f).
[0094] It can be understood that in an ideal state, the domain-specific features and domain-irrelevant features include all the features of the sample fundus image. Then, the reconstructed image generated based on the overall features of the sample fundus image should be the same as the input sample fundus image. The reconstruction error calculated in this step is used to indicate the error between the sample fundus image and the reconstructed image.
[0095] Then, by calculating the reconstruction error L recon between the sample fundus image and the reconstructed image, and requiring the reconstruction error to converge, it can be ensured that the extracted domain-specific features and domain-irrelevant features do not lose any information of the sample fundus image, that is, to constrain that the domain-specific features and domain-irrelevant features include all the features of the sample fundus image.
[0096] Among them, the reconstruction error L recon between the sample fundus image and the reconstructed image can be the square of the L2 norm between the sample fundus image and the reconstructed image, denoted as ||X pred - X|| 2 . Or, other loss functions can also be used, and this application does not make any limitations in this regard.
[0097] In one implementation, obtaining the sample fundus image and the annotation information of the sample fundus image further includes obtaining the classification information of the domain to which the sample fundus image belongs; then, according to the independent component constraint loss value, the target loss value, and the reconstruction error, perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, and the decoder until the independent component constraint loss value, the target loss value, and the reconstruction error converge, including:
[0098] Input domain-specific features into a classifier to obtain a classification result for the domain to which the sample fundus image belongs; the classification result is used to indicate whether the sample fundus image is an image taken by a common model or an image taken by a rare model; calculate the classification loss value between the classification result and the classification information; according to the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value, perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, the decoder, and the classifier until the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value converge.
[0099] As can be seen from the foregoing, domain-specific features refer to features related to the domain to which the sample fundus image belongs. Then, in this step, the domain to which the sample fundus image belongs can be classified based on the domain-specific features to determine the domain to which the sample fundus image belongs, that is, whether the sample fundus image is an image taken by a common model or an image taken by a rare model.
[0100] Then, by calculating the classification loss value L between the classification result and the classification information Domain , and requiring the classification loss value to converge, it can be ensured that the extracted domain-specific features are features related to the domain to which the sample fundus image belongs, that is, the feature extraction model can be constrained to extract only features related to the domain to which the sample fundus image belongs.
[0101] Among them, the classification loss value can be the cross-entropy between the classification result and the classification information, or other loss functions can also be used. This application does not make a limitation in this regard.
[0102] In this application, according to the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value, perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, the decoder, and the classifier until the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value converge, including:
[0103] Perform weighted summation on the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value to obtain an overall loss value; the weights of the independent component constraint loss value, the reconstruction error, and the classification loss value are less than the weight of the target loss value; according to the overall loss value, perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, the decoder, and the classifier until the overall loss value converges.
[0104] That is to say, in the process of backpropagation, weighted summation can be performed on the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value to obtain an overall loss value L, and then the specific feature extraction model, the irrelevant feature extraction model, the predictor, the decoder, and the classifier are trained based on the overall loss value.
[0105] Then, the overall loss value formula can be expressed as:
[0106] L = L Task + α * L Domain + β * L indept + γ * L recon
[0107] Among them, α, β, and γ respectively represent the weights corresponding to the classification loss value, the independent component constraint loss value, and the reconstruction error. The values of all three are not greater than 1, and the weight value of the target loss value is 1.
[0108] Among them, any common gradient descent method can be used to train the overall network, including but not limited to SGD (Stochastic Gradient Descent), ADAM (Adaptive Moment Estimation), etc. This application does not make any limitations in this regard.
[0109] In step S17, after obtaining the fundus image to be processed, input the fundus image to be processed into the target feature extraction model and the target task model to obtain the analysis result of the fundus image to be processed.
[0110] Through the foregoing steps, the trained target feature extraction model and the target task model are obtained. In this step, the obtained fundus image to be processed can be input into the target feature extraction model and the target task model to obtain the analysis result of the fundus image to be processed. Among them, the fundus image to be processed is the fundus image that needs to be analyzed. Since the target feature extraction model is a trained irrelevant feature extraction model.
[0111] Therefore, whether the fundus image to be processed is an image taken by a common model or an image taken by a rare model, features irrelevant to its field can be obtained for performing the target task. Correspondingly, the analysis result of the fundus image to be processed will not be affected by the shooting model and has good robustness and accuracy.
[0112] As Figure 2 shown, it is a model schematic diagram of a fundus image analysis method. Among them, the feature extractor includes a specific feature extraction model and an irrelevant feature extraction model.
[0113] During the training process, the specific feature extraction model and the irrelevant feature extraction model respectively extract domain-specific features and domain-irrelevant features. Then, the independent component constraint loss value between the domain-specific features and the domain-irrelevant features is calculated; the domain classifier determines the classification result of the domain to which the sample fundus image belongs based on the domain-specific features, and then calculates the classification loss value; the extracted domain-specific features and domain-irrelevant features are added together to obtain the overall feature of the sample fundus image, and the overall feature is input into the decoder to be converted into a reconstructed image, obtaining the reconstruction error between the sample fundus image and the reconstructed image; the target predictor analyzes the domain-irrelevant features of the sample fundus image to obtain the predicted value of the sample fundus image, and further calculates the target loss value between the predicted value and the annotation information.
[0114] Furthermore, according to the independent component constraint loss value, the classification loss value, the reconstruction error, and the target loss value, backpropagation is performed on the feature extractor, the classifier, the predictor, and the decoder until the independent component constraint loss value, the classification loss value, the reconstruction error, and the target loss value converge, obtaining the irrelevant feature extraction model as the target feature extraction model and the predictor as the target task model.
[0115] Then, if a fundus image to be processed is obtained, it is only necessary to sequentially input the fundus image to be processed into the target feature extraction model and the target task model, and the domain-irrelevant features of the fundus image to be processed and the corresponding target task processing results can be obtained.
[0116] As can be seen from the above, the technical solution provided by the embodiments of the present disclosure performs backpropagation on the model parameters of the specific feature extraction model and the irrelevant feature extraction model according to the independent component constraint value between the domain-specific features and the domain-irrelevant features, so that the extracted domain-related features and domain-irrelevant features are decoupled. Among them, the domain-irrelevant features have nothing to do with the domain to which the sample fundus image belongs, but are related to the target task. Then, the trained irrelevant feature extraction model is used as the target feature extraction model, which can extract information related to the target task of the fundus image to be processed, and then be used to analyze the fundus image to be processed, thereby improving the robustness and accuracy of the analysis results of the target feature extraction model and the target task model for the fundus images to be processed captured by different models.
[0117] Figure 3 It is a block diagram of a fundus image analysis device shown according to an exemplary embodiment, including:
[0118] An acquisition module 201, configured to acquire a sample fundus image and the annotation information of the sample fundus image;
[0119] A specific feature extraction module 202, configured to input the sample fundus image into a specific feature extraction model for feature extraction to obtain the domain-specific features of the sample fundus image;
[0120] An irrelevant feature extraction module 203 is configured to input the sample fundus image into an irrelevant feature extraction model for feature extraction to obtain domain-irrelevant features of the sample fundus image;
[0121] An independent component constraint loss value calculation module 204 is configured to calculate an independent component constraint loss value between the domain-specific features and the domain-irrelevant features;
[0122] A target loss value calculation module 205 is configured to determine a predicted value of the sample fundus image according to the domain-irrelevant features by a predictor; calculate a target loss value between the predicted value and the annotation information;
[0123] An iteration module 206 is configured to perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, and the predictor according to the independent component constraint loss value and the target loss value until the independent component constraint loss value and the target loss value converge, so as to obtain the irrelevant feature extraction model as a target feature extraction model and the predictor as a target task model;
[0124] An analysis module 207 is configured to, after obtaining a fundus image to be processed, input the fundus image to be processed into the target feature extraction model and the target task model to obtain an analysis result of the fundus image to be processed.
[0125] As can be seen from the above, for the technical solution provided by the embodiment of the present disclosure, according to the independent component constraint value between the domain-specific features and the domain-irrelevant features, backpropagation is performed on the model parameters of the specific feature extraction model and the irrelevant feature extraction model, so that the extracted domain-related features and the domain-irrelevant features are decoupled. Among them, the domain-irrelevant features are irrelevant to the domain to which the sample fundus image belongs, but are related to the target task. Then, the trained irrelevant feature extraction model is used as the target feature extraction model, which can extract information related to the target task of the fundus image to be processed, and further be used to analyze the fundus image to be processed, thereby improving the robustness and accuracy of the analysis results of the target feature extraction model and the target task model for the fundus images to be processed captured by different models.
[0126] Figure 4 It is a block diagram of an electronic device for fundus image analysis shown according to an exemplary embodiment.
[0127] In an exemplary embodiment, a computer-readable storage medium including instructions, such as a memory including instructions, is also provided. The instructions can be executed by a processor of an electronic device to complete the method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0128] In an exemplary embodiment, a computer program product is also provided. When it runs on a computer, it enables the computer to implement the method for analyzing fundus images.
[0129] As can be seen from the above, the technical solution provided by the embodiments of the present disclosure performs backpropagation on the model parameters of the specific feature extraction model and the irrelevant feature extraction model according to the independent component constraint values between the domain-specific features and the domain-independent features, so that the extracted domain-related features and domain-independent features are decoupled. Among them, the domain-independent features are irrelevant to the domain to which the sample fundus image belongs, but are related to the target task. Then, the trained irrelevant feature extraction model is used as the target feature extraction model, which can extract information related to the target task from the fundus image to be processed, and then be used to analyze the fundus image to be processed, thereby improving the robustness and accuracy of the analysis results of the target feature extraction model and the target task model for the fundus images to be processed captured by different models.
[0130] Figure 5 FIG. 10 is a block diagram of an apparatus 800 for analyzing fundus images shown according to an exemplary embodiment.
[0131] For example, the apparatus 800 can be a mobile phone, a computer, a digital broadcast electronic device, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0132] Referring to Figure 5 , the apparatus 800 can include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.
[0133] The processing component 802 generally controls the overall operation of the apparatus 800, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete all or part of the steps of the method. In addition, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0134] The memory 804 is configured to store various types of data to support the operation of the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, and the like. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.
[0135] The power component 807 provides power to various components of the device 800. The power component 807 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.
[0136] The multimedia component 808 includes a screen that provides an output interface between the device 800 and an account. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from an account. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of the touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.
[0137] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.
[0138] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a power-on button, and a lock button.
[0139] The sensor assembly 814 includes one or more sensors for providing a status assessment of various aspects of the device 800. For example, the sensor assembly 814 can detect the on / off state of the device 800, the relative positioning of components, such as the display and keypad of the device 800. The sensor assembly 814 can also detect a change in the position of the device 800 or a component of the device 800, the presence or absence of contact of an object with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0140] The communication component 816 is configured to facilitate communication between the device 800 and other devices in a wired or wireless manner. The device 800 can access a wireless network based on communication standards, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0141] In an exemplary embodiment, the device 800 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the methods described in the first aspect and the second aspect.
[0142] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, is also provided. The instructions can be executed by the processor 820 of the device 800 to complete the method. Optionally, for example, the storage medium can be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.
[0143] In an exemplary embodiment, there is also provided a computer program product including instructions that, when running on a computer, cause the computer to execute the fundus image analysis method according to any one of the embodiments.
[0144] As can be seen from the above, the technical solution provided by the embodiments of the present disclosure backpropagates the model parameters of the specific feature extraction model and the irrelevant feature extraction model according to the independent component constraint values between the domain-specific features and the domain-independent features, so that the extracted domain-related features and domain-independent features are decoupled. Among them, the domain-independent features are irrelevant to the domain to which the sample fundus image belongs, but are related to the target task. Then, the trained irrelevant feature extraction model is used as the target feature extraction model, which can extract the information related to the target task from the fundus image to be processed, and further be used to analyze the fundus image to be processed, thereby improving the robustness and accuracy of the analysis results of the target feature extraction model and the target task model for the fundus images to be processed captured by different models.
[0145] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the present disclosure. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.
[0146] It should be understood that the present disclosure is not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.
Claims
1. A fundus image analysis method, characterized in that, Including: Obtaining a sample fundus image and annotation information of the sample fundus image; Inputting the sample fundus image into a domain-specific feature extraction model for feature extraction to obtain domain-specific features of the sample fundus image; Inputting the sample fundus image into an irrelevant feature extraction model for feature extraction to obtain domain-irrelevant features of the sample fundus image; Calculating an independent component constraint loss value between the domain-specific features and the domain-irrelevant features; Based on the domain-irrelevant features, a predictor determines a predicted value of the sample fundus image; calculating an objective loss value between the predicted value and the annotation information; According to the independent component constraint loss value and the objective loss value, backpropagating the model parameters of the specific feature extraction model, the irrelevant feature extraction model, and the predictor until the independent component constraint loss value and the objective loss value converge, obtaining the irrelevant feature extraction model as the target feature extraction model, and the predictor as the target task model; After obtaining a fundus image to be processed, inputting the fundus image to be processed into the target feature extraction model and the target task model to obtain an analysis result of the fundus image to be processed.
2. The fundus image analysis method according to claim 1, characterized in that, The backpropagating the model parameters of the specific feature extraction model, the irrelevant feature extraction model, and the predictor according to the independent component constraint loss value and the objective loss value until the independent component constraint loss value and the objective loss value converge includes: Inputting the domain-specific features and the domain-irrelevant features into a decoder to obtain a reconstructed image; Calculating a reconstruction error between the sample fundus image and the reconstructed image; According to the independent component constraint loss value, the objective loss value, and the reconstruction error, backpropagating the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, and the decoder until the independent component constraint loss value, the objective loss value, and the reconstruction error converge.
3. The fundus image analysis method according to claim 2, characterized in that, The obtaining the sample fundus image and the annotation information of the sample fundus image further includes: Obtaining classification information of the domain to which the sample fundus image belongs; The backpropagating the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, and the decoder according to the independent component constraint loss value, the objective loss value, and the reconstruction error until the independent component constraint loss value and the objective loss value converge includes: Inputting the domain-specific features into a classifier to obtain a classification result of the domain to which the sample fundus image belongs; the classification result is used to indicate that the sample fundus image is an image taken by a common model or an image taken by a rare model; Calculating a classification loss value between the classification result and the classification information; Based on the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value, perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, the decoder, and the classifier until the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value converge.
4. The fundus image analysis method according to claim 3, characterized in that, The performing backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, the decoder, and the classifier based on the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value until the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value converge includes: Perform weighted summation on the independent component constraint loss value, the target loss value, the reconstruction error, and the classification loss value to obtain an overall loss value; the weights of the independent component constraint loss value, the reconstruction error, and the classification loss value are less than the weight of the target loss value; Perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, the predictor, the decoder, and the classifier according to the overall loss value until the overall loss value converges.
5. The fundus image analysis method according to claim 1, characterized in that, The calculating the independent component constraint loss value between the domain-specific features and the domain-irrelevant features includes: Calculate the minimum KL divergence, the minimum JS divergence, and / or the orthogonal loss value between the domain-specific features and the domain-irrelevant features as the independent component constraint loss value.
6. The fundus image analysis method according to claim 1, characterized in that, The predictor is a multi-task learning model, and based on the domain-irrelevant features, the predictor determines the predicted value of the sample fundus image; The calculating the target loss value between the predicted value and the annotation information includes: Based on the domain-irrelevant features, the multi-task learning model determines the predicted values of the sample fundus image under multiple tasks; Calculate the sub-loss values between the predicted value and the annotation information for each task respectively; Based on the sub-loss values, determine the target loss value between the predicted value and the annotation information.
7. A fundus image analysis device, characterized in that, Includes: An acquisition module for acquiring a sample fundus image and the annotation information of the sample fundus image; A specific feature extraction module for inputting the sample fundus image into a specific feature extraction model for feature extraction to obtain the domain-specific features of the sample fundus image; An irrelevant feature extraction module for inputting the sample fundus image into an irrelevant feature extraction model for feature extraction to obtain the domain-irrelevant features of the sample fundus image; An independent component constraint loss value calculation module for calculating the independent component constraint loss value between the domain-specific features and the domain-irrelevant features; A target loss value calculation module for, based on the domain-irrelevant features, the predictor determining the predicted value of the sample fundus image; calculating the target loss value between the predicted value and the annotation information; An iterative module, configured to perform backpropagation on the model parameters of the specific feature extraction model, the irrelevant feature extraction model, and the predictor according to the independent component constraint loss value and the target loss value until the specific feature extraction model, the irrelevant feature extraction model, and the predictor converge, so as to obtain the irrelevant feature extraction model as the target feature extraction model and the predictor as the target task model; An analysis module, configured to, after obtaining a fundus image to be processed, input the fundus image to be processed into the target feature extraction model and the target task model to obtain an analysis result of the fundus image to be processed.
8. An electronic device, characterized in that, Comprising: A processor; A memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the instructions to implement the fundus image analysis method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the fundus image analysis electronic device, the fundus image analysis electronic device is enabled to execute the fundus image analysis method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the fundus image analysis method according to any one of claims 1 to 6.