Image lesion attribute prediction model training method, prediction method and related device

Abstract

The invention provides an image lesion attribute prediction model training method, a prediction method and a related device, and the method comprises the steps: carrying out the visual coding of a fundus image sample through a visual encoder, and obtaining a pseudo-image visual feature; performing text coding on each lesion attribute sub-category description text to obtain lesion attribute sub-category features; performing matching according to the visual features of the pseudo image and the lesion attribute sub-category features to obtain a pseudo matching probability; performing visual coding on the fundus image sample through the initial image lesion attribute prediction model to obtain initial image visual features; performing matching according to the initial image visual features and the lesion attribute sub-category features to obtain a predicted matching probability; carrying out loss calculation according to the pseudo matching probability and the predicted matching probability to obtain a target matching loss function; and updating the initial model based on the target matching loss function to obtain an image lesion attribute prediction model. In conclusion, the lesion analysis accuracy of the fundus image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image lesion attribute prediction model training method, a prediction method, and related devices. Background Art

[0002] Currently, the use of AI, especially self-supervised learning (SSL), to analyze fundus images to assist in the diagnosis of ophthalmic diseases is an important direction, aiming to overcome the bottleneck of traditional manual labeling. However, existing mainstream SSL methods face challenges when applied to fundus images. Although methods based on contrastive learning (such as SimCLR) and reconstruction learning (such as MAE) are effective in general visual tasks, they often have difficulty capturing fine-grained lesion information (such as lesion type, size, and morphological details) in fundus images that are crucial for diagnosis, because these lesions are usually morphologically diverse and have blurred boundaries. At the same time, although the Contrastive Language-Image Pre-training (CLIP) model can associate images and text, the representation space learned when it is applied to fundus images is relatively coarse, making it difficult to distinguish subtle lesion differences.

[0003] These methods either focus on macroscopic features while ignoring details, have difficulty reconstructing complex lesions, or lack a detailed representation space, resulting in limited performance in ophthalmic tasks that require precise lesion identification and analysis. Therefore, improving the accuracy of lesion analysis in fundus images has become a pressing technical challenge. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to provide an image lesion attribute prediction model training method, prediction method and related devices, which can improve the accuracy of lesion analysis of fundus images, thereby improving the performance in downstream ophthalmic disease diagnosis and analysis tasks, and then provide doctors with more accurate diagnostic support, with high applicability.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a method for training an image lesion attribute prediction model, the method comprising:

[0006] obtaining fundus image samples;

[0007] Visually encoding the fundus image sample using a preset visual encoder to obtain a pseudo image visual feature;

[0008] Acquire multiple lesion attribute subcategory description texts of respective multiple lesion attribute categories, and perform text encoding on each of the lesion attribute subcategory description texts to obtain lesion attribute subcategory features;

[0009] Matching the pseudo image visual features with the features of the plurality of lesion attribute subcategories of each lesion attribute category to obtain pseudo matching probabilities for the plurality of lesion attribute subcategories;

[0010] Visually encoding the fundus image sample using a preset initial image lesion attribute prediction model to obtain initial image visual features;

[0011] Matching the initial image visual features with the features of the plurality of lesion attribute subcategories of each lesion attribute category to obtain predicted matching probabilities for the plurality of lesion attribute subcategories;

[0012] performing loss calculation based on the pseudo matching probabilities and the predicted matching probabilities of the plurality of lesion attribute subcategories of each of the plurality of lesion attribute categories to obtain a target matching loss function;

[0013] The initial image lesion attribute prediction model is updated based on the target matching loss function to obtain an image lesion attribute prediction model.

[0014] In some embodiments, performing loss calculation based on the pseudo-matching probabilities and the predicted matching probabilities of the plurality of lesion attribute subcategories of each of the plurality of lesion attribute categories to obtain a target matching loss function includes:

[0015] Performing loss calculation based on the pseudo-matching probability and the predicted matching probability to obtain a subcategory matching loss function for each of the lesion attribute subcategories;

[0016] Performing function fusion based on the plurality of subcategory matching loss functions of each lesion attribute category to obtain a lesion attribute category matching loss function;

[0017] The target matching loss function is obtained by performing weighted summation on the multiple lesion attribute category matching loss functions.

[0018] In some embodiments, after matching the pseudo image visual features with the features of the plurality of lesion attribute subcategories of each lesion attribute category to obtain pseudo matching probabilities for the plurality of lesion attribute subcategories, the method further includes:

[0019] Obtaining the pseudo-matching probability of each of the lesion attribute subcategories in the plurality of fundus image samples to obtain a plurality of probabilities to be corrected;

[0020] The multiple probabilities to be corrected are corrected according to a preset dynamic depolarizer to obtain the corrected pseudo-matching probability.

[0021] In some embodiments, correcting the plurality of probabilities to be corrected according to a preset dynamic depolarizer to obtain corrected false match probabilities includes:

[0022] Obtaining an average historical false matching probability for each lesion attribute subcategory;

[0023] Calculating an average value of the false matching probability of each lesion attribute subcategory in a plurality of fundus image samples to obtain an average false matching probability;

[0024] Update the historical average value of pseudo matching probabilities according to the average value of pseudo matching probabilities to obtain a current average value of pseudo matching probabilities;

[0025] The multiple probabilities to be corrected are corrected according to the current average value of the pseudo-match probabilities and a debiasing factor to obtain the corrected pseudo-match probabilities.

[0026] In some embodiments, the initial image lesion attribute prediction model includes a target network and a predictor, and visually encoding the fundus image sample using the preset initial image lesion attribute prediction model to obtain initial image visual features includes:

[0027] The target network is constructed according to the visual encoder, and the fundus image sample is input into the target network to obtain a reference image visual feature;

[0028] The reference image visual features are input into the predictor to obtain the initial image visual features.

[0029] In some embodiments, before updating the initial image lesion attribute prediction model based on the target matching loss function to obtain the image lesion attribute prediction model, the method further includes:

[0030] Taking each of the fundus image samples as a target fundus image sample one by one, performing different data enhancements on the target fundus image samples to obtain a plurality of different views;

[0031] Inputting the plurality of views into the target network respectively to obtain positive contrast image visual features;

[0032] Obtaining negative contrast image visual features based on the reference image visual features of the other fundus image samples, wherein the other fundus image samples are all the fundus image samples except the target fundus image sample;

[0033] Performing loss calculation based on the visual features of the reference image, the positive contrast image, and the negative contrast image to obtain a self-supervised loss function;

[0034] The target matching loss function is updated according to the self-supervised loss function.

[0035] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a method for predicting image lesion attributes, the method comprising:

[0036] Acquire target fundus image;

[0037] Predicting lesion attributes of the target fundus image using an image lesion attribute prediction model to obtain target fundus image features; the image lesion attribute prediction model is trained using the method described in the first aspect;

[0038] Matching the target fundus image features with the features of the plurality of lesion attribute subcategories of the plurality of lesion attribute categories to obtain target matching probabilities of the plurality of lesion attribute subcategories of each lesion attribute category;

[0039] The plurality of lesion attribute subcategories of each lesion attribute category are screened according to the target matching probability to obtain a target predicted lesion attribute subcategory of the lesion attribute category.

[0040] To achieve the above-mentioned objectives, a third aspect of the embodiments of the present application provides a device for training an image lesion attribute prediction model, the device comprising:

[0041] A data acquisition module, used for acquiring fundus image samples;

[0042] a pseudo image visual feature generation module, configured to perform visual encoding on the fundus image sample using a preset visual encoder to obtain a pseudo image visual feature;

[0043] a lesion attribute subcategory feature generation module, configured to obtain a plurality of lesion attribute subcategory description texts for respective lesion attribute categories, and perform text encoding on each of the lesion attribute subcategory description texts to obtain lesion attribute subcategory features;

[0044] a pseudo-matching probability generating module, configured to match the pseudo-image visual features with the features of the plurality of lesion attribute subcategories of each lesion attribute category to obtain pseudo-matching probabilities for each of the plurality of lesion attribute subcategories;

[0045] An initial image visual feature generation module is used to visually encode the fundus image sample using a preset initial image lesion attribute prediction model to obtain initial image visual features;

[0046] a predicted matching probability generating module, configured to match the visual features of the initial image with the features of the plurality of lesion attribute subcategories of each lesion attribute category to obtain predicted matching probabilities for each of the plurality of lesion attribute subcategories;

[0047] a target matching loss function generating module, configured to perform loss calculation based on the pseudo matching probabilities and the predicted matching probabilities of the plurality of lesion attribute subcategories of each of the plurality of lesion attribute categories to obtain a target matching loss function;

[0048] A model updating module is used to update the initial image lesion attribute prediction model based on the target matching loss function to obtain an image lesion attribute prediction model.

[0049] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory storing a computer program, and the processor implementing the method described in the first aspect or the method described in the second aspect when executing the computer program.

[0050] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect or the method described in the second aspect.

[0051] The present application proposes a method for training an image lesion attribute prediction model, a prediction method, and related devices. The method uses a preset visual encoder to extract pseudo-image visual features from fundus image samples and lesion attribute subcategory features obtained by encoding multiple lesion attribute subcategory description texts to accurately generate a pseudo-matching probability as a supervisory signal. Then, during the model training process, the pseudo-matching probability is used together with the predicted matching probability predicted by the initial model to jointly construct a target matching loss function to optimize and adjust the initial image lesion attribute prediction model, thereby prompting the model to learn the precise mapping relationship from image visual features to specific lesion attribute subcategories, so as to make up for the defects of insufficient learning of fine-grained lesion attributes or insufficient semantic understanding in traditional training methods, and improve the model's prediction accuracy and detail for various lesion attributes in fundus images. Ultimately, the embodiments of the present application can improve the accuracy of lesion analysis of fundus images, thereby improving performance in downstream ophthalmic disease diagnosis and analysis tasks, and thus can provide doctors with more accurate diagnostic support, with high applicability.

[0052] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1This is a flowchart of a method for training an image lesion attribute prediction model provided by an embodiment of the present application;

[0054] Figure 2 This is a flowchart of a method for training an image lesion attribute prediction model provided by another embodiment of the present application;

[0055] Figure 3 This is a flowchart of a method for training an image lesion attribute prediction model provided by another embodiment of the present application;

[0056] Figure 4 This is a flowchart of a method for training an image lesion attribute prediction model provided by another embodiment of the present application;

[0057] Figure 5 This is a flowchart of a method for training an image lesion attribute prediction model provided by another embodiment of the present application;

[0058] Figure 6 This is a flowchart of a method for training an image lesion attribute prediction model provided by another embodiment of the present application;

[0059] Figure 7 This is a flowchart of a method for predicting image lesion attributes provided by another embodiment of the present application;

[0060] Figure 8 This is a schematic diagram of an image lesion attribute prediction model training device provided in an embodiment of the present application;

[0061] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0065] First, let’s analyze some of the terms used in this application:

[0066] Self-supervised learning (SSL) is a machine learning paradigm that eliminates the need for human-labeled models. Instead, it trains models by automatically generating supervisory signals from the data itself. Its core concept is to leverage the internal structure or correlations of the data to construct pre-training tasks, such as predicting missing parts in images (Masked Image Modeling), sorting video frames (Temporal Order Verification), or filling in text context (e.g., Masked Language Modeling in BERT). These tasks force the model to learn meaningful representations that can then be transferred to downstream tasks through fine-tuning. Unlike supervised learning, which relies on external annotations, SSL's supervisory signal comes from the data's own pseudo-labels, making it efficient in utilizing massive amounts of unlabeled data. Typical approaches include contrastive learning (e.g., SimCLR), generative modeling (e.g., VAE), and mask-based pre-training (e.g., MAE). While self-supervised learning offers the advantage of reducing reliance on labeled data, its performance is highly dependent on the design of the pre-training task and the characteristics of the data. Currently, SSL has become an important research direction in fields such as computer vision (CV) and natural language processing (NLP).

[0067] Contrastive Language-Image Pretraining (CLIP) is a multimodal self-supervised learning model proposed by OpenAI. It maps images and text into a unified semantic space through contrastive learning, thereby achieving cross-modal semantic alignment. The core idea of ​​CLIP is to train the model on large-scale image-text pairs (such as images and their descriptions from the internet), enabling it to understand the relationship between visual content and natural language. During training, CLIP uses a contrastive loss to optimize the model: for a batch of image-text pairs, the model calculates the similarity between each image and all text descriptions, encouraging matching pairs to have higher similarity and reducing the similarity for non-matching pairs. This training method enables CLIP to learn universal visual-language representations without relying on task-specific labeled data. CLIP's architecture consists of two main components: a visual encoder (typically based on ViT or ResNet) and a text encoder (typically based on Transformer). During inference, CLIP can perform zero-shot classification, directly classifying images based on textual cues (e.g., "a photo of a dog") without the need for additional fine-tuning. Furthermore, CLIP excels in tasks such as image retrieval, cross-modal generation, and visual question answering, becoming a key benchmark model in the multimodal field. Its open-source version, OpenCLIP, has also spurred subsequent research.

[0068] Vision Language Model (VLM): This model focuses primarily on the natural world and aims to establish a mapping between visual information (such as images and videos) and linguistic descriptions. Its core is to learn the semantic features inherent in visual data and convert them into appropriate textual representations. Typical applications include image description generation and visual question answering. For example, given a photograph of a natural landscape, it can generate a textual description such as "a tranquil lake surrounded by lush mountains," helping people better understand the information conveyed by the visual content.

[0069] Currently, the use of AI, especially self-supervised learning (SSL), to analyze fundus images to assist in the diagnosis of ophthalmic diseases is an important direction, aiming to overcome the bottleneck of traditional manual labeling. However, existing mainstream SSL methods face challenges when applied to fundus images. Although methods based on contrastive learning (such as SimCLR) and reconstruction learning (such as MAE) are effective in general visual tasks, they often have difficulty capturing fine-grained lesion information (such as lesion type, size, and morphological details) in fundus images that are crucial for diagnosis, because these lesions are usually morphologically diverse and have blurred boundaries. At the same time, although the Contrastive Language-Image Pre-training (CLIP) model can associate images and text, the representation space learned when it is applied to fundus images is relatively coarse, making it difficult to distinguish subtle lesion differences.

[0070] These methods either focus on macroscopic features while ignoring details, have difficulty reconstructing complex lesions, or lack a detailed representation space, resulting in limited performance in ophthalmic tasks that require precise lesion identification and analysis. Therefore, improving the accuracy of lesion analysis in fundus images has become a pressing technical challenge.

[0071] Based on this, the embodiment of the present application uses pseudo-image visual features extracted from fundus image samples by a preset visual encoder and lesion attribute subcategory features obtained by encoding multiple lesion attribute subcategory description texts to accurately generate a pseudo-matching probability as a supervisory signal. Then, during the model training process, the pseudo-matching probability and the predicted matching probability predicted by the initial model are used to jointly construct a target matching loss function to optimize and adjust the initial image lesion attribute prediction model, thereby prompting the model to learn the precise mapping relationship from image visual features to specific lesion attribute subcategories, so as to make up for the defects of insufficient learning of fine-grained lesion attributes or insufficient semantic understanding in traditional training methods, and improve the model's prediction accuracy and detail for various lesion attributes in fundus images; ultimately, the embodiment of the present application can improve the accuracy of lesion analysis of fundus images, thereby improving performance in downstream ophthalmic disease diagnosis and analysis tasks, and thus can provide doctors with more accurate diagnostic support, with high applicability.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multi-processor systems, microprocessor-based systems, set-top boxes, network PCs, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0073] The embodiments of the present application provide an image lesion attribute prediction model training method, a prediction method and related devices, which are specifically illustrated by the following embodiments. First, a method for training an image lesion attribute prediction model in the embodiments of the present application is described.

[0074] Figure 1 This is an optional flowchart of a method for training an image lesion attribute prediction model provided in an embodiment of the present application, which may include but is not limited to steps S101 to S108.

[0075] Step S101, obtaining a fundus image sample;

[0076] Step S102, visually encoding the fundus image sample using a preset visual encoder to obtain a pseudo image visual feature;

[0077] Step S103, obtaining a plurality of lesion attribute subcategory description texts of respective lesion attribute categories, and performing text encoding on each lesion attribute subcategory description text to obtain lesion attribute subcategory features;

[0078] Step S104, matching the pseudo image visual features with the features of multiple lesion attribute subcategories of each lesion attribute category to obtain pseudo matching probabilities for each of the multiple lesion attribute subcategories;

[0079] Step S105, visually encoding the fundus image sample using a preset initial image lesion attribute prediction model to obtain initial image visual features;

[0080] Step S106, matching the initial image visual features with the features of multiple lesion attribute subcategories of each lesion attribute category to obtain predicted matching probabilities for each of the multiple lesion attribute subcategories;

[0081] Step S107 , performing loss calculation based on the pseudo matching probabilities and predicted matching probabilities of the multiple lesion attribute subcategories of each of the multiple lesion attribute categories to obtain a target matching loss function;

[0082] Step S108 : updating the initial image lesion attribute prediction model based on the target matching loss function to obtain the image lesion attribute prediction model.

[0083] In steps S101 to S108 shown in the embodiment of the present application, the pseudo-image visual features obtained by encoding the fundus image through a preset visual encoder are matched with the lesion attribute subcategory text features to obtain the pseudo-matching probability as a pseudo-label or training target for model training, and the loss is calculated based on the predicted matching probability and the pseudo-matching probability predicted by the model to be trained (initial image lesion attribute prediction model), and the model is updated based on the loss. By aligning the predicted output of the model with the fine-grained pseudo-matching probability obtained based on the preset encoder and text feature matching, the method of this embodiment enables the trained image lesion attribute prediction model to learn the correspondence between more refined lesion attribute subcategory features and image features, thereby improving the accuracy of lesion analysis of fundus images, thereby improving the performance in downstream ophthalmic disease diagnosis and analysis tasks, and thus providing doctors with more accurate diagnostic support, with high applicability.

[0084] In step S101 of some embodiments, fundus image samples serve as the data foundation for training and evaluating image lesion attribute prediction models. These samples typically originate from professional ophthalmic medical institutions or publicly available fundus image datasets, such as EyePACS and Ocular Disease Intelligent Recognition (ODIR). Fundus images are typically captured using a fundus camera and can be color fundus images, fluorescein angiography images, optical coherence tomography (OCT) images, and other modalities. Fundus images may also contain various lesion information, such as the presence of hemorrhage, exudation, and microaneurysms. The quantity and quality of the acquired fundus image samples are crucial to the effectiveness of model training. It is typically necessary to collect a sufficient number of representative samples to ensure the model's generalization capabilities. In practical applications, the appropriate sample source and quantity can be selected based on the specific lesion prediction task and available data resources. Furthermore, to improve the robustness and generalization capabilities of the model, the acquired fundus image samples typically require preprocessing, such as image resizing and data augmentation.

[0085] In step S102 of some embodiments, the acquired fundus image samples are first input into a preset visual encoder. The visual encoder can be various deep learning models that can extract image features, such as convolutional neural networks (CNN, such as ResNet, EfficientNet), visual transformers (ViT), etc. These models can convert the input fundus image samples into a feature vector containing rich visual information, that is, pseudo-image visual features. For example, if ResNet is used as a visual encoder, it will gradually extract low-level features of the image (such as edges, textures) to high-level features (such as the shape and position of the lesion) through multiple layers of convolution and pooling operations. Ultimately, the output of the last layer of ResNet will be used as pseudo-image visual features. These pseudo-image visual features are used to guide the training of the initial image lesion attribute prediction model in subsequent steps, so that it can learn more effective feature representations.

[0086] In step S103 of some embodiments, the lesion attribute category refers to a high-level, abstract classification of lesions in the fundus image, such as lesion type, lesion number, etc. Each lesion attribute category contains multiple specific lesion attribute subcategory description texts. For example, under the "lesion type" category, there may be description texts such as "white lesions with fuzzy edges", "small yellow lesions", and "small red lesions"; under the "lesion number" category, there may be description texts such as "no lesions", "a small number of lesions", "some lesions", and "a large number of lesions". The text encoding process typically uses a pre-trained language model (such as BERT) or a cross-modal pre-trained model such as the text portion of CLIP to convert these description texts into fixed-dimensional feature vectors, namely lesion attribute subcategory features, for subsequent matching and calculation with image features (pseudo-image visual features and original image visual features). This process converts text information into numerical feature representations, enabling the image lesion attribute prediction model to understand and process the semantic information of lesion attributes, laying the foundation for subsequent prediction tasks.

[0087] In step S104 of some embodiments, the pseudo-image visual features include basic visual information extracted from the fundus image sample, while the lesion attribute subcategory features are semantic encodings of the description texts of each type of lesion attribute subcategory. The matching process is usually achieved by calculating the similarity between the two feature vectors, for example, by using dot product or cosine similarity. The higher the similarity, the more the pseudo-image visual features match the lesion attribute subcategory features, and the higher the corresponding pseudo-matching probability. For example, if the similarity calculation result of the pseudo-image visual features of a fundus image sample and the lesion attribute subcategory feature of "white lesions with fuzzy edges" is high, it indicates that the image sample is preliminarily judged to have a high probability of having "white lesions with fuzzy edges", and the corresponding pseudo-matching probability will show a higher value. This step provides a reference basis for subsequent more accurate prediction of the matching probability through the pseudo-matching probability, and preliminarily establishes the correlation between the image features and the lesion attributes.

[0088] See also Figure 2 In some embodiments, after step S104, the following steps may also be performed, including but not limited to steps S201 to S202:

[0089] Step S201, obtaining a pseudo-matching probability of each lesion attribute subcategory in a plurality of fundus image samples to obtain a plurality of probabilities to be corrected;

[0090] Step S202 , correcting the multiple probabilities to be corrected according to a preset dynamic depolarizer to obtain corrected false matching probabilities.

[0091] In step S201 of some embodiments, the pseudo-match probability for each lesion attribute subcategory in multiple fundus image samples is obtained to obtain multiple probabilities to be corrected. This step is performed after step S104 is completed, i.e., the pseudo-match probability for each fundus image sample with all lesion attribute subcategories is calculated. In actual model training, a batch of fundus image samples is typically processed at a time. This step aims to collect the corresponding pseudo-match probabilities for all fundus image samples in the batch and for each specific lesion attribute subcategory. For example, if a batch contains N fundus images and there are M lesion attribute subcategories, then for a specific lesion attribute subcategory j, this step collects the N pseudo-match probability values ​​obtained by matching these N images with subcategory j. These sets of pseudo-match probabilities for the same lesion attribute subcategory, spanning multiple samples within the batch, are defined as the "multiple probabilities to be corrected" for that subcategory. It is easy to understand that there are a total of N×M probabilities to be corrected. This is done so that in subsequent steps, these pseudo-match probabilities can be uniformly or dynamically corrected based on the statistical information of the batch.

[0092] In step S202 of some embodiments, the preset dynamic depolarizer is a module for adjusting or correcting the probability of false matches. Since the probability of false matches obtained in step S104 may have certain systematic deviations or uneven distribution problems, especially for some rare or difficult-to-identify lesion subcategories, their false match probabilities may be generally low or high. At the same time, the dynamic depolarizer aims to dynamically adjust these probabilities based on the overall distribution of the probabilities to be corrected in the current batch or other preset rules to eliminate or reduce such deviations. The correction process can adopt a variety of methods, for example, normalization based on the distribution of false match probabilities within the batch, temperature scaling, or nonlinear transformation through a learnable correction network. The corrected false match probabilities are more reliable or more balanced pseudo-label signals processed by the dynamic depolarizer, which will be used to replace the original false match probabilities for subsequent loss calculations.

[0093] After obtaining the false match probability through steps S201 and S202, this embodiment further introduces a dynamic depolarizer to correct it, effectively alleviating potential bias issues and making the false match probability more stable and reliable, thereby providing a higher-quality supervisory signal for subsequent model training. This correction helps improve the effectiveness and generalization of model learning, further enhancing the accuracy of lesion analysis in fundus images, thereby improving performance in downstream ophthalmic disease diagnosis and analysis tasks, and thus providing more accurate diagnostic support for doctors, with high applicability.

[0094] See also Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S304:

[0095] Step S301, obtaining the average historical false matching probability of each lesion attribute subcategory;

[0096] Step S302, calculating the average value of the false matching probability of each lesion attribute subcategory in multiple fundus image samples to obtain the average false matching probability;

[0097] Step S303, updating the historical average value of pseudo matching probabilities according to the average value of pseudo matching probabilities to obtain the current average value of pseudo matching probabilities;

[0098] Step S304 , correcting the multiple probabilities to be corrected according to the current average value of the pseudo match probabilities and the debiasing factor to obtain corrected pseudo match probabilities.

[0099] In step S301 of some embodiments, the historical average false match probability is the long-term average of the false match probabilities accumulated and calculated over previous training iterations or batches for a specific lesion attribute subcategory. This is a value input into the dynamic depolarizer and represents a relatively stable historical statistical benchmark for the false match probability of that subcategory. This historical average is obtained so that during the correction process of the current batch, a more stable statistic can be referenced, rather than relying solely on the instantaneous statistics of the current batch, thereby improving the stability of the correction.

[0100] In some embodiments, step S302 is to collect statistics on the false match probabilities for each lesion attribute subcategory in the currently processed set of fundus image samples in the current training batch, and calculate their average value to obtain the average level of false match probabilities for each lesion attribute subcategory in the batch of samples, i.e., the average false match probability. This helps to understand the feature distribution of the current data batch. This value is the local statistical information of the current batch.

[0101] In some embodiments, in step S303, the average pseudo-match probability of the current batch calculated in step S302 is combined with the average historical pseudo-match probability corresponding to the subcategory obtained in step S301 to perform an update operation. This is typically performed using an exponential moving average (EMA) or other method, for example, where the new historical average is set to the weighted sum of the old historical average and the current batch average. The current average pseudo-match probability is the updated historical average, which more smoothly reflects the long-term trend of the pseudo-match probability of the subcategory and gradually evolves as training progresses.

[0102] In step S304 of some embodiments, the probability to be corrected, the current average value of the pseudo-match probability, and a debiasing factor (the factor can be a preset constant or a parameter learned during the model training process) are input into the "correction" module. The correction process can adopt a variety of mathematical transformations. For example, each probability value to be corrected is adjusted according to the degree of deviation from the current average value of the pseudo-match probability, and the debiasing factor is combined to perform overall scaling or translation. This enables the corrected pseudo-match probability to better reflect the true association strength of the subcategory, while reducing the deviation caused by uneven data distribution or inherent characteristics of the preset encoder. The final output is the "corrected pseudo-match probability".

[0103] Through steps S301 to S304, this embodiment provides a mechanism for dynamically correcting the probability of false matches based on historical statistical information. The mechanism uses the statistical mean of the current batch (the mean of the probability of false matches) to update a more stable historical mean, and adjusts the original probability of false matches based on this stable historical mean and a debiasing factor. This correction method based on cumulative statistics is more robust than relying solely on the statistics of the current batch, and can effectively remove non-systematic deviations, making the final corrected false match probability more reliable and stable as a training signal. This improves the efficiency of model training and the prediction accuracy of the final model, thereby improving the accuracy of lesion analysis of fundus images, thereby improving performance in downstream ophthalmic disease diagnosis and analysis tasks, and thus providing doctors with more accurate diagnostic support, with higher applicability.

[0104] In step S105 of some embodiments, the fundus image sample is visually encoded using a preset initial image lesion attribute prediction model to obtain initial image visual features. Unlike step S102, which uses a preset visual encoder to extract pseudo-image visual features, this step uses a preset initial image lesion attribute prediction model for visual encoding. This initial model can be a randomly initialized model or a pre-trained model. Its structure can be the same as or different from the visual encoder used in step S102. For example, the initial model can be a convolutional neural network (CNN), such as ResNet, Inception, etc., or a visual transformer (ViT). The fundus image sample is input into the initial model, and after a series of operations such as convolution, pooling, or attention mechanisms, the initial image visual features are finally obtained. These initial image visual features represent the initial model's understanding of the fundus image and will be used to subsequently match the lesion attribute subcategory features to calculate the predicted matching probability. By comparing with the pseudo-matching probability, a loss function can be calculated and used to update the parameters of the initial model so that it gradually learns to more accurately identify lesion attributes in the fundus image.

[0105] See also Figure 4 In some embodiments, the initial image lesion attribute prediction model includes a target network and a predictor. Step S105 may include but is not limited to steps S401 to S402:

[0106] Step S401: construct a target network based on the visual encoder, input the fundus image sample into the target network to obtain the visual features of the reference image;

[0107] Step S402: Input the reference image visual features into the predictor to obtain the initial image visual features.

[0108] In step S401 of some embodiments, a target network is constructed based on a visual encoder, and the fundus image samples are input into the target network to obtain reference image visual features. The target network is one of the main components of the initial image lesion attribute prediction model, which is responsible for extracting basic visual features from the input fundus image. It is constructed based on a visual encoder, which is usually a deep convolutional neural network or a model such as Vision Transformer, and its function is to convert raw pixel data into a higher-level feature representation. The fundus image samples are input into the target network for processing to obtain reference image visual features. The reference image visual features are the feature vectors output by the fundus image samples after encoding and processing by the target network, which is the first feature abstraction representation of the fundus image sample content.

[0109] In step S402 of some embodiments, the reference image visual features are input into the predictor to obtain the initial image visual features. The predictor is another important component of the initial image lesion attribute prediction model, which receives the reference image visual features from the target network as input. The predictor usually includes several layers of neural networks (for example, fully connected layers or nonlinear activation functions), whose function is to further transform or map the reference image visual features to obtain feature representations that are more suitable for subsequent tasks (i.e., matching prediction with text features in step S106). The initial image visual features are the final image feature vectors obtained after the reference image visual features are processed by the predictor. This feature vector represents the image representation used to predict lesion attributes in the current state of the initial image lesion attribute prediction model.

[0110] Through steps S401 to S402, this embodiment describes in detail how, in step S105, the initial image lesion attribute prediction model processes and converts the fundus image samples into initial image visual features for subsequent prediction and matching through its internal target network and predictor structure. This structured processing flow (first extracting basic features by the target network, and then refining or mapping by the predictor) helps the model learn more effective image representations, thereby improving the model's ability to capture image lesion information. This can further improve the accuracy of lesion analysis of fundus images, thereby improving performance in downstream ophthalmic disease diagnosis and analysis tasks, and thus can provide doctors with more accurate diagnostic support, with higher applicability.

[0111] In step S106 of some embodiments, after obtaining the initial image visual features in step S105, this step matches these features with the lesion attribute subcategory features obtained in step S103. The matching method can be the same as that used in step S104, such as calculating cosine similarity or dot product similarity. The initial image visual features represent the model's understanding of the fundus image, while the lesion attribute subcategory features represent textual descriptions of specific lesion types. By calculating the similarity between them, predicted matching probabilities for each of the multiple lesion attribute subcategories can be obtained. The higher the predicted matching probability, the more likely the model believes that the fundus image contains the corresponding lesion attribute subcategory. These predicted matching probabilities are compared with the pseudo-match probabilities obtained in step S104 and used to calculate a loss function, thereby guiding the training process of the initial image lesion attribute prediction model. For example, if the initial image visual features of a fundus image have a high similarity with the lesion attribute subcategory features of "small red lesions," the predicted matching probability for this subcategory will increase accordingly, indicating that the model has preliminarily determined a high probability of "small red lesions" in the fundus image sample.

[0112] In step S107 of some embodiments, the core is to calculate the loss function, which is used to measure the difference between the predicted matching probability and the pseudo matching probability. The pseudo matching probability can be regarded as a "supervisory signal", and the predicted matching probability is generated by the initial image lesion attribute prediction model. The goal of the loss function is to make the predicted matching probability as close to the pseudo matching probability as possible. Commonly used loss functions include cross entropy loss function, KL divergence, etc. For example, a binary cross entropy loss function can be used to calculate the loss of each lesion attribute subcategory, and then the losses of all subcategories are summed to obtain the final target matching loss function. The smaller the value of the loss function, the closer the predicted matching probability is to the pseudo matching probability, and the better the prediction effect of the model. This target matching loss function will be used to update the initial image lesion attribute prediction model in subsequent steps to guide the model training process so that it can learn more accurate feature representations, thereby improving the accuracy of lesion analysis of fundus images.

[0113] See also Figure 5 In some embodiments, step S107 may include but is not limited to steps S501 to S503:

[0114] Step S501 , performing loss calculation based on the pseudo-matching probability and the predicted matching probability to obtain a subcategory matching loss function for each lesion attribute subcategory;

[0115] Step S502 , performing function fusion based on the multiple sub-category matching loss functions of each lesion attribute category to obtain a lesion attribute category matching loss function;

[0116] Step S503 , performing weighted summation based on multiple lesion attribute category matching loss functions to obtain a target matching loss function.

[0117] In step S501 of some embodiments, a loss calculation is performed based on the pseudo-matching probability and the predicted matching probability to obtain a subcategory matching loss function for each lesion attribute subcategory. Loss calculation is performed for each specific lesion attribute subcategory. For a specific lesion attribute subcategory, its corresponding corrected pseudo-matching probability and the predicted matching probability predicted by the initial image lesion attribute prediction model are used as inputs, and the difference between the two is calculated using a preset loss function (such as cross entropy loss, mean square error, etc.). The subcategory matching loss function is the loss value calculated for the specific lesion attribute subcategory, which quantifies the model's prediction accuracy for the single subcategory. This step is performed independently for all lesion attribute subcategories to obtain a loss value for each subcategory.

[0118] In step S502 of some embodiments, function fusion is performed based on the multiple subcategory matching loss functions of each lesion attribute category to obtain a lesion attribute category matching loss function. The subcategory matching loss functions of all lesion attribute subcategories under the same lesion attribute category (such as lesion type) calculated in step S501 are aggregated or fused. Multiple "subcategory matching loss functions" belonging to the same category are input for "fusion" operation. Function fusion can be achieved in a variety of ways, such as simply averaging the loss values ​​of these subcategories, or performing weighted summation (the weights can be set according to factors such as the clinical importance of the subcategory or the amount of data). The lesion attribute category matching loss function is the loss value obtained after fusion, which represents the prediction accuracy of the entire lesion attribute category. This step enables the loss calculation to reflect the prediction performance of the initial image lesion attribute prediction model on higher-level lesion categories.

[0119] In step S503 of some embodiments, a weighted sum is performed based on multiple lesion attribute category matching loss functions to obtain a target matching loss function. This step is the link for the final aggregation of the total loss. Figure 5 As shown, multiple "lesion attribute category matching loss functions" calculated in step S502 for different lesion attribute categories (e.g., lesion types) are input for a "weighted summation" operation. Weighted summation refers to multiplying these lesion attribute category matching loss functions by their respective preset weight coefficients (these weights can reflect the clinical significance, data distribution, or training focus of different lesion categories, etc.), and then adding up the weighted loss values. The target matching loss function is the final calculated total loss value, which comprehensively reflects the overall prediction performance of the initial image lesion attribute prediction model on all lesion attribute categories and subcategories, and is the final scalar value used to guide the update of model parameters.

[0120] Through steps S501 to S503, this embodiment provides a hierarchical loss calculation method. First, the loss is calculated at the finest sub-category level, and then the sub-category matching loss functions under the same lesion attribute category are fused to obtain the lesion attribute category matching loss function, and finally the losses of different categories are weighted and summed to obtain the final target matching loss function. This hierarchical loss design enables the model to focus on both fine-grained lesion sub-category predictions and macroscopic lesion category predictions during the training process, thereby helping the model to comprehensively learn the lesion information in the fundus image and improve the model's predictive ability. This can improve the accuracy of lesion analysis of fundus images, thereby improving performance in downstream ophthalmic disease diagnosis and analysis tasks, and thus provide doctors with more accurate diagnostic support, with high applicability.

[0121] In step S108 of some embodiments, the initial image lesion attribute prediction model is updated based on the target matching loss function to obtain the image lesion attribute prediction model. After the target matching loss function is calculated in step S107, this step uses the loss function to update the parameters of the initial image lesion attribute prediction model. The model parameters are usually updated using a gradient-based optimization algorithm, such as stochastic gradient descent (SGD), Adam optimizer, etc. These optimization algorithms adjust the parameters of the model based on the gradient information of the loss function so that the value of the loss function gradually decreases. By continuously iterating steps S105 to S108, the model can gradually learn to more effectively extract fundus image features and more accurately predict the lesion attributes of the fundus image. The updated model can better associate the visual features of the fundus image with the text description of the lesion attribute subcategory, thereby improving the accuracy of lesion analysis of the fundus image. After multiple iterative training, the image lesion attribute prediction model finally obtained can be used for various downstream ophthalmic disease diagnosis and analysis tasks.

[0122] See also Figure 6 In some embodiments, before step S108, the following steps may also be performed, but not limited to, steps S601 to S605:

[0123] Step S601, taking each fundus image sample as a target fundus image sample one by one, performing different data enhancements on the target fundus image sample to obtain multiple different views;

[0124] Step S602: Input the multiple views into the target network to obtain positive contrast image visual features;

[0125] Step S603, obtaining negative contrast image visual features based on reference image visual features of other fundus image samples, where the other fundus image samples are all fundus image samples except the target fundus image sample;

[0126] Step S604, performing loss calculation based on the visual features of the reference image, the visual features of the positive contrast image, and the visual features of the negative contrast image to obtain a self-supervised loss function;

[0127] Step S605: Update the target matching loss function according to the self-supervision loss function.

[0128] In step S601 of some embodiments, each fundus image sample is taken as a target fundus image sample one by one, and different data enhancements are performed on the target fundus image sample to obtain multiple different views. The focus is on the utilization of the internal information of a single fundus image sample. Specifically, an original "fundus image sample" is selected as the "target fundus image sample". Data enhancement refers to applying a series of random or preset image transformation operations to the target fundus image sample, such as cropping, rotation, color jittering, blurring, etc., each time applying a different combination of transformations, thereby generating multiple image versions that are different in appearance but semantically related in content from the same original image. Multiple different views are variants of the same fundus image obtained after different data enhancement operations. This process is intended to force the initial image lesion attribute prediction model to learn the image content of the fundus image sample itself, rather than just the memory of specific surface features.

[0129] In step S602 of some embodiments, the "multiple different views" generated in step S601 are respectively input into the "target network" for feature extraction. Since these views all originate from the same original fundus image, they are semantically related. After being processed by the target network, each view will generate a corresponding feature vector. The positive contrast image visual features are the set of feature vectors extracted from different views of the same original image. In the context of self-supervised contrastive learning, these feature vectors are regarded as "positive sample pairs" because they originate from the same source.

[0130] In step S603 of some embodiments, other fundus image samples refer to all fundus image samples in the current training batch except the “target fundus image sample” selected in step S601. Figure 6 As shown in Figure 2, the "reference image visual features" obtained by passing these other fundus image samples through the target network are considered negative sample features. Negative contrast image visual features are a set of feature vectors derived from these other images that are different from the target sample. In self-supervised contrastive learning, these feature vectors are considered "negative samples" because they differ from the original image content of the target sample.

[0131] In step S604 of some embodiments, the reference image visual features, the positive contrast image visual features, and the negative contrast image visual features are input into the "self-supervised loss calculation" module. The self-supervised loss function usually adopts contrastive learning loss (for example, InfoNCE loss, NT-Xent loss, etc.), the basic idea of ​​which is to maximize the similarity between positive sample pairs (features of different views of the same original image) while minimizing the similarity between positive samples and negative samples (features of the target image view and features of other images). The self-supervised loss function is the calculated loss value, which prompts the target network to learn an image representation so that different transformations of the same image have similar features, while the features of different images are far apart.

[0132] It should be noted that in some embodiments, the specific construction method of the self-supervised loss function is not fixed and its form may vary depending on the self-supervised model adopted. For example, when using contrastive learning-based methods such as SimCLR, MoCo, or BYOL, the self-supervised loss function is generally based on the core idea of ​​narrowing the distance in feature space between features of different enhanced views of the same original image (i.e., positive sample pairs) while simultaneously pushing the distance between features of different image views (i.e., negative sample pairs) further apart. A specific implementation can be the InfoNCE loss, which measures the similarity of positive sample pairs relative to the negative sample set. In contrast, when using reconstruction learning-based methods such as MAE (Masked Autoencoder), the self-supervised loss function is to train the model to predict or reconstruct the masked parts of the input image, for example by calculating the mean squared error or L1 loss between the reconstructed pixel values ​​and the original pixel values. There are also clustering-based methods (such as SwAV), whose loss function may involve the correspondence between the predicted samples and their assigned cluster centers. The self-supervised loss function calculated in step S604 of the embodiment of the present application can be implemented according to the actually selected self-supervised learning paradigm and its corresponding specific loss calculation formula, but its fundamental purpose is to train the target network to learn high-quality visual feature representation by utilizing the relationship within the fundus image samples themselves or between batch samples.

[0133] In step S605 of some embodiments, the "self-supervised loss function" is used to "update" or more accurately, is combined with the "target matching loss function". This combination is usually a weighted summation of the two loss functions to form a joint loss function, and then the model parameters are updated based on this joint loss function. For example, the total loss ultimately used to update the model can be total loss = target matching loss + self-supervised loss. By adding self-supervised loss, the training of the initial image lesion attribute prediction model no longer relies solely on the probability of pseudo-matching, but also utilizes the intrinsic structure and relationship of the image itself through contrastive learning, thereby further improving the quality of the image visual features learned by the target network. This enables the model to focus more on diagnostic features, thereby showing higher accuracy in downstream tasks (such as fundus image classification).

[0134] Through steps S601 to S605, this embodiment not only performs supervised learning based on the pseudo-matching probability, but also introduces a self-supervised training mechanism based on contrastive learning. This mechanism enables the target network to learn a more robust and discriminative visual representation of the image content through data enhancement, feature extraction and contrast loss calculation. Combining the self-supervised loss with the target matching loss can make full use of the structural information in the unlabeled or weakly labeled data and enhance the feature extraction ability of the initial image lesion attribute prediction model. This multi-task or joint training method helps to train a more powerful visual encoder, thereby improving the accuracy of the model's lesion analysis of fundus images, thereby improving the performance in downstream ophthalmic disease diagnosis and analysis tasks, and thus providing doctors with more accurate diagnostic support, with high applicability.

[0135] Through steps S101 to S108, the embodiment of the present application first obtains a fundus image sample and encodes the fundus image sample using a preset visual encoder to obtain a pseudo-image visual feature. At the same time, the description text of multiple lesion attribute subcategories is obtained and text encoded to obtain lesion attribute subcategory features. Then, the pseudo-image visual feature is matched with the lesion attribute subcategory features to obtain a pseudo-matching probability. The pseudo-matching probability can be understood as a pseudo-label or training target that reflects the correlation between the image and the text description of each fine-grained lesion subcategory constructed based on a preset image-text encoder (such as CLIP). In addition, in some embodiments, the pseudo-matching probability can also pass through a dynamic depolarizer, which can dynamically correct the pseudo-matching probability based on historical statistical information to obtain a more stable and reliable corrected pseudo-matching probability. Then, the present application visually encodes the same fundus image sample through a preset initial image lesion attribute prediction model (including a target network and a predictor) to obtain the initial image visual feature. The initial image visual feature is matched with the lesion attribute subcategory features to obtain the model's current predicted matching probability. Subsequently, the loss is calculated based on the pseudo-matching probability (corrected pseudo-matching probability) and the predicted matching probability to obtain the target matching loss function, which is calculated and aggregated in a hierarchical manner, thereby focusing on the prediction accuracy of subcategories and categories at the same time. In some embodiments, before updating the model based on the target matching loss, an additional self-supervised learning process is introduced to further optimize the robustness and discriminability of the image representation learned by the target network through data enhancement, feature extraction and contrast loss calculation (the construction method of the self-supervised loss can be adjusted according to the specific model). Finally, based on the calculated target matching loss function (joint loss function), the parameters of the initial image lesion attribute prediction model are updated to obtain a trained image lesion attribute prediction model. The embodiment of the present application utilizes cross-modal fine-grained text information as a pseudo-label for training, and combines it with a self-supervised learning mechanism to enhance the feature extraction capability of the model, so that the trained model can overcome the challenge that existing methods are difficult to capture subtle lesion information in fundus images. In particular, pseudo-match probabilities provide a finer-grained supervisory signal than coarse-grained image labels, guiding the model to learn the association between image features and specific lesion attribute subcategories; while optional dynamic debiasing and self-supervised learning further improve training efficiency and model performance. This training method enables the model to more accurately identify and distinguish lesions with diverse morphologies and fuzzy boundaries in fundus images, thereby improving the accuracy of lesion analysis in fundus images, thereby improving performance in downstream ophthalmic disease diagnosis and analysis tasks (such as disease grading, lesion detection, follow-up assessment, etc.), thereby providing doctors with more accurate diagnostic support and having high applicability.

[0136] In addition, the present application embodiment proposes a method for predicting image lesion attributes. Figure 7, which is an optional flow chart of the image lesion attribute prediction method provided in an embodiment of the present application, Figure 7 The method in the embodiment may include but is not limited to steps S701 to S704. Figure 7 The order of step S701 to step S704 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0137] Step S701, acquiring a target fundus image;

[0138] Step S702: predicting the lesion attributes of the target fundus image using the image lesion attribute prediction model to obtain target fundus image features; the image lesion attribute prediction model is trained using the image lesion attribute prediction model training method described above;

[0139] Step S703 , matching the target fundus image features with the features of the multiple lesion attribute subcategories of the multiple lesion attribute categories to obtain target matching probabilities of the multiple lesion attribute subcategories of each lesion attribute category;

[0140] Step S704 : screening multiple lesion attribute subcategories of each lesion attribute category according to the target matching probability to obtain a target predicted lesion attribute subcategory of the lesion attribute category.

[0141] In some embodiments, first, a target fundus image is obtained as the starting point of the analysis. Next, the image lesion attribute prediction model trained by the aforementioned image lesion attribute prediction model training method is used to predict the lesion attributes of the target fundus image, thereby extracting target fundus image features that can characterize the image lesion characteristics. Then, based on these features, multiple lesion attribute categories and their subcategory features are matched to calculate the target matching probability corresponding to each lesion attribute subcategory. Finally, each lesion attribute subcategory is screened according to the target matching probability, and the most likely lesion attribute subcategory is determined as the prediction result (target predicted lesion attribute subcategory). This process is not only comprehensive, but also highly flexible and adaptable. It can effectively respond to different fundus image samples and provide strong support for auxiliary diagnosis of ophthalmic diseases.

[0142] See also Figure 8 The present application also provides an image lesion attribute prediction model training device, which can implement the above-mentioned image lesion attribute prediction model training method. Figure 8The present invention provides a schematic diagram of a basic image lesion attribute prediction model training device provided in an embodiment of the present invention. The system includes: a data acquisition module 801, a pseudo-image visual feature generation module 802, a lesion attribute subcategory feature generation module 803, a pseudo-matching probability generation module 804, an initial image visual feature generation module 805, a predicted matching probability generation module 806, a target matching loss function generation module 807, and a model updating module 808. The data acquisition module 801 is used to acquire fundus image samples; the pseudo-image visual feature generation module 802 is used to visually encode the fundus image samples using a preset visual encoder to obtain pseudo-image visual features; the lesion attribute subcategory feature generation module 803 is used to acquire multiple lesion attribute subcategory description texts for each of multiple lesion attribute categories, and perform text encoding on each lesion attribute subcategory description text to obtain lesion attribute subcategory features; the pseudo-matching probability generation module 804 is used to match the pseudo-image visual features with the multiple lesion attribute subcategory features of each lesion attribute category to obtain pseudo-matching probabilities for each of the multiple lesion attribute subcategories; the initial image visual feature generation module 805 is used to generate a pseudo-matching probability for each of the multiple lesion attribute subcategories. Used to visually encode fundus image samples through a preset initial image lesion attribute prediction model to obtain initial image visual features; a predicted matching probability generation module 806 is used to match the initial image visual features with multiple lesion attribute subcategory features of each lesion attribute category to obtain the predicted matching probability of each of the multiple lesion attribute subcategories; a target matching loss function generation module 807 is used to calculate the loss based on the pseudo-matching probability and predicted matching probability of the multiple lesion attribute subcategories of each of the multiple lesion attribute categories to obtain the target matching loss function; a model updating module 808 is used to update the initial image lesion attribute prediction model based on the target matching loss function to obtain the image lesion attribute prediction model.

[0143] It should be noted that the specific implementation of the image lesion attribute prediction model training device is basically the same as the specific embodiment of the above-mentioned image lesion attribute prediction model training method, and will not be repeated here.

[0144] An embodiment of the present application further provides an electronic device, including:

[0145] at least one memory;

[0146] at least one processor;

[0147] at least one program;

[0148] The program is stored in the memory, and the processor executes the at least one program to implement the above-mentioned image lesion attribute prediction model training method implemented in this application. The electronic device can be any smart terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), an in-vehicle computer, etc.

[0149] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0150] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0151] The memory 902 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the image lesion attribute prediction model training method or image lesion attribute prediction method of the embodiments of this application;

[0152] Input / output interface 903, used to implement information input and output;

[0153] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0154] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0155] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0156] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned image lesion attribute prediction model training method or image lesion attribute prediction method.

[0157] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The present application proposes a method for training an image lesion attribute prediction model, an image lesion attribute prediction method and related devices. First, a fundus image sample is visually encoded by a preset visual encoder to obtain a pseudo-image visual feature, and multiple lesion attribute subcategory description texts of multiple lesion attribute categories are obtained at the same time, and each lesion attribute subcategory description text is text-encoded to obtain a lesion attribute subcategory feature; then, the pseudo-image visual feature is matched with the multiple lesion attribute subcategory features of each lesion attribute category to obtain a pseudo-matching probability of each of the multiple lesion attribute subcategories, and the fundus image sample is visually encoded by a preset initial image lesion attribute prediction model to obtain an initial image visual feature; then, the initial image visual feature is matched with the multiple lesion attribute subcategory features of each lesion attribute category to obtain a predicted matching probability of each of the multiple lesion attribute subcategories; finally, a loss is calculated based on the pseudo-matching probabilities and predicted matching probabilities of the multiple lesion attribute subcategories of the multiple lesion attribute categories to obtain a target matching loss function; and the initial image lesion attribute prediction model is updated based on the target matching loss function to obtain an image lesion attribute prediction model. The embodiment of the present application utilizes pseudo-image visual features extracted from fundus image samples by a preset visual encoder and lesion attribute subcategory features obtained by encoding multiple lesion attribute subcategory description texts to accurately generate a pseudo-matching probability as a supervisory signal. Then, during the model training process, the pseudo-matching probability is used together with the predicted matching probability predicted by the initial model to construct a target matching loss function to optimize and adjust the initial image lesion attribute prediction model, thereby prompting the model to learn the precise mapping relationship from image visual features to specific lesion attribute subcategories, so as to make up for the defects of insufficient learning of fine-grained lesion attributes or insufficient semantic understanding in traditional training methods, and improve the model's prediction accuracy and detail for various lesion attributes in fundus images. Ultimately, the embodiment of the present application can improve the accuracy of lesion analysis of fundus images, thereby improving performance in downstream ophthalmic disease diagnosis and analysis tasks, and thus can provide doctors with more accurate diagnostic support, with high applicability.

[0159] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0160] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0161] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0162] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0163] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0164] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0166] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0169] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for training an image lesion attribute prediction model, characterized in that: The method comprises: obtaining fundus image samples; Visually encoding the fundus image sample using a preset visual encoder to obtain a pseudo image visual feature; Acquire multiple lesion attribute subcategory description texts of respective multiple lesion attribute categories, and perform text encoding on each of the lesion attribute subcategory description texts to obtain lesion attribute subcategory features; Matching the pseudo image visual features with the features of the plurality of lesion attribute subcategories of each lesion attribute category to obtain pseudo matching probabilities for the plurality of lesion attribute subcategories; Visually encoding the fundus image sample using a preset initial image lesion attribute prediction model to obtain initial image visual features; Matching the initial image visual features with the features of the plurality of lesion attribute subcategories of each lesion attribute category to obtain predicted matching probabilities for the plurality of lesion attribute subcategories; performing loss calculation based on the pseudo matching probabilities and the predicted matching probabilities of the plurality of lesion attribute subcategories of each of the plurality of lesion attribute categories to obtain a target matching loss function; The initial image lesion attribute prediction model is updated based on the target matching loss function to obtain an image lesion attribute prediction model.

2. The method according to claim 1, characterized in that The performing loss calculation based on the pseudo matching probabilities and the predicted matching probabilities of the plurality of lesion attribute subcategories of each of the plurality of lesion attribute categories to obtain a target matching loss function includes: Performing loss calculation based on the pseudo-matching probability and the predicted matching probability to obtain a subcategory matching loss function for each of the lesion attribute subcategories; Performing function fusion based on the plurality of subcategory matching loss functions of each lesion attribute category to obtain a lesion attribute category matching loss function; The target matching loss function is obtained by performing weighted summation on the multiple lesion attribute category matching loss functions.

3. The method according to claim 1, characterized in that After matching the pseudo image visual features with the features of the plurality of lesion attribute subcategories of each lesion attribute category to obtain pseudo matching probabilities for the plurality of lesion attribute subcategories, the method further includes: Obtaining the pseudo-matching probability of each of the lesion attribute subcategories in the plurality of fundus image samples to obtain a plurality of probabilities to be corrected; The multiple probabilities to be corrected are corrected according to a preset dynamic depolarizer to obtain the corrected pseudo-matching probability.

4. The method according to claim 3, characterized in that Correcting the plurality of probabilities to be corrected according to a preset dynamic depolarizer to obtain corrected pseudo-match probabilities includes: Obtaining an average historical false matching probability for each lesion attribute subcategory; Calculating an average value of the false matching probability of each lesion attribute subcategory in a plurality of fundus image samples to obtain an average false matching probability; Update the historical average value of pseudo matching probabilities according to the average value of pseudo matching probabilities to obtain a current average value of pseudo matching probabilities; The multiple probabilities to be corrected are corrected according to the current average value of the pseudo-match probabilities and a debiasing factor to obtain the corrected pseudo-match probabilities.

5. The method according to claim 1, wherein The initial image lesion attribute prediction model includes a target network and a predictor. The fundus image sample is visually encoded using the preset initial image lesion attribute prediction model to obtain initial image visual features, including: The target network is constructed according to the visual encoder, and the fundus image sample is input into the target network to obtain a reference image visual feature; The reference image visual features are input into the predictor to obtain the initial image visual features.

6. The method according to claim 5, characterized in that Before updating the initial image lesion attribute prediction model based on the target matching loss function to obtain the image lesion attribute prediction model, the method further includes: Taking each of the fundus image samples as a target fundus image sample one by one, performing different data enhancements on the target fundus image samples to obtain a plurality of different views; Inputting the plurality of views into the target network respectively to obtain positive contrast image visual features; Obtaining negative contrast image visual features based on the reference image visual features of the other fundus image samples, wherein the other fundus image samples are all the fundus image samples except the target fundus image sample; Performing loss calculation based on the visual features of the reference image, the positive contrast image, and the negative contrast image to obtain a self-supervised loss function; The target matching loss function is updated according to the self-supervised loss function.

7. A method for predicting image lesion attributes, characterized in that: The method comprises: Acquire target fundus image; Predicting lesion attributes of the target fundus image using an image lesion attribute prediction model to obtain target fundus image features; the image lesion attribute prediction model is trained using the method according to any one of claims 1 to 6; Matching the target fundus image features with the features of the plurality of lesion attribute subcategories of the plurality of lesion attribute categories to obtain target matching probabilities of the plurality of lesion attribute subcategories of each lesion attribute category; The plurality of lesion attribute subcategories of each lesion attribute category are screened according to the target matching probability to obtain a target predicted lesion attribute subcategory of the lesion attribute category.

8. A device for training an image lesion attribute prediction model, characterized in that: The device comprises: A data acquisition module, used for acquiring fundus image samples; a pseudo image visual feature generation module, configured to perform visual encoding on the fundus image sample using a preset visual encoder to obtain a pseudo image visual feature; a lesion attribute subcategory feature generation module, configured to obtain a plurality of lesion attribute subcategory description texts for respective lesion attribute categories, and perform text encoding on each of the lesion attribute subcategory description texts to obtain lesion attribute subcategory features; a pseudo-matching probability generating module, configured to match the pseudo-image visual features with the features of the plurality of lesion attribute subcategories of each lesion attribute category to obtain pseudo-matching probabilities for each of the plurality of lesion attribute subcategories; An initial image visual feature generation module is used to visually encode the fundus image sample using a preset initial image lesion attribute prediction model to obtain initial image visual features; a predicted matching probability generating module, configured to match the visual features of the initial image with the features of the plurality of lesion attribute subcategories of each lesion attribute category to obtain predicted matching probabilities for each of the plurality of lesion attribute subcategories; a target matching loss function generating module, configured to perform loss calculation based on the pseudo matching probabilities and the predicted matching probabilities of the plurality of lesion attribute subcategories of each of the plurality of lesion attribute categories to obtain a target matching loss function; A model updating module is used to update the initial image lesion attribute prediction model based on the target matching loss function to obtain an image lesion attribute prediction model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 or the method according to claim 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 or the method according to claim 7 is implemented.

Images