Dry eye detection method and system based on multi-modal prior and generalized contrast learning

Abstract

This invention relates to the field of medical image analysis technology, specifically to a method and system for dry eye detection based on multimodal prior and generalized contrastive learning. The method includes: first, acquiring typical images, descriptive text, and relationship graphs of various dry eye diseases; extracting and stitching multimodal features to construct a medical prior knowledge base; second, extracting visual features from the anterior segment of the eye image and retrieving corresponding semantic features from the knowledge base according to disease labels, then projecting and aligning the two; third, constructing a medical similarity matrix based on multimodal features, mining difficult negative samples, and performing generalized contrastive learning on positive and ordinary negative samples to train a visual feature extraction network; finally, inputting the test image into the network for tear river segmentation, calculating the tear river height and pupil-tear river distance, and combining the cosine similarity of visual features and various semantic projection features to output the probability that the test image belongs to each dry eye disease level. This invention improves the accuracy of dry eye detection.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis technology, and in particular to a method and system for detecting dry eye syndrome based on multimodal priors and generalized contrastive learning. Background Technology

[0002] Dry eye syndrome is a common chronic ocular surface disease. Current clinical testing mainly relies on highly subjective tests such as tear film breakup time, resulting in poor reproducibility. Automated analysis methods based on tear film parameters face technical challenges such as sensitivity to image quality, scarcity of labeled data, poor model generalization ability, and lack of guidance from medical knowledge.

[0003] There are four major technical bottlenecks in existing intelligent dry eye detection methods: traditional tear river parameter measurement relies heavily on doctors' subjective experience, leading to inconsistent diagnostic standards; existing automated image analysis methods are not robust enough to low-quality images such as blurred tear river edges and have large measurement errors; depth segmentation models suffer from severe overfitting and poor generalization ability in scenarios with scarce labeled data; and there is a semantic gap between the algorithm and clinical medical knowledge, resulting in a lack of interpretability in the decision-making process. To address these issues, we propose an intelligent dry eye detection method based on multimodal prior embedding and generalized contrastive learning. Summary of the Invention

[0004] This invention provides a method and system for detecting dry eye syndrome based on multimodal priors and generalized contrastive learning, which addresses the problems of strong measurement subjectivity, low automation accuracy, severe overfitting with few samples, and lack of medical knowledge guidance in existing dry eye syndrome detection methods.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] The first aspect of this invention is to provide a dry eye detection method based on multimodal priors and generalized contrastive learning, comprising: We acquire typical images of each disease in dry eye syndrome, disease description texts for all diseases, and disease relationship graphs, and extract multimodal medical semantic features. Then, we concatenate the multimodal medical semantic features to obtain the semantic feature vector for each disease. We construct a medical prior knowledge base using the semantic feature vectors of all disease types. Acquire an anterior segment image of the eye, label it with disease category tags, and extract visual feature vectors; based on the disease category tags, retrieve and extract semantic feature vectors of the corresponding disease category from the medical prior knowledge base, and project and align them with the visual feature vectors to obtain visual projection feature vectors and semantic projection feature vectors. Anchor samples are obtained, and a medical similarity matrix between diseases is constructed based on multimodal medical semantic features. Based on the medical similarity matrix, samples with high similarity to anchor samples are identified and denoted as difficult negative samples. Positive samples and ordinary negative samples are constructed using anchor samples and difficult negative samples. Through comparative learning using anchor samples, difficult negative samples, positive samples, and ordinary negative samples, a well-trained visual feature extraction network is obtained. The test image of the anterior segment of the eye is input into a trained visual feature extraction network to extract visual features and perform segmentation to obtain a tear river segmentation mask; the tear river height and the distance from the pupil to the tear river are determined by the tear river segmentation mask. Visual features are projected and aligned, and the cosine similarity between the visual features and the semantic projection feature vectors of each disease is calculated. The probability that the tested anterior segment image of the eye belongs to each dry eye syndrome level is obtained by using the cosine similarity, tear river height, and pupil-to-tear river distance.

[0007] Furthermore, the process of obtaining the semantic feature vector for each type of disease includes: By having experts score typical images of each disease, a multi-dimensional visual attribute vector is obtained; the visual attribute vector is then input into a fully connected network, and a visual attribute feature vector is obtained through the ReLU activation function. The pre-trained BioClinicalBERT model encodes the disease description text for each category and outputs text feature vectors. Based on the disease relationship graph, a graph convolutional network is used to encode the structural relationships between nodes to obtain graph relationship features; Among them, multimodal medical semantic features include visual attribute feature vectors, text feature vectors, and graph relationship features; Visual attribute feature vectors, text feature vectors, and graph relation features are concatenated and fused to obtain a joint feature vector. Then, a linear dimensionality reduction mapping operation is performed through a fully connected layer and layer normalization to finally output the structured medical semantic feature vector corresponding to each disease, denoted as the semantic feature vector of each disease.

[0008] Furthermore, the process of extracting the visual feature vector includes: The anterior segment image of the eye is input into an improved ResNet-34 network, which outputs a visual feature vector. The improved ResNet-34 network is obtained by adding a spatial attention module and a channel attention module after the fourth and fifth stage functional modules of the ResNet-34 network, respectively.

[0009] Furthermore, the process of obtaining the medical similarity matrix includes:

[0010] In the formula, Indicates the first Class of diseases and the first Cosine similarity of text feature vectors between disease classes Indicates the first Class of diseases and the first Cosine similarity of visual attribute feature vectors between disease classes In the disease relationship map, the first... Disease-like nodes and the first The distance between disease-related nodes; Indicates the first Class of diseases and the first Medical similarity between similar diseases; This represents the preset text feature weights. To preset the visual attribute feature weights, Preset the weights of the graph relationship features; A medical similarity matrix is ​​constructed by using the medical similarity between any two disease categories; the rows and columns of the medical similarity matrix represent the disease categories.

[0011] Furthermore, the process of obtaining the positive samples and ordinary negative samples includes: Positive samples are obtained by randomly enhancing the anterior segment image of the eye in the anchor point sample; Select samples from the current batch that are different from the anchor point category and are not in the hard negative sample set, and denot them as ordinary negative samples.

[0012] Furthermore, the trained visual feature extraction network includes: The anterior segment images of the eyes in the batch are input into the visual feature extraction network for comparative learning. The parameters are updated by the AdamW optimizer according to the total loss function and trained. The trained visual feature extraction network parameters are output. That is, the trained visual feature extraction network is determined by the trained visual feature extraction network parameters. The anterior segment images of the eye in the batch include anchor point samples, positive samples, ordinary negative samples, and difficult negative samples; The total loss function is specifically expressed by the following formula:

[0013] In the formula, The weights represent the visual and semantic alignment loss. This represents the total loss value. Represents the visual and semantic alignment loss values. This represents the contrast loss value; where the weights for visual and semantic alignment losses are empirical values. The visual and semantic alignment loss functions are specifically expressed by the following formulas:

[0014] In the formula, Indicates batch size, Indicates the index of the anchor sample. The visual projection features of the anchor point sample. Indicates the category corresponding to the anchor sample semantic projection features, It is the square of the L2 norm, that is, the square of the Euclidean distance; The contrastive loss function is specifically expressed by the following formula:

[0015] In the formula, The visual projection features representing positive samples. The visual projection features representing ordinary negative samples, Visual projection features representing difficult negative samples, This represents a temperature parameter that controls the smoothness of the similarity distribution. Represents the set of ordinary negative samples. Represents the set of difficult negative samples. This represents the penalty weight for difficult negative samples. This represents an exponential function that maps similarity to positive numbers; Represents the logarithmic function. represents the vector dot product, and represents the cosine similarity.

[0016] Furthermore, the process of obtaining the tear river height and the distance from the pupil to the tear river includes: Based on the tear river segmentation mask, the lower and upper boundary points of the tear river are obtained by vertical projection. Then, the pupil center point is found from the central region of the tear river segmentation mask by Hough circle detection. The central region of the tear river segmentation mask is a region set based on experience. The height of the tear river is obtained based on the vertical difference between its lower and upper boundaries, and the distance from the pupil to the tear river is obtained based on the vertical difference between its lower boundary and the center of the pupil. The specific formulas for the height of the tear river and the distance from the pupil to the tear river are as follows:

[0017]

[0018] In the formula, This represents the vertical coordinates of the point at the upper boundary of the Tears River. This represents the vertical coordinates of the lower boundary point of the Tears River. Represents the vertical coordinates of the pupil's center point. Indicates the calibration conversion factor; Indicates the height of the River of Tears. This indicates the distance from the pupil to the tear duct.

[0019] Furthermore, the probability that the anterior segment images of the eye tested belong to various levels of dry eye syndrome includes: The visual features corresponding to the tested anterior segment image of the eye are projected into the contrast space to obtain the test visual projection feature vector; the cosine similarity between the test visual projection feature vector and the semantic projection feature vector of each disease is calculated. Based on the cosine similarity between the test visual projection feature vector and the semantic projection feature vector of each disease, the tear river height, and the distance from the pupil to the tear river, the probability that the tested anterior segment image of the eye belongs to each dry eye syndrome level is obtained.

[0020] In the formula, Indicates the test visual projection feature vector and the first Cosine similarity between semantic projection feature vectors of disease classes Indicates the height of the River of Tears. Indicates the distance from the pupil to the tear duct. This indicates that the anterior segment image of the eye in the test belongs to the first... The possibility of it being at the level of dry eye syndrome. This indicates an indicator function that states if the tear river height or pupil-to-tear river distance of the current test image falls within the specified range. If the value falls within the normal range for dry eye syndrome, the value is 1; otherwise, the value is 0. This is the activation function used for data normalization. To preset similarity weights, 2 represents the preset feature weights.

[0021] A second aspect of the present invention is to provide a dry eye detection system based on multimodal priors and generalized contrastive learning, comprising: The medical prior knowledge base construction module is used to acquire typical images of each disease in dry eye syndrome, disease description texts of all diseases, and disease relationship graphs, and extract multimodal medical semantic features. Then, the multimodal medical semantic features are concatenated to obtain the semantic feature vector of each disease. The medical prior knowledge base is constructed through the semantic feature vectors of all disease types. Visual and semantic feature extraction and alignment module: used to acquire anterior segment images of the eye, label them with disease category tags and extract visual feature vectors; based on the disease category tags, retrieve and extract semantic feature vectors of the corresponding disease category from the medical prior knowledge base, and project and align them with the visual feature vectors to obtain visual projection feature vectors and semantic projection feature vectors; Generalized contrastive learning training module: used to acquire anchor samples, construct a medical similarity matrix between diseases based on multimodal medical semantic features; based on the medical similarity matrix, mine samples with high similarity to anchor samples, and denot them as difficult negative samples; construct positive samples and ordinary negative samples through anchor samples and difficult negative samples; obtain a trained visual feature extraction network through contrastive learning of anchor samples, difficult negative samples, positive samples and ordinary negative samples; The tear river geometry parameter determination module is used to input the tested anterior segment image of the eye into a trained visual feature extraction network to extract visual features and perform segmentation to obtain a tear river segmentation mask; the tear river height and the distance from the pupil to the tear river are determined through the tear river segmentation mask. The probability analysis module for dry eye syndrome levels is used to project and align visual features and calculate the cosine similarity between the feature vector and the semantic projection feature vector of each disease. Based on the cosine similarity, tear river height, and pupil-to-tear river distance, the probability that the tested anterior segment image of the eye belongs to each dry eye syndrome level is obtained.

[0022] A third aspect of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the dry eye detection method based on multimodal priors and generalized contrastive learning.

[0023] Compared with existing technologies, the beneficial effects of this invention are as follows: Typical images of each type of dry eye disease, disease description texts for all diseases, and disease relationship graphs are acquired, and multimodal medical semantic features are extracted. These multimodal medical semantic features are then concatenated to obtain semantic feature vectors for each type of disease. A medical prior knowledge base is constructed using the semantic feature vectors of all disease types. By integrating multimodal medical knowledge, the problem of the algorithm lacking medical reasoning ability is solved, improving the interpretability of the diagnosis. Anterior segment images of the eye are acquired, and disease category labels are marked and visual feature vectors are extracted. Based on the disease category labels, semantic feature vectors for the corresponding disease category are retrieved and extracted from the medical prior knowledge base, and projected and aligned with the visual feature vectors to obtain visual projection feature vectors and semantic projection feature vectors. By aligning visual features with medical semantic features, the semantic gap between modalities is eliminated, improving the accuracy of feature expression. Anchor point samples are acquired, and a medical similarity matrix between diseases is constructed based on multimodal medical semantic features. Based on the medical similarity matrix, samples with high similarity to anchor point samples are identified and recorded as difficult negative samples. Through anchor point samples… The process involves constructing positive and ordinary negative samples from difficult negative samples; obtaining a well-trained visual feature extraction network through comparative learning using anchor point samples, difficult negative samples, positive samples, and ordinary negative samples; significantly improving the ability to discriminate subtle pathological changes by mining medically relevant difficult negative samples and assigning them higher weights; inputting the tested anterior segment image of the eye into the trained visual feature extraction network to extract visual features and perform segmentation to obtain a tear river segmentation mask; determining the tear river height and the distance from the pupil to the tear river using the tear river segmentation mask; and avoiding error accumulation through end-to-end joint optimization. This approach reduces TMH measurement errors; it projects and aligns visual features, calculating the cosine similarity between the visual features and the semantic projection feature vectors for each disease category. Using this cosine similarity, tear river height, and pupil-to-tear river distance, the probability that the tested anterior segment image belongs to each dry eye syndrome level is obtained. Combining geometric parameters with semantic feature output probabilities ensures high accuracy in grading while providing doctors with a quantitative confidence reference. This addresses the problems of strong measurement subjectivity, low automation accuracy, severe overfitting with few samples, and lack of medical knowledge guidance in existing dry eye syndrome detection methods. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1A flowchart illustrating the steps of the dry eye detection method based on multimodal prior and generalized contrastive learning provided for this invention; Figure 2 This invention provides a schematic diagram of the module flow of a dry eye detection system based on multimodal priors and generalized contrastive learning. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] To address the problems existing in the background technology, a dry eye detection method and system based on multimodal prior and generalized contrastive learning was designed, which has important practical significance.

[0029] like Figure 1 As shown, the first aspect of the present invention is to provide a dry eye detection method based on multimodal priors and generalized contrastive learning, comprising the following steps: Step S1: Obtain typical images of each type of dry eye disease, disease description texts of all diseases, and disease relationship graphs, and extract multimodal medical semantic features. Then, concatenate the multimodal medical semantic features to obtain the semantic feature vector of each type of disease; construct a medical prior knowledge base through the semantic feature vectors of all types of diseases.

[0030] It should be noted that existing deep learning methods learn visual patterns only from image pixels, lacking an understanding of medical knowledge related to dry eye syndrome, resulting in limited model discrimination ability and poor interpretability. Therefore, by constructing a multimodal prior knowledge base containing text descriptions, visual attribute scores, and disease relationship graphs, and encoding this unstructured medical knowledge into structured semantic feature vectors, we can provide knowledge guidance for subsequent visual and semantic comparative learning. This allows the model to reason using prior knowledge such as anatomy and pathophysiology, just like a clinician, fundamentally solving the semantic gap between algorithms and medical knowledge.

[0031] Specifically, the process involves acquiring typical images of each type of dry eye disease, disease description text for all diseases, and disease relationship graphs (nodes representing disease categories and edges representing medical relevance). Multimodal medical semantic features are then extracted and concatenated to obtain a semantic feature vector for each disease type. These multimodal medical semantic features include visual attribute feature vectors, text feature vectors, and graph relationship features. The specific process for obtaining the semantic feature vector for each disease type is as follows: By having experts score typical images of each disease, a multi-dimensional visual attribute vector is obtained. This visual attribute vector is then input into a 3-layer fully connected network and encoded into a 256-dimensional visual attribute feature vector through the ReLU activation function (Rectified Linear Unit).

[0032] In this embodiment, the dimensions of the visual attribute vector include shape regularity, edge sharpness, texture uniformity, brightness consistency, etc.; each item's score ranges from 0 to 1. In this embodiment, the visual attribute vector has 12 dimensions, but this is not specifically limited and can be determined by the implementer according to specific circumstances.

[0033] The pre-trained BioClinicalBERT model encodes the disease description text for each class, outputting a 768-dimensional text feature vector. Based on the disease relationship graph, a two-layer graph convolutional network (GCN) is used to encode the structural relationships between nodes, resulting in 256-dimensional graph relationship features.

[0034] Visual attribute feature vectors, text feature vectors, and graph relation features are concatenated and fused to obtain a 1280-dimensional joint feature vector. Then, a linear dimensionality reduction mapping operation is performed through a fully connected layer and layer normalization. Finally, a 1024-dimensional structured medical semantic feature vector corresponding to each disease is output, denoted as the semantic feature vector of each disease.

[0035] Among them, the BioClinicalBERT model (Bio+Clinical Bidirectional Encoder Representations from Transformers) is a bidirectional encoder representation technology for both biological and clinical applications.

[0036] The semantic feature vectors of all disease categories are used to form a medical prior knowledge base.

[0037] Thus, a medical prior knowledge base was obtained.

[0038] Step S2: Acquire an anterior segment image of the eye, label it with disease category tags, and extract visual feature vectors; based on the disease category tags, retrieve and extract the semantic feature vectors of the corresponding disease category from the medical prior knowledge base, and project and align them with the visual feature vectors to obtain visual projection feature vectors and semantic projection feature vectors.

[0039] It should be noted that although the medical semantic features (text, attributes, and graphs) constructed above contain rich prior knowledge, they reside in a high-dimensional space different from image features and cannot be directly used to guide image analysis. Therefore, a visual feature extraction network is used to obtain visual representations from eye images, and these visual representations are projected onto a common contrast space for alignment with the medical semantic features. The core purpose of this is to eliminate the "heterogeneous gap" between the visual and semantic modalities, enabling the model to establish a direct mapping relationship between visual patterns in the image (such as the shape and edges of tear rivers) and expert-defined medical concepts (such as edge sharpness and disease categories), laying the foundation for medical knowledge-guided contrastive learning in subsequent steps.

[0040] It should be further clarified that ocular images can be divided into anterior segment images and posterior segment images. Anterior segment images refer to the anterior half of the eye's structure, mainly including the cornea, conjunctiva, iris, pupil, lens, tear film, and tear meniscus. Posterior segment images include the vitreous body, retina, choroid, and optic nerve. Since the core pathological changes of dry eye syndrome (decreased tear film stability, abnormal tear meniscus morphology, and ocular surface epithelial damage) all occur in ocular surface structures, they fall under the anterior segment category. The posterior segment of the eye (retina, optic nerve, etc.) is not directly related to tear secretion and tear film function, and therefore is not used as an image source for the diagnosis of dry eye syndrome. Therefore, only anterior segment images are collected.

[0041] Specifically, an anterior segment image of the eye is acquired, and the anterior segment image is labeled with disease category labels (i.e., labeled with 0, 1, 2, 3, 4).

[0042] By adding spatial attention and channel attention modules (to enhance attention to the tear river region) after conv4_x (fourth-stage functional module) and conv5_x (fifth-stage functional module) of the ResNet-34 network, respectively, an improved ResNet-34 network is obtained. The ResNet-34 network comprises five functional modules: shallow feature extraction module (first-stage functional module), low-level semantic module (second-stage functional module), mid-level semantic module (third-stage functional module), high-level semantic module (third-stage functional module), and deep discriminative module (fifth-stage functional module).

[0043] The anterior segment image of the eye is input into an improved ResNet-34 network, which outputs a visual feature vector (512-dimensional in this case).

[0044] Based on the disease category labels of the anterior segment image of the eye, semantic feature vectors of the corresponding disease category are retrieved and extracted from the medical prior knowledge base.

[0045] Two independent linear projection layers are used to map the 512-dimensional visual feature vector and the 1024-dimensional semantic feature vector to a 256-dimensional common contrast space. The projected features are then L2 normalized to obtain aligned visual and semantic projected feature vectors for subsequent similarity calculations.

[0046] Thus, the visual projection feature vector and the semantic projection feature vector are obtained.

[0047] Step S3: Obtain anchor samples and construct a medical similarity matrix between diseases based on multimodal medical semantic features; based on the medical similarity matrix, mine samples with high similarity to anchor samples and record them as difficult negative samples; construct positive samples and ordinary negative samples through anchor samples and difficult negative samples; obtain a trained visual feature extraction network through comparative learning of anchor samples, difficult negative samples, positive samples and ordinary negative samples.

[0048] It should be noted that although the model can map images and semantic features to the same space through visual-semantic feature alignment projection, conventional contrastive learning relies solely on random data augmentation to construct positive and negative samples, which cannot distinguish between medically similar but different cases (such as DEWS level I and II). Therefore, this step introduces a medically-aware hard negative sample mining mechanism. This involves constructing a medical similarity matrix using the three features from step S1, actively selecting samples with similar medical features but different categories from the anchor sample as "hard negative samples," and assigning them a higher penalty weight in the contrastive loss, forcing the model to focus on subtle pathological discriminative features. Simultaneously, a visual-semantic alignment loss is added to strengthen cross-modal consistency. Ultimately, through this medical knowledge-guided generalized contrastive learning, the model's discriminative ability and generalization performance are significantly improved in scenarios with few samples and minimal inter-class differences.

[0049] Step S31: Mining of difficult negative samples.

[0050] Specifically, based on visual attribute feature vectors, text feature vectors, and disease relationship graphs, the medical similarity between any two disease classes is calculated; whereby the medical similarity between any two disease classes is expressed by the formula:

[0051] In the formula, Indicates the first Class of diseases and the first Cosine similarity of text feature vectors between disease classes Indicates the first Class of diseases and the first Cosine similarity of visual attribute feature vectors between disease classes In the disease relationship map, the first... Disease-like nodes and the first The distance between disease nodes (i.e., the number of edges in the shortest path between two disease nodes in the relationship graph). Indicates the first Class of diseases and the first Medical similarity between similar diseases; This represents the preset text feature weights. To preset the visual attribute feature weights, The preset graph relationship feature weights are used. In this embodiment... , , In this embodiment, for , and No specific restrictions are imposed; implementers can decide based on the specific circumstances.

[0052] A medical similarity matrix is ​​constructed by using the medical similarity between any two disease categories; the rows and columns of the medical similarity matrix represent the disease categories.

[0053] Based on the medical similarity matrix, samples with high similarity to the anchor sample are identified and denoted as difficult negative samples; the specific process is as follows: The system quickly identifies several other disease categories with the highest medical similarity to the anchor sample category using a medical similarity matrix. The anterior segment images corresponding to these other disease categories are then used as difficult negative samples. In this embodiment, there are three other disease categories, but this is not specifically limited; the implementer can determine the specific categories based on the circumstances. The anchor sample category refers to the disease category of the currently processed image (e.g., DEWS Level I); DEWS (Dry Eye Workshop) Level I corresponds to Level I (mild dry eye) in the dry eye severity grading standard.

[0054] Step S32: Construct positive and negative sample pairs.

[0055] Specifically, positive samples are obtained by randomly enhancing the anterior segment image of the eye in the anchor point sample; random enhancement involves rotating the image within a small range and randomly adjusting the brightness within a small range (rotation range is ±5°; brightness adjustment range is ±10%).

[0056] Select samples from the current batch that are different from the anchor point category and are not in the hard negative sample set, and denot them as ordinary negative samples.

[0057] It should be noted that the anchor sample is the single sample currently being processed, and the current batch contains the anchor sample and several other different samples.

[0058] At this point, we have obtained positive samples and ordinary negative samples.

[0059] Step S33: Calculate the generalized contrast loss.

[0060] Specifically, the contrastive loss function is expressed by the following formula:

[0061] In the formula, Indicates batch size, Indicates the index of the anchor sample. The visual projection features of the anchor point sample. Represents the visual projection features of positive samples (enhanced views of the same image). The visual projection features representing ordinary negative samples, Visual projection features representing difficult negative samples, This represents a temperature parameter that controls the smoothness of the similarity distribution. Represents the set of ordinary negative samples. Represents the set of difficult negative samples. This represents the penalty weight for difficult negative samples. This represents an exponential function that maps similarity to positive numbers; Represents the logarithmic function. represents the vector dot product, and represents the cosine similarity (since the features have been L2 normalized). This indicates the comparison loss value.

[0062] All visual projection features are 256-dimensional and have been normalized.

[0063] In this embodiment and The values ​​are 0.07 and 2, both chosen based on experience.

[0064] Step S34: Visual-semantic alignment loss.

[0065] Specifically, the visual and semantic alignment loss functions are expressed by the following formula:

[0066] In the formula, Indicates batch size, Indicates the index of the anchor sample. The visual projection features of the anchor point sample. Indicates the category corresponding to the anchor sample semantic projection features, It is the square of the L2 norm, that is, the square of the Euclidean distance; This represents the visual and semantic alignment loss value.

[0067] Step S35: Total Loss and Optimization.

[0068] Specifically, the total loss function is expressed by the formula:

[0069] In the formula, The weights for visual and semantic alignment loss are represented (0.5 is taken as an empirical value). This represents the total loss value.

[0070] Images of the anterior segment of the eye from the batch are input into a visual feature extraction network for comparative learning. The results are then processed using the AdamW optimizer (with an initial learning rate) based on the total loss function. Weight decay The parameters are updated and trained for 100 rounds, and the trained visual feature extraction network parameters (improved ResNet-34 + projection head) are output. That is, the trained visual feature extraction network is determined by the trained visual feature extraction network parameters; it is used for tear river segmentation, parameter measurement and dry eye grading in the next step.

[0071] The batch of anterior segment images for the eye includes anchor point samples, positive samples, ordinary negative samples, and difficult negative samples. The projection head is a fully connected layer (linear layer) that maps the 512-dimensional features output by the visual feature extraction network to a 256-dimensional contrast space, typically followed by L2 normalization for calculating the contrast loss. After training in step three, the projection head is usually discarded in the inference phase of step four, and only the backbone network is used to extract visual features.

[0072] Step S4: Input the tested anterior segment image of the eye into the trained visual feature extraction network to extract visual features and perform segmentation to obtain a tear river segmentation mask; determine the tear river height and the distance from the pupil to the tear river using the tear river segmentation mask; project and align the visual features, and calculate the cosine similarity between the visual features and the semantic projection feature vector of each disease type; obtain the probability that the tested anterior segment image of the eye belongs to each dry eye syndrome level using the cosine similarity, tear river height, and distance from the pupil to the tear river.

[0073] It should be noted that while the trained visual feature extraction network possesses medical discrimination capabilities, it has not yet outputted clinically applicable diagnostic indicators. Therefore, this step utilizes the network to segment the tear meniscus region of the test image, and then automatically measures two core geometric parameters: tear meniscus height (TMH) and pupil-tear meniscus distance (PTMD, distance from the center of the pupil to the lower edge of the tear meniscus). Based on this, and combined with visual-semantic similarity and the normal range constraints of TMH, the model ultimately outputs the probability of a DEWS (Dry Eye Workshop; the grading standards published by the workshop are often referred to as DEWS grading) classification. This step realizes the transformation from "feature representation" to "clinically quantitative diagnosis," making the model output directly correspond to the measurement indicators and grading standards in clinical practice, thus meeting practical application needs.

[0074] Specifically, a test image of the anterior segment of the eye is acquired and input into a trained visual feature extraction network to extract visual features. After the visual features are extracted, a tear river segmentation mask is generated through a segmentation head (a 3-layer deconvolutional network). The tear river segmentation mask is a 512×512 pixel image, where the values ​​are 0 and 1.

[0075] Based on the tear river segmentation mask, the lower and upper boundary points of the tear river are obtained using the vertical projection method. Then, the pupil center point is found from the central region of the tear river segmentation mask using Hough circle detection. Both the vertical projection method and Hough circle detection are well-known techniques and will not be elaborated upon here. The central region of the tear river segmentation mask is an empirically defined area of ​​240×240 pixels.

[0076] The height of the tear river is obtained based on the vertical difference between its lower and upper boundaries, and the distance from the pupil to the tear river is obtained based on the vertical difference between its lower boundary and the center of the pupil. The specific formulas for the height of the tear river and the distance from the pupil to the tear river are as follows:

[0077]

[0078] In the formula, This represents the vertical coordinates of the point at the upper boundary of the Tears River. This represents the vertical coordinates of the lower boundary point of the Tears River. Represents the vertical coordinates of the pupil's center point. This indicates the calibration conversion factor, in mm / pixel; Indicates the height of the River of Tears. This represents the distance from the pupil to the tear duct. In this embodiment, the conversion coefficient is calibrated. The value is 0.01. In this embodiment, the calibration conversion coefficient is not specifically limited, and the implementer can determine it according to the specific situation.

[0079] The visual features corresponding to the tested anterior segment image of the eye are projected into the contrast space to obtain the test visual projection feature vector; the cosine similarity between the test visual projection feature vector and the semantic projection feature vector of each disease is calculated.

[0080] Based on the cosine similarity between the test visual projection feature vector and the semantic projection feature vector of each disease type, the tear river height, and the distance from the pupil to the tear river, the probability that the tested anterior segment image belongs to each dry eye syndrome level is obtained; whereby the probability that the tested anterior segment image belongs to each dry eye syndrome level is specifically expressed by the formula:

[0081] In the formula, Indicates the test visual projection feature vector and the first Cosine similarity between semantic projection feature vectors of disease classes Indicates the height of the River of Tears. Indicates the distance from the pupil to the tear duct. This indicates that the anterior segment image of the eye in the test belongs to the first... The possibility of it being at the level of dry eye syndrome. This indicates an indicator function that states if the tear river height or pupil-to-tear river distance of the current test image falls within the specified range. If the value falls within the normal range for dry eye syndrome, the value is 1; otherwise, the value is 0. This is the activation function used for data normalization. To preset similarity weights, These are preset feature weights. In this embodiment, the preset similarity weight is used. Preset feature weights In this embodiment, a preset similarity weight is used. and preset feature weights No specific restrictions are imposed; implementers can decide based on the specific circumstances.

[0082] It should be noted that each type of disease corresponds to a different level of dry eye syndrome.

[0083] At this point, the probability of the current test image belonging to each dry eye syndrome level can be determined.

[0084] Finally, the system provides doctors with the probability distribution of the current test images belonging to each level of dry eye syndrome, serving as a quantitative reference to assist doctors in making a comprehensive judgment based on clinical experience and determining the final classification result.

[0085] This concludes the embodiment.

[0086] like Figure 2 As shown, a second aspect of the present invention is to provide a dry eye detection system based on multimodal priors and generalized contrastive learning, comprising: Medical Prior Knowledge Base Construction Module 101: This module is used to acquire typical images of each type of dry eye disease, disease description texts of all diseases, and disease relationship graphs, and to extract multimodal medical semantic features. Then, the multimodal medical semantic features are concatenated to obtain the semantic feature vector of each type of disease; and a medical prior knowledge base is constructed using the semantic feature vectors of all types of diseases. Visual and semantic feature extraction and alignment module 102: used to acquire anterior segment image of the eye, and to label disease category and extract visual feature vectors; according to the disease category label, to retrieve and extract semantic feature vectors of the corresponding disease category from the medical prior knowledge base, and to project and align them with the visual feature vectors to obtain visual projection feature vectors and semantic projection feature vectors. Generalized contrastive learning training module 103: used to acquire anchor samples, construct a medical similarity matrix between diseases based on multimodal medical semantic features; based on the medical similarity matrix, mine samples with high similarity to anchor samples, and record them as difficult negative samples; construct positive samples and ordinary negative samples through anchor samples and difficult negative samples; obtain a trained visual feature extraction network through contrastive learning of anchor samples, difficult negative samples, positive samples and ordinary negative samples; Tear River Geometric Parameter Determination Module 104: This module is used to input the tested anterior segment image of the eye into a trained visual feature extraction network to extract visual features and perform segmentation to obtain a tear river segmentation mask; and to determine the tear river height and the distance from the pupil to the tear river through the tear river segmentation mask. The probability analysis module 105 for dry eye syndrome level is used to project and align visual features and calculate the cosine similarity between the visual features and the semantic projection feature vector of each disease. The probability of the tested anterior segment image of the eye belonging to each dry eye syndrome level is obtained by using the cosine similarity, tear river height and pupil-to-tear river distance.

[0087] A third aspect of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements a dry eye detection method based on multimodal priors and generalized contrastive learning.

[0088] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0089] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A dry eye detection method based on multimodal priors and generalized contrastive learning, characterized in that, include: We acquire typical images of each disease in dry eye syndrome, disease description texts for all diseases, and disease relationship graphs, and extract multimodal medical semantic features. Then, we concatenate the multimodal medical semantic features to obtain the semantic feature vector for each disease. We construct a medical prior knowledge base using the semantic feature vectors of all disease types. Acquire images of the anterior segment of the eye, label them with disease category tags, and extract visual feature vectors; Based on the disease category label, the semantic feature vector of the corresponding disease category is retrieved and extracted from the medical prior knowledge base, and projected and aligned with the visual feature vector to obtain the visual projection feature vector and the semantic projection feature vector. Anchor samples are obtained, and a medical similarity matrix between diseases is constructed based on multimodal medical semantic features. Based on the medical similarity matrix, samples with high similarity to anchor samples are identified and denoted as difficult negative samples. Positive samples and ordinary negative samples are constructed using anchor samples and difficult negative samples. Through comparative learning using anchor samples, difficult negative samples, positive samples, and ordinary negative samples, a well-trained visual feature extraction network is obtained. The test image of the anterior segment of the eye is input into a trained visual feature extraction network to extract visual features and perform segmentation to obtain a tear river segmentation mask; the tear river height and the distance from the pupil to the tear river are determined by the tear river segmentation mask. Visual features are projected and aligned, and the cosine similarity between the visual features and the semantic projection feature vectors of each disease is calculated. The probability that the tested anterior segment image of the eye belongs to each dry eye syndrome level is obtained by using the cosine similarity, tear river height, and pupil-to-tear river distance.

2. The dry eye detection method based on multimodal prior and generalized contrastive learning according to claim 1, characterized in that, The process of obtaining the semantic feature vector for each type of disease includes: By having experts score typical images of each disease, a multi-dimensional visual attribute vector is obtained; the visual attribute vector is then input into a fully connected network, and a visual attribute feature vector is obtained through the ReLU activation function. The pre-trained BioClinicalBERT model encodes the disease description text for each category and outputs text feature vectors. Based on the disease relationship graph, a graph convolutional network is used to encode the structural relationships between nodes to obtain graph relationship features; Among them, multimodal medical semantic features include visual attribute feature vectors, text feature vectors, and graph relationship features; Visual attribute feature vectors, text feature vectors, and graph relation features are concatenated and fused to obtain a joint feature vector. Then, a linear dimensionality reduction mapping operation is performed through a fully connected layer and layer normalization to finally output the structured medical semantic feature vector corresponding to each disease, denoted as the semantic feature vector of each disease.

3. The dry eye detection method based on multimodal prior and generalized contrastive learning according to claim 1, characterized in that, The process of extracting the visual feature vector includes: The anterior segment image of the eye is input into an improved ResNet-34 network, which outputs a visual feature vector. The improved ResNet-34 network is obtained by adding a spatial attention module and a channel attention module after the fourth and fifth stage functional modules of the ResNet-34 network, respectively.

4. The dry eye detection method based on multimodal prior and generalized contrastive learning according to claim 1, characterized in that, The process of obtaining the medical similarity matrix includes: In the formula, Indicates the first Class of diseases and the first Cosine similarity of text feature vectors between disease classes Indicates the first Class of diseases and the first Cosine similarity of visual attribute feature vectors between disease classes In the disease relationship map, the first... Disease-like nodes and the first The distance between disease-related nodes; Indicates the first Class of diseases and the first Medical similarity between similar diseases; This represents the preset text feature weights. To preset the visual attribute feature weights, Preset the weights of the graph relationship features; A medical similarity matrix is ​​constructed by using the medical similarity between any two disease categories; the rows and columns of the medical similarity matrix represent the disease categories.

5. The dry eye detection method based on multimodal prior and generalized contrastive learning according to claim 1, characterized in that, The process of obtaining the positive samples and ordinary negative samples includes: Positive samples are obtained by randomly enhancing the anterior segment image of the eye in the anchor point sample; Select samples from the current batch that are different from the anchor point category and are not in the hard negative sample set, and denot them as ordinary negative samples.

6. The dry eye detection method based on multimodal prior and generalized contrastive learning according to claim 1, characterized in that, The trained visual feature extraction network includes: The anterior segment images of the eyes in the batch are input into the visual feature extraction network for comparative learning. The parameters are updated by the AdamW optimizer according to the total loss function and trained. The trained visual feature extraction network parameters are output. That is, the trained visual feature extraction network is determined by the trained visual feature extraction network parameters. The anterior segment images of the eye in the batch include anchor point samples, positive samples, ordinary negative samples, and difficult negative samples; The total loss function is specifically expressed by the following formula: In the formula, The weights represent the visual and semantic alignment loss. This represents the total loss value. Represents the visual and semantic alignment loss values. This represents the contrast loss value; where the weights for visual and semantic alignment losses are empirical values. The visual and semantic alignment loss functions are specifically expressed by the following formulas: In the formula, Indicates batch size, Indicates the index of the anchor sample. The visual projection features of the anchor point sample. Indicates the category corresponding to the anchor sample semantic projection features, It is the square of the L2 norm, that is, the square of the Euclidean distance; The contrastive loss function is specifically expressed by the following formula: In the formula, The visual projection features representing positive samples. The visual projection features representing ordinary negative samples, Visual projection features representing difficult negative samples, This represents a temperature parameter that controls the smoothness of the similarity distribution. Represents the set of ordinary negative samples. Represents the set of difficult negative samples. This represents the penalty weight for difficult negative samples. This represents an exponential function that maps similarity to positive numbers; Represents the logarithmic function. represents the vector dot product, and represents the cosine similarity.

7. The dry eye detection method based on multimodal prior and generalized contrastive learning according to claim 1, characterized in that, The process of obtaining the tear river height and the distance from the pupil to the tear river includes: Based on the tear river segmentation mask, the lower and upper boundary points of the tear river are obtained by vertical projection. Then, the pupil center point is found from the central region of the tear river segmentation mask by Hough circle detection. The central region of the tear river segmentation mask is a region set based on experience. The height of the tear river is obtained based on the vertical difference between its lower and upper boundaries, and the distance from the pupil to the tear river is obtained based on the vertical difference between its lower boundary and the center of the pupil. The specific formulas for the height of the tear river and the distance from the pupil to the tear river are as follows: In the formula, This represents the vertical coordinates of the point at the upper boundary of the Tears River. This represents the vertical coordinates of the lower boundary point of the Tears River. Represents the vertical coordinates of the pupil's center point. Indicates the calibration conversion factor; Indicates the height of the River of Tears. This indicates the distance from the pupil to the tear duct.

8. The dry eye detection method based on multimodal prior and generalized contrastive learning according to claim 1, characterized in that, The probability that the anterior segment images of the eye tested belong to each level of dry eye syndrome includes: The visual features corresponding to the tested anterior segment image of the eye are projected into the contrast space to obtain the test visual projection feature vector; the cosine similarity between the test visual projection feature vector and the semantic projection feature vector of each disease is calculated. Based on the cosine similarity between the test visual projection feature vector and the semantic projection feature vector of each disease, the tear river height, and the distance from the pupil to the tear river, the probability that the tested anterior segment image of the eye belongs to each dry eye syndrome level is obtained. In the formula, Indicates the test visual projection feature vector and the first Cosine similarity between semantic projection feature vectors of disease classes Indicates the height of the River of Tears. Indicates the distance from the pupil to the tear duct. This indicates that the anterior segment image of the eye in the test belongs to the first... The possibility of it being at the level of dry eye syndrome. This indicates an indicator function that states if the tear river height or pupil-to-tear river distance of the current test image falls within the specified range. If the value falls within the normal range for dry eye syndrome, the value is 1; otherwise, the value is 0. This is the activation function used for data normalization. To preset similarity weights, 2 represents the preset feature weights.

9. A dry eye detection system based on multimodal priors and generalized contrastive learning, characterized in that, include: The medical prior knowledge base construction module is used to acquire typical images of each disease in dry eye syndrome, disease description texts of all diseases, and disease relationship graphs, and extract multimodal medical semantic features. Then, the multimodal medical semantic features are concatenated to obtain the semantic feature vector of each disease. The medical prior knowledge base is constructed through the semantic feature vectors of all disease types. Visual and semantic feature extraction and alignment module: used to acquire anterior segment images of the eye, label them with disease category tags and extract visual feature vectors; based on the disease category tags, retrieve and extract semantic feature vectors of the corresponding disease category from the medical prior knowledge base, and project and align them with the visual feature vectors to obtain visual projection feature vectors and semantic projection feature vectors; Generalized contrastive learning training module: used to acquire anchor samples, construct a medical similarity matrix between diseases based on multimodal medical semantic features; based on the medical similarity matrix, mine samples with high similarity to anchor samples, and denot them as difficult negative samples; construct positive samples and ordinary negative samples through anchor samples and difficult negative samples; obtain a trained visual feature extraction network through contrastive learning of anchor samples, difficult negative samples, positive samples and ordinary negative samples; The tear river geometry parameter determination module is used to input the tested anterior segment image of the eye into a trained visual feature extraction network to extract visual features and perform segmentation to obtain a tear river segmentation mask; the tear river height and the distance from the pupil to the tear river are determined through the tear river segmentation mask. The probability analysis module for dry eye syndrome levels is used to project and align visual features and calculate the cosine similarity between the feature vector and the semantic projection feature vector of each disease. Based on the cosine similarity, tear river height, and pupil-to-tear river distance, the probability that the tested anterior segment image of the eye belongs to each dry eye syndrome level is obtained.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the dry eye detection method based on multimodal priors and generalized contrastive learning as described in any one of claims 1-8.

Images