A dry eye grading and typing monitoring system based on multi-modal feature fusion

Abstract

The application provides a dry eye grading and typing monitoring system based on multi-modal feature fusion, tongue image of a target user is subjected to semantic segmentation and feature extraction to obtain tongue features; blood vessel texture enhancement and color threshold segmentation are performed on an eye region of the target user to obtain eye features; physiological values of the target user are subjected to normalization processing to obtain physiological metabolism features; the tongue features, the eye features and the physiological metabolism features are concatenated and spliced to obtain a total feature vector; a pre-trained classifier is used to calculate confidence degrees of different dry eye subtypes based on a preset pathological mapping rule and the total feature vector to obtain a plurality of confidence score; and a diagnosis result is determined according to a comparison result of the plurality of confidence scores; the blinking behavior, the tongue visual features and the physiological metabolism indexes are fused to improve the accuracy and stability of dry eye subtype recognition; and the preset pathological mapping rule provides a direct basis for individualized treatment.

Description

Technical Field

[0001] This invention relates to the field of computer vision and medical image analysis technology, and in particular to a dry eye disease grading and classification monitoring system based on multimodal feature fusion. Background Technology

[0002] In clinical practice, dry eye syndrome is a syndrome with significant individual differences and heterogeneous etiologies. Different patients exhibit considerable variations in tear secretion function, tear film evaporation rate, degree of ocular surface inflammation, and response to treatment. Some patients with predominantly tear film lipid layer dysfunction are better suited to meibomian gland-related interventions to improve excessive evaporation; while patients with significant inflammatory responses or systemic metabolic abnormalities usually require combined anti-inflammatory or metabolic regulation methods. Therefore, etiological classification and grading of dry eye syndrome are crucial prerequisites for achieving precise treatment and improving treatment outcomes.

[0003] Currently, the diagnosis of dry eye syndrome mainly relies on traditional methods, such as the Schirmer test, tear film breakup time (TBUT) measurement, and subjective questionnaire assessment (such as the OSDI questionnaire). These methods have significant limitations: First, invasive tests (such as Schirmer test strips) easily induce reflexive tearing, and the results are affected by the operator's experience and the patient's condition, resulting in poor repeatability; second, although questionnaire assessments can reflect the severity of symptoms, they lack objective quantitative indicators and cannot distinguish the etiological type.

[0004] In recent years, non-invasive detection solutions based on computer vision, such as blink frequency analysis, have improved convenience by extracting eye features through cameras. However, these technologies are mostly limited to a single visual modality, such as analyzing only blinking behavior or ocular surface structure, and it is difficult to integrate multi-source information such as tongue image features and physiological parameters.

[0005] From a physiological and pathological perspective, the condition of the ocular surface is not isolated but closely related to factors such as the body's overall metabolic level, inflammatory state, and lifestyle. For example, some obese individuals or those with metabolic abnormalities are more prone to meibomian gland dysfunction and chronic inflammation, while those with irregular work and rest schedules or high levels of mental stress are more likely to experience excessive tear evaporation. This systemic information is often difficult to reflect directly through a single ocular indicator but may be indirectly reflected through tongue features or basic physiological parameters. In non-invasive detection schemes based on computer vision, ocular features, tongue information, and physiological parameters are usually collected and analyzed separately, lacking a unified multimodal joint modeling and decision-making mechanism. This leads to insufficient information utilization and makes it difficult to provide a reliable basis for the refined classification of dry eye syndrome. Summary of the Invention

[0006] This invention provides a dry eye syndrome grading and classification monitoring system based on multimodal feature fusion, which addresses the shortcomings of existing technologies that lack a unified multimodal joint modeling and decision-making mechanism, resulting in insufficient information utilization and difficulty in providing a reliable basis for the refined classification of dry eye syndrome. It realizes joint modeling of eye, tongue, and body, improving the accuracy and stability of dry eye syndrome subtype identification; and based on preset pathological mapping rules, it provides a direct basis for personalized treatment, enhancing clinical applicability.

[0007] This invention provides a dry eye syndrome grading and classification monitoring system based on multimodal feature fusion, which includes the following modules.

[0008] The multimodal feature extraction module is used to perform semantic segmentation and feature extraction on the target user's tongue image to obtain tongue features; to enhance blood vessel texture and perform color threshold segmentation on the target user's eye region to obtain eye features; and to normalize the target user's physiological values ​​to obtain physiological metabolic features. The feature fusion module is used to concatenate and splice tongue features, eye features, and physiological metabolic features to obtain a total feature vector; The diagnostic module is used to calculate the confidence scores of different dry eye subtypes based on the total feature vector using a pre-trained classifier and preset pathological mapping rules, thereby obtaining multiple confidence scores; and to determine the diagnostic result based on the comparison of multiple confidence scores.

[0009] A dry eye syndrome grading and classification monitoring system based on multimodal feature fusion provided by the present invention further includes: The target user identification module is used to identify target users based on the user's blink frequency and average blink interval calculated in real time.

[0010] According to the present invention, a dry eye syndrome grading and classification monitoring system based on multimodal feature fusion determines target users based on real-time calculated user blink frequency and average blink interval, including: The face alignment network processes real-time captured facial images to obtain the coordinates of multiple key points; the aspect ratio of the eyes is calculated based on the coordinates of multiple key points using a preset geometric formula. Within a preset number of frames, the effective number of blinks is determined based on the comparison between the eye aspect ratio and the preset eye-closing threshold; the blink frequency and average blink interval are calculated based on the effective number of blinks within the preset number of frames. If the blinking frequency is greater than a preset frequency threshold, or the average blinking interval is less than a preset time threshold, the user corresponding to the real-time captured facial image will be identified as the target user.

[0011] According to the present invention, a dry eye syndrome grading and classification monitoring system based on multimodal feature fusion performs semantic segmentation and feature extraction on the tongue image of the target user to obtain tongue features, including: Using a pre-trained U-Net network, semantic segmentation is performed on the tongue image of the target user to obtain the region of interest of the tongue. The mean values ​​of the hue and saturation channels are calculated for the region of interest on the tongue to obtain the mean hue and saturation values. If the mean hue is within the preset red range and the mean saturation is greater than the preset saturation value, the red tongue feature is determined. Using a Gabor filter, multi-scale texture features are extracted from the region of interest on the tongue to obtain texture contrast and energy values. Based on the texture contrast and energy values, dryness-type or inflammation-type features are determined.

[0012] According to the present invention, a dry eye syndrome grading and classification monitoring system based on multimodal feature fusion is provided. The ocular features include vascular density values ​​and ocular secretion characteristics. The system performs vascular texture enhancement and color threshold segmentation on the target user's ocular region to obtain the ocular features, including: The vascular texture of the sclera region of the target user is enhanced and calculated to obtain the vascular density value; Color threshold segmentation is performed on the inner canthus region of the target user to identify bright yellowish pixel blocks; based on the area ratio of the bright yellowish pixel blocks to the region of interest in the eye, the characteristics of eye secretions are obtained.

[0013] The present invention provides a dry eye disease grading and classification monitoring system based on multimodal feature fusion. Through a pre-trained classifier, based on preset pathological mapping rules, it calculates the confidence scores for different dry eye disease subtypes according to the total feature vector, obtaining multiple confidence scores, including: Using a pre-trained classifier, based on preset pathological mapping rules and the total feature vector, the confidence scores for the dry type and the inflammation type are calculated to obtain the dry type score and the inflammation type score.

[0014] According to the present invention, a dry eye syndrome grading and classification monitoring system based on multimodal feature fusion determines the diagnostic result based on the comparison results of multiple confidence scores, including: By comparing the scores for the dryness type and the inflammation type, the dry eye subtype corresponding to the larger score is determined as the diagnostic result.

[0015] This invention provides a dry eye syndrome grading and classification monitoring system based on multimodal feature fusion. It obtains tongue features by performing semantic segmentation and feature extraction on the target user's tongue image; obtains eye features by enhancing vascular texture and performing color threshold segmentation on the target user's eye region; obtains physiological metabolic features by normalizing the target user's physiological values; and concatenates the tongue, eye, and physiological metabolic features to obtain a total feature vector. Using a pre-trained classifier based on preset pathological mapping rules, it calculates the confidence scores for different dry eye syndrome subtypes based on the total feature vector, obtaining multiple confidence scores. The diagnosis is determined based on the comparison of these multiple confidence scores. Compared to existing technologies that lack a unified multimodal joint modeling and decision-making mechanism, leading to insufficient information utilization and difficulty in providing reliable evidence for refined classification of dry eye syndrome, this invention achieves eye-tongue-body joint modeling by fusing blinking behavior, tongue visual features, and physiological metabolic indicators, improving the accuracy and stability of dry eye syndrome subtype identification. Furthermore, based on preset pathological mapping rules, it provides a direct basis for personalized treatment, enhancing clinical applicability. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of the dry eye syndrome grading and classification monitoring system based on multimodal feature fusion provided by the present invention.

[0018] Figure 2 This is a logical architecture diagram of the dry eye syndrome grading and classification monitoring system based on multimodal feature fusion provided by the present invention during monitoring.

[0019] Figure 3 This is a flowchart illustrating the process of multimodal monitoring and classification decision-making for dry eye syndrome during monitoring, as provided in a specific embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] The following is combined Figures 1 to 3 This invention is described.

[0022] Figure 1 This is a flowchart illustrating the dry eye syndrome grading and classification monitoring device based on multimodal feature fusion provided by the present invention, as shown below. Figure 1 As shown, the device includes the following: The multimodal feature extraction module 401 is used to perform semantic segmentation and feature extraction on the target user's tongue image to obtain tongue features; to perform vascular texture enhancement and color threshold segmentation on the target user's eye region to obtain eye features; and to normalize the target user's physiological values ​​to obtain physiological metabolic features.

[0023] The feature fusion module 402 is used to concatenate and splice tongue features, eye features and physiological metabolic features to obtain a total feature vector.

[0024] The diagnostic module 403 is used to calculate the confidence scores of different dry eye subtypes based on the total feature vector using a pre-trained classifier and preset pathological mapping rules, and obtain multiple confidence scores; and to determine the diagnostic result based on the comparison results of multiple confidence scores.

[0025] This invention provides a dry eye syndrome grading and classification monitoring device based on multimodal feature fusion. It obtains tongue features by performing semantic segmentation and feature extraction on the target user's tongue image; obtains eye features by enhancing vascular texture and performing color threshold segmentation on the target user's eye region; obtains physiological metabolic features by normalizing the target user's physiological values; and concatenates the tongue, eye, and physiological metabolic features to obtain a total feature vector. Using a pre-trained classifier based on preset pathological mapping rules, it calculates the confidence scores for different dry eye syndrome subtypes based on the total feature vector, obtaining multiple confidence scores. The diagnosis is determined based on the comparison of these multiple confidence scores. Compared to existing technologies that lack a unified multimodal joint modeling and decision-making mechanism, leading to insufficient information utilization and difficulty in providing reliable evidence for refined classification of dry eye syndrome, this invention achieves eye-tongue-body joint modeling by fusing blinking behavior, tongue visual features, and physiological metabolic indicators, improving the accuracy and stability of dry eye syndrome subtype identification. Furthermore, based on preset pathological mapping rules, it provides a direct basis for personalized treatment, enhancing clinical applicability.

[0026] The dry eye syndrome grading and classification monitoring system based on multimodal feature fusion provided by this invention performs the following steps during monitoring: Step 101: Perform semantic segmentation and feature extraction on the target user's tongue image to obtain tongue features; perform vascular texture enhancement and color threshold segmentation on the target user's eye region to obtain eye features; normalize the target user's physiological values ​​to obtain physiological metabolic features.

[0027] In step 101 above, the tongue features include: tongue color features and tongue texture features.

[0028] This patent application aims to address several technical problems existing in current dry eye detection and classification technologies. Existing dry eye assessment methods largely rely on tear secretion tests or questionnaires, which are somewhat invasive or subjective, making them unsuitable for home environments and long-term continuous monitoring. Furthermore, dry eye detection methods based on non-contact visual information generally suffer from insufficient feature utilization and low stability in identifying different etiologies. In addition, existing automated detection technologies primarily focus on analyzing single ocular behaviors or local signs, failing to effectively link with relevant signs reflecting the overall state of the body. The results lack clear etiological indications, making it difficult to support subsequent personalized interventions. To address these issues, this patent proposes a dry eye classification and monitoring method and system based on multimodal feature fusion. By integrating ocular behavioral features, static ocular surface signs, and physiologically relevant information, it achieves automatic identification of different etiologies of dry eye, thus integrating dry eye monitoring and classification under non-invasive conditions.

[0029] Figure 2 The logical architecture diagram of this invention is shown. The system mainly consists of four core parts: an image acquisition module, a blink dynamics analysis module, a multimodal feature extraction and fusion module, and a classification decision output module. Initial screening is achieved through the blink dynamics analysis module, and fine classification is achieved through the multimodal feature extraction and fusion module.

[0030] Optionally, the dry eye syndrome grading and classification monitoring system based on multimodal feature fusion provided by the present invention further includes: a target user determination module, used to determine the target user based on the user's blink frequency and average blink interval calculated in real time.

[0031] Optionally, the target user determination module may perform the following steps: Step A11: Process the real-time acquired facial images using a face alignment network to obtain the coordinates of multiple key points; calculate the aspect ratio of the eyes based on the coordinates of multiple key points using a preset geometric formula.

[0032] Step A12: Within a preset number of frames, determine the effective number of blinks based on the comparison between the eye aspect ratio and the preset eye-closing threshold; calculate the blink frequency and average blink interval based on the effective number of blinks within the preset number of frames.

[0033] Step A13: If the blinking frequency is greater than a preset frequency threshold, or the average blinking interval is less than a preset time threshold, the user corresponding to the real-time facial image is identified as the target user.

[0034] In steps A11 to A12 above, this stage is mainly performed by the blink dynamics analysis module. The system first controls the camera unit to capture the user's facial video stream at a frame rate of 30fps via the image acquisition module. Then, a lightweight face alignment network is used to process each frame of the image, locating the coordinates of six key points around the eyes in real time, denoted as... to .

[0035] Based on the key positioning points mentioned above, the system calculates the eye aspect ratio (EAR) using a geometric formula. The formula is as follows: ; in, The numerator represents the horizontal width of the palpebral fissure, and the denominator represents the vertical height of the palpebral fissure. To accurately capture blinking movements, the system sets a closure threshold. (Recommended value: 0.2). When continuous frame, Detected Each time a blink occurs, the system records a valid blink. Based on this, the system further calculates the user's blink rate and average blink interval. If the detected blink rate exceeds a preset frequency threshold... For example, 15 blinks per minute, or an average blink interval less than a time threshold. For example, if the test takes 4 seconds, the system will determine the user as a preliminary positive, suspected of having dry eye, and automatically trigger the subsequent detailed typing process; otherwise, it will determine the user as negative and end the test.

[0036] For users who test positive in the initial screening, the feature extraction and fusion module will perform the following steps: collect and extract deep features in three dimensions to construct a complete pathological profile.

[0037] Optionally, step 101 involves semantic segmentation and feature extraction of the target user's tongue image to obtain tongue features, including steps B1 to B3: Step B1: Using a pre-trained U-Net network, perform semantic segmentation on the target user's tongue image to obtain the region of interest for the tongue.

[0038] Step B2: Calculate the mean values ​​of the hue and saturation channels for the region of interest on the tongue to obtain the mean hue and saturation values; determine the red tongue feature if the mean hue is within the preset red range and the mean saturation is greater than the preset saturation value.

[0039] Step B3: Using a Gabor filter, multi-scale texture features are extracted from the region of interest on the tongue to obtain texture contrast and energy values; based on the texture contrast and energy values, dryness-type or inflammation-type features are determined.

[0040] Specifically, in steps B1 to B3 above, the first step is tongue image feature extraction ( The system guides the user to capture an image of their tongue, and uses a trained U-Net network to perform semantic segmentation on the image, accurately extracting the region of interest (ROI) of the tongue. Regarding color features, the system converts the segmented tongue ROI from RGB color space to HSV color space and calculates the mean value of the H channel (hue). and the mean of the S channel (saturation) .like Falling within a specific red zone and Higher contrast values ​​are extracted as a "red tongue" feature, which in traditional Chinese medicine pathology usually indicates yin deficiency and excessive fire. Regarding texture features, the system uses a Gabor filter to extract multi-scale texture features and calculates texture contrast and energy. High-contrast, low-energy features are mapped to a cracked tongue or no coating, indicating a dry type; while low-contrast, high-homogeneity features are mapped to a thick, greasy coating, indicating an inflammatory type.

[0041] Optionally, the eye features include vascular density values ​​and eye secretion features; step 101 involves enhancing the vascular texture and performing color threshold segmentation on the target user's eye region to obtain the eye features, including steps C1 to C2: Step C1: Enhance and calculate the vascular texture of the sclera region of the target user to obtain the vascular density value.

[0042] Step C2: Perform color threshold segmentation on the inner canthus region of the target user to identify bright yellow pixel blocks; obtain the characteristics of eye secretions based on the area ratio of the bright yellow pixel blocks to the region of interest in the eye.

[0043] In steps C1 to C2 above, static features of the ocular surface are extracted ( The system extracts the sclera region from the eye's ROI, applies a blood vessel segmentation algorithm (such as Frangi filtering) to enhance blood vessel texture, and calculates blood vessel density values. A higher value indicates a more severe inflammatory response. Simultaneously, the system performs color thresholding in the inner canthus region, identifying bright and yellowish pixel blocks and calculating their area proportion within the eye's region of interest (ROI). It is used to help determine whether there is an increase in secretions.

[0044] Physiological metabolic feature extraction ( The system acquires the user's physiological values ​​(such as BMI, uric acid (UA), and triglycerides (TG), performs normalization processing, and constructs a physiological feature vector. The key to this step is to introduce systemic metabolic indicators, because high BMI and abnormal blood lipids are often highly correlated with inflammatory dry eye caused by meibomian gland dysfunction.

[0045] Step 102: Concatenate and splice the tongue features, eye features, and physiological metabolic features to obtain the total feature vector.

[0046] In step 102 above, this invention employs a late fusion strategy to enhance the interpretability of the model. The system first concatenates the feature vectors from the three dimensions to form a total feature vector. .

[0047] Step 103: Using the pre-trained classifier, based on the preset pathological mapping rules, calculate the confidence scores of different dry eye subtypes according to the total feature vector to obtain multiple confidence scores; determine the diagnosis result based on the comparison results of multiple confidence scores.

[0048] Optionally, in step 103, the pre-trained classifier calculates the confidence scores for different dry eye subtypes based on preset pathological mapping rules and the total feature vector, resulting in multiple confidence scores, including step D1: Step D1: Using the pre-trained classifier, based on the preset pathological mapping rules and the total feature vector, calculate the confidence scores for the dry type and the inflammation type to obtain the dry type score and the inflammation type score.

[0049] Optionally, the diagnostic result determined in step 103 based on the comparison of multiple confidence scores includes step E1: Step E1: Compare the scores for the dryness type and the inflammation type, and determine the dry eye subtype corresponding to the larger score as the diagnostic result.

[0050] In step 103 above, Input a pre-trained classifier. The classifier calculates confidence scores for each subtype based on pre-defined pathological mapping rules: Dry type score... The main positive correlation was with high-frequency blinking fluctuations, a red tongue, and the absence of tongue coating or cracks.22 The inflammatory type score was positively correlated with these characteristics. It is mainly positively correlated with excessive eye discharge, conjunctival congestion, thick and greasy tongue coating, high BMI, and abnormal blood lipid / uric acid characteristics.

[0051] Finally, the system compares and The value of the value corresponds to the category of the highest value, which is then used as the final diagnostic result. If the result is classified as dry, it indicates that the patient primarily has a problem with excessively rapid tear evaporation; if it is classified as inflammatory, it indicates that the patient has meibomian gland dysfunction or systemic metabolic inflammation. Through the above process, this invention achieves a leap from monitoring single ocular symptoms to a holistic correlation analysis of the "eye-tongue-body" system.

[0052] The present invention will be further explained through the following specific embodiments. Specific Implementation Example 1 In one specific embodiment of the present invention, a dry eye syndrome grading and classification monitoring system based on multimodal feature fusion is provided. Figure 3 This is a schematic diagram of a multimodal monitoring and classification decision-making system for dry eye syndrome provided in an embodiment of the present invention. The system includes a facial video acquisition module, a blink dynamics analysis module, a dry eye syndrome initial screening and discrimination module, a multimodal feature acquisition module, a feature fusion and weighted discrimination module, and a classification decision output module. The system first acquires a continuous sequence of images of the user's eyes through facial video acquisition, calculates dynamic indicators such as blink frequency, and determines the risk of dry eye syndrome based on these indicators. If the initial screening result is positive, it further acquires multimodal information such as tongue features, ocular surface static features, and physiological metabolic features. Through feature fusion and weighted discrimination, it achieves refined identification of the dry eye syndrome type and finally outputs the corresponding classification decision result to provide a basis for subsequent intervention and management.

[0054] The following is combined Figure 3 Each step is explained in detail.

[0055] Facial Video Acquisition: In this embodiment, the facial video acquisition module operates based on an image acquisition unit located at the front end of the terminal device. This image acquisition unit is preferably a built-in front-facing camera or an external high-definition camera module, used to continuously acquire video sequences of the user's face under natural light or supplementary lighting conditions. The acquired video data mainly includes temporal image information of the eye area, which can fully reflect the user's blinking behavior, eyelid opening and closing state, and local eye movement characteristics during the detection process, providing basic data support for subsequent blink dynamics analysis and initial screening for dry eye syndrome.

[0056] Blinking dynamics analysis module: In this embodiment, the blinking dynamics analysis module is used to perform real-time analysis of the subject's blinking behavior on the end-side device, extract dynamic indicators such as blinking frequency and blinking rhythm, and complete the preliminary screening and judgment of dry eye syndrome based on the indicators.

[0057] This module mainly includes: an eye key point localization unit, a blink signal construction unit, a blink event determination unit, and a preliminary screening determination unit.

[0058] The blink dynamics analysis module first processes the acquired facial video sequence frame by frame, locates the facial region using a facial key point detection algorithm, and further precisely extracts the eyelid key points of the eye region. In this way, the geometric structure information of the eyes can be obtained in each video frame, providing basic data for subsequent blink amplitude and state analysis.

[0059] After obtaining the coordinates of key eye points, the system constructs a blink dynamics time series based on the eye aspect ratio (EAR). The definition is as follows: ; The EAR values ​​calculated from consecutive video frames are arranged in chronological order to form the original signal sequence of blink dynamics.

[0060] Due to factors such as changes in lighting, head movements, and image noise that may affect the video acquisition process, the original EAR time series contains jitter and high-frequency noise.

[0061] To improve the stability of blink detection, this embodiment performs smoothing processing on the EAR time series, specifically including: Continuous EAR signals are subjected to low-delay smoothing filtering to suppress burst noise; While maintaining the characteristics of blink edge changes, the amplitude of signal fluctuations is reduced.

[0062] The smoothed EAR signal is used to determine subsequent blink events.

[0063] Blinking event identification and index calculation. After obtaining the smoothed EAR time series, the blink dynamics analysis module identifies blinking events based on a state machine model. Eyelid change events with excessively short or long durations are judged as invalid events and discarded.

[0064] After identifying a valid blink event, the system collects blink dynamics indicators within a preset time window, including but not limited to: the number of blinks per unit time; the time interval between two adjacent blinks; and the stability indicators of the blink rhythm.

[0065] Initial screening for dry eye syndrome. The calculated blink frequency and blink interval are compared to preset thresholds: if the blink frequency is higher than the threshold or the blink interval is lower than the threshold, it is considered an abnormal blink, and a positive initial screening result for dry eye syndrome is output; if the blink dynamics indicators are within the normal range, it is considered a negative initial screening result, and the subsequent multimodal feature acquisition process is terminated. Through the above blink dynamics analysis module, low-computing-power, real-time initial screening for dry eye syndrome can be achieved on edge devices, providing triggering conditions for subsequent multimodal feature analysis.

[0066] Feature Extraction: After completing blink dynamics analysis and initial screening, the system enters the dry eye syndrome grading and classification detection stage. This embodiment extracts multidimensional features from blink dynamics, ocular surface static signs, tongue appearance, and physiological metabolic features based on facial video images and multimodal physiological and physical information, and then performs fusion analysis.

[0067] Tongue Feature Extraction: The system first processes the acquired tongue images, extracting the regions of interest (ROIs) through image segmentation to eliminate background interference. Then, color and texture analysis is performed within these regions. The tongue image is converted to a color space, and the color distribution is statistically analyzed to obtain the proportion and saturation of red areas, reflecting the reddish hue and color variations of the tongue. Simultaneously, multi-scale texture analysis is used to extract the structural features of the tongue coating surface, calculating texture contrast and energy indices to characterize the thickness, dryness, cracking, or greasy texture of the coating. These tongue features reflect the body's fluid status and inflammatory tendency.

[0068] Static feature extraction of the ocular surface: The system performs contrast enhancement processing on the ocular image to highlight the fine structures within the sclera. The scleral vascular region is extracted using vascular enhancement and segmentation methods, and the proportion of vascular pixels within the sclera is calculated to obtain scleral vascular density features, which reflect the degree of ocular surface inflammation. Simultaneously, color clustering analysis is performed on the ocular surface region, dividing the pixels into background, normal tissue, and secretion regions. The area proportion of the secretion region is calculated to characterize the accumulation of ocular surface secretions.

[0069] Physiological metabolic feature extraction: The system acquires basic physiological metabolic indicators related to dry eye syndrome and standardizes these indicators based on medical norms to reflect the degree to which an individual's metabolic state deviates from the normal range. The standardized metabolic features are used to help determine the inflammation-related type of dry eye syndrome.

[0070] Through the above multimodal feature extraction process, the system obtains a feature set that can comprehensively reflect local ocular signs, overall condition and related pathological features, providing a reliable feature basis for the classification and grading of dry eye syndrome.

[0071] Multimodal fusion and classification decision: In this embodiment, the multimodal fusion and classification decision module is used to perform unified modeling and comprehensive discrimination on the multi-source features extracted by module 2, solve the classification problem of dry eye syndrome between dry and inflammatory types, and output interpretable classification results.

[0072] In the feature fusion stage, the system adopts a feature-level fusion approach to unify and integrate features from different modalities. Specifically, blink dynamics features, tongue image features, ocular surface static features, and physiological metabolic features are concatenated in a preset order to construct a total feature vector.

[0073] ; In the classification decision stage, the system inputs the fused feature vector into the classification decision model for classification judgment. The classification decision model employs a tree-based ensemble learning method, which has excellent modeling capabilities for structured feature data. This model uses a combination of multiple decision trees to non-linearly partition different feature dimensions, thereby achieving the differentiation of dry eye syndrome types.

[0074] During model training, the classification decision model is configured with multi-class probability output, capable of simultaneously outputting the probability values ​​of a subject belonging to normal, dry-type, and inflammatory-type dry eye. Classification error is used as the optimization objective during model training to achieve a balance between overall classification accuracy and stability.

[0075] To enhance the interpretability of diagnostic results, the classification decision model, while outputting classification results, can also analyze the contribution of each feature in the decision-making process, thereby identifying key features that significantly influence the current classification result. In this way, the system can clearly identify the main symptom factors leading to the classification result, providing a reference for medical auxiliary diagnosis.

[0076] During the decision output phase, the system sets confidence thresholds based on the probability values ​​of each category output by the model. When the predicted probabilities of the dry or inflammatory type are both lower than the preset confidence thresholds, the system outputs a mixed type or a judgment result requiring manual review; when the predicted probability of a certain type is higher than the confidence threshold, the system outputs the dry eye type with the highest probability value as the final classification result. Specific Implementation Example 2 This specific embodiment provides a multimodal feature fusion-based dry eye syndrome grading and classification monitoring system applicable to hospital ophthalmology waiting areas. Compared to Specific Embodiment 1, in the initial detection stage, this embodiment allows patients to scan the barcode on their registration slip, and the system automatically retrieves recent blood test reports from the patient's electronic medical record, directly obtaining accurate TG, UA, and blood glucose values ​​without manual input. Simultaneously, the system is integrated with the hospital's information system, allowing nurses to use a handheld tablet with a simple slit-lamp lens to capture images of the patient's cornea and sclera. The results are then directly pushed to the doctor's computer screen, thereby improving the accuracy of information collection and the clarity of feature extraction. This system is suitable for professional-level pre-diagnosis screening in high-traffic medical settings.

[0078] like Figure 3 As shown, the system in this embodiment also includes a facial video acquisition module, a blink dynamics analysis module, a dry eye syndrome screening and discrimination module, a multimodal feature acquisition module, a feature fusion and weight discrimination module, and a classification decision output module, but it expands the information acquisition and data interaction by adding a hospital information system integration and data interaction module.

[0079] 1) Patient Identity Binding and Data Acquisition: This example uses a terminal to scan the barcode on the patient's registration slip or the QR code on the electronic medical insurance voucher to extract the patient's unique identifier. The system initiates a query to the laboratory information system or electronic medical record database through a standard medical data interface, requesting the patient's recently specified biochemical indicators. The query request includes the patient's identifier, authorization token, and the code of the required test item. The system receives the structured JSON format data, automatically parses it, and extracts the values. Subsequently, according to the Z-Score standardization formula defined in the multimodal fusion and classification decision module, the system uses pre-set medical norms (μ_medical, σ_medical) to calculate indicators such as TG and BMI in real time, generating standardized physiological metabolic feature vectors Z_BMI and Z_TG, which are directly injected into the multimodal feature fusion process, completely replacing manual data entry.

[0080] 2) Information Collection and Multimodal Analysis: After patient identification and automatic acquisition of test data are completed, facial video acquisition is initiated. The user's face is continuously captured by the terminal's front-facing camera under natural light conditions, obtaining a video sequence containing temporal dynamics of the eyes, providing basic data for subsequent analysis. Subsequently, the blink dynamics analysis module processes the video in real time on the device: First, the eyes are located through facial key point detection, and the eye aspect ratio (EAR) of each frame is calculated to construct the original blink signal; then, the EAR signal is smoothed and filtered to suppress noise; next, a state machine model is used to identify valid blink events and calculate dynamic indicators such as blink frequency and interval; finally, if the indicators exceed the normal threshold (e.g., blink frequency > 15 times / minute or average interval < 4 seconds), it is judged as a positive initial screening result, automatically triggering the subsequent detailed examination process.

[0081] If the initial screening is positive, the system proceeds to the multimodal feature extraction stage. This stage involves three simultaneous analyses: tongue feature extraction, which segments the tongue image, analyzes the proportion and saturation of the red area in the HSV color space to assess the tendency of "yin deficiency and excessive fire," and uses a multi-scale Gabor filter to analyze texture contrast and energy to determine whether the tongue coating is thick and greasy or dry and cracked; ocular surface static feature extraction, which enhances and segments the blood vessels in high-resolution ocular images, calculates scleral vessel density to assess the degree of inflammation, and quantifies the proportion of secretions through LAB color space clustering; simultaneously, physiological indicators (such as TG and BMI) previously obtained from the hospital information system have been Z-score standardized according to medical norms to generate metabolic feature vectors.

[0082] Once all features are ready, the system concatenates the feature vectors from four dimensions—blink dynamics, tongue appearance, ocular surface, and metabolism—to form a total feature vector, which is then input into a pre-trained XGBoost classification model. The model outputs the probability distributions for "normal," "dry," and "inflammatory" types and makes a final decision based on a pre-set confidence threshold: if the probability of a certain type exceeds the threshold, the classification result and key feature basis are directly output; otherwise, a suggestion of "mixed type / requires manual review" is output.

[0083] 3) System-to-HIS / EMR Bidirectional Interface: This example uses an HTTPS-based RESTful API as the primary interaction method. Simultaneously, to address potential traditional systems within the hospital, a message interface based on the HL7 V2.x standard is configured in parallel, with message conversion handled by the hospital information platform. After analysis, the system encapsulates the structured pre-screening results and writes them back to the designated module of the EMR system via a secure interface.

[0084] 4) Results Push and Integration with Clinical Workstations: This example uses server push event technology to establish a persistent connection between the doctor's workstation and the system's server. When a patient's pre-examination analysis is completed, the server proactively pushes a notification message to the patient's attending physician or the workstation in the current consultation room. A new "Dry Eye Pre-examination AI Report" tab is added to the patient details page of the doctor's workstation software. The results display view of this system is directly embedded in the front-end component, allowing doctors to view it directly within the system without needing to navigate.

[0085] This embodiment improves the convenience and accuracy of professional-level dry eye screening in medical settings by combining data interaction and multimodal analysis. It is applicable to hospital ophthalmology waiting areas and helps doctors complete the basic classification of patients before face-to-face consultation, thereby improving consultation efficiency.

[0086] This invention establishes a multimodal feature fusion-based system for precise monitoring of dry eye syndrome classification and grading, offering significant advantages over existing technologies. The invention employs a cascaded, non-invasive detection method combining initial screening based on blink dynamics with refined multimodal classification to monitor dry eye syndrome subtypes. This avoids the eye irritation, reflexive tearing, and subjective diagnostic bias associated with traditional Schirmer test strips. Furthermore, the convenience of the detection significantly reduces the time users spend deciding whether to seek medical attention after experiencing symptoms, effectively lowering the risk of missed diagnoses and misdiagnoses associated with single-modal detection, as well as reducing the computational burden of continuous monitoring. By integrating multimodal physiological features such as blink temporal behavioral characteristics, tongue visual features, ocular surface static features, and physiological metabolic indicators, and introducing a post-fusion joint decision-making model, adaptive weighting of multi-source complementary information is achieved, improving the accuracy and stability of dry eye syndrome subtype identification. Finally, through digital quantitative modeling of TCM tongue diagnosis features, precise etiological feature mapping rules, and hierarchical processing optimization of the cascaded architecture, the overall personalized treatment intervention effect and system reliability are enhanced.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dry eye grading and subtyping monitoring system based on multi-modal feature fusion, characterized in that, include: The multimodal feature extraction module is used to perform semantic segmentation and feature extraction on the target user's tongue image to obtain tongue features; The target user's eye region is subjected to vascular texture enhancement and color threshold segmentation to obtain eye features; The physiological values ​​of the target user are normalized to obtain physiological metabolic characteristics. The feature fusion module is used to concatenate and splice the tongue features, eye features, and physiological metabolic features to obtain a total feature vector; The diagnostic module is used to calculate the confidence scores of different dry eye subtypes based on the total feature vector using a pre-trained classifier and preset pathological mapping rules, thereby obtaining multiple confidence scores; and to determine the diagnostic result based on the comparison results of the multiple confidence scores.

2. The dry eye grading and typing monitoring system based on multi-modal feature fusion according to claim 1, characterized in that, Also includes: The target user identification module is used to identify target users based on the user's blink frequency and average blink interval calculated in real time.

3. The dry eye syndrome grading and classification monitoring system based on multimodal feature fusion according to claim 2, characterized in that, The process of determining the target user based on the user's blink frequency and average blink interval calculated in real time includes: The face alignment network processes real-time captured facial images to obtain the coordinates of multiple key points; the aspect ratio of the eyes is calculated based on the coordinates of multiple key points using a preset geometric formula. Within a preset number of frames, the effective number of blinks is determined based on the comparison between the eye aspect ratio and the preset eye-closing threshold; the blink frequency and average blink interval are calculated based on the effective number of blinks within the preset number of frames. If the blinking frequency is greater than a preset frequency threshold, or the average blinking interval is less than a preset time threshold, the user corresponding to the real-time captured facial image will be identified as the target user. 4.The dry eye grading and subtyping monitoring system based on multi-modal feature fusion according to claim 1, characterized in that, The step of performing semantic segmentation and feature extraction on the tongue image of the target user to obtain tongue features includes: Using a pre-trained U-Net network, semantic segmentation is performed on the tongue image of the target user to obtain the region of interest of the tongue. The mean values ​​of the hue channel and saturation channel are calculated for the region of interest of the tongue to obtain the mean hue value and the mean saturation value; when the mean hue value is within a preset red range and the mean saturation value is greater than a preset saturation value, the red tongue feature is determined; Using a Gabor filter, multi-scale texture features are extracted from the region of interest of the tongue to obtain texture contrast and energy values; based on the texture contrast and energy values, dryness-type or inflammation-type features are determined. 5.The dry eye grading and subtyping monitoring system based on multi-modal feature fusion according to claim 1, characterized in that, The eye features include vascular density values ​​and eye secretion characteristics; the process of enhancing vascular texture and performing color threshold segmentation on the target user's eye region to obtain eye features includes: The vascular texture of the sclera region of the target user is enhanced and calculated to obtain the vascular density value; Color threshold segmentation is performed on the inner canthus region of the target user to identify bright yellowish pixel blocks; based on the area ratio of the bright yellowish pixel blocks to the region of interest in the eye, the characteristics of eye secretions are obtained. 6.The dry eye grading and subtyping monitoring system based on multi-modal feature fusion according to claim 1, characterized in that, The pre-trained classifier, based on preset pathological mapping rules, calculates confidence scores for different dry eye subtypes according to the total feature vector, resulting in multiple confidence scores, including: Using a pre-trained classifier, based on preset pathological mapping rules and the total feature vector, the confidence scores for the dry type and the inflammation type are calculated to obtain the dry type score and the inflammation type score. 7.The dry eye grading and subtyping monitoring system based on multi-modal feature fusion according to claim 1, characterized in that, The process of determining the diagnostic result based on the comparison of multiple confidence scores includes: By comparing the scores for the dryness type and the inflammation type, the dry eye subtype corresponding to the larger score is determined as the diagnostic result.

Images