An eye sign feature extraction and analysis system based on multi-modal external eye image and clinical text

Abstract

This invention belongs to the field of medical technology and discloses a system for extracting and analyzing ocular signs in orbital diseases based on multimodal external ocular images and clinical text. By constructing a multimodal model-based system for extracting and analyzing external ocular signs, this invention upgrades the technology from single-view analysis to collaborative analysis of "multi-view images—quantified signs—clinical text." This invention overcomes the reliance on single-modal or single-view images in existing technologies by performing cross-modal alignment and correlation analysis on standardized multi-view external ocular images, key quantitative parameters of external ocular signs, and clinical text information, thereby achieving systematic modeling of orbital-related external ocular signs. This approach better aligns with the actual process of clinicians comprehensively evaluating multi-source information during visual and oral examinations, helping to improve the completeness and stability of the analysis of complex or early atypical external ocular manifestations.

Description

Technical Field

[0001] This invention belongs to the field of medical technology, and in particular relates to a system for extracting and analyzing ocular signs and characteristics of orbital diseases based on multimodal external eye images and clinical text. Background Technology

[0002] Orbital diseases (including inflammatory, neoplastic, and traumatic orbital diseases) are blinding and disabling diseases that seriously impair visual function and facial appearance. In clinical practice, early identification and systematic evaluation of orbital-related external ocular abnormalities are crucial for their early diagnosis and accurate classification.

[0003] External eye assessment for orbital diseases requires comprehensive, multi-dimensional information, including multi-angle external eye inspection, quantitative measurement of eyelid and eyeball position, objective observation of eye movement range and symmetry, and overall analysis combined with the patient's chief complaint and medical history. However, current clinical practice faces the following core challenges: First, early signs of orbital diseases are often subtle and atypical, requiring the experience of senior experts for identification and differentiation, which is difficult to popularize in grassroots and resource-limited areas; second, key signs of orbital diseases (such as exophthalmos, eyelid retraction, conjunctival edema, and eye movement disorders) often overlap with different diseases, making reliable differentiation difficult based solely on visual observation or two-dimensional images.

[0004] Existing technical solutions also have obvious limitations: most rely on subjective human assessment and lack systematic feature modeling and multi-view information fusion for orbit-related signs; most current artificial intelligence models are still based on single-modal design and fail to effectively integrate patients' multi-angle external eye images with key text information such as chief complaint and present medical history, resulting in the system lacking deep semantic understanding of the condition and medical logical reasoning ability, and insufficient generalization and clinical applicability.

[0005] Therefore, there is an urgent need for a feature extraction and analysis system that can standardize the processing of multi-view external eye images, automatically identify and quantify key ocular signs, locate abnormal areas, and perform correlation analysis and interpretation with clinical text information to improve the objectivity, automation, and interpretability of early identification and assessment of orbital diseases.

[0006] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0007] Most current artificial intelligence models rely on subjective human assessment and lack systematic feature modeling and multi-view information fusion for orbit-related signs. Most current artificial intelligence models are still based on single-modal design and fail to effectively integrate patients' multi-angle external eye images with key text information such as chief complaint and present medical history. This results in the system lacking deep semantic understanding of the condition and medical logical reasoning ability, and has insufficient generalization and clinical applicability. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a system for extracting and analyzing ocular signs and characteristics of orbital diseases based on multimodal external eye images and clinical text.

[0009] This invention is implemented as follows: A system for extracting and analyzing ocular signs and characteristics of orbital diseases based on multimodal external eye images and clinical text includes:

[0010] Data acquisition module, input module, standardization module, recognition module, detection module, structuring module, analysis module, and report generation module;

[0011] The data acquisition module, connected to the input module, is used for acquiring multimodal information and images;

[0012] The input module, connected to the standardized module, is used for dynamic input mechanisms;

[0013] The standardization module, connected to the input module, recognition module, detection module, structuring module, analysis module, and report generation module, is used for the standardization of the external eye region.

[0014] The recognition module, connected to the standardization module, is used for the recognition of ten external ocular signs.

[0015] The detection module, connected to the standardization module, is used for the automatic localization and detection of abnormal areas in the eye.

[0016] The structured module, connected to the standardized module, is used for structuring clinical text;

[0017] The analysis module, connected to the standardization module, is used for multimodal feature alignment and correlation analysis.

[0018] The report generation module, connected to the standardization module, is used to generate interpretable multimodal feature reports.

[0019] Furthermore, the data acquisition module:

[0020] Input includes:

[0021] Multi-angle 2D external eye photographs (up to 16 images)

[0022] Clinical textual information (chief complaint, medical history, self-reported signs, etc.)

[0023] Among them, the requirements for acquiring 16 standardized external eye images are as follows: Figure 1 The details are as follows:

[0024] ① Images from nine angles: front view and human eye. When shooting from the lower right, lower left, and lower left angles, the eyelids need to be pulled open to ensure the eyeballs are captured well.

[0025] ② Images viewed from 45° and 90° to the left and right respectively;

[0026] ③ Frontal closed-eye image and frontal bottom view image

[0027] ④ Image from an upward position;

[0028] The shooting logic follows these rules:

[0029] The photos were taken in three groups based on whether the eyes were moving or closed:

[0030] ① The first group consists of 5 images acquired simultaneously at an upward, 45° and 90° angles to the left and right, without eye movement.

[0031] ②The second set consists of two images: a frontal view, a view with eyes closed, and a bottom view.

[0032] ③ The third group consists of nine images from the frontal view, showing nine different eye positions.

[0033] Furthermore, the input module:

[0034] Considering that patients may not be able to acquire 16 images at once, this invention adopts a dynamic input priority strategy;

[0035] (1) Priority View Scheduler

[0036] The system prioritizes the contributions of different perspectives to sign recognition and disease classification:

[0037] Priority 1 (Required): Frontal view, used for baseline pupillary distance, palpebral fissure measurement, overall ocular symmetry analysis, and comprehensive assessment of eyelid position, eyelid shape, ocular surface morphology, and ocular position characteristics.

[0038] Priority 2: 45° / 90° to the left and right sides, used for estimating exophthalmos, assessing eye position, and identifying lateral scleral exposure.

[0039] Priority 3: Nine-view eye position images, used for the identification of limited eye movement and strabismus.

[0040] Priority 4: Eyes closed, looking down, and tilting back, used for identifying abnormal eyelid movements such as incomplete eyelid closure and delayed eyelid drooping.

[0041] If a viewpoint is missing, the scheduler automatically records the missing viewpoint ID, adjusts the combination of available viewpoints, and provides the corresponding modal confidence.

[0042] (2) Missing-Modality Mask (MMM)

[0043] For each expected input image, the system generates a binary mask:

[0044] 1: This angle image exists.

[0045] 0: Missing

[0046] This mask is input into the multimodal large model along with the image features, enabling the MLLM to "sense" the missing modalities and dynamically adjust the inference path;

[0047] (3) Construction of View-wise Feature Tensor

[0048] Each image is processed by a visual encoder, which outputs a feature tensor Fi ∈ RC×H×W; (Fi: the feature tensor output by the visual encoder for the i-th viewpoint image).

[0049] The system stitches together features from multiple perspectives into:

[0050] (F: Overall features after multi-view feature stitching)

[0051] Missing modal positions are filled with zero vectors and subsequent attention weights are controlled by MMM masks;

[0052] (4) Dynamic View Re-weighting (DVR)

[0053] The system learns a trainable weight vector:

[0054] w = {w1, w2, ..., w16} (wi: weight of the i-th viewpoint)

[0055] If the viewpoint is missing, then: wi=0, Fi=0

[0056] If a viewpoint exists, the system automatically assigns different weights based on the correlation between the viewpoint and the vital signs, for example:

[0057] The left and right lateral perspectives increase the weight of the "protruding eye" symptom.

[0058] The downward viewing angle increases the weight of "eyelid retardation".

[0059] 5) Uncertainty range estimation

[0060] The system automatically calculates confidence intervals based on: the number of missing views, weight recalibration values, and model output variance.

[0061] (y^: model prediction of a certain trait (or quantified value of the trait); Var(y^): prediction variance; CI: confidence interval, CI=[y^-δ, y^+δ]; δ: half-width of the interval calculated from the number of missing views, output variance, etc.)

[0062] The more missing perspectives, the wider the confidence interval.

[0063] Furthermore, the standardized module:

[0064] In the aforementioned 16 images, the system automatically performs facial landmark detection, binocular region cropping, horizontal / pose calibration, illumination correction, angle-by-angle resampling, periorbital segmentation, and structured measurement of the eye fissure, providing highly standardized input for ten vital sign recognitions. Specifically, it utilizes real-time facial landmark detection algorithms based on convolutional neural networks (CNNs) (such as Face++ or Dlib in OpenCV) to identify the accurate locations of key regions such as the eyes, eyelids, and periorbital area; and performs three-dimensional facial pose estimation (Head Pose)... Estimation is performed to obtain (roll, pitch, yaw) and perform roll rotation correction, orbital region alignment to a standard template, and illumination normalization (Retinex or Gamma correction). Multi-angle image standardization resampling: For 16 images from different viewpoints, geometric transformations are used to align them to a uniform scale, constructing a multi-view consistency dataset to ensure the comparability of the same feature under different viewpoints. On the aligned images, the system automatically performs periorbital region segmentation and quantifies a series of basic anatomical parameters, including palpebral fissure height and width, pupil diameter, and corneal light-reflecting point position, providing standardized input and reference benchmarks for subsequent feature recognition.

[0065] Furthermore, the identification module:

[0066] The system includes a network for recognizing ten external ocular signs, specifically: proptosis, sunken eyeball, strabismus, conjunctival congestion, conjunctival edema, eyelid congestion, eyelid retraction, eyelid edema, ptosis, and caruncle swelling.

[0067] The ten vital signs recognition adopts a multi-task learning structure (MTL), sharing the image backbone, but setting up an independent classification / regression head for each vital sign.

[0068] (1) Vital signs network structure

[0069] F = Backbone(Ii) (Ii: the i-th input external eye image)

[0070] zk=Headk(F) (zk: model output for the k-th individual feature task; Head / Head_k: task head / k-th individual feature task head, used for feature-specific prediction of shared features.)

[0071] Vital signs output include: presence (sigmoid output); left / right eye; proptosis is continuously quantified (mm) and estimated when there is an input photo with a sufficient angle; ptosis is automatically measured and graded as MRD1 (distance from the upper eyelid margin to the corneal reflective point): mild: 1–2 mm; moderate: 3–4 mm; severe: ≥4 mm; and restricted eye movement is output along with the restricted direction (e.g., restricted upward gaze).

[0072] (2) Construction of trait vectors

[0073] The ten vital signs are combined into a structured vector:

[0074] T=[t1,t2,...,t10] (t k : The first in the structured feature vector (Code results of the physical characteristics)

[0075] Furthermore, the detection module:

[0076] Based on the aforementioned image feature and vital sign feature extraction results, the system uses a target detection network to locate local regions in the image that are related to specific vital signs; the target detection network can be a detector based on the anchor box mechanism or a detector based on key point regression; for each detected region, its bounding box coordinates and the vital sign type identifier associated with that region are output;

[0077] Visual saliency heatmap generation: Using techniques such as class activation mapping (e.g., Grad-CAM), the model is calculated to determine the degree of attention paid to different pixel regions during the extraction of vital signs, and a pixel-level weight distribution map is generated; the heatmap is used to indicate pixel regions in the image that are highly correlated with specific vital signs.

[0078] By overlaying the heatmap with the original image or detection box results, the spatial distribution of vital signs-related areas in the image can be presented intuitively, providing a basis for subsequent manual analysis or system verification.

[0079] The structured module:

[0080] Medical information is extracted from the input text, including: chief complaint (bulging eyes, diplopia, decreased vision, etc.), present medical history (cause, disease progression, accompanying symptoms, etc.), comorbid systemic diseases (such as thyroid disease), treatment history, trauma history, etc. Key diagnostic clues are extracted from the natural language description, and medical semantic embedding Etext is generated using medical-specific LLM (such as Med-BERT, Clinical LLaMA) for alignment and fusion with image features.

[0081] The analysis module:

[0082] By using a visual encoder, a stereo encoder, and a text encoder for cross-modal alignment, the model can achieve joint understanding of images and text across modalities, and deep semantic alignment and correlation analysis between abnormal regions and text information.

[0083] (1) Cross-modal alignment

[0084] The inputs include: image features F, feature vector T, text embedding Etext, and view mask MMM. Cross-modal attention is achieved through a cross-modal transformer.

[0085] H = CrossAttn(F, T, Etext, MMM) (H: Latent variable representation after cross-modal fusion; CrossAttn / Cross-modal transformer: Cross-modal attention module)

[0086] (2) Association analysis and evidence chain construction:

[0087] After completing cross-modal alignment, the system further analyzes and establishes the correspondence between the following information: the sign-related regions detected in the image and their spatial localization results, the corresponding quantitative parameters or status identifiers in the sign feature vectors, the symptom descriptions related to the above signs in the clinical text, and time information or degree descriptions.

[0088] For example, the system can semantically match the locations of significant changes in the conjunctival region in the image, the corresponding conjunctival state parameters in the sign feature vector, with descriptions such as "red eyes" and "congestion" in the text, and calculate the consistency index among the three.

[0089] Ultimately, the system constructs a structured multimodal chain of evidence to clearly demonstrate the correspondence and strength of association between image evidence, physical evidence, and textual evidence.

[0090] The report generation module:

[0091] This module integrates all the above processing and analysis results, automatically generating a well-structured, evidence-related, and interpretable feature analysis report; the report content includes:

[0092] ①Standardized Images and Basic Structural Information: Displays standardized key view images and calculated basic anatomical parameters;

[0093] ② Quantitative results of vital signs: The identification results, quantitative values, and status indicators of ten key vital signs are clearly listed in tabular form;

[0094] ③ Abnormal area visualization: Overlay detection boxes or region masks of vital signs-related areas onto the original or standardized image, and simultaneously display the corresponding visual saliency heatmap to intuitively present the spatial distribution of vital signs-related areas;

[0095] ④ Multimodal Association Evidence Chain: Presents the association analysis results between image features, vital signs parameters and text information in a structured and visual manner, illustrating the correspondence and consistency between intermediate evidence from different modalities.

[0096] Another objective of this invention is to provide a method for extracting and analyzing ocular signs and characteristics of orbital diseases based on multimodal external eye images and clinical text, including:

[0097] Step 1: Acquire multimodal information and images through the data acquisition module;

[0098] Step 2: Implement a dynamic input mechanism through the input module;

[0099] Step 3: Standardize the external eye region using the standardization module;

[0100] Step 4: Identify ten external ocular signs using the recognition module;

[0101] Step 5: Automatically locate and detect abnormal areas of the eye using the detection module;

[0102] Step 6: Structuralize clinical texts using the structured module;

[0103] Step 7: Perform multimodal feature alignment and correlation analysis using the analysis module;

[0104] Step 8: Generate an interpretable multimodal feature report using the report generation module.

[0105] Another object of the present invention is to provide a computer device, the computer device including a memory and a processor, the memory storing a computer program, the computer program being executed by the processor causing the processor to perform the steps of the method for extracting and analyzing ocular signs and features of orbital diseases based on multimodal external eye images and clinical text.

[0106] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method for extracting and analyzing ocular signs and features of orbital diseases based on multimodal external eye images and clinical text.

[0107] Another objective of this invention is to provide an information data processing terminal for implementing the orbital disease ocular sign feature extraction and analysis system based on multimodal external eye images and clinical text.

[0108] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0109] This invention, by constructing a multimodal model-based system for extracting and analyzing external ocular features, achieves a technological upgrade from single-view analysis to collaborative analysis of "multi-view images—quantified signs—clinical text." Its beneficial effects are mainly reflected in the following three aspects:

[0110] 1. Improved ability and interpretability in analyzing orbital-related signs

[0111] (1) Collaborative modeling and analysis of multimodal physical characteristics

[0112] This invention breaks through the reliance on single-modality or single-viewpoint images in existing technologies. Targeting the characteristics of orbital external ocular signs, such as "viewpoint sensitivity, diverse manifestations, early atypicality, and significant influence from subjective experience," it adopts standardized multi-view external ocular images as the input basis and performs joint modeling and cross-modal alignment of image information with key sign quantification parameters and clinical text information, thereby achieving systematic modeling and comprehensive analysis of orbital-related external ocular signs.

[0113] Specifically, this invention, on the one hand, employs a multi-view acquisition / standardization strategy to ensure that the external ocular manifestations of the same subject are fully recorded from multiple observation angles, avoiding omissions or biases caused by relying solely on a single frontal image; on the other hand, it uses a multi-task learning structure to identify and quantify multiple signs in parallel, outputting information including the presence or absence of the sign, side (left / right eye or both eyes), key quantitative indicators (such as continuous numerical estimation of exophthalmos, automatic measurement of ptosis MRD-1, etc.), and necessary direction / location information (such as the location of restricted eye movement). Simultaneously, the system performs structured extraction of symptom descriptions, key medical history points, and examination records from clinical texts, performing semantic alignment and consistency analysis with the sign output from the image side, thereby establishing an interpretable association between the three types of information: "image appearance—quantified signs—text description."

[0114] This approach aligns more closely with the actual process of clinicians comprehensively evaluating multi-source information during visual and oral examinations: doctors typically combine observations of appearance from different angles, quantifiable examination indicators, and patient complaints / medical history descriptions to make judgments. This invention solidifies this process into an algorithm and system, achieving a more complete representation of complex or early atypical external ocular manifestations, reducing random fluctuations caused by single perspectives or single modalities, and helping to improve the stability and consistency of vital sign analysis results.

[0115] (2) Provide transparent and traceable intermediate results of vital sign analysis

[0116] Unlike existing technologies that only output uninterpretable single classification conclusions, the system output of this invention is geared towards clinical review needs, forming a traceable chain of results consisting of "structured vital signs parameters - abnormal area localization - multimodal association evidence", which significantly improves interpretability and verifiability.

[0117] Specifically, the intermediate results output by the system should include at least:

[0118] ①Structured vital sign parameters: The presence probability, lateral information, quantification value (mm), grading results, and necessary orientation / location labels for each vital sign are recorded in the form of structured vectors;

[0119] ② Abnormal area location information: Provides location results such as detection boxes / key points / region masks for local areas related to vital signs, so that doctors can intuitively see the "area of ​​interest" of the model;

[0120] ③ Visual evidence visualization: Output visual saliency heatmap or attention weight map to show key image evidence that affects the judgment of physical signs;

[0121] ④ Cross-modal association evidence: Align entities / phrases related to signs in clinical text (such as "ptosis", "diplopia", "pain", "swelling", etc.) with the sign output on the image side, and give an association consistency score or association relationship description.

[0122] By simultaneously providing quantitative results of vital signs, image detection boxes, and heat maps, and combining them with correlation analysis based on text descriptions, this invention can clearly demonstrate the basis and corresponding relationship for "the model's judgment of this vital sign," enabling doctors to quickly verify key evidence when using it, reducing the black box risk of "only providing conclusions without knowing the basis," and thus making it more suitable for scenarios with higher interpretability requirements, such as screening, follow-up, consultation, and remote assessment.

[0123] 2. Improved efficiency and accessibility of vital sign assessment

[0124] (1) Reduce reliance on specialized resources and improve accessibility of physical examination.

[0125] In current assessments of orbital disease signs, some key indicators often rely on the experience of specialist physicians, specialized measuring instruments, or high-cost imaging equipment, making standardized assessments difficult to conduct in grassroots settings or resource-constrained environments. This invention utilizes algorithmic identification and quantification of external ocular signs in images obtainable from ordinary imaging equipment to automatically extract, quantify, and output structured orbit-related external ocular signs.

[0126] Specifically, this invention requires no expensive imaging equipment or complex operations; it only requires conventionally taken external eye photographs for vital sign analysis. The system can automatically generate key quantitative results such as exophthalmos degree and palpebral fissure-related indicators (such as MRD-1) and output structured reports. This reduces reliance on high-level specialist resources, minimizes subjective differences and repetitive work caused by manual measurements, and enables primary care institutions to obtain relatively objective and comparable vital sign data, providing a low-cost technical means for hierarchical diagnosis and treatment, initial screening and referral, and long-term follow-up.

[0127] (2) Supports flexible and robust multi-perspective analysis workflow

[0128] In real-world scenarios such as physical examinations, follow-up records, or remote data acquisition, issues such as missing viewpoints, inconsistent image quality, and head tilt often arise. Traditional methods typically require complete and rigorous input; otherwise, stable results are difficult to achieve. This invention introduces a dynamic viewpoint scheduling mechanism and a missing modality labeling strategy, enabling the system to continue operating even with partially missing viewpoints, significantly improving scenario adaptability and engineering usability.

[0129] Specifically, the system can identify the current set of input viewpoints, mark missing viewpoints, and dynamically adjust the contribution weights of different viewpoints during the model fusion stage. When key viewpoints are insufficient to support certain tasks requiring continuous quantification of angle information (such as precise quantification of bulging), the system can correspondingly reduce the output confidence level of that item or prompt "increased uncertainty / suggestion to reshoot viewpoints," avoiding the output of misleading "pseudo-precise" results when information is insufficient. This mechanism enhances the system's robustness under various acquisition conditions, enabling it to adapt to high-missing and high-noise scenarios such as primary care physical examinations, follow-up records, and remote consultations, thereby improving overall application accessibility and clinical usability.

[0130] 3. The advanced nature and scalability of the technical framework

[0131] (1) Longitudinal quantitative and comparative analysis of orbital disease-related signs

[0132] Patients with orbital diseases often require long-term follow-up and efficacy evaluation, with particular clinical focus on the trends of physical signs over time and differences before and after treatment. Existing methods relying on subjective descriptions or non-standardized records struggle to achieve rigorous comparisons. This invention provides a unified data foundation for longitudinal comparisons by standardizing, quantifying, and structuring the output of multiple key external ocular signs.

[0133] Specifically, the system encodes the multi-view images and clinical text information collected each time into comparable sign vectors and structured reports, enabling the same subject to record and compare indicators such as exophthalmos, MRD-1, and direction of movement restriction at different time points on a consistent scale. This can be directly used for follow-up trend analysis, efficacy evaluation, auxiliary judgment of disease stage, and objective comparison of external ocular manifestations at different stages, thereby improving the standardization and repeatability of follow-up records.

[0134] (2) Possesses good potential for disease expansion and platformization.

[0135] The multimodal feature extraction and correlation analysis framework employed in this invention is universal: its core lies in aligning "appearance image evidence—quantified physical signs parameters—clinical text description" with a unified structure and outputting the evidence chain. This framework is not only applicable to the analysis of external ocular signs in orbital diseases, but can also be transferred to other oculofacial related issues that rely on appearance and text description for assessment.

[0136] For example, for other ocular surface, eyelid, or facial appearance features (such as abnormal eyelid margins, changes in palpebral fissure morphology, conjunctival hyperemia / edema, and abnormal caruncle), the system can reuse the overall architecture of standardized acquisition, MTL feature modeling, cross-modal alignment, and interpretable evidence chain output. Expansion can be completed simply by extending the feature task head or updating the feature definition, thus enabling platform-based deployment and continuous iteration capabilities, laying the technical foundation for building a scalable ocular and facial feature analysis platform.

[0137] This invention addresses the assessment needs of external ocular signs in orbital diseases and related oculofacial diseases. It standardizes, structures, and automates the "visual diagnosis + consultation" process, which is highly dependent on the experience of specialists in clinical practice. It enables cross-modal alignment, joint reasoning, and evidence-based output of multi-view images, quantitative signs, and clinical text information, and has a clear path to productization and platformization.

[0138] In terms of expected benefits and commercial value, this invention can complete the quantification of vital signs and the output of structured reports using only conventional external eye imaging equipment, significantly reducing reliance on expensive imaging equipment such as CT / MRI and manual measurements, shortening assessment time and reducing the labor costs of repeated examinations and follow-up records. The system has dynamic perspective scheduling and robust reasoning capabilities for missing perspectives, which can adapt to real-world scenarios such as primary care clinics, physical examinations, and remote follow-ups where "perspectives are incomplete and acquisition conditions are inconsistent," which is conducive to the standardization of assessment capabilities and the expansion of service coverage. At the same time, it outputs unified structured vital sign vectors and traceable evidence chains, which can be directly used for follow-up trend analysis, efficacy evaluation, and clinical research data accumulation, with continuous data value-added potential, and provides a foundation for compliant, interpretable, and auditable clinical applications.

[0139] In terms of technological innovation, existing solutions are mostly single-view image classification or single-sign detection, or only use text as weak cue information. They lack a complete system for unified representation, cross-modal alignment, and joint reasoning of multi-view images, quantifiable sign indicators, and clinical text. This invention is the first to achieve collaborative modeling and evidence chain output of three types of information in the same system, so that the results not only provide conclusions, but also provide verifiable multimodal evidence.

[0140] In terms of overcoming technical challenges, this invention addresses the long-standing problems of "vital signs being highly dependent on experience, difficult to standardize, and difficult to compare during follow-up." It introduces multi-perspective standardized data collection, multi-task quantitative modeling, cross-modal alignment, and uncertainty alerting mechanisms. Even in real-world environments with missing perspectives, information noise, and complex coupling of vital signs, it can still stably output structured results, making this complex comprehensive assessment process engineering-feasible for the first time.

[0141] Furthermore, this invention breaks through the traditional technical bias of "one part is indispensable, otherwise it is unusable". In the case of incomplete information, it achieves an output strategy that is "operable and controllable" through weight recalibration and uncertainty prompts, which significantly improves the clinical adaptability and promotion value of the system. Attached Figure Description

[0142] Figure 1 This is a structural block diagram of the orbital disease ocular sign feature extraction and analysis system based on multimodal external eye images and clinical text provided in this embodiment of the invention.

[0143] Figure 2 These are 16 standardized external eye images provided in this embodiment of the invention.

[0144] Figure 3 This is a flowchart of a method for extracting and analyzing ocular signs and characteristics of orbital diseases based on multimodal external eye images and clinical text, provided in an embodiment of the present invention.

[0145] Figure 4 This is a bar chart showing the performance of the present invention in classifying ocular signs.

[0146] Figure 5 This is an interpretable heatmap output by the present invention when classifying vital signs.

[0147] Figure 1 The module consists of: 1. Data acquisition module; 2. Input module; 3. Standardization module; 4. Recognition module; 5. Detection module; 6. Structured module; 7. Analysis module; and 8. Report generation module. Detailed Implementation

[0148] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0149] like Figure 1 As shown in the figure, an embodiment of the present invention provides a system for extracting and analyzing ocular signs of orbital diseases based on multimodal external eye images and clinical text, comprising:

[0150] Data acquisition module 1, input module 2, standardization module 3, identification module 4, detection module 5, structuring module 6, analysis module 7, report generation module 8;

[0151] Data acquisition module 1, connected to input module 2, is used for multimodal information and image acquisition;

[0152] Input module 2, connected to standardization module 3, is used for dynamic input mechanism;

[0153] Standardization module 3, connected to input module 2, recognition module 4, detection module 5, structuring module 6, analysis module 7, and report generation module 8, is used for standardization of the external eye region;

[0154] The recognition module 4 is connected to the standardization module 3 and is used for the recognition of ten external ocular signs.

[0155] Detection module 5, connected to standardization module 3, is used for automatic localization and detection of abnormal areas in the eye;

[0156] Structured module 6, connected to standardized module 3, is used for structuring clinical text;

[0157] Analysis module 7, connected to standardization module 3, is used for multimodal feature alignment and correlation analysis;

[0158] The report generation module 8, connected to the standardization module 3, is used for generating interpretable multimodal feature reports.

[0159] This invention provides a system for extracting and analyzing ocular signs in orbital diseases based on multimodal external ocular images and clinical text. Through joint modeling of multi-view external ocular image information and clinical text information, it achieves automatic identification, quantification, correlation analysis, and interpretable output of ocular signs. The system starts with data acquisition module 1 and ends with report generation module 8. All modules work collaboratively under the unified constraints and coordinate system of standardized module 3, forming a complete closed loop from raw data to structured results and evidentiary output.

[0160] Data acquisition module 1 is used to collect multimodal raw data, including external eye image information from different shooting angles and corresponding clinical text information. The image information can come from a regular camera, mobile phone, or outpatient standard imaging system, and the text information includes medical history descriptions, chief complaints, or examination records, etc. The acquired multimodal data is transmitted to input module 2.

[0161] Input module 2 is used for unified access and dynamic management of multi-source data, supporting organization by perspective, time, or patient instance, and sending the accessed data to standardization module 3. The dynamic input mechanism of input module 2 is used to adapt to different quantities, perspectives, and quality of data input to ensure that the system can still operate normally under real clinical data collection conditions.

[0162] Standardization module 3 is used to perform unified processing on the outer eye region. This includes scale normalization, pose correction, and region alignment for images from different sources, as well as format standardization and field mapping for text data, thereby transforming heterogeneous inputs into a standardized representation that the system can process. The standardized data serves as the unified input basis for subsequent modules.

[0163] Based on the external eye image data output by the standardized module 3, the recognition module 4 automatically identifies ten predefined external eye signs and outputs the corresponding signs' presence status or quantitative description results to depict the overall appearance of the eye.

[0164] The detection module 5 automatically locates and detects abnormal areas based on the standardized external eye image, which is used to determine the spatial location and distribution characteristics of the lesion area in the image, thereby providing spatial constraints and evidence support for subsequent analysis.

[0165] The structuring module 6 performs structuring processing on the clinical text information output by the standardization module 3, converting unstructured text descriptions into computable structured fields, enabling the text information to participate in subsequent feature alignment and association analysis processes.

[0166] Analysis module 7 performs multimodal feature alignment and correlation analysis on the results from identification module 4, detection module 5 and structuring module 6, establishes correspondence and consistency constraints between different modalities, and is used to comprehensively judge the correlation between signs and their degree of matching with clinical descriptions.

[0167] Based on the output of the analysis module 7, the report generation module 8 generates an interpretable multimodal feature report that includes quantitative results of vital signs, location information of abnormal areas, textual correlation evidence, and uncertainty prompts, thereby providing doctors with traceable and verifiable auxiliary decision-making information.

[0168] Through the collaborative work of the above modules, this invention realizes a complete process from multimodal data acquisition, standardized processing, sign identification and detection, multimodal correlation analysis to evidence output, transforming the assessment process of external ocular signs of orbital diseases from experience-dependent manual judgment to a standardized, structured, and interpretable automated process.

[0169] The data acquisition module provided in this embodiment of the invention:

[0170] Input includes:

[0171] Multi-angle 2D external eye photographs (up to 16 images)

[0172] Clinical textual information (chief complaint, medical history, self-reported signs, etc.)

[0173] Among them, the requirements for acquiring 16 standardized external eye images are as follows: Figure 2 The details are as follows:

[0174] ① Images from nine angles: front view and human eye. When shooting from the lower right, lower left, and lower left angles, the eyelids need to be pulled open to ensure the eyeballs are captured well.

[0175] ② Images viewed from 45° and 90° to the left and right respectively;

[0176] ③ Frontal closed-eye image and frontal bottom view image

[0177] ④ Image from an upward position;

[0178] The shooting logic follows these rules:

[0179] The photos were taken in three groups based on whether the eyes were moving or closed:

[0180] ①The first group consists of 5 images acquired at a time, taken without eye movement, in the supine, 45° and 90° left and right lateral positions;

[0181] ②The second set consists of two images: a frontal view, a view with eyes closed, and a bottom view.

[0182] ③ The third group consists of nine images from the frontal view, showing nine different eye positions.

[0183] The input module provided in this embodiment of the invention:

[0184] Considering that patients may not be able to acquire 16 images at once, this invention adopts a dynamic input priority strategy;

[0185] (1) Priority View Scheduler

[0186] The system prioritizes the contributions of different perspectives to sign recognition and disease classification:

[0187] Priority 1 (Required): Frontal view, used for baseline pupillary distance, palpebral fissure measurement, overall ocular symmetry analysis, and comprehensive assessment of eyelid position, eyelid shape, ocular surface morphology, and ocular position characteristics.

[0188] Priority 2: 45° / 90° to the left and right sides, used for estimating exophthalmos, assessing eye position, and identifying lateral scleral exposure.

[0189] Priority 3: Nine-view eye position images, used for the identification of limited eye movement and strabismus.

[0190] Priority 4: Eyes closed, looking down, and tilting back, used for identifying abnormal eyelid movements such as incomplete eyelid closure and delayed eyelid drooping.

[0191] If a viewpoint is missing, the scheduler automatically records the missing viewpoint ID, adjusts the combination of available viewpoints, and provides the corresponding modal confidence.

[0192] (2) Missing-Modality Mask (MMM)

[0193] For each expected input image, the system generates a binary mask:

[0194] 1: This angle image exists.

[0195] 0: Missing

[0196] This mask is input into the multimodal large model along with the image features, enabling the MLLM to "sense" the missing modalities and dynamically adjust the inference path;

[0197] (3) Construction of View-wise Feature Tensor

[0198] Each image is processed by a visual encoder, which outputs a feature tensor Fi ∈ RC×H×W;

[0199] The system stitches together features from multiple perspectives into:

[0200]

[0201] Missing modal positions are filled with zero vectors and subsequent attention weights are controlled by MMM masks;

[0202] (4) Dynamic View Re-weighting (DVR)

[0203] The system learns a trainable weight vector:

[0204] w={w1,w2,...,w16}

[0205] If the viewpoint is missing, then: wi=0, Fi=0

[0206] If a viewpoint exists, the system automatically assigns different weights based on the correlation between the viewpoint and the vital signs, for example:

[0207] The left and right lateral perspectives increase the weight of the "protruding eye" symptom.

[0208] The downward viewing angle increases the weight of "eyelid retardation".

[0209] 5) Uncertainty range estimation

[0210] The system automatically calculates confidence intervals based on: the number of missing views, weight recalibration values, and model output variance.

[0211]

[0212] The more missing perspectives, the wider the confidence interval.

[0213] The standardized module provided in this embodiment of the invention:

[0214] In the aforementioned 16 images, the system automatically performs facial landmark detection, binocular region cropping, horizontal / pose calibration, illumination correction, angle-by-angle resampling, periorbital segmentation, and structured measurement of the eye fissure, providing highly standardized input for ten vital sign recognitions. Specifically, it utilizes real-time facial landmark detection algorithms based on convolutional neural networks (CNNs) (such as Face++ or Dlib in OpenCV) to identify the accurate locations of key regions such as the eyes, eyelids, and periorbital area; and performs three-dimensional facial pose estimation (Head Pose)... Estimation is performed to obtain (roll, pitch, yaw) and perform roll rotation correction, orbital region alignment to a standard template, and illumination normalization (Retinex or Gamma correction). Multi-angle image standardization resampling: For 16 images from different viewpoints, geometric transformations are used to align them to a uniform scale, constructing a multi-view consistency dataset to ensure the comparability of the same feature under different viewpoints. On the aligned images, the system automatically performs periorbital region segmentation and quantifies a series of basic anatomical parameters, including palpebral fissure height and width, pupil diameter, and corneal light-reflecting point position, providing standardized input and reference benchmarks for subsequent feature recognition.

[0215] The identification module provided in this embodiment of the invention:

[0216] The system includes a network for recognizing ten external ocular signs, specifically: proptosis, sunken eyeball, strabismus, conjunctival congestion, conjunctival edema, eyelid congestion, eyelid retraction, eyelid edema, ptosis, and caruncle swelling.

[0217] The ten vital signs recognition adopts a multi-task learning structure (MTL), sharing the image backbone, but setting up an independent classification / regression head for each vital sign.

[0218] (1) Vital signs network structure

[0219] F = Backbone(Ii)

[0220] zk=Headk(F)

[0221] Vital signs output include: presence (sigmoid output); left / right eye; proptosis is continuously quantified (mm) and estimated when there is an input photo with a sufficient angle; ptosis is automatically measured and graded as MRD1 (distance from the upper eyelid margin to the corneal reflective point): mild: 1–2 mm; moderate: 3–4 mm; severe: ≥4 mm; and restricted eye movement is output along with the restricted direction (e.g., restricted upward gaze).

[0222] (2) Construction of trait vectors

[0223] The ten vital signs are combined into a structured vector:

[0224] T=[t1,t2,...,t10].

[0225] The detection module provided in this embodiment of the invention:

[0226] Based on the aforementioned image feature and vital sign feature extraction results, the system uses a target detection network to locate local regions in the image that are related to specific vital signs; the target detection network can be a detector based on the anchor box mechanism or a detector based on key point regression; for each detected region, its bounding box coordinates and the vital sign type identifier associated with that region are output;

[0227] Visual saliency heatmap generation: Using techniques such as class activation mapping (e.g., Grad-CAM), the model is calculated to determine the degree of attention paid to different pixel regions during the extraction of vital signs, and a pixel-level weight distribution map is generated; the heatmap is used to indicate pixel regions in the image that are highly correlated with specific vital signs.

[0228] By overlaying the heatmap with the original image or detection box results, the spatial distribution of vital signs-related areas in the image can be presented intuitively, providing a basis for subsequent manual analysis or system verification.

[0229] The structured module:

[0230] Medical information is extracted from the input text, including: chief complaint (bulging eyes, diplopia, decreased vision, etc.), present medical history (cause, disease progression, accompanying symptoms, etc.), comorbid systemic diseases (such as thyroid disease), treatment history, trauma history, etc. Key diagnostic clues are extracted from the natural language description, and medical semantic embedding Etext is generated using medical-specific LLM (such as Med-BERT, ClinicalLLaMA) for alignment and fusion with image features.

[0231] The analysis module:

[0232] By using a visual encoder, a stereo encoder, and a text encoder for cross-modal alignment, the model can achieve joint understanding of images and text across modalities, and deep semantic alignment and correlation analysis between abnormal regions and text information.

[0233] (1) Cross-modal alignment

[0234] The inputs include: image features F, feature vector T, text embedding Etext, and view mask MMM. Cross-modal attention is achieved through a cross-modal transformer.

[0235] H=CrossAttn(F, T, Etext, MMM)

[0236] (2) Association analysis and evidence chain construction:

[0237] After completing cross-modal alignment, the system further analyzes and establishes the correspondence between the following information: the sign-related regions detected in the image and their spatial localization results, the corresponding quantitative parameters or status identifiers in the sign feature vectors, the symptom descriptions related to the above signs in the clinical text, and time information or degree descriptions.

[0238] For example, the system can semantically match the locations of significant changes in the conjunctival region in the image, the corresponding conjunctival state parameters in the sign feature vector, with descriptions such as "red eyes" and "congestion" in the text, and calculate the consistency index among the three.

[0239] Ultimately, the system constructs a structured multimodal chain of evidence to clearly demonstrate the correspondence and strength of association between image evidence, physical evidence, and textual evidence.

[0240] The report generation module:

[0241] This module integrates all the above processing and analysis results, automatically generating a well-structured, evidence-related, and interpretable feature analysis report; the report content includes:

[0242] ①Standardized Images and Basic Structural Information: Displays standardized key view images and calculated basic anatomical parameters;

[0243] ② Quantitative results of vital signs: The identification results, quantitative values, and status indicators of ten key vital signs are clearly listed in tabular form;

[0244] ③ Abnormal area visualization: Overlay detection boxes or region masks of vital signs-related areas onto the original or standardized image, and simultaneously display the corresponding visual saliency heatmap to intuitively present the spatial distribution of vital signs-related areas;

[0245] ④ Multimodal Association Evidence Chain: Presents the association analysis results between image features, vital signs parameters and text information in a structured and visual manner, illustrating the correspondence and consistency between intermediate evidence from different modalities.

[0246] like Figure 3 As shown in the embodiment of the present invention, a method for extracting and analyzing ocular signs of orbital diseases based on multimodal external eye images and clinical text includes:

[0247] S101 acquires multimodal information and images through a data acquisition module;

[0248] S102, a dynamic input mechanism is implemented through the input module;

[0249] S103, standardization of the external eye region is achieved through a standardized module;

[0250] S104, uses the recognition module to identify ten external ocular signs;

[0251] S105, automatically locates and detects abnormal areas of the eye through the detection module;

[0252] S106, structuring clinical text through a structured module;

[0253] S107, performs multimodal feature alignment and correlation analysis through the analysis module;

[0254] S108, interpretable multimodal feature reports are generated through the report generation module.

[0255] This invention constructs a progressively hierarchical analysis process for orbital disease signs by collaboratively modeling multimodal external ocular images and clinical text, moving from the perception layer and semantic layer to the decision-making layer. This achieves automated, standardized, structured, and interpretable analysis of orbital disease-related signs. Its overall working principle involves mapping heterogeneous external ocular image information and clinical text information with significantly different expressions to the same analysis space through a unified data acquisition and standardization mechanism. Within this space, sign identification, anomaly detection, semantic alignment, and causal correlation analysis are performed, resulting in interpretable outputs for clinical decision-making.

[0256] During the data acquisition phase, the system simultaneously acquires multi-angle images of the external eye and corresponding clinical text records through the data acquisition module, ensuring consistency between visual and semantic information across time and case dimensions. The input module introduces a dynamic input mechanism, enabling the system to adapt to different imaging devices, resolutions, and text formats, achieving adaptive access to heterogeneous inputs. The standardization module performs unified localization, scale normalization, and posture correction of the external eye region, ensuring that subsequent recognition and detection operations are based on the same anatomical semantic reference system, eliminating systemic biases caused by differences in acquisition posture, lighting, and equipment at the source.

[0257] At the visual information processing level, the recognition module performs structured recognition of ten external ocular signs, transforming traditional subjective descriptions relying on human experience into quantifiable and comparable standard feature expressions. The detection module further precisely locates abnormal regions, actively separating abnormal signs from the overall appearance and preventing them from being masked by the averaging of overall features. At the semantic information processing level, the structuring module transforms unstructured clinical text into standard fields and semantic tags, turning natural language descriptions that could not be directly used in calculations into structured medical semantic features that can be used for analysis.

[0258] Building upon this foundation, the analysis module performs semantic alignment and association analysis on features from both visual and textual modalities. This establishes cross-modal mapping relationships not only at the feature level but also at the pathological semantic level, creating a correspondence between physical signs and clinical descriptions. This achieves automatic association from "visible features" to "medical meaning," avoiding the problem of existing technologies that rely solely on image recognition or text analysis and fail to form a closed-loop interpretation. Finally, the report generation module transforms the multimodal analysis results into interpretable feature reports, ensuring that each diagnostic conclusion is traceable to corresponding image signs and textual evidence, enhancing the system's credibility and usability in clinical applications.

[0259] This invention breaks through the existing technical paradigm of separating external eye image analysis from clinical text analysis. For the first time in the field of orbital disease analysis, it constructs a multimodal collaborative analysis mechanism based on "standardization—structuring—semantic alignment—interpretable output." Existing technologies typically only reach the level of single-modal recognition or simple information overlay, failing to address issues such as semantic inconsistencies, scale discrepancies, and untraceable interpretations between different modalities. This invention, by introducing a unified standardization mechanism at the front end, a cross-modal alignment mechanism in the middle, and an interpretable reporting mechanism at the back end, achieves a closed-loop design from data acquisition to clinical decision-making. This enables the system not only to "identify signs" but also to "understand signs" and "interpret signs." This systematic collaborative design, from the data layer to the semantic layer and then to the decision layer, cannot be directly derived by those skilled in the art using conventional methods. It possesses significant structural innovation and functional advancement, thus this invention has substantial characteristics and significant progress compared to existing technologies.

[0260] Example 1: Joint acquisition, integrity verification, and standardization processing of multi-view external eye images and clinical text

[0261] In this embodiment, multiple external eye images of the examinee are acquired in an outpatient environment using a regular digital camera or equivalent imaging device. The acquired viewing angles include at least one or more of the following: frontal, left / right 45°, left / right 90°, closed eyes, downward gaze, upward gaze, and nine-position eye images. The number of images acquired does not exceed 16. Each image is de-identified with the examinee's identity information and bound to a unified timestamp. A viewing angle tag (view_id) and metadata of the shooting conditions (such as resolution, focal length, lighting level, whether a flash was used, etc.) are also recorded.

[0262] The system synchronously collects clinical text information filled in by patients. The text includes at least one or more of the following: chief complaint, present illness, and past medical history, and is also linked to the examinee's identifier with a timestamp. The system performs integrity checks on both images and text.

[0263] (1) Image side verification includes: whether there is a valid face / eye area, whether the minimum resolution threshold is met, whether there is severe occlusion or a closed eye state that contradicts the viewpoint label, etc.

[0264] (2) Text-side validation includes: field missing detection, prompts for empty key symptom fields, etc.

[0265] The system generates a missing viewpoint identifier vector MMM, where MMM =1 indicates that the j-th viewpoint exists, MMM =0 indicates a missing value, and this value is used as one of the inputs for subsequent fusion modules.

[0266] The system performs keypoint localization and eye region cropping on the images. Keypoints include at least the inner / outer canthus, upper / lower eyelid margin keypoints, and pupil / corneal reflection points. Based on this, it performs pose correction (e.g., rotational normalization based on the binocular line) and illumination normalization (e.g., histogram / luminance normalization) to form a standardized external eye image dataset. The system performs word segmentation, medical entity recognition, and semantic encoding on the text, outputting structured text fields (e.g., symptom entities, time entities, severity entities) and corresponding semantic vector representations. Subsequently, the system inputs multi-view image features, physical sign features, and text features into a joint analysis module, outputting structured analysis results for subsequent report generation.

[0267] Example 2: Dynamic Input, Viewpoint Weight Recalibration, and Uncertainty Output under Missing Viewpoint Conditions

[0268] In this embodiment, the subject is unable to complete the upward-facing image capture. During the acquisition phase, the system records the missing upward-facing viewpoint, resulting in the corresponding upward-facing identifier being 0 in the MMM. During the input phase, the system dynamically adjusts the multi-view feature fusion structure: the feature tensor corresponding to the missing viewpoint is set to zero, and the corresponding weight is set to 0 during the weight allocation phase. The weights of the remaining viewpoints are determined jointly based on preset priorities and model learning weights, ensuring that effective viewpoint information is fully utilized while the missing viewpoint does not introduce noise.

[0269] Furthermore, to avoid outputting misleading "pseudo-precise" quantification results when information is insufficient, the system constructs an uncertainty estimation mechanism based on the number of missing perspectives, the status of missing key perspectives, and the variance of the model output:

[0270] When a missing perspective affects the key angle required for quantifying a certain physical characteristic (for example, quantifying exophthalmos requires information from the side or a specific angle), the system automatically reduces the confidence level of that result.

[0271] The system outputs a confidence interval CI for this vital sign. δ increases with the number of missing viewpoints, or with the output variance.

[0272] The system outputs a "list of missing perspectives", a "list of affected signs", and "confidence interval / confidence level" in the final report, enabling doctors to identify the source of uncertainty and decide whether to reshoot key perspectives.

[0273] Example 3: Multi-task learning joint recognition and quantification / hierarchical output of ten external ocular signs (revised version, with added output fields)

[0274] In this embodiment, the system employs a multi-task learning structure (MTL) to jointly identify ten external ocular signs: Features F are extracted from a shared image backbone, and an independent classification / regression head (Head_k) is set for each sign, outputting z_k. For each sign, the system output includes at least:

[0275] (1) The probability of the presence of the physical sign p_k (sigmoid output);

[0276] (2) Lateral information s_k (left eye / right eye / both eyes / uncertain);

[0277] (3) Quantization value v_k (if applicable, in mm or direction label);

[0278] (4) Grading / Category u_k (if applicable, such as light / medium / heavy, etc.).

[0279] Taking quantifiable physical signs as an example:

[0280] Protrusion: Outputs continuous quantization values ​​(mm) when there is an input photo that meets the angle requirements; if the key angle is insufficient, it outputs a confidence interval or prompts that a retake is needed.

[0281] Ptosis: MRD-1 (distance from the upper eyelid margin to the corneal reflective point, mm) is automatically measured and graded according to the threshold (mild 1–2 mm; moderate 3–4 mm; severe ≥4 mm).

[0282] Eye movement restriction: Whether output is restricted and the location of restriction (e.g., restricted upward / downward / adduction / abduction, etc.), and the degree of restriction if necessary.

[0283] The system organizes the ten vital signs into a structured vital sign vector T= The t_k uses a unified data structure and maintains an indexed association with the original image, key points, and detection box / heatmap results to ensure that subsequent modules can locate the image evidence and numerical basis corresponding to each vital sign.

[0284] Example 4: Automatic localization and visualization of abnormal areas related to vital signs

[0285] In this embodiment, based on the vital sign recognition results, the system automatically locates the regions in the image related to conjunctival congestion, eyelid edema, and lacrimal caruncle swelling, and outputs the corresponding region boundary information. Simultaneously, the system generates a visual saliency heatmap to identify pixel regions that the model focuses on most when judging vital signs.

[0286] The system overlays the boundary of the abnormal area with a heatmap on the original image, allowing doctors to visually see the areas of interest of the model and use them for manual review or teaching examples.

[0287] Example 5: Clinical text structuring, cross-modal alignment, and construction of structured evidence chains

[0288] In this embodiment, the system performs structured extraction of the patient's clinical text: extracting medical entities such as "exophthalmos," "diplopia," "red eyes," "pain," "disease progression," and "thyroid history" from the chief complaint and medical history, and standardizing the time and severity descriptions into structured fields (such as duration and severity). The system encodes the text into a semantic vector E and retains the position index (text_span) of the entity in the original text.

[0289] Subsequently, the system inputs multi-view image features, feature vector T, and text semantic vector E into a cross-modal attention model to achieve alignment between features, image regions, and text entities. The model outputs association weights or similarity scores to measure the strength and consistency of the association between evidence.

[0290] Based on this, the system constructs a structured chain of evidence. For example, it maps the entity "red eyes" in the text to the conjunctival hyperemia area in the image and outputs a consistency score; it maps "diplopia" to the direction of restricted eye movement and outputs related evidence. Through the chain of evidence structure, the system can clearly identify the source of evidence for each vital sign conclusion, improving the interpretability and traceability of the output.

[0291] Example 6: Generation of Interpretable Vital Signs Analysis Report

[0292] In this embodiment, the system automatically generates a comprehensive analysis report. The report includes at least:

[0293] (1) Standardized multi-view images and view labels;

[0294] (2) Structured results table of ten vital signs (including probability of presence, laterality, quantification value, classification and confidence interval / confidence level);

[0295] (3) Abnormal area location results and heat map visualization (corresponding one-to-one with the vital signs items);

[0296] (4) Summary of cross-modal evidence chain (correspondence and consistency score between physical signs, image evidence and text evidence).

[0297] (5) List of missing perspectives and uncertainty warnings (if there are missing perspectives or insufficient image quality).

[0298] The report presents the quantitative results of vital signs in a table and image overlay format, and provides clickable / locatable evidence indexes for each vital sign to assist doctors in classifying orbital diseases, assessing the condition and keeping follow-up records. It also supports archiving as supplementary material to medical records and communication with patients.

[0299] Evidence related to the technical effects obtained by the embodiments of the present invention.

[0300] The observability of external ocular signs exhibits a significant "viewpoint-dependent" characteristic. For example:

[0301] Signs such as exophthalmos and eyelid retraction are more likely to form quantifiable geometric cues from a lateral or oblique perspective.

[0302] The direction of restricted eye movement requires nine-dimensional eye position images to be fully covered;

[0303] Local signs such as conjunctival congestion are clearer in frontal high-resolution images.

[0304] When different perspectives provide complementary information, multi-perspective joint inference is equivalent to estimating the result with a larger amount of effective information, and its estimation variance can theoretically be reduced. Intuitively speaking, a single-perspective model may be in a state of "insufficient information or insensitivity to perspective" for certain features, resulting in large fluctuations in output; while multi-perspective joint inference can supplement the information "not visible from one perspective" with other perspectives, improving the stability and completeness of feature recognition.

[0305] Therefore, this invention uses standardized multi-view external eye image input and performs joint modeling, which theoretically enables:

[0306] (1) Reduce misjudgments caused by single-view occlusion, head tilt, and loss of local details;

[0307] (2) Improve the detection stability of early atypical manifestations and mild signs;

[0308] (3) Provide more reliable geometric clues and references for quantification tasks (bulging, MRD-1).

[0309] The following are some of the vital signs recognition results:

[0310]

[0311] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0312] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A system for extracting and analyzing ocular signs and characteristics of orbital diseases based on multimodal external eye images and clinical text, characterized in that, The system includes a data acquisition module, an input module, a standardization module, a recognition module, a detection module, a structuring module, an analysis module, and a report generation module, wherein: The data acquisition module is used to acquire multi-view external eye images and clinical text information; The input module is used to perform viewpoint priority scheduling, missing modality identification, and multi-viewpoint feature construction on the multi-view external eye images; The standardization module is used to perform key point detection, region cropping, pose calibration, illumination normalization, angle resampling, and anatomical parameter quantization on the input external eye image; The recognition module is used to identify multiple external ocular features based on standardized image features and output feature vectors; The detection module is used to automatically locate the image region corresponding to the vital signs and generate region association information; The structuring module is used to convert clinical text into structured medical semantic vectors; The analysis module is used to perform cross-modal alignment and correlation analysis on image features, feature vectors, and text semantic vectors, and to construct chains of evidence; The report generation module is used to generate an interpretable vital sign analysis report based on the chain of evidence.

2. The system as described in claim 1, characterized in that, The data acquisition module acquires no more than 16 two-dimensional external eye images from different perspectives, including at least a frontal view, a 45-degree side view, a 90-degree side view, a nine-directional view, a closed-eye view, a downward-looking view, and an upward-looking view.

3. The system as described in claim 1, characterized in that, The input module includes a viewpoint priority scheduling unit, a missing modality mask generation unit, a multi-view feature splicing unit, and a dynamic weight recalibration unit, which are used to allocate weights according to viewpoint importance, identify missing views, and adjust the participation level of multi-view features.

4. A method for extracting and analyzing ocular signs and characteristics of orbital diseases based on multimodal external eye images and clinical text, characterized in that, Includes the following steps: Step 1: Acquire multi-view external eye images and clinical text information; Step 2: Perform viewpoint priority scheduling, missing modality identification, and multi-viewpoint feature construction on the external eye image; Step 3: Perform key point detection, region cropping, pose calibration, illumination normalization, angle resampling, and anatomical parameter quantification on the external eye image; Step 4: Identify multiple external ocular features and construct feature vectors; Step 5: Automatically locate the image region corresponding to the vital signs and generate region association information; Step 6: Convert the clinical text into structured medical semantic vectors; Step 7: Perform cross-modal alignment and correlation analysis on image features, feature vectors, and text semantic vectors, and construct a chain of evidence; Step 8: Generate an interpretable vital signs analysis report.

5. The method as described in claim 4, characterized in that, In step 2, a missing modality mask is generated for each expected viewpoint, and the feature weights corresponding to the missing viewpoints are set to 0.

6. The method as described in claim 4, characterized in that, The external ocular signs mentioned in step 4 include at least exophthalmos, enophthalmos, strabismus, conjunctival hyperemia, conjunctival edema, eyelid hyperemia, eyelid retraction, eyelid edema, ptosis, and caruncle swelling.

7. The method as described in claim 4, characterized in that, The evidence chain constructed in step 7 includes the correspondence between the location of the sign region in the image, the sign vector parameters, and the clinical text description of symptoms.

8. A method for generating an interpretable vital sign analysis report for implementing the system of claim 1 or the method of claim 4, characterized in that, include: Generate standardized external eye images and basic anatomical parameters; Generate vital sign recognition results and their quantification parameters; Generate location results and saliency heatmaps for regions related to vital signs; Generate a chain of evidence linking image evidence, physical evidence, and textual evidence, and demonstrate their consistency.

9. The method as described in claim 8, characterized in that, The associated chain of evidence records the strength of each piece of evidence in a structured form.

10. The method as described in claim 8, characterized in that, The report is used to assist in the classification analysis and disease assessment of orbital diseases.

Images