Method, product and medium for slit lamp image analysis based on multi-agent collaboration

Abstract

The embodiment of the application discloses a kind of slit lamp image analysis method, product and medium based on multi-agent cooperation, comprising: according to the imaging condition analysis of slit lamp image according to predetermined visual language model;Through abnormal perception model, visual saliency channel anomaly detection and semantic guide channel anomaly detection are carried out on slit lamp image based on analysis result, and structured abnormal candidate set is obtained;Based on clinical description model and standard terminology constraint, the standardized clinical description corresponding to structured abnormal candidate set is generated;Through the multidimensional review of predetermined review model to standardized clinical description, based on multidimensional review result, standardized clinical description is iteratively optimized, and target standardized description is obtained;Based on target standardized description, slit lamp image examination report is generated.The high-quality analysis report of description specification and structure uniformity can be obtained.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a slit lamp image analysis method, product and medium based on multi-agent collaboration. Background Technology

[0002] Slit-lamp microscopy is the most frequently used basic examination equipment in ophthalmological clinics, widely used for routine examination and preoperative assessment of anterior segment structures such as the cornea, anterior chamber, iris, pupil, and lens. By adjusting different illumination methods (diffuse illumination, direct focal illumination, backlighting, and specular reflection, etc.) and magnification, slit lamps can reveal rich tissue details and pathological features. Given the continuously increasing number of outpatient visits and the insufficient ophthalmological resources at the grassroots level, leveraging artificial intelligence technology to achieve automatic anomaly detection and standardized report generation for slit-lamp images is of great significance for improving examination efficiency and diagnostic consistency.

[0003] In recent years, artificial intelligence and deep learning have made significant progress in the field of medical image analysis. One approach involves directly inputting slit-lamp images into a large, general visual language model and generating clinical descriptive text. While this method possesses some image understanding and text generation capabilities, the general-purpose model's learning ability for ophthalmological expertise is limited, resulting in incomplete and inaccurate analysis of slit-lamp images. The generated descriptions often suffer from inaccurate terminology and inconsistent descriptive structures. Furthermore, the single-step reasoning method using a large model cannot self-verify the detection results, failing to identify inconsistencies between descriptions and images, or omissions of abnormal regions, leading to inaccurate output. Summary of the Invention

[0004] This invention provides a slit lamp image analysis method, product, and storage medium based on multi-agent collaboration, which can perform more comprehensive, accurate, and professional analysis and detection of slit lamp images, improve the completeness of slit lamp image anomaly detection, the standardization of clinical description, and the reliability of output, and ensure that a high-quality analysis report with standardized description and uniform structure is obtained.

[0005] In a first aspect, embodiments of the present invention provide a slit-lamp image analysis method based on multi-agent collaboration, comprising: Acquire slit lamp images and perform imaging condition analysis on the slit lamp images according to a pre-determined visual language model to obtain analysis results; wherein, the analysis results include visual channel weights and semantic guidance channel weights; Using a pre-determined anomaly perception model, visual saliency channel anomaly detection and semantic guidance channel anomaly detection are performed on the slit lamp image based on the analysis results. The results of visual saliency channel anomaly detection and semantic guidance channel anomaly detection are then fused and analyzed to obtain a structured anomaly candidate set. Standardized clinical descriptions corresponding to the structured anomaly candidate set are generated based on a pre-determined clinical description model and preset standard terminology constraints. The standardized clinical description is reviewed in multiple dimensions using a pre-determined review model. Based on the results of the multi-dimensional review, the results of the visual saliency channel anomaly detection, the results of the semantic guidance channel anomaly detection, and the candidate set of structured anomalies, the standardized clinical description is iteratively optimized to obtain the target standardized description. A slit lamp image inspection report is generated based on the standardized description of the target.

[0006] Secondly, embodiments of the present invention also provide an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the slit lamp image analysis method based on multi-agent collaboration as described in any of the embodiments of the present invention.

[0007] Thirdly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the slit lamp image analysis method based on multi-agent collaboration as described in any of the embodiments of the present invention.

[0008] Fourthly, embodiments of the present invention provide a computer program product, including a computer program, which, when executed by a processor, implements an emergency response method as described in any of the embodiments of the present invention.

[0009] The method of this invention employs a dual-channel detection strategy for more comprehensive anomaly detection in slit-lamp images, effectively covering the detection blind spots of a single channel and improving the model's detection performance under different imaging conditions. By using standardized terminology constraints, it generates descriptions that better conform to medical clinical standards, solving the problems of inaccurate terminology and inconsistent structure in text generated by general visual language models. Iterative optimization of the standardized clinical description based on the detection results at each stage effectively identifies potential misjudgments, ensuring the reliability of the target standardized description. In other words, this invention, through multi-model collaborative report generation, enables more comprehensive, accurate, and professional analysis and detection of slit-lamp images, improving the completeness of anomaly detection, the standardization of clinical descriptions, and the reliability of output, ensuring a high-quality analysis report with standardized descriptions and a consistent structure. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A first flowchart of a slit lamp image analysis method based on multi-agent collaboration provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the dual-channel anomaly detection process provided in an embodiment of the present invention; Figure 3 This is a collaborative architecture diagram of the multi-intelligent model provided in the embodiments of the present invention; Figure 4 The second flowchart of a slit lamp image analysis method based on multi-agent collaboration provided in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the process of iteratively optimizing standardized clinical descriptions provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the invention and not all structures. The acquisition, storage, use, and processing of data in the technical solutions of this application comply with relevant national laws and regulations. It should be noted that, in the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already used or necessarily used the relevant content of such solutions.

[0013] Figure 1This is a first flowchart of a slit-lamp image analysis method based on multi-agent collaboration provided by an embodiment of the present invention. The method of this embodiment can perform more comprehensive, accurate, and professional analysis and detection of slit-lamp images, improving the completeness of slit-lamp image anomaly detection, the standardization of clinical description, and the reliability of output, ensuring a high-quality analysis report with standardized description and unified structure. This method can be executed by a slit-lamp image analysis device based on multi-agent collaboration provided by an embodiment of the present invention, which can be implemented in software and / or hardware. The following embodiments will illustrate this using the integration of the device into an electronic device as an example. The electronic device can be a server, cloud server, or computer device, etc., used to implement a slit-lamp image analysis method based on multi-agent collaboration. (Refer to...) Figure 1 The method may specifically include the following steps:

[0014] Step 101: Acquire slit lamp images and perform imaging condition analysis on the slit lamp images according to a pre-determined visual language model to obtain analysis results.

[0015] The slit-lamp images are ophthalmic images acquired using a slit-lamp microscope. After the microscope acquires the slit-lamp image, the server automatically retrieves it. The visual language model is a large-scale deep learning model capable of processing both visual information and natural language simultaneously. Analysis results include illumination mode labels, observation region labels, visual channel weights, semantic guidance channel weights, and strictness parameters. Illumination mode labels identify the illumination mode used by the slit-lamp microscope during image acquisition. Illumination modes include one or more combinations of diffuse illumination, direct focal illumination, backlighting, and specular reflection. Observation region labels identify the main ocular anatomical structures in focus within the slit-lamp image. Observation regions include five basic types and their combinations: cornea, anterior chamber, iris, pupil, and lens.

[0016] After obtaining the slit lamp image, imaging condition analysis can be performed on the slit lamp image using a visual language model. In this scheme, the imaging condition analysis of the slit lamp image is performed according to a pre-determined visual language model, and the analysis results include: standardizing the slit lamp image, analyzing the standardized slit lamp image using the visual language model to obtain illumination mode labels and observation area labels; and determining the visual channel weights and semantic guidance channel weights based on the illumination mode labels and pre-defined imaging condition weight information.

[0017] Specifically, after obtaining the slit lamp image, it can be standardized, including illumination normalization to map images acquired under different exposure conditions to a unified brightness and contrast range. Adaptive contrast enhancement is then performed, for example, by dividing the image into several local regions and calculating the contrast index for each region. Stronger enhancement algorithms are used for regions with low contrast, while weaker algorithms are used for regions with high contrast. The region of interest is then cropped to remove invalid dark areas at the image edges, resulting in a standard-format slit lamp image that can be directly input into the visual language model. The standardized slit lamp image and corresponding model prompts are then input into the visual language model. The prompts guide the model to output the analysis results of imaging conditions, including illumination mode labels and observation region labels. The model prompts can be input by the user or automatically generated by the visual language model.

[0018] The imaging condition weight information can be a predefined imaging condition weight mapping table, which includes the allocation of visual channel weights and semantic guidance channel weights for each illumination mode. For example, the allocation might include: higher visual channel weights (visual channel weight ≥ 0.6) for backlighting and specular reflection, and higher semantic channel weights (semantic guidance channel weight ≥ 0.6) for diffuse and direct focus illumination. After determining the illumination mode label, the weight allocation corresponding to the illumination mode label is looked up in the imaging condition weight mapping table, and a channel weight allocation vector is generated based on the weight allocation. ;in For visual channel weights, For semantically guided channel weights, .

[0019] The stringency parameter in this scheme also includes a strictness parameter. After determining the illumination mode label, the stringency parameter corresponding to the illumination mode label can be determined according to a predefined imaging condition threshold mapping table. The imaging condition threshold mapping table includes the stringency parameter corresponding to each illumination mode. For example, specular reflection method corresponds to a higher review stringency due to more artifacts (stringency parameter ≥ 0.8), while diffuse illumination corresponds to a moderate review stringency due to uniform overall illumination (0.5 ≤ stringency parameter < 0.8). Automatically adjusting the dual-channel fusion weights and review stringency parameters according to the illumination mode improves the accuracy of subsequent anomaly detection. Imaging condition analysis of slit lamp images is performed through a visual language model to dynamically determine the visual channel weights and semantic guidance channel weights, improving the matching degree between the dual-channel detection strategy and specific imaging conditions.

[0020] Step 102: Using a pre-determined anomaly perception model, perform visual saliency channel anomaly detection and semantic guidance channel anomaly detection on the slit lamp image based on the analysis results, and perform fusion analysis on the visual saliency channel anomaly detection results and semantic guidance channel anomaly detection results to obtain a structured anomaly candidate set.

[0021] The anomaly perception model is used for visual saliency channel detection and semantically guided channel detection in slit-lamp images. The anomaly perception model includes a backbone network of the visual language model, which shares parameters with the visual language model in step 101. However, while keeping the network parameters of the visual language model frozen, a small number of trainable parameters guide the anomaly perception model to focus on the spatial features of abnormal regions, resulting in a fine-tuned backbone network. In other words, the anomaly perception model in this scheme possesses both the visual understanding capabilities of the visual language model and the ability to perform anomaly detection tasks. Visual saliency channel anomaly detection employs a bottom-up detection strategy, which does not rely on prior knowledge or predefined disease classifications, but rather analyzes potential lesions in slit-lamp images based on differences in visual features. Semantically guided channel anomaly detection employs a top-down detection strategy, which can utilize prior knowledge to guide the anomaly perception model for targeted detection. Semantically guided channel anomaly detection can simulate a doctor's process of checking an examination checklist item by item, accurately identifying known types of anomalies.

[0022] In one optional implementation, multi-scale visual feature maps of slit lamp images are extracted using the visual language backbone network of an anomaly perception model; anomaly saliency scores for each pixel position of the multi-scale visual feature maps are calculated based on predetermined prototype features using the visual saliency channel of the anomaly perception model; a set of visual saliency candidate regions is obtained based on the anomaly saliency scores and a preset score threshold; and a set of semantically guided candidate regions is obtained by performing targeted detection on the slit lamp images based on illumination mode labels, observation area labels, and a preset clinical examination database using the semantic guidance channel of the anomaly perception model.

[0023] The structured anomaly candidate set is a collection of anomaly candidates obtained by fusing and deduplicating the anomaly detection results from the visual saliency channel and the semantic guidance channel. Each candidate in the structured anomaly candidate set may have fused the detection results from both channels (visual saliency channel and semantic guidance channel). That is, each candidate in the structured anomaly candidate set may come only from the detection results from the visual channel, only from the detection results from the semantic guidance channel, or a fusion of the detection results from both channels. In one optional implementation, the visual saliency candidate region set and the semantic guidance candidate region set are fused using a first projection network and a second projection network of the anomaly perception model to obtain a fused feature space; the network parameters of the first projection network and the second projection network are shared; the fused feature space is clustered and deduplicated based on the visual channel weights and the semantic guidance channel weights to obtain a fused candidate region set; and each candidate region in the fused candidate region set is structurally encapsulated to obtain the structured anomaly candidate set.

[0024] For example, Figure 2 This is a schematic diagram illustrating the dual-channel anomaly detection process provided in an embodiment of the present invention. Figure 2 As shown, anomaly detection is performed on multi-scale visual features using both the visual saliency channel and the semantic guidance channel. Visual saliency channel anomaly detection involves calculating the distance between pixels of the multi-scale visual features and the prototype features, then applying a threshold based on the distance to obtain a set of visual saliency candidate regions. , This represents the i-th visual saliency candidate region. Anomaly detection in the semantic guidance channel involves targeted detection of slit-lamp images based on clinical examination databases and visual-language networks to obtain a set of semantic guidance candidate regions. , This represents the i-th semantic guidance candidate region. The contrastive learning fusion module is used to fuse features between the visual saliency candidate region set and the semantic guidance candidate region set to obtain a set of structured anomaly candidates.

[0025] Step 103: Generate standardized clinical descriptions corresponding to the structured abnormality candidate set based on the pre-determined clinical description model and preset standard terminology constraints.

[0026] The clinical description model generates standardized clinical descriptions for each candidate region of structured abnormalities in the candidate set of structured abnormalities, based on standard terminology constraints. This clinical description model also includes a backbone network of a visual language model, which shares parameters with the visual language model. Standardized clinical descriptions are descriptive texts that conform to ophthalmological clinical writing standards and accurately reflect the pathological characteristics of abnormal regions in slit-lamp images, using terminology and structure. Standard terminology constraints guide the clinical description model to generate standardized clinical descriptions that are accurate in terminology, consistent in expression, and highly consistent with clinical norms. Standard terminology constraints can be predefined standard terminology constraint tables, including: a standard terminology set for anatomical locations (cornea, corneal epithelium, corneal stroma, corneal endothelium, anterior chamber, iris, pupil, lens, anterior lens capsule, lens cortex, and lens nucleus, etc.), a standard terminology set for abnormality types (opacity, infiltration, edema, neovascularization, pigmentation, atrophy, adhesion, defects, and deposits, etc.), a standard descriptive terminology set for morphological characteristics (punctate, patchy, diffuse, localized, annular, dendritic, and discoid, etc.), and a standard descriptive terminology set for extent (involving the entire thickness, involving a portion, central area, peripheral area, superior, inferior, nasal, and temporal, etc.).

[0027] In one optional implementation, the structured anomaly candidate set is encoded with multi-granularity visual features through the visual language network of the clinical description model to obtain the joint visual representation vector of each structured anomaly candidate region in the structured anomaly candidate set; standardized descriptive text corresponding to the joint visual representation vector is generated based on the clinical description model and standard terminology constraints; and standardized clinical description is determined based on the standardized descriptive text and its corresponding structured anomaly candidate region.

[0028] Step 104: Perform multi-dimensional review of the standardized clinical description using a pre-determined review model. Based on the multi-dimensional review results, visual saliency channel anomaly detection results, semantic guidance channel anomaly detection results, and structured anomaly candidate set, iteratively optimize the standardized clinical description to obtain the target standardized description.

[0029] The multi-dimensional review includes visual text consistency review, missed detection review, adversarial counterexample hypothesis testing, and cross-regional correlation reasoning. Text consistency review checks whether the textual descriptions of standardized clinical descriptions match the actual conditions of the corresponding slit-lamp image regions. Missed detection review determines whether the dual-channel anomaly detection model has missed any anomalies. Adversarial counterexample hypothesis testing determines whether the dual-channel anomaly detection has misjudged any cases. Cross-regional correlation reasoning examines the completeness and clinical rationality of the detection results from a global perspective.

[0030] The review model is primarily used for multi-dimensional review and iterative optimization of standardized clinical descriptions. It also includes a backbone network that shares parameters with the visual language model. In this approach, a low-rank adaptation strategy can be used to fine-tune the review model during the model training phase. This strategy excels at assessing the matching degree between image regions and text descriptions, as well as identifying potential errors in the descriptions. During training, the low-rank adaptation strategy uses group-based relative policy optimization. This involves generating a set of candidate review strategies for the same input sample, calculating the reward value of each candidate strategy according to a defined reward function, recording the relative ranking of reward values ​​within the group, and then updating the policy gradient based on the ranking advantage. For example, the reward function... for:

[0031] ;in Bonuses will be awarded for higher accuracy in detecting inconsistencies. Rewards will be given for discovering missed detections. Penalty for false alarm rate To implement penalties for each round of audits, , , and The weighting coefficients are set. The reward function balances four dimensions: accuracy of inconsistency detection, ability to detect missed detections, false positive control, and review efficiency. This enables the review model to form an efficient review strategy, and further allows for high-quality review judgments to be executed with fixed parameters during the inference phase.

[0032] In one optional implementation, throughout the slit-lamp image processing, the server maintains a dynamically updated evidence accumulation board for each structured anomaly candidate (including the structured anomaly candidate region and its corresponding standardized clinical description, etc.). This board records all relevant anomaly evidence from the candidate's entire lifecycle, from generation to convergence, such as source tags, round numbers, detection channel identifiers, and individual confidence scores output by the review models from different models (anomaly perception model, description generation model, and review model, etc.) and different review rounds. The data recorded on the accumulation board is then fused and analyzed to obtain the final comprehensive detection result. Based on this result, the standardized clinical description is iteratively optimized to obtain the target standardized description used to generate the slit-lamp image examination report.

[0033] In one optional implementation, a single confidence score is determined based on multi-dimensional review results, visual saliency channel anomaly detection results, semantic guidance channel anomaly detection results, and a structured anomaly candidate set. A comprehensive confidence score is calculated based on the single confidence score, a preset source weight coefficient, and a preset source independence coefficient. Whether the standardized clinical description meets preset conditions is determined based on the rigor parameter and the comprehensive confidence score. If the standardized clinical description does not meet the preset conditions, a correction instruction is generated based on the clinical description. The correction instruction includes a correction step; the correction step is either an anomaly perception step or a description generation step. If the correction step is an anomaly perception step, the correction instruction and analysis results are input into the anomaly perception model, and the steps of detecting visual saliency channel anomalies and semantic guidance channel anomalies in the slit-lamp image based on the analysis results using a pre-determined anomaly perception model are repeated until a target standardized description that meets the preset conditions is obtained. If the correction step is a description generation step, the correction instruction and analysis results are input into the clinical description model, and the steps of generating standardized clinical descriptions corresponding to the structured anomaly candidate set based on a pre-determined clinical description model and preset standard terminology constraints are repeated until a target standardized description that meets the preset conditions is obtained.

[0034] Figure 3 This is a collaborative architecture diagram of the multi-intelligent model provided in an embodiment of the present invention. (See diagram below.) Figure 3 As shown, the anomaly perception model, clinical description model, and review model share a single visual-language backbone network. Independent low-rank adaptation modules (LoRA) are configured for each of the three models for task-specific fine-tuning: the LoRA for the anomaly perception model is for anomaly region localization, the LoRA for the clinical description model is for clinical text generation, and the LoRA for the review model is for visual-text consistency judgment. The three intelligent models share the same visual-language backbone network, achieving task differentiation only through lightweight LoRA modules. Compared to deploying three independent, complete models, this significantly reduces the number of parameters and computational resources.

[0035] Step 105: Generate a slit lamp image inspection report based on the target standardized description.

[0036] The slit-lamp image examination report is a file generated by the server based on the target standardized description. It records anterior segment structural abnormalities corresponding to the acquired slit-lamp images and provides a basis for diagnostic and treatment decisions. Specifically, after obtaining the target standardized description, the server can sort the target standardized description according to the standard order of anterior segment anatomical structures. For cases where multiple abnormal descriptions exist for the same anatomical region, the server can perform secondary sorting according to the severity of the lesion or clinical importance. For example, the order could be: conjunctiva → cornea (corneal epithelium → corneal stroma → corneal endothelium) → anterior chamber → iris → pupil → lens (anterior lens capsule → lens cortex → lens nucleus). For anatomical regions where no abnormalities are found in the standard order, the standard description "no obvious abnormalities found" can be automatically filled in. Furthermore, the sequentially arranged and fully filled sub-descriptions (descriptions of examination findings in the slit-lamp imaging report, categorized by anatomical structure) are integrated into sub-description segments. These sub-description segments, organized by anatomical structure, record the examination findings of each region, forming the main content of the slit-lamp imaging report. Based on this, the knowledge integration and reasoning capabilities of the visual language model can be utilized to summarize the scattered sub-description segments into a clinical diagnostic impression. The clinical diagnostic impression includes a preliminary clinical diagnostic impression and recommendations for further examination or follow-up. The preliminary clinical diagnostic impression includes the inferred possible disease diagnosis, while the recommendations for further examination or follow-up include subsequent treatment plans proposed based on the diagnostic impression.

[0037] The technical solution of this embodiment involves acquiring a slit-lamp image and performing imaging condition analysis on the slit-lamp image based on a pre-determined visual language model to obtain analysis results. These results include visual channel weights and semantic guidance channel weights. Using a pre-determined anomaly perception model, the slit-lamp image undergoes visual saliency channel anomaly detection and semantic guidance channel anomaly detection based on the analysis results. The results of visual saliency channel anomaly detection and semantic guidance channel anomaly detection are then fused and analyzed to obtain a structured anomaly candidate set. A standardized clinical description corresponding to the structured anomaly candidate set is generated based on a pre-determined clinical description model and preset standard terminology constraints. The standardized clinical description undergoes multi-dimensional review using a pre-determined review model. Based on the multi-dimensional review results, visual saliency channel anomaly detection results, semantic guidance channel anomaly detection results, and the structured anomaly candidate set, the standardized clinical description is iteratively optimized to obtain a target standardized description. A slit-lamp image inspection report is generated based on the target standardized description. This technical solution, through a dual-channel detection strategy, enables more comprehensive anomaly detection of slit-lamp images, effectively covering the detection blind spots of a single channel and improving detection performance under different imaging conditions. By using standardized terminology constraints, descriptions that better conform to medical clinical standards can be generated, solving the problems of inaccurate terminology and inconsistent structure in text generated by general visual language models. Iterative optimization of the standardized clinical description based on the detection results at each stage can effectively identify potential misjudgments, ensuring the reliability of the target standardized description. In other words, this embodiment of the invention, through multi-model collaborative report generation, enables more comprehensive, accurate, and professional analysis and detection of slit-lamp images, improving the completeness of slit-lamp image anomaly detection, the standardization of clinical descriptions, and the reliability of output, ensuring a high-quality analysis report with standardized descriptions and a consistent structure.

[0038] Figure 4 This is a second flowchart of a slit lamp image analysis method based on multi-agent collaboration provided in an embodiment of the present invention. This embodiment is a refinement based on the above embodiment. The specific method can be as follows: Figure 4 As shown, the method may include the following steps:

[0039] Step 401: Acquire slit lamp images and perform imaging condition analysis on the slit lamp images according to a pre-determined visual language model to obtain analysis results.

[0040] The analysis results include visual channel weights and semantic guidance channel weights.

[0041] Step 402: Extract multi-scale visual feature maps of slit lamp images through the visual language network of the anomaly perception model; calculate the anomaly saliency score of each pixel position of the multi-scale visual feature map based on the pre-determined prototype features through the visual saliency channel of the anomaly perception model; obtain a set of visual saliency candidate regions based on the anomaly saliency score and the preset score threshold.

[0042] The multi-scale visual feature map is a hierarchical feature representation extracted from slit-lamp images by the visual language network. It includes shallow and deep features. Shallow features include low-level visual information such as image texture and color. Deep features include high-level abstract information such as semantic concepts. Prototype features are typical visual feature vectors of normal anterior segment tissues obtained through pre-training. Prototype features can represent the standard patterns of normal structures such as the cornea, anterior chamber, iris, or lens in feature space. The anomaly saliency score is a numerical indicator that quantifies the degree of deviation from normality at various locations in the slit-lamp image. The visual saliency candidate region set is a set of connected regions extracted from the slit-lamp image based on the anomaly saliency score. Each visual saliency candidate region is accompanied by spatial location coordinates (bounding box) and a corresponding visual feature vector.

[0043] Specifically, the visual language network of the anomaly perception model shares parameters with the visual language model itself. The slit lamp image is input into the anomaly perception model, and its visual language network extracts multi-scale visual feature maps of the slit lamp image. The distance between each pixel location and its corresponding prototype feature in the multi-scale visual feature map is calculated; for example, the Euclidean or cosine distance between the visual feature and the prototype feature can be calculated. Anomaly saliency scores for each pixel location are determined according to a predefined distance scoring mapping function, where greater distances result in higher anomaly saliency scores, and closer distances result in lower anomaly saliency scores. Anomaly pixels with anomaly saliency scores greater than a preset score threshold are extracted, and connected component analysis is performed on these pixels. Spatially adjacent anomaly pixels are formed into connected regions, each representing a potential lesion. Further, candidate visual saliency regions are determined based on the spatial coordinates of each connected region and its corresponding visual feature vector, resulting in a set of candidate visual saliency regions. , This represents the i-th visual saliency candidate region. Each visual saliency candidate region is accompanied by spatial location coordinates (bounding box) and the corresponding visual feature vector.

[0044] Step 403: Through the semantic guidance channel of the anomaly perception model, targeted detection is performed on the slit lamp image based on the illumination mode label, observation area label, and preset clinical examination database to obtain a set of semantically guided candidate regions.

[0045] The clinical examination database stores ophthalmic clinical examination knowledge. This database includes examination checklists arranged according to combinations of illumination methods and observation areas. Each checklist contains a list of items to be emphasized for each illumination method, along with descriptions of visual characteristics and detection parameters for each item. For example, the checklist for corneal observation using the backlight method includes items such as corneal dystrophy (crystalline reflection, polymorphism) and corneal endothelial cell morphology (hexagonal mosaic structure under specular reflection). The semantic guidance channel employs a top-down detection strategy, using ophthalmic clinical examination knowledge to guide the detection process, thereby simulating the standardized behavior of clinicians checking each item on the checklist. Illumination method labels identify the illumination method used by the slit-lamp microscope when acquiring images. Different illumination methods present different characteristics of the anterior segment structure; therefore, the illumination method label determines the direction of subsequent checklist searches. For example, diffuse illumination is suitable for observing the general surface morphology, direct focal illumination displays optical section layers, backlighting highlights the translucent shadows of opaque lesions, and specular reflection reveals fine structures such as endothelial cells. The observation area label is used to identify the main ocular anatomical structures in focus within a slit-lamp image. This label allows for a focused examination area, and together with the illumination method label, it defines the specific items on the examination checklist. For example, when using posterior illumination, the focus may be on scarring and malnutrition when examining the cornea, and on opacity and positional abnormalities when examining the lens.

[0046] Specifically, in the semantic guidance channel of the anomaly perception model, a list of key observation items corresponding to the illumination mode label and observation area label is determined from the clinical examination database. The clinical examination database is then injected into the semantic guidance channel of the anomaly perception model in the form of natural language cues. These cues include imaging conditions, observation areas, and key examination items. The visual language network of the anomaly perception model receives the natural language cues and slit-lamp images, performs cross-modal targeted analysis, and outputs each semantic guidance candidate region. Each semantic guidance candidate region may include spatial location coordinates, semantic category labels, and confidence scores. Further, a set of semantic guidance candidate regions is determined based on each semantic guidance candidate region. , This represents the i-th semantic guidance candidate region.

[0047] Step 404: Perform a fusion analysis on the anomaly detection results of the visual saliency channel and the semantic guidance channel to obtain a structured anomaly candidate set.

[0048] The structured anomaly candidate set is a collection of anomaly candidates obtained by fusing and deduplicating the anomaly detection results from the visual saliency channel and the semantic guidance channel. Each candidate in the structured anomaly candidate set may incorporate detection results from both channels (visual saliency channel and semantic guidance channel). That is, each candidate in the structured anomaly candidate set may originate solely from the visual channel detection results, solely from the semantic guidance channel detection results, or may incorporate detection results from both channels. In this scheme, optionally, the visual saliency channel anomaly detection results and the semantic guidance channel anomaly detection results are fused and analyzed to obtain the structured anomaly candidate set, including the following steps A1-A2:

[0049] Step A1: The visual saliency candidate region set and the semantic guidance candidate region set are fused using the first projection network and the second projection network of the anomaly perception model to obtain the fused feature space.

[0050] The first and second projection networks are used to process the visual saliency candidate region set and the semantic guidance candidate region set, respectively. The network parameters of the first and second projection networks are shared, ensuring that the features of both sets are mapped through the same transformation function, guaranteeing the consistency and comparability of the mapping. The projection network can employ a multilayer perceptron structure, using linear transformations and nonlinear activations to compress high-dimensional sparse visual features into a low-dimensional dense semantic space, while retaining key information distinguishing different lesions. Through the first and second projection networks, the candidate region features of the visual saliency channel and the semantic guidance channel are mapped to the same fused feature space, allowing candidate regions of homologous lesions to naturally cluster, laying the foundation for subsequent clustering and deduplication.

[0051] During the model training phase, a pre-defined contrastive loss function can be used to train the projection network, maximizing the cosine similarity between candidate regions pointing to the same actual lesion and minimizing the cosine similarity between candidate regions corresponding to different lesions. The pre-defined contrastive loss function is: ;in and For positive sample pairs of the same lesion, the projection vector is... Let be the negative sample projection vector, and sim(·,·) be the cosine similarity function. The temperature coefficient is predetermined.

[0052] Step A2: Cluster the fusion feature space based on visual channel weights and semantic guidance channel weights to obtain a fusion candidate region set; encapsulate each candidate region in the fusion candidate region set in a structured manner to obtain a structured anomaly candidate set.

[0053] Specifically, the fusion feature space comprises a set of points consisting of the projected features of all visually saliency candidate regions and semantically guided candidate regions. The anomaly perception model can perform density-based clustering on the feature points, grouping densely connected feature points into the same cluster, and merging each cluster to obtain the fusion candidate regions. Intra-cluster weights are determined based on the visual channel weights and semantically guided channel weights, and a weighted average of the projected features of all candidate regions within a cluster is calculated based on these intra-cluster weights. For example, if the visually saliency channel weight is higher, the visually saliency candidate region features will have a larger proportion in the feature fusion.

[0054] Each fusion candidate region in this scheme includes a detection source identifier, which can include detection by only the visual saliency channel, detection by only the semantic guidance channel, or detection by both channels. If a cluster contains only visual saliency candidate regions, the detection source identifier is the visual saliency channel; if a cluster contains only semantic guidance candidate regions, the detection source identifier is the semantic guidance channel; if a cluster contains candidate regions by both channels, the detection source identifier is detection by both channels. After obtaining the fusion candidate regions, a preliminary anomaly saliency score can be calculated for each fusion candidate region to quantify the degree of abnormality of the corresponding lesion. For example, a preliminary anomaly saliency score can be calculated based on the density characteristics of the cluster, channel weights, and the original single-channel score. Clusters with higher density correspond to higher preliminary anomaly saliency scores, providing a data foundation for subsequent calculation of the comprehensive confidence score.

[0055] Furthermore, each fusion candidate region is encapsulated as a structured anomaly candidate, which includes spatial location coordinates, fusion feature vector, detection source identifier (visual saliency channel / semantic guidance channel / dual-channel co-detection), preliminary anomaly saliency score, and imaging condition label (corresponding to the imaging conditions in step 201). All structured anomaly candidates constitute a structured anomaly candidate set. , This represents the i-th structured anomaly candidate.

[0056] Step 405: Encode the structured anomaly candidate set with multi-granularity visual features using the visual language network of the clinical description model to obtain the joint visual representation vector of each structured anomaly candidate region in the structured anomaly candidate set.

[0057] Multi-granularity visual feature encoding is a technique that simultaneously extracts detailed features of the lesion's local area and its positional context features within the overall image, fusing the two to form a complete visual representation. Specifically, for each structured abnormality candidate region, a local region patch is extracted from the original slit-lamp image, centered on its spatial coordinates. The patch size can be larger than the bounding box of the corresponding candidate region to include sufficient contextual edge information. The visual language network of the clinical description model shares backbone parameters with the visual language model, but is fine-tuned using a low-rank adaptation strategy for clinical text generation tasks. The extracted local region patches are input into the visual language network of the clinical description model to obtain the local visual features output by the visual language network. Simultaneously, the positional context information of the candidate is input into the visual language network to obtain global contextual features. The positional context information includes the candidate's relative position in the overall image, its spatial relationship with surrounding anatomical structures, and imaging condition labels. Furthermore, the local visual features and global contextual features are concatenated and fused to obtain a joint visual representation vector.

[0058] Step 406: Generate standardized descriptive text corresponding to the joint visual representation vector based on the clinical description model and standard terminology constraints; determine standardized clinical description based on the standardized descriptive text and its corresponding structured abnormality candidate regions.

[0059] The clinical description model incorporates a predefined standard terminology constraint table. This table includes: a standard terminology set for anatomical locations (cornea, corneal epithelium, corneal stroma, corneal endothelium, anterior chamber, iris, pupil, lens, anterior lens capsule, lens cortex, and lens nucleus, etc.), a standard terminology set for abnormality types (opacity, infiltration, edema, neovascularization, pigmentation, atrophy, adhesions, defects, and deposits, etc.), a standard descriptive terminology set for morphological features (punctate, patchy, diffuse, localized, ring-shaped, dendritic, and discoid, etc.), and a standard descriptive terminology set for extent (involving the entire thickness, involving a portion, central area, peripheral area, superior, inferior, nasal, and temporal, etc.). The standardized description text in this scheme can be a four-tuple structure: anatomical location + abnormality type + morphological feature + extent.

[0060] After obtaining the joint visual representation vector, the clinical description model selects the data that best matches the joint visual representation vector from the standard terminology constraint table, and obtains a complete descriptive text based on the matched anatomical location and abnormality type. The generated standardized descriptive text is then associated with its corresponding structured abnormality candidate region to form a standardized clinical description. All pairings of standardized clinical descriptions with their corresponding structured abnormality candidate regions constitute a standardized clinical description. .in It is a candidate for structured anomaly. Its corresponding standardized clinical description.

[0061] Step 407: Perform multi-dimensional review of the standardized clinical description using a pre-determined review model. Based on the multi-dimensional review results, visual saliency channel anomaly detection results, semantic guidance channel anomaly detection results, and structured anomaly candidate set, iteratively optimize the standardized clinical description to obtain the target standardized description.

[0062] The standardized description of the target can be directly used to generate a slit-lamp image inspection report. Multi-dimensional review includes visual-text consistency review, missed detection review, adversarial counterexample hypothesis testing, and cross-regional correlation reasoning.

[0063] Specifically, the standardized clinical description text and the original slit-lamp image region corresponding to the candidate are input into the review model. The visual language network of the review model determines the consistency score between the standardized clinical description text and the actual situation (original slit-lamp image region). The consistency score is a value greater than or equal to 0 and less than or equal to 1.

[0064] All detected and preliminarily approved candidate regions are masked on the original slit-lamp image, for example, by covering the corresponding regions with black blocks or blurring. The masked slit-lamp image is then re-input into the anomaly detection model for dual-channel anomaly detection. If previously undetected anomalies are found during the re-detection, the result of the missed detection review is determined to be a missed detection. The missed regions are marked as "re-detected" and added to the structured anomaly candidate set for subsequent generation of standardized clinical descriptions.

[0065] Specifically, cross-regional correlation reasoning includes: a predefined clinical correlation rule table in the server, which describes the accompanying relationships between different abnormalities. Examples include "corneal ulcers are often accompanied by anterior chamber reaction" and "posterior synechiae are often accompanied by anterior lens capsule pigmentation." During review, the review model checks whether the detected abnormal combination triggers the antecedent of the clinical correlation rule (e.g., corneal ulcer). If so, it checks whether the consequent of the rule (accompanying signs, such as anterior chamber reaction) is also in the detection results. If it finds that the antecedent of a rule is satisfied but the consequent is missing, it determines that there may be a missed detection. Further, structured correction instructions can be generated based on the missed detection situation. The correction instructions include a correction type identifier, coordinates of the involved region, and correction confidence level.

[0066] Optionally, the standardized clinical description can be validated against adversarial counterexamples using an auditing model. This includes: evaluating the consistency between the standardized clinical description and its corresponding slit-lamp image in local areas to obtain a description consistency score; evaluating the consistency between the counterexample hypothesis and the local areas to obtain a counterexample consistency score; and validating the standardized clinical description against adversarial counterexamples based on the description consistency score and the counterexample consistency score.

[0067] Counterexample hypotheses are alternative clinical interpretations of the same image region that compete with the original standardized clinical description. Specifically, the standardized clinical description and its corresponding local region from the slit-lamp image are input into the review model. Utilizing the cross-modal understanding capability of the review model's visual language network, the model assesses whether the standardized clinical description accurately reflects the actual visual representation of the image and outputs a consistency score. The review model can generate at least one counterexample hypothesis based on the standardized clinical description and clinical differential diagnostic knowledge. For example, if the standardized clinical description is "lens nucleus opacity," its counterexample hypothesis can be "lens nucleus optical section artifact." Further, the consistency between the generated counterexample hypothesis and the corresponding local region of the slit-lamp image is evaluated to obtain a counterexample consistency score. If the description consistency score is higher than the counterexample consistency score and the difference is significant, it indicates that the standardized clinical description is superior to the counterexample hypothesis, meaning the standardized clinical description passes the adversarial counterexample hypothesis check. If the description consistency score is lower than the counterexample consistency score and the difference is significant, it indicates that the counterexample hypothesis is superior to the standardized clinical description, meaning the standardized clinical description fails the adversarial counterexample hypothesis check. This can further trigger a correction mechanism to re-perform dual-channel anomaly detection or generate standardized clinical descriptions.

[0068] In this scheme, optionally, the target standardized description is obtained by iteratively optimizing the standardized clinical description based on the multi-dimensional review results, the visual saliency channel anomaly detection results, the semantic guidance channel anomaly detection results, and the structured anomaly candidate set, including the following steps B1-B3: Step B1: Determine the individual confidence score based on the multi-dimensional review results, the visual saliency channel anomaly detection results, the semantic guidance channel anomaly detection results, and the structured anomaly candidate set; calculate the comprehensive confidence score based on the individual confidence score, the preset source weight coefficient, and the preset source independence coefficient.

[0069] Specifically, the single-item confidence score includes four scores provided by the multi-dimensional review results: consistency score for visual text consistency verification, supplementary detection confidence score for missed detection review, description relative advantage score for adversarial counterexample hypothesis verification, rule satisfaction score for cross-regional correlation reasoning, and anomaly salience score for each candidate region provided by the visual salience channel anomaly detection results; detection confidence score for each candidate region provided by the semantic guidance channel anomaly detection results; and preliminary anomaly salience score and detection source identifier provided by the structured anomaly candidate set.

[0070] In one optional implementation, a dynamically updated evidence accumulation board is maintained for each structured anomaly candidate, recording source labels, round numbers, detection channel identifiers, and individual confidence scores from different models (anomaly perception model, description generation model, and review model, etc.) and different review rounds. The data recorded on the accumulation board is fused and analyzed to obtain the final comprehensive detection result. Based on the comprehensive detection result, the standardized clinical description is iteratively optimized to obtain the target standardized description used to generate slit-lamp image examination reports. The comprehensive confidence score is calculated based on the source weight coefficient and the source independence coefficient.

[0071] ;in, To calculate the overall confidence score, Score the confidence level of the j-th piece of evidence individually. For source weighting coefficients, This represents the independence coefficient of the evidence source.

[0072] Step B2: Determine whether the standardized clinical description meets the preset conditions based on the rigor parameter and the overall confidence score; if the standardized clinical description does not meet the preset conditions, generate a correction instruction based on the clinical description.

[0073] The correction instructions are used to determine how to iteratively optimize standardized clinical descriptions. These instructions include correction steps, which can be either an anomaly detection step or a description generation step. If the overall confidence score reaches or exceeds the stringency parameter, the standardized clinical description meets the preset conditions and a slit-lamp image examination report can be directly generated from it. If the overall confidence score does not reach the stringency parameter, the standardized clinical description does not meet the preset conditions and requires iterative optimization. If the overall confidence score does not reach the stringency parameter, the reasons for not meeting the preset conditions can be analyzed based on the review results, and structured correction instructions can be generated based on the specific reasons. For example, if the reason for not meeting the preset conditions is incomplete detection, inaccurate localization, or missed detection, the correction step is determined to be an anomaly detection step. If the reason for not meeting the preset conditions is inaccurate description, inappropriate terminology, or semantic deviation, the correction step is a description generation step. Correction instructions can also include fields involving region coordinates, correction type, and correction confidence, thereby providing precise guidance for the iterative optimization process.

[0074] Step B3: If the correction step is an anomaly detection step, the correction instructions and analysis results are input into the anomaly detection model. The steps of visual saliency channel anomaly detection and semantically guided channel anomaly detection of the slit lamp image based on the analysis results using the pre-determined anomaly detection model are repeated until a target standardized description that meets the preset conditions is obtained. If the correction step is a description generation step, the correction instructions and analysis results are input into the clinical description model. The steps of generating standardized clinical descriptions corresponding to the structured anomaly candidate set based on the pre-determined clinical description model and preset standard terminology constraints are repeated until a target standardized description that meets the preset conditions is obtained.

[0075] If the correction step is an anomaly detection step, the correction instructions and analysis results are input into the anomaly detection model, which then performs dual-channel anomaly detection again based on the correction instructions. For example, the detection threshold can be lowered to detect more subtle anomalies, a finer scan can be performed near specified coordinates, or the retrieval range of prior knowledge can be expanded based on the correction instructions. Further, standardized clinical descriptions are generated from the structured anomaly candidate set obtained after the re-detection, resulting in new standardized clinical descriptions. These new standardized clinical descriptions undergo multi-dimensional review until a standardized clinical description that meets preset conditions is obtained, or the preset maximum number of iterations is reached. At this point, iteration stops, and the latest standardized clinical description is determined as the target standardized description.

[0076] If the correction step is a description generation step, the correction instructions and analysis results are input into the clinical description model. The clinical description model then regenerates the standardized clinical description based on the correction instructions. For example, the correction instructions can be used to adjust terminology selection, morphological description, or reassess the scope to obtain a new standardized clinical description. The new standardized clinical description undergoes multi-dimensional review until a standardized clinical description that meets the preset conditions is obtained, or the number of iterations has reached the preset maximum number of iterations. At this point, iteration stops, and the latest standardized clinical description is determined as the target standardized description.

[0077] For example, Figure 5 This is a schematic diagram illustrating the process of iteratively optimizing standardized clinical descriptions as provided in an embodiment of the present invention. Figure 5As shown, standardized clinical descriptions undergo visual text consistency review, missed detection review, adversarial counterexample hypothesis testing, and cross-regional correlation reasoning. The process data is recorded on an evidence accumulation board, which includes source labels, round numbers, detection channels, and individual confidence scores. Multi-source evidence fusion is performed based on the evidence accumulation board data to obtain a comprehensive confidence score. If the comprehensive confidence score is greater than or equal to the stringency parameter, the standardized clinical description is determined as the target standardized description. If the comprehensive confidence score is less than the stringency parameter, iterative optimization is performed, and multi-source evidence fusion is re-performed based on the evidence accumulation board data to obtain a new comprehensive confidence score. Through multi-dimensional review by the review model and multiple rounds of evidence accumulation and confidence fusion, closed-loop quality control can be achieved. Furthermore, the adversarial testing simulates the differential diagnosis exclusion method, effectively identifying potential misjudgments.

[0078] Step 408: Generate a slit lamp image inspection report based on the target standardized description.

[0079] The technical solution of this embodiment discovers unknown abnormalities through a visual saliency channel and identifies known types of abnormalities through a semantic guidance channel. By fusing the anomaly detection results from both channels through clustering, it eliminates the detection blind spots of a single channel, and has better coverage, especially for early subtle lesions and atypical manifestations. By limiting the range of values ​​for anatomical location, abnormality type, morphological features, and extent through preset standard terminology constraints, and by generating standardized clinical descriptions using a four-tuple structure, it solves the problems of inaccurate terminology and inconsistent structure in text generated by general visual language models. A comprehensive confidence score is calculated based on multi-dimensional review results, visual saliency channel anomaly detection results, semantic guidance channel anomaly detection results, and a structured anomaly candidate set. Iterative optimization is then performed based on the comprehensive confidence score, which can simulate the review behavior of doctors, effectively identify potential misjudgments, and significantly improve the reliability of the model output.

[0080] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, with reference to... Figure 5 , Figure 5 The electronic device 12 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application. Figure 5 As shown, the electronic device 12 is represented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0081] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0082] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0083] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0084] A program / utility 40 having a set (at least one) of program modules 46 may be stored, for example, in system memory 28. Such program modules 46 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 46 typically perform the functions and / or methods described in the embodiments of this application.

[0085] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 12, and / or with any device that enables electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that, although... Figure 5 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0086] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28. For example, it implements a slit lamp image analysis method based on multi-agent collaboration provided in this embodiment of the invention: acquiring a slit lamp image and performing imaging condition analysis on the slit lamp image according to a predetermined visual language model to obtain analysis results; using a predetermined anomaly perception model, performing visual saliency channel anomaly detection and semantic guidance channel anomaly detection on the slit lamp image based on the analysis results, and performing fusion analysis on the visual saliency channel anomaly detection results and semantic guidance channel anomaly detection results to obtain a structured anomaly candidate set; generating a standardized clinical description corresponding to the structured anomaly candidate set based on a predetermined clinical description model and preset standard terminology constraints; performing multi-dimensional review of the standardized clinical description through a predetermined review model, and iteratively optimizing the standardized clinical description based on the multi-dimensional review results, visual saliency channel anomaly detection results, semantic guidance channel anomaly detection results, and structured anomaly candidate set to obtain a target standardized description; and generating a slit lamp image inspection report based on the target standardized description.

[0087] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a multi-agent collaborative slit-lamp image analysis method as provided in all embodiments of this invention: acquiring a slit-lamp image and performing imaging condition analysis on the slit-lamp image according to a pre-determined visual language model to obtain analysis results; using a pre-determined anomaly perception model, performing visual saliency channel anomaly detection and semantic guidance channel anomaly detection on the slit-lamp image based on the analysis results, and performing fusion analysis on the visual saliency channel anomaly detection results and semantic guidance channel anomaly detection results to obtain a structured anomaly candidate set; generating standardized clinical descriptions corresponding to the structured anomaly candidate set based on a pre-determined clinical description model and preset standard terminology constraints; performing multi-dimensional review of the standardized clinical descriptions using a pre-determined review model, and iteratively optimizing the standardized clinical descriptions based on the multi-dimensional review results, visual saliency channel anomaly detection results, semantic guidance channel anomaly detection results, and the structured anomaly candidate set to obtain a target standardized description; and generating a slit-lamp image inspection report based on the target standardized description. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, semiconductor electronic devices, apparatuses, or devices, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an electronic device, apparatus, or device that can be executed by instructions.

[0088] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in conjunction with an electronic device, apparatus, or device that executes instructions.

[0089] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0090] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0091] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A slit-lamp image analysis method based on multi-agent collaboration, characterized in that, The method includes: Acquire slit lamp images and perform imaging condition analysis on the slit lamp images according to a pre-determined visual language model to obtain analysis results; wherein, the analysis results include visual channel weights and semantic guidance channel weights; Using a pre-determined anomaly perception model, visual saliency channel anomaly detection and semantic guidance channel anomaly detection are performed on the slit lamp image based on the analysis results. The results of visual saliency channel anomaly detection and semantic guidance channel anomaly detection are then fused and analyzed to obtain a structured anomaly candidate set. Standardized clinical descriptions corresponding to the structured anomaly candidate set are generated based on a pre-determined clinical description model and preset standard terminology constraints. The standardized clinical description is reviewed in multiple dimensions using a pre-determined review model. Based on the results of the multi-dimensional review, the results of the visual saliency channel anomaly detection, the results of the semantic guidance channel anomaly detection, and the candidate set of structured anomalies, the standardized clinical description is iteratively optimized to obtain the target standardized description. A slit lamp image inspection report is generated based on the standardized description of the target.

2. The method according to claim 1, characterized in that, The analysis results include illumination mode labels and observation area labels; based on a pre-determined visual language model, the slit lamp image is subjected to imaging condition analysis to obtain the analysis results, including: The slit lamp image is standardized, and the standardized slit lamp image is analyzed using the visual language model to obtain the lighting mode label and the observation area label. The visual channel weight and the semantic guidance channel weight are determined based on the illumination mode label and the preset imaging condition weight information.

3. The method according to claim 2, characterized in that, Based on the analysis results, visual saliency channel anomaly detection and semantic guidance channel anomaly detection are performed on the slit lamp image, including: The visual language network of the anomaly perception model is used to extract multi-scale visual feature maps of the slit lamp image; The anomaly saliency score of each pixel position in the multi-scale visual feature map is calculated based on the pre-determined prototype features through the visual saliency channel of the anomaly perception model; a set of visual saliency candidate regions is obtained based on the anomaly saliency score and a preset score threshold. The anomaly perception model uses a semantic guidance channel to perform targeted detection on the slit lamp image based on the lighting mode label, the observation area label, and a preset clinical examination database, thereby obtaining a set of semantically guided candidate regions.

4. The method according to claim 3, characterized in that, The results of anomaly detection from the visual saliency channel and the semantic guidance channel are fused and analyzed to obtain a candidate set of structured anomalies, including: The visual saliency candidate region set and the semantic guidance candidate region set are fused using the first projection network and the second projection network of the anomaly perception model to obtain a fused feature space; the network parameters of the first projection network and the second projection network are shared. Based on the visual channel weights and the semantic guidance channel weights, the fusion feature space is clustered and deduplicated to obtain a fusion candidate region set; The candidate regions of the fusion candidate region set are structurally encapsulated to obtain the structured anomaly candidate set.

5. The method according to claim 1, characterized in that, Based on a pre-determined clinical description model and preset standard terminology constraints, standardized clinical descriptions corresponding to the structured anomaly candidate set are generated, including: The structured anomaly candidate set is encoded with multi-granularity visual features using the visual language network of the clinical description model to obtain the joint visual representation vector of each structured anomaly candidate region in the structured anomaly candidate set. Based on the clinical description model and the standard terminology constraints, a standardized descriptive text corresponding to the joint visual representation vector is generated; The standardized clinical description is determined based on the standardized descriptive text and its corresponding structured abnormality candidate regions.

6. The method according to claim 1, characterized in that, The multi-dimensional review includes visual text consistency review, missed detection review, adversarial counterexample hypothesis testing, and cross-regional correlation reasoning; the standardized clinical description is reviewed in multiple dimensions using a pre-defined review model, including: Generate counterexample hypotheses for the standardized clinical description; A consistency evaluation is performed on the standardized clinical description and its corresponding local region of the slit-lamp image to obtain a description consistency score; a consistency evaluation is performed on the counterexample hypothesis and the local region to obtain a counterexample consistency score. The standardized clinical description is subjected to the adversarial counterexample hypothesis test based on the description consistency score and the counterexample consistency score.

7. The method according to claim 1, characterized in that, The analysis results include a stringency parameter; based on the multi-dimensional review results, the visual saliency channel anomaly detection results, the semantic guidance channel anomaly detection results, and the structured anomaly candidate set, the standardized clinical description is iteratively optimized to obtain the target standardized description, including: Based on the multi-dimensional review results, the visual saliency channel anomaly detection results, the semantic guidance channel anomaly detection results, and the structured anomaly candidate set, a single confidence score is determined; a comprehensive confidence score is calculated based on the single confidence score, a preset source weight coefficient, and a preset source independence coefficient. Based on the rigor parameter and the comprehensive confidence score, it is determined whether the standardized clinical description meets the preset conditions; if the standardized clinical description does not meet the preset conditions, a correction instruction is generated based on the clinical description; the correction instruction includes a correction step; the correction step is an anomaly detection step or a description generation step. If the correction step is the anomaly perception step, then the correction instruction and the analysis result are input into the anomaly perception model, and the steps of performing visual saliency channel anomaly detection and semantically guided channel anomaly detection on the slit lamp image based on the analysis result through the predetermined anomaly perception model are repeated until the target standardized description that meets the preset conditions is obtained. If the correction step is the description generation step, then the correction instruction and the analysis result are input into the clinical description model, and the step of generating the standardized clinical description corresponding to the structured abnormality candidate set based on the predetermined clinical description model and preset standard terminology constraints is repeated until the target standardized description that meets the preset conditions is obtained.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the slit lamp image analysis method based on multi-agent cooperation as described in any one of claims 1-7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the slit lamp image analysis method based on multi-agent collaboration as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the slit lamp image analysis method based on multi-agent collaboration as described in any one of claims 1-7.

Images