A corneal fungal infection diagnosis method and system based on multi-modal time sequence information

Abstract

The present application relates to the technical field of medical image processing and intelligent auxiliary diagnosis, and particularly discloses a corneal fungal infection diagnosis method and system based on multi-modal time sequence information, which comprises the following steps: acquiring an image sequence and clinical data; inputting the image sequence and the clinical data into a multi-modal fusion analysis model, which extracts spatial form and space-time dynamic characteristics of the image and a clinical feature vector; fusing the two types of features to generate diagnosis features; and finally outputting a diagnosis result containing mycelium existence determination and judgment basis. The present application has higher diagnosis accuracy, stronger robustness, and good interpretability and interactivity, and simulates and enhances the comprehensive diagnosis thinking process of doctors by fusing space-time dynamic information of the image sequence and patient clinical context.

Description

Technical Field

[0001] This invention relates to the field of medical image processing and intelligent assisted diagnosis technology, and in particular to a method and system for diagnosing corneal fungal infections based on multimodal temporal information. Background Technology

[0002] Fungal keratitis is an eye infection with a high rate of blindness. Early and accurate diagnosis is crucial for treatment. Confocal microscopy is a key in vivo, non-invasive examination method. However, in corneal confocal images, fungal hyphae and corneal nerve fibers are highly similar in morphology, both being long, branched linear structures, which can easily lead to misdiagnosis even by experienced doctors.

[0003] Existing technologies (such as the patent with publication number CN113591601B) propose segmenting hyphae and nerve fibers using deep learning models and combining morphological rules for post-processing correction, achieving automated recognition. However, existing learning models generally have the following limitations: First, they rely solely on a single static image, failing to utilize the dynamic characteristics of hyphae such as growth and extension, and also failing to integrate rich patient clinical information, resulting in a gap with doctors' comprehensive decision-making models. Second, post-processing correction relies on preset fixed morphological thresholds, which have insufficient generalization ability for different imaging devices and image qualities. Third, the model output is a segmentation mask, lacking interpretable explanations of the diagnostic decision-making basis, and does not consider interaction with doctors' workflows.

[0004] Therefore, there is a need for an intelligent diagnostic solution that can integrate multi-dimensional information, has dynamic analysis capabilities, and is more clinically applicable. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide an intelligent diagnostic method and system for corneal fungal infections with higher diagnostic accuracy, stronger robustness, and good interpretability and interactivity.

[0006] To achieve the above objectives, in one aspect, the present invention provides a method for diagnosing corneal fungal infections based on multimodal temporal information, comprising:

[0007] Acquire confocal microscopy image sequences of the cornea to be diagnosed and clinical information data of the patient;

[0008] The image sequences and clinical information data are input into a trained multimodal fusion analysis model, wherein the multimodal fusion analysis model is configured as follows:

[0009] Based on the image sequence, image fusion features containing spatial morphological features and spatiotemporal dynamic features are extracted; the spatiotemporal dynamic features include at least one of the growth vector, texture evolution consistency and branch event detection features calculated from the image sequence.

[0010] Based on the aforementioned clinical information data, extract clinical feature vectors;

[0011] The image fusion features are fused with the clinical feature vector to generate diagnostic features;

[0012] Based on the diagnostic features, a diagnostic result is output. The diagnostic result includes at least a determination of whether hyphae exist in the image sequence, and supporting information for the determination. The supporting information includes key discriminant factors based on the image fusion features and / or clinical feature vectors.

[0013] Furthermore, the step of extracting image fusion features containing spatial morphological features and spatiotemporal dynamic features based on the image sequence specifically includes:

[0014] The image sequence is subjected to inter-frame registration and preprocessing;

[0015] The preprocessed image sequence is input into a dual-path 3D convolutional neural network, where:

[0016] The network structure of the first path is used to extract the spatial morphological features of a single frame image;

[0017] The network structure of the second path is used to extract the spatiotemporal dynamic features of cross-frame images;

[0018] The image fusion feature is obtained by fusing the features output from the first path and the second path.

[0019] Furthermore, the network structure of the second path includes a three-dimensional convolutional layer or a convolutional long short-term memory network layer.

[0020] Furthermore, after outputting the diagnostic results, the method further includes:

[0021] If the confidence score in the diagnostic result is lower than a preset threshold, the case is marked as requiring manual review.

[0022] Provides an interactive interface to receive user input for corrections to the hyphal identification region in the diagnostic results;

[0023] Based on the corrected input, the diagnostic results of the current case are optimized in real time through an online adaptation model.

[0024] Furthermore, the method also includes:

[0025] The corrected input and the corresponding image sequences and clinical information data are stored in the feedback database;

[0026] The multimodal fusion analysis model is incrementally learned periodically using data from the feedback database.

[0027] Furthermore, the clinical information data includes at least one of the following: patient age, history of trauma, history of contact lens wear, chief complaint, and past medical history;

[0028] The step of extracting clinical feature vectors based on the clinical information data includes performing natural language processing on the chief complaint text to extract key symptom features.

[0029] Furthermore, the output diagnostic results also include:

[0030] Generate hyphae segmentation maps and nerve fiber segmentation maps corresponding to the image sequence;

[0031] A visualization heatmap is generated to highlight the region that contributes most to the spatiotemporal dynamic features in at least one frame of the image sequence.

[0032] On the other hand, the present invention also provides an intelligent diagnostic system for corneal fungal infections based on multimodal temporal information, comprising:

[0033] The data acquisition module is used to acquire the corneal confocal microscopy image sequence to be diagnosed and the patient's clinical information data;

[0034] The multimodal fusion analysis module, including a trained multimodal fusion analysis model, is used for:

[0035] Based on the image sequence, image fusion features containing spatial morphological features and spatiotemporal dynamic features are extracted; the spatiotemporal dynamic features include at least one of the growth vector, texture evolution consistency and branch event detection features calculated from the image sequence.

[0036] Based on the aforementioned clinical information data, extract clinical feature vectors;

[0037] The image fusion features are fused with the clinical feature vector to generate diagnostic features;

[0038] The result output module is used to output a diagnostic result based on the diagnostic features. The diagnostic result includes at least a determination of whether hyphae exist in the image sequence and supporting information for the determination. The supporting information includes key discriminant factors based on the image fusion features and / or clinical feature vectors.

[0039] Furthermore, it also includes an interaction optimization module, used for:

[0040] If the confidence score included in the diagnostic results is lower than a preset threshold, the case is marked as requiring manual review.

[0041] The system receives user feedback on the mycelium identification area through an interactive interface.

[0042] An online adaptation model is invoked to optimize the diagnosis of the current case in real time based on the corrected input;

[0043] The corrected data is stored in the feedback database for incremental learning of the multimodal fusion analysis model.

[0044] Compared with existing technologies, the corneal fungal infection diagnosis method and system based on multimodal temporal information proposed in this invention have the following beneficial technical effects:

[0045] I. By fusing the temporal dynamic features of images with the patient's clinical information, it provides richer criteria for discrimination than a single static image, effectively reducing the confusion rate between hyphae and nerve fibers, and is particularly effective in recognizing atypical hyphal morphology, thus significantly improving diagnostic accuracy.

[0046] Second, spatiotemporal analysis has a higher tolerance for the quality of single-frame images (such as slight motion blur), and decision-making combined with clinical information reduces misjudgments caused by fluctuations in image quality, thus enhancing the robustness of the system.

[0047] Third, the judgment criteria for the output (static morphology + dynamic evidence + clinical indications) make the AI ​​decision-making process more transparent, easier for doctors to understand and trust, meet the development requirements of medical AI, and have clinical interpretability.

[0048] Fourth, the introduction of a human-computer interaction feedback mechanism enables the system to continuously learn and optimize from actual clinical use, and adapt to the diagnostic habits of different medical institutions or doctors, achieving closed-loop learning and personalized adaptation, thus making it more practical. Attached Figure Description

[0049] Figure 1 This is a flowchart of the diagnostic method of the present invention;

[0050] Figure 2 This is a diagram showing the composition of the diagnostic system of the present invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Example 1

[0053] This embodiment provides a method for diagnosing corneal fungal infections based on multimodal temporal information, including the following steps:

[0054] Acquire confocal microscopy image sequences of the cornea to be diagnosed and clinical information data of the patient;

[0055] Image sequences and clinical information data are input into a trained multimodal fusion analysis model, which is configured as follows:

[0056] Based on image sequences, image fusion features containing spatial morphological features and spatiotemporal dynamic features are extracted; the spatiotemporal dynamic features include at least one of the following: growth vector calculated from the image sequence, texture evolution consistency, and branch event detection features;

[0057] Extract clinical feature vectors based on clinical information data;

[0058] By fusing image fusion features with clinical feature vectors, diagnostic features are generated.

[0059] Based on diagnostic features, a diagnostic result is output. The diagnostic result includes at least the determination of whether hyphae exist in the image sequence, and the supporting information for the determination. The supporting information includes key discriminant factors based on image fusion features and / or clinical feature vectors.

[0060] Specifically, during the data acquisition phase, image sequences can be acquired by using confocal microscopy to perform volumetric scanning of the patient's cornea, obtaining multiple consecutive two-dimensional images of the same region along either the depth (Z-axis) or time (T-axis), forming a three-dimensional image stack (X, Y, Z / T). Clinical data acquisition involves extracting structured electronic medical record data from the hospital information system. Clinical information data includes at least one of the following: patient age, history of trauma, history of contact lens wear, chief complaint text, and past medical history.

[0061] When training the multimodal fusion analysis model, the preferred loss function is the focal loss function, and its calculation formula for the loss term Li for pixel i is as follows:

[0062] ;

[0063] Where, p i α is the probability that the model predicts that the pixel belongs to the true class. t The weighting factor is set according to the category frequency. γ is a focusing parameter that is greater than zero, used to reduce the weight of easily classified samples, so that the model training focuses on the difficult-to-classify hyphae and nerve fiber pixels.

[0064] During preprocessing, the image sequence is registered between frames to eliminate displacement caused by the patient's eye movements; image denoising (such as non-local mean denoising) is performed to improve the signal-to-noise ratio; and clinical data is standardized, such as normalizing age and one-hot encoding categorical variables (such as "history of vegetative trauma": yes / no).

[0065] Multimodal feature extraction is the core operation of the model. Image feature extraction can be achieved through a dual-path 3D convolutional neural network. The spatial path can use 3D convolutional kernels to operate in the local spatial region of a single frame and adjacent frames, mainly extracting the static morphological features of hyphae / nerve fibers, such as line thickness, curvature, edge sharpness, and local texture. The spatiotemporal path uses 3D convolutional or ConvLSTM (convolutional long short-term memory) layers, whose convolutional kernels have a larger span in the time dimension, and are specifically used to capture dynamic evolution features across frames. For example, growth vectors are calculated by tracking the displacement of suspected structural endpoints in the sequence, and texture evolution consistency is calculated by calculating the temporal similarity changes of texture features around linear structures. This path enables the model to present the growth, branching, or morphological change process of structures. The features extracted by the two paths are fused in the deep layers of the network to form a unified image fusion feature vector.

[0066] In an image sequence that has undergone inter-frame registration, for the endpoint P identified as a linear skeleton in frame t... t Find its matching endpoint P within the predetermined search radius R in frame t+1. t+1 Then the growth vector of this segment structure Through formula The calculations show that the modulus and orientation angle represent the growth rate and direction, respectively.

[0067] In an image sequence, a strip-shaped region of interest is defined for the linear structure, and the texture feature vector F(t) of this region is extracted for each frame. The temporal stability of the texture is quantified by calculating the similarity S(t) of the feature vectors between consecutive frames, such as cosine similarity or correlation coefficient. .

[0068] Perform topological analysis on the linear skeleton of each frame in the image sequence to identify branch points; through cross-frame matching, record a new branch point that appears in the (t+1)th frame and has no matching branch point within the tolerance range of the corresponding position in the tth frame as a branch event.

[0069] During clinical feature extraction, numerical / categorical data (such as age and medical history) are encoded and directly used as a subset of features. For textual data (such as "chief complaint: eye pain, photophobia for 3 days"), key symptom features are extracted using natural language processing techniques (such as word embedding models). All processed clinical data are then concatenated to form a clinical feature vector.

[0070] Feature fusion involves concatenating the image fusion feature vector obtained above with the clinical feature vector along the feature dimensions to form a more comprehensive multimodal joint feature vector. For example, this joint feature vector is input into a fully connected decision network. This network learns complex weights during training, weights and combines the features, and finally outputs a binary classification result (presence / absence of hyphae) and its confidence level, as well as the basis for the judgment: the system can identify the feature dimensions that contribute the most to the final judgment; for example, it can be traced back to find that the directional growth feature detected by the spatiotemporal path and the history of plant-related trauma in the clinical data jointly led to the "positive" judgment. These key discriminant factors are organized into readable basis information.

[0071] As a preferred approach, feature fusion is implemented through a cross-modal attention module. Specifically, the clinical feature vector is used as the query vector Q, and the image fusion features are used as the key vector K and value vector V. After calculating the attention weights, the output is:

[0072] ;

[0073] Where, d k The dimension of the key vector is denoted as ; the output of the attention weights represents the image features weighted by clinical information, which are then fused with the original features to generate diagnostic features.

[0074] Finally, a structured diagnostic report is generated, which clearly includes the diagnostic conclusion and the key evidence supporting the conclusion. The evidence section will cite both image-found evidence (e.g., "a bamboo-like, directional linear structure was detected") and clinical evidence (e.g., "the patient has a clear history of crop injury").

[0075] This method introduces spatiotemporal dynamic features into automated diagnosis for the first time. The growth of hyphae is one of the most fundamental biological differences between hyphae and static nerve fibers. By analyzing image sequences to capture this characteristic, the fundamental problem of the extreme similarity in morphology between the two in a single image is solved, greatly reducing misjudgment. Clinical data provides crucial context for image analysis. For example, when image features are ambiguous, a medical history of "wearing contact lenses" will significantly increase the system's vigilance for "infection". This cross-validation of images and clinical information simulates the comprehensive decision-making thinking of senior experts, upgrading diagnosis from simply reading pictures to comprehensive judgment.

[0076] Most existing deep learning models are black box systems, while this method requires outputting the basis for judgment. This allows doctors not only to see the results, but also to understand the "reasons" behind the AI's judgment (whether it is based on growth signs in the image or on a clear history of trauma). This transparency greatly increases clinicians' trust and acceptance of AI-assisted diagnosis, which is in line with the core development requirements of medical AI.

[0077] Relying solely on image morphology, atypical hyphae (such as those that are too curved or too short) are easily missed. However, in this method, clinical feature vectors act as risk indicators. Even if the image features are weak, strong clinical indications (such as a high-risk medical history) will prompt the system to conduct in-depth analysis or provide a "uncertain, review recommended" prompt, thereby reducing the risk of missed diagnosis and significantly improving the overall robustness of the system.

[0078] In this embodiment, based on the image sequence, image fusion features containing spatial morphological features and spatiotemporal dynamic features are extracted, which may specifically include:

[0079] First, the image sequence undergoes inter-frame registration and preprocessing to ensure spatial alignment and improve image quality, providing stable and clean input for subsequent quantitative analysis. During image acquisition, minor eye movements are unavoidable, leading to non-rigid deformations such as translation and rotation between consecutive frames. This step calculates the transformation model (e.g., affine transformation) between adjacent frames using image registration algorithms (e.g., feature point-based matching such as SIFT, ORB, or optical flow), aligning all frames to a common reference coordinate system. This is a prerequisite for calculating "spatiotemporal dynamic features" (e.g., growth vectors); otherwise, displacement will be misinterpreted as growth. Preprocessing primarily involves image denoising, employing filtering algorithms suitable for medical images, such as nonlocal mean filtering or wavelet thresholding, to suppress noise while preserving as much edge and texture detail as possible in hyphae and nerve fibers.

[0080] The preprocessed image sequence is input into a dual-path 3D convolutional neural network, where: the network structure of the first path is used to extract the spatial morphological features of a single frame image; and the network structure of the second path is used to extract the spatiotemporal dynamic features of images spanning multiple frames.

[0081] The first path mainly consists of multiple stacked 3D convolutional layers, activation functions, and pooling layers. The 3D convolutional kernels slide along the three dimensions (X, Y, Z). The 3D convolutional kernels have a large receptive field in the spatial dimension (X, Y), used to capture the cross-sectional morphology (such as thickness and edges), local texture (such as bamboo-like appearance), and orientation of hyphae / nerve fibers. In the temporal dimension (Z), the receptive field is smaller (usually 1 or 2), its main function being to integrate contextual information from adjacent 1-2 frames to enhance single-frame features; the core objective is to extract the static or quasi-static appearance attributes of the target. The second path also uses 3D convolutional layers, but its convolutional kernel design has a larger span in the temporal dimension (Z). The temporal span of 3D convolutions or ConvLSTM can learn the patterns of feature evolution between frames, thereby directly capturing dynamic processes such as "structural extension," "brightness propagation," and "branching," and quantifying them into features such as "growth vectors" and "texture evolution." Preferably, the network structure of the second path includes three-dimensional convolutional layers or convolutional long short-term memory network layers.

[0082] Then, the features output by the first path and the second path are fused to obtain the image fusion features. The two paths are usually fused in the deep layers of the network (e.g., at the end of the encoder). The feature maps output by the two paths are spliced ​​in the channel dimension or weighted and fused through an attention mechanism to form a unified image fusion feature that contains both rich spatial details and temporal evolution information.

[0083] In this embodiment, after outputting the diagnostic results, the method further includes:

[0084] First, if the confidence score in the diagnosis is lower than a preset threshold, the case is marked as requiring manual review.

[0085] Secondly, an interactive interface is provided to receive user input for corrections to the hyphal recognition area in the diagnostic results;

[0086] Furthermore, based on the corrected input, the diagnostic results of the current case are optimized in real time through an online adaptation model.

[0087] Specifically, after outputting the diagnostic result, the model outputs a scalar value between 0 and 1 (e.g., obtained through the Softmax function) representing its confidence level in judging "the presence of hyphae," i.e., the confidence score. An adjustable confidence threshold is preset (e.g., 0.85), which can be set by analyzing the model error rate on the validation set, with the goal of capturing most of the potentially erroneous "fuzzy" cases. The system automatically compares the confidence score of the diagnostic result with the preset threshold. If the score is lower than the threshold, it is determined that the model's judgment uncertainty for that case is high, and its status is automatically marked as "requiring manual review," and pushed to the doctor's interactive workflow queue.

[0088] The system provides a graphical user interface that displays the following side-by-side to doctors: the original / preprocessed image sequences, the AI-generated preliminary hyphae / nerve fiber segmentation map (often displayed in overlays of different colors), and the diagnostic conclusion. On this interface, doctors can use tools (such as a pen and eraser) to fine-tune the AI's segmentation results. For example, they can relabel nerve fiber regions that the AI ​​misidentified as hyphae as "nerve fibers" or "background," or manually outline small or blurry hyphae regions that the AI ​​failed to recognize and label them as "hyphae." Every edit performed by the user is captured by the system in real time, recorded as pixel-level category correction labels, and associated with the corresponding image region coordinates.

[0089] The online adaptation model is a lightweight neural network (e.g., a small convolutional network or a set of adaptation layers) whose parameters can be updated in a very short time. It can re-input the image sequence of the current case into the multimodal fusion analysis model to obtain the feature maps of the intermediate layers. It can also use the region corrected by the doctor as the "gold standard" ground value to calculate the prediction loss of the current model in that region (such as cross-entropy loss). This allows the system to learn in real time for the special circumstances of the current patient (such as rare hyphal morphology or special image artifacts). After optimization, the system immediately uses the fine-tuned model weights to re-infer the entire image of the case, generating updated and more accurate segmentation and diagnosis results, and providing real-time feedback on the interface.

[0090] Furthermore, each manual correction is a high-quality, labeled dataset targeting difficult examples. Systematically collecting this data allows for periodic incremental learning of the main model, enabling it to continuously learn from its own errors and expert corrections. This makes the entire system smarter with use, creating a virtuous cycle of continuous performance improvement. Therefore, this method also includes the following steps:

[0091] First, the corrected input and the corresponding image sequences and clinical information data are stored in the feedback database;

[0092] Then, the multimodal fusion analysis model is incrementally learned periodically using data from the feedback database.

[0093] In this embodiment, the output diagnostic results also include:

[0094] First, hyphae and nerve fiber segmentation maps corresponding to the image sequence are generated. To facilitate observation by doctors, pseudo-color overlay display technology can be used. All pixels identified as "hyphae" by the model are highlighted with a striking color (such as bright green or red), and all pixels identified as "nerve fibers" are highlighted with another distinct color (such as blue or yellow). The generated segmentation mask (pseudo-color) is overlaid on the original corneal confocal grayscale image with a certain transparency (such as 50%). In this way, doctors can simultaneously see the original image details and the two structures automatically delineated by AI, and intuitively evaluate the completeness and accuracy of the AI ​​recognition results.

[0095] Furthermore, after obtaining the hyphal segmentation diagram, morphological optimization can be performed: the segmentation diagram is converted into a graph structure G=(V,E), where the vertex set V represents the skeleton intersections and endpoints, and the edge set E represents the skeleton segments connecting the vertices; an energy function is defined.

[0096] ;

[0097] Here, ϕ(v) is a data term, which is related to the predicted probability of the model at vertex v; ψ(u,v) is a smoothing term, which penalizes discontinuous abrupt changes in curvature and width of adjacent edges (u,v); by minimizing the energy function E(G), the hyphal segmentation map is optimized, thereby effectively removing isolated noise points and connecting broken hyphae, making the segmentation results more consistent with medical understanding, and improving the usability and accuracy of the segmentation results.

[0098] Then, a visualization heatmap is generated to highlight the regions that contribute most to the spatiotemporal dynamic features in at least one frame of the image sequence. After the model performs forward propagation on the hyphae, the feature maps of the convolutional layers at the end of the spatiotemporal path are obtained. By calculating the gradient of the category relative to these feature maps, the weight of each feature map is obtained. The larger the weight, the greater the contribution of the feature map to the judgment of "hyphae presence". These weighted feature maps are summed and upsampled to restore them to the original image size to generate a heatmap. In this heatmap, the highlighted areas (such as red) represent those parts in the image sequence whose cross-frame dynamic change patterns (such as growth, texture evolution) play a key supporting role in the model's determination of "hyphae presence". These are usually areas of active hyphal growth or new branching.

[0099] Example 2

[0100] Corresponding to Example 1, this example provides an intelligent diagnostic system for corneal fungal infections based on multimodal temporal information, comprising:

[0101] The data acquisition module is used to acquire the corneal confocal microscopy image sequence to be diagnosed and the patient's clinical information data;

[0102] The multimodal fusion analysis module, including a trained multimodal fusion analysis model, is used for:

[0103] Based on image sequences, image fusion features containing spatial morphological features and spatiotemporal dynamic features are extracted; the spatiotemporal dynamic features include at least one of the following: growth vector calculated from the image sequence, texture evolution consistency, and branch event detection features;

[0104] Extract clinical feature vectors based on clinical information data;

[0105] By fusing image fusion features with clinical feature vectors, diagnostic features are generated.

[0106] The results output module is used to output diagnostic results based on diagnostic features. The diagnostic results include at least the determination of whether hyphae exist in the image sequence and the basis information used to support the determination. The basis information includes key discriminant factors based on image fusion features and / or clinical feature vectors.

[0107] Specifically, the data acquisition module serves as the interface between the system and external data sources. It is responsible for receiving and standardizing all input data, including the software interface integrated into the confocal microscope workstation for directly reading and parsing image sequence files output by the microscope, and the data interface connected to the hospital information system (HIS / EMR) to automatically acquire structured patient clinical information via standard protocols (such as HL7, FHIR) or database access. This module internally includes data validation and caching logic to ensure the integrity and format of the input data meet the requirements of downstream processing modules.

[0108] The multimodal fusion analysis module carries the core algorithm of the entire intelligent diagnosis, responsible for feature extraction, fusion, and inference. It exists in software form and runs on hardware with powerful parallel computing capabilities, such as servers, workstations, or dedicated AI computing cards equipped with GPUs (graphics processing units). The core of this module is a pre-trained and fixed multimodal fusion analysis model, which integrates: an image feature extraction sub-model, which can be a dual-path 3D convolutional neural network capable of extracting spatial and spatiotemporal features in parallel; a clinical feature extraction sub-model, which includes an encoder for structured data and a natural language processing engine for text; and a feature fusion and decision sub-model, which can be a fully connected neural network responsible for fusing the feature vectors output by the former two and mapping them to the final diagnostic features and results.

[0109] After receiving the data, the multimodal fusion analysis module automatically calls the corresponding sub-models and sequentially performs image preprocessing, dual-path feature extraction, clinical feature vectorization, feature fusion, and decision analysis to finally generate intermediate results containing diagnostic conclusions and confidence levels.

[0110] The output module is responsible for transforming the raw data results generated by the analysis module into a final form that doctors can understand and use; receiving the diagnostic features output by the analysis module and decoding them into explicit diagnostic conclusion text (such as "suggests fungal keratitis") and confidence scores; generating hyphae segmentation maps and nerve fiber segmentation maps by calling image processing libraries based on pixel-level segmentation results; generating a visual heatmap using class activation mapping technology to highlight the key areas that contribute the most to dynamic features; and automatically generating a structured electronic diagnostic report with illustrations by integrating diagnostic conclusions, confidence scores, visual images, and key discriminant factors parsed from diagnostic features.

[0111] The three modules are connected in series via a predefined internal data bus or API interface; the output of the data acquisition module is the input of the multimodal fusion analysis module; the output of the analysis module is the input of the result output module, and the whole system forms an automated processing pipeline from data input to report output.

[0112] In this embodiment, an interaction optimization module can also be set up for:

[0113] If the confidence score in the diagnostic results is lower than a preset threshold, the case is marked as requiring manual review.

[0114] The system receives user feedback on the mycelium identification area through an interactive interface.

[0115] An online adaptation model is invoked to optimize the diagnosis of the current case in real time based on the corrected input;

[0116] The corrected data is stored in the feedback database for incremental learning of the multimodal fusion analysis model.

[0117] The interaction optimization module facilitates human-machine collaboration and system self-evolution, and may include:

[0118] The confidence assessment and review triggering submodule listens to the diagnostic results from the result output module, extracts the confidence score, and compares the score with a preset, configurable threshold (e.g., 0.85). If the score is lower than the threshold, the module automatically generates a "review required" task and packages all relevant data of the case (images, clinical data, preliminary diagnostic results) and pushes them to the task queue of the human-computer interaction interface submodule. It may also display a prominent prompt on the system interface.

[0119] The human-computer interaction interface submodule is designed in the style of a medical image reader. It supports side-by-side or overlay display of original images, AI-generated segmentation maps (hyphae and nerve fibers are highlighted in different colors), and diagnostic reports. It also integrates a set of pixel-level annotation tools, such as different colored pens (used to change the background or nerve fibers to hyphae) and an eraser (used to erase misjudged hyphae). When the doctor makes corrections, the interface accurately records the coordinates of each modified pixel and its corrected category label, forming a correction annotation map for the current case.

[0120] The online adaptation model calls a submodule, which can be a copy of the main model, but only its last few layer parameters can be quickly adjusted. This submodule is triggered after the doctor completes and confirms the correction. Using the doctor's corrected annotation map as the target, the prediction loss of the current main model on that case is calculated. Only a few rounds of gradient descent optimization are performed on the parameters of the online adaptation model, aiming to make its output quickly approximate the doctor's corrected results. After optimization, this fine-tuned model is immediately used to re-infer the current case, and the optimized segmentation and diagnosis results are refreshed and displayed on the interactive interface in real time. The entire process should be completed within seconds, providing immediate feedback to the doctor.

[0121] The feedback data storage submodule automatically takes the current case's "input data (image sequence, clinical data) - corrected standard answer (corrected annotation image) - final diagnosis" as a high-quality training data pair after the doctor confirms the final correction result. After anonymization, it stores the data in a dedicated feedback database. The database is designed to support version management and data traceability to ensure the data quality of incremental learning.

[0122] The incremental learning scheduling submodule can be set as a scheduled task (such as weekly / monthly). When the amount of data in the feedback database accumulates to a certain scale, this submodule automatically starts a training task. It samples data from the feedback database and performs additional training rounds (i.e., incremental learning or fine-tuning) on ​​the basis of the existing main model, so that the main model can continuously learn from past errors and expert corrections, thereby performing better on future new cases.

[0123] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0124] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A diagnostic method for corneal fungal infection based on multimodal temporal information, characterized in that, include: Acquire confocal microscopy image sequences of the cornea to be diagnosed and clinical information data of the patient; The image sequences and clinical information data are input into a trained multimodal fusion analysis model, wherein the multimodal fusion analysis model is configured as follows: Based on the image sequence, image fusion features containing spatial morphological features and spatiotemporal dynamic features are extracted; the spatiotemporal dynamic features include at least one of the growth vector, texture evolution consistency and branch event detection features calculated from the image sequence. Based on the aforementioned clinical information data, extract clinical feature vectors; The image fusion features are fused with the clinical feature vector to generate diagnostic features; Based on the diagnostic features, a diagnostic result is output. The diagnostic result includes at least a determination of whether hyphae exist in the image sequence, and supporting information for the determination. The supporting information includes key discriminant factors based on the image fusion features and / or clinical feature vectors.

2. The method according to claim 1, characterized in that, The step of extracting image fusion features containing spatial morphological features and spatiotemporal dynamic features based on the image sequence specifically includes: The image sequence is subjected to inter-frame registration and preprocessing; The preprocessed image sequence is input into a dual-path 3D convolutional neural network, where: The network structure of the first path is used to extract the spatial morphological features of a single frame image; The network structure of the second path is used to extract the spatiotemporal dynamic features of cross-frame images; The image fusion feature is obtained by fusing the features output from the first path and the second path.

3. The method according to claim 2, characterized in that, The network structure of the second path includes three-dimensional convolutional layers or convolutional long short-term memory network layers.

4. The method according to claim 1, characterized in that, After outputting the diagnostic results, the method further includes: If the confidence score in the diagnostic result is lower than a preset threshold, the case is marked as requiring manual review. Provides an interactive interface to receive user input for corrections to the hyphal identification region in the diagnostic results; Based on the corrected input, the diagnostic results of the current case are optimized in real time through an online adaptation model.

5. The method according to claim 4, characterized in that, The method further includes: The corrected input and the corresponding image sequences and clinical information data are stored in the feedback database; The multimodal fusion analysis model is incrementally learned periodically using data from the feedback database.

6. The method according to claim 1, characterized in that, The clinical information data includes at least one of the following: patient age, history of trauma, history of contact lens wear, chief complaint, and past medical history; The step of extracting clinical feature vectors based on the clinical information data includes performing natural language processing on the chief complaint text to extract key symptom features.

7. The method according to claim 1, characterized in that, The output diagnostic results also include: Generate hyphae segmentation maps and nerve fiber segmentation maps corresponding to the image sequence; A visualization heatmap is generated to highlight the region that contributes most to the spatiotemporal dynamic features in at least one frame of the image sequence.

8. A diagnostic system for corneal fungal infections based on multimodal temporal information, characterized in that, include: The data acquisition module is used to acquire the corneal confocal microscopy image sequence to be diagnosed and the patient's clinical information data; The multimodal fusion analysis module, including a trained multimodal fusion analysis model, is used for: Based on the image sequence, image fusion features containing spatial morphological features and spatiotemporal dynamic features are extracted; the spatiotemporal dynamic features include at least one of the growth vector, texture evolution consistency and branch event detection features calculated from the image sequence. Based on the aforementioned clinical information data, extract clinical feature vectors; The image fusion features are fused with the clinical feature vector to generate diagnostic features; The result output module is used to output a diagnostic result based on the diagnostic features. The diagnostic result includes at least a determination of whether hyphae exist in the image sequence and supporting information for the determination. The supporting information includes key discriminant factors based on the image fusion features and / or clinical feature vectors.

9. The system according to claim 8, characterized in that, It also includes an interaction optimization module, used for: If the confidence score included in the diagnostic results is lower than a preset threshold, the case is marked as requiring manual review. The system receives user feedback on the mycelium identification area through an interactive interface. An online adaptation model is invoked to optimize the diagnosis of the current case in real time based on the corrected input; The corrected data is stored in the feedback database for incremental learning of the multimodal fusion analysis model.

Images