A small sample tympanic membrane image recognition method based on meta prompt and knowledge driving

Abstract

The application provides a small sample tympanic membrane image recognition method based on meta prompt and knowledge driving. First, the small sample training set is input into an initial multi-modal pre-training model to obtain a predicted category and compare it with a real category, and misclassified samples are screened by combining confidence to construct a meta task set. Then, the meta task set is input into a primary diagnosis model to output a preliminary diagnosis report. Then, a knowledge refinement model is used to optimize medical description text samples based on the preliminary diagnosis report, replace the original text samples, and obtain updated samples. Finally, the initial multi-modal pre-training model is iteratively trained using the updated samples until the termination condition is met. The method constructs a closed-loop optimization system composed of a primary diagnosis model and a knowledge refinement model. Under the condition of a small sample, the system dynamically optimizes the visual-semantic understanding ability of the model by using the error samples generated by the model itself, effectively improving the accuracy of small sample tympanic membrane image recognition.

Description

Technical Field

[0001] This application relates to the field of image recognition technology. Specifically, this application relates to a few-sample tympanic membrane image recognition method based on meta-cues and knowledge-driven approaches. Background Technology

[0002] With the deepening application of artificial intelligence technology in the medical field, the demand for accuracy and efficiency in tympanic membrane disease identification (such as acute otitis media, tympanic membrane perforation, and tympanic sclerosis) is becoming increasingly prominent. However, current tympanic membrane image recognition technology faces the dilemma of data scarcity and insufficient learning efficiency with small samples. Under limited sample resources, traditional methods struggle to maintain high performance, resulting in low accuracy in tympanic membrane disease identification. Summary of the Invention

[0003] Therefore, it is necessary to provide a few-sample tympanic membrane image recognition method based on meta-hints and knowledge-driven approaches to address the aforementioned technical problems, including:

[0004] Obtain a small training set; wherein the training set consists of multiple training samples, each training sample contains input data and its corresponding label, the input data contains tympanic membrane image samples and their corresponding medical description text samples, the label is the true category of the tympanic membrane image sample, and the medical description text sample is the specific category description of the true category;

[0005] The training set is input into an initial multimodal pre-trained model, which predicts the category of the tympanic membrane image samples and outputs the predicted category of the tympanic membrane image samples.

[0006] Based on the comparison results between the predicted category and the true category, and combined with the confidence level corresponding to the predicted category, misclassified samples are selected from the training set, and a meta-task set is constructed based on the selected misclassified samples.

[0007] The meta-task set is input into a preset primary diagnostic model, which processes the meta-task set and outputs a preliminary diagnostic report. The preliminary diagnostic report includes semantic conflict samples detected from the meta-task set, image features of tympanic membrane image samples among the semantic conflict samples, preliminary diagnostic categories generated by the primary diagnostic model for the semantic conflict samples, confidence distributions, and preliminary correction suggestions. The preliminary diagnostic categories are the results of the primary diagnostic model's category prediction for the semantic conflict samples, and the confidence distributions are the probabilities of the primary diagnostic model predicting different categories for the semantic conflict samples.

[0008] The preliminary diagnosis report is input into a preset knowledge refinement model, which processes the relevant information in the preliminary diagnosis report and outputs an optimized medical description text sample.

[0009] The original medical description text samples in the training set are replaced with the optimized medical description text samples to obtain the updated training samples.

[0010] The updated training samples are then input again into the initial multimodal pre-trained model for iterative training until a preset termination condition is met; wherein, the trained multimodal pre-trained model is used for actual tympanic membrane image recognition tasks.

[0011] The aforementioned few-sample tympanic membrane image recognition method based on meta-hints and knowledge-driven approaches first inputs a few-sample training set containing tympanic membrane image samples, medical descriptive text samples, and true class labels into an initial multimodal pre-trained model. This model obtains predicted classes and compares them with the true classes, while simultaneously filtering misclassified samples based on confidence scores to construct a meta-task set. Next, the meta-task set is input into a primary diagnostic model, which outputs a preliminary diagnostic report containing semantically conflicting samples, image features, preliminary diagnostic classes, confidence distributions, and preliminary correction suggestions. Then, a knowledge refinement model optimizes the medical descriptive text samples based on the preliminary diagnostic report, replacing the original text samples to obtain updated samples. Finally, the updated samples are used to iteratively train the initial multimodal pre-trained model until the termination condition is met. This method constructs a closed-loop optimization system consisting of a primary diagnostic model and a knowledge refinement model. Under few-sample conditions, this system utilizes the model's own generated error samples (meta-tasks) and dynamically optimizes the model's visual-semantic understanding capabilities through an iterative "diagnosis-verification-correction-re-diagnosis" process. This improves the model's few-sample generalization ability and fine-grained recognition accuracy, thereby effectively improving the accuracy of few-sample tympanic membrane image recognition. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the system architecture in one embodiment;

[0013] Figure 2 This is a schematic diagram of a web interaction system in one embodiment;

[0014] Figure 3 This is a topological diagram of tympanic membrane pathology knowledge in one embodiment;

[0015] Figure 4 This is a schematic diagram of the knowledge dynamic update process in one embodiment;

[0016] Figure 5 This is a schematic diagram of the template selection engine process in one embodiment;

[0017] Figure 6 This is a schematic diagram of the overall framework in another embodiment;

[0018] Figure 7This is a medical context dynamic prompt template in another embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] In one embodiment, please refer to, for example Figure 1 The system architecture shown in this invention comprises four innovative modules: a dynamic medical knowledge base, a meta-feature alignment engine, a dual-core optimization of metaprompt, and an interpretable diagnostic system. Among these,

[0021] The dynamic medical knowledge base, as the core knowledge hub of the system, continuously integrates and updates structured and multi-source medical knowledge (such as clinical guidelines, expert consensus, scientific research literature, and case reports), constructs a semantic network and quantitative standards for tympanic membrane pathology, and provides diagnostic logic, feature definitions, and rule constraints for subsequent modules.

[0022] The meta-feature alignment engine processes the input tympanic membrane image. Its core task is to accurately align the image features (such as the shape, texture, and color distribution of the lesion area) with semantic features in the knowledge base (such as "blunt and rounded perforation edges" and "turbidity of effusion >70%)). Through advanced visual encoding (spatial attention, multi-scale feature fusion) and semantic encoding (medical context enhancement, dynamic terminology expansion) techniques, and by utilizing dynamic cue generation algorithms, it transforms complex visual information into structured feature representations rich in medical semantics, providing high-quality input for diagnostic models.

[0023] Metaprompt dual-core optimization is the core innovation of this invention. It constructs a closed-loop optimization system consisting of a primary diagnostic model (Meta-Diagnoser) and a knowledge-refining model (Knowledge-Refiner), coordinated by a dynamic iterative engine (Description Updater). Under small sample conditions, this system utilizes a knowledge base and misclassified samples (meta-tasks) generated by the model itself. Through an iterative "diagnosis-verification-correction-rediagnosis" process, it dynamically optimizes the model's visual-semantic understanding capabilities and diagnostic logic, significantly improving small-sample generalization ability and fine-grained recognition accuracy. The primary diagnostic model is used for preliminary diagnosis and feature conflict localization; the knowledge-refining model, based on the knowledge base, performs logical verification and semantic refinement of the preliminary diagnosis, generating more accurate feature descriptions that better conform to medical standards.

[0024] The interpretable diagnostic system transforms the analysis results from the aforementioned modules into diagnostic reports and visualizations that clinicians can understand. It comprehensively displays the diagnostic basis, key pathological features, decision-making logic, and model confidence through a four-level interpretation system (regional heatmap, feature-level quantification panel, decision-level inference path tree, and risk-level confidence indicator), addressing the "black box" problem of AI diagnosis and improving clinical acceptance. The web-based interactive system provides a user-friendly interface, supporting image uploading, analysis, result display, and report generation.

[0025] In one embodiment, a few-sample tympanic membrane image recognition method based on meta-cues and knowledge-driven methods is provided, the method comprising the following steps:

[0026] Step S102: Obtain a small sample training set.

[0027] The training set consists of multiple training samples. Each training sample contains input data and its corresponding label. The input data includes tympanic membrane image samples and their corresponding medical description text samples. The label is the true category of the tympanic membrane image sample, and the medical description text sample is the specific category description of the true category.

[0028] Specifically, the first step is to build the Meta-Task Constructor.

[0029] When constructing this module, its input is the prepared small sample training set. This can be expressed mathematically as:

[0030]

[0031] Among them, I i Let y represent the i-th tympanic membrane image sample. i The label represents the label corresponding to the tympanic membrane image sample, and N represents the total number of samples in the training set.

[0032] Step S104: Input the training set into the initial multimodal pre-trained model, and the model outputs the predicted category of the tympanic membrane image sample.

[0033] Optionally, the multimodal pre-trained model is a Contrastive Language-Image Pre-training (CLIP) model.

[0034] Specifically, the training set is used as input through an error sampler to run the initial CLIP model to generate prediction results.

[0035] Step S106: Based on the comparison results between the predicted category and the true category, and combined with the confidence level corresponding to the predicted category, misclassified samples are selected from the training set to construct the meta-task set.

[0036] Specifically, if the predicted category does not match the true category, and the confidence level of the predicted category exceeds a preset confidence threshold, the corresponding training sample is labeled as a misclassified sample, and a meta-task set is constructed based on this labeled misclassified sample.

[0037] In a specific example, an error sampler is used to filter out predictions from all generated predictions. With real label y i Inconsistency, i.e. And confidence level Highly certain misclassified samples are labeled as misclassified samples.

[0038] Step S108: Input the meta-task set into the preset primary diagnostic model (Meta-Diagnoser), and the model will output a preliminary diagnostic report.

[0039] The preliminary diagnostic report includes semantic conflict samples detected from the meta-task set, image features corresponding to the semantic conflict samples, preliminary diagnostic categories and confidence distributions generated by the primary diagnostic model, and preliminary correction suggestions.

[0040] Specifically, step S108 is implemented through the following steps:

[0041] Step S1081: For each misclassified sample in the meta-task set, generate a structured prompt file based on the preset prompt template.

[0042] The prompt file contains the image identifier, true category, predicted category, confidence distribution, image features, and medical logic rules corresponding to the misclassified samples.

[0043] In step S1081, the multimodal cue generator is used to generate cue for each... The system uses a dynamic template engine to generate a structured Prompt file. This file can optionally be in JSON format.

[0044] Here is an example of a Prompt file:

[0045]

[0046] Medical logic rules (i.e., knowledge_constraints) can be matched with a pre-built medical logic rule library to obtain corresponding medical logic rules. This matching process includes the true category of the tympanic membrane image sample in the misclassified sample, the predicted category generated by the initial multimodal pre-trained model for class prediction of the misclassified sample, the confidence distribution of different category predictions, and keywords contained in at least one of the image features. Alternatively, medical logic rules can be pre-configured content in a prompt file.

[0047] Step S1082: For each misclassified sample in the meta-task set, calculate the cosine similarity between the corresponding image features and text features.

[0048] Specifically, for each misclassified sample in the meta-task set, the cosine similarity between image features and text features is calculated.

[0049] Step S1083: Misclassified samples with cosine similarity less than a preset similarity threshold are marked as semantic conflict samples.

[0050] Specifically, when sim < δ (threshold δ = 0.4), it is marked as a semantic conflict sample.

[0051] Step S1084: Extract medical logic rules from the prompt file corresponding to the semantic conflict sample using the dynamic prompt parsing algorithm, and convert the medical logic rules into verification instructions.

[0052] Specifically, the following is the dynamic suggestion parsing algorithm:

[0053]

[0054] Step S1085: Process the semantic conflict samples according to the verification instructions to generate a preliminary diagnostic report.

[0055] Specifically, semantic conflict samples are processed according to verification instructions, and correction suggestions containing three elements are output, as follows: First, feature correction items, such as correcting "tympanic membrane not visible" to "tympanic membrane partially visible (visible area <35%)"; second, quantitative parameter supplementation, such as "adding a perforation diameter grading index, specifically divided into small (<2mm), medium (2-4mm), and large (>4mm)"; third, misjudgment attribution encoding, classifying misjudgments as data quality or semantic gap, and outputting the corresponding probability distribution. Furthermore, semantic conflict samples are used as input to run a preliminary diagnostic model to generate preliminary diagnostic categories. This preliminary diagnostic model incorporates a cross-modal attention enhancement layer, which inserts learnable weight matrices. The formula for locating pathological areas is as follows:

[0056]

[0057] Among them, Attention cross (Q,K,V) represents the characteristic response heatmap focused on the tympanic membrane perforation region, where Q = φ(I i ) indicates from image I i The features extracted from the image are denoted by φ, which represents a feature extraction function used to map the image to a feature vector space. Represents image I i The corresponding text features, where ψ represents the function that maps the text to the feature vector space. This represents the scaling factor, preventing the dot product result from becoming too large.

[0058] Finally, the image features corresponding to the semantic conflict samples, the preliminary diagnosis categories, the confidence distribution generated by the primary diagnosis model, the preliminary correction suggestions, and the semantic conflict samples detected from the meta-task set are integrated to generate a preliminary diagnosis report.

[0059] Step S110: Input the preliminary diagnosis report into the preset knowledge refinement model, and the model outputs an optimized medical description text sample.

[0060] Specifically, the knowledge refinement model plays the role of "expert review," receiving preliminary diagnostic reports (including image I) from the preliminary diagnostic model. i The preliminary diagnostic results (e.g., "normal tympanic membrane"), confidence distribution, detected visual-semantic conflict terms (e.g., sim(...) < 0.4), and correction suggestions proposed by Meta-Diagnoser (e.g., feature correction terms, quantization parameter supplementation) are analyzed. Based on validated medical knowledge (disease definitions, diagnostic criteria, differential diagnosis rules, quantization parameter relationships, etc.) stored in the dynamic medical knowledge base, these preliminary results undergo in-depth verification, logical reasoning, and semantic refinement. As one implementation method, step S110 is specifically achieved through the following steps:

[0061] Step S1101: Based on the keywords in the preliminary diagnosis category, the true category, and the semantic conflict samples, retrieve relevant knowledge subgraphs from the preset dynamic medical knowledge base.

[0062] Specifically, the system receives the output of the preliminary diagnostic model, which includes semantic conflict samples detected from the meta-task set (e.g., sim(...) < 0.4), the image features corresponding to the semantic conflict samples, the preliminary diagnostic category generated by the preliminary diagnostic model (e.g., "normal tympanic membrane"), the confidence distribution, and preliminary correction suggestions (e.g., feature correction terms, quantization parameter supplementation). Based on the preliminary diagnostic results, the true labels from the meta-task, and the keywords in the semantic conflict samples (e.g., "chronic otitis media," "tympanic membrane perforation," "light reflex integrity"), the system retrieves relevant knowledge subgraphs from the dynamic medical knowledge base.

[0063] Step S1102: Check whether the preliminary correction suggestions are consistent with the rules in the knowledge subgraph.

[0064] Specifically, when performing logical rule verification, the first step is to check whether the preliminary correction suggestions output by the preliminary diagnostic model are consistent with the knowledge subgraph retrieved from the dynamic medical knowledge base. The rules in the knowledge base must be consistent. For example, if it is suggested to "revise 'tympanic membrane not visible' to 'tympanic membrane partially visible (visible area <35%)", it is necessary to verify whether there are rules in the knowledge base that support "visible area <35%" as a key diagnostic indicator (such as the degree of external auditory canal obstruction). Secondly, it is necessary to verify whether the preliminary diagnosis results meet the diagnostic criteria in the knowledge base. For example, the knowledge base defines "chronic otitis media" as requiring "tympanic membrane perforation lasting >3 months" or "presence of granulation tissue". If the image features and description do not reflect these conditions, the diagnosis results should be questioned. Finally, differential diagnosis should be performed. With the help of the differential diagnosis edge in the knowledge base, it is necessary to analyze whether the current features are more consistent with other diseases (such as tympanosclerosis).

[0065] Step S1103: If yes, then optimize the medical description text sample corresponding to the semantic conflict sample based on the knowledge subgraph to obtain the optimized medical description text sample.

[0066] Specifically, based on the terminology definitions, quantification standards, and relationships in the knowledge base, the original or revised medical description text samples generated by the preliminary diagnostic model are standardized and refined. For example, the original description "a yellowish-brown area visible on the tympanic membrane" is refined according to the knowledge base to "calcifications seen in the posterosuperior quadrant of the tympanic membrane (associated with the typical manifestation of 'tympanic sclerosis' in the knowledge base), with supplementary descriptions: diameter 1.8±0.3mm (referencing the knowledge base quantification standard: common size range of punctate calcifications), average CT value >120HU (based on the threshold for distinguishing calcifications from other substances in the knowledge base). Tympanic sclerosis needs to be excluded (according to the knowledge base differential diagnosis rule: if the density of calcifications is >5 / mm²). 2This suggests tympanic sclerosis, requiring further measurement or examination. Simultaneously, based on the quantified relationships (quantified_by edges) in the knowledge base, quantification parameters are automatically supplemented, adding necessary measurement indicators, classification standards, and their error ranges, such as grading perforation diameter as small (<2mm), medium (2-4mm), and large (>4mm ±0.2mm). Furthermore, an interpretable reasoning step from image features to the final diagnosis is generated using knowledge graph paths, forming a diagnostic reasoning chain. Finally, the medical description text sample, refined and validated by knowledge verification, the precise diagnostic category (which may be corrected or maintained), the updated confidence level, and the diagnostic reasoning path are passed to the dynamic iteration engine.

[0067] Step S112: Replace the original medical description text sample with the optimized medical description text sample to obtain the updated training sample.

[0068] Step S114: Input the updated training samples back into the initial multimodal pre-trained model for iterative training until the preset termination condition is met.

[0069] The trained multimodal pre-trained model is used for actual tympanic membrane image recognition tasks.

[0070] Specifically, a dynamic iterative engine receives the output from the knowledge refinement model (i.e., refined diagnostic descriptions and conclusions conforming to knowledge base standards) and feeds these optimized descriptions back to the meta-feature alignment engine and / or the primary diagnostic model to update the model's internal representation or parameters, thereby improving the model's diagnostic accuracy and semantic alignment ability when encountering similar images again. This process is performed on the selected meta-task sample set. The iteration continues. The iteration steps are as follows:

[0071] First, the system receives optimized medical description text samples and precise diagnostic categories from the knowledge refinement model for the current batch of meta-task samples. Then, parameter updates are performed, treating the optimized medical description text samples as more accurate and richer supervisory signals. Losses (such as contrastive loss or cross-entropy loss) are calculated, and the weights of relevant models (mainly the visual / semantic encoder in the meta-feature alignment engine, the dynamic cue generator, and the parameters in the preliminary diagnosis model) are updated according to a meta-learning-based optimization strategy. The expression for the meta-learning-based optimization strategy is as follows:

[0072]

[0073] Where, θ t+1 θ represents the model parameters at time step t+1. t This represents the model parameters at time step t, and α represents the meta-learning rate. Represents the meta-task set. I represents the gradient. i This represents the i-th tympanic membrane image sample. This represents the corresponding optimized medical description text sample. This represents the alignment loss between image features and text features.

[0074] Finally, an evaluation and termination decision are made. Alternatively, evaluate the performance of the updated model (e.g., diagnostic accuracy Acc) on a small validation set. Set a convergence threshold ∈ = 0.03. When the accuracy improvement ΔAcc < ∈ for three consecutive iterations, the optimization process is considered to have converged and the iteration is terminated. Otherwise, return to step 102 for the next iteration (you can choose to resample using the updated model or construct a new meta-task set for subsequent iterations).

[0075] In one embodiment, the method further includes the following steps:

[0076] Receive images of the tympanic membrane to be identified uploaded by the user;

[0077] The tympanic membrane image to be identified is identified by a multimodal pre-trained model deployed on a server, and the predicted category, the corresponding category description, and the confidence level of each category are output.

[0078] The predicted category, the corresponding specific category description, and the confidence distribution of each category are returned to the user terminal for visualization.

[0079] Specifically, such as Figure 2 As shown, this invention also provides a web interaction system. This system adopts a front-end / back-end separation architecture, using the Flask framework as its core to build a RESTful API on the server side, and combining HTML5 / CSS3 / JavaScript (or alternatively, frameworks such as Vue.js / React) to achieve a responsive front-end interface. The aim is to provide clinicians with an intuitive and efficient intelligent diagnostic tool for tympanic membrane images.

[0080] The backend (Flask Server) is deployed on a Linux server.

[0081] Core functions: Receive tympanic membrane images uploaded from the front end; process them using the meta-feature alignment engine and Metaprompt dual-core optimization mechanism; generate structured data containing the original image, diagnostic results, four-level interpretation information, and a diagnostic report (which can be exported as PDF); and access a dynamic medical knowledge base to provide relevant medical knowledge query support.

[0082] Implement the API interfaces: / upload (image upload), / analyze (start analysis), / result / <image_id> (Get results), / report / <image_id> (Generate report).

[0083] Front-end (Web UI): Adapted to mainstream browsers (Chrome / Firefox / Edge / Safari), core interaction flow includes:

[0084] User login / registration: Doctor identity verification, recording operation history.

[0085] Image upload: Supports drag-and-drop or file selector upload of single or multiple tympanic membrane images (common formats: JPG, PNG, DICOM).

[0086] Analysis Startup and Status: Analysis is automatically or manually triggered after uploading, and a processing progress bar or status prompt is displayed.

[0087] Diagnostic results displayed:

[0088] Main view area: Displays the original uploaded image and a visual image overlaid with a regional heatmap side-by-side. The heatmap color map (e.g., red-yellow-green) visually indicates the model confidence level of lesion areas (probability > 0.8 displays a red high-confidence lesion area). Image zooming and panning are supported.

[0089] Feature-level panel: Clearly lists key pathological features and their quantitative values ​​in table or card format. For example: 1) Perforation diameter: 3.2 mm (medium) [According to the knowledge base definition: 2-4 mm]; 2) Fluid turbidity: 78% (severely turbid) [Threshold reference: >70% is severe]; 3) Calcification density: 4.1 calcifications / mm² 2 [Reference: Threshold value for tympanic sclerosis is 5 mm / mm] 2 4) Light cone integrity loss rate: 12% [Rule: <10% can rule out acute otitis media]

[0090] Decision-level reasoning tree: Displays the complete logical chain of diagnostic reasoning in a collapsible / expandable tree diagram. The root node is the final diagnostic conclusion, the child nodes are the key features supporting the conclusion and the knowledge base rules they satisfy, and the leaf nodes can be associated with the corresponding image evidence areas.

[0091] For example:

[0092] Risk Level Confidence Indicator: Employs a 3D visual coding system to display the overall confidence level of the model for the final diagnosis. 1) Shape: 3D prism (e.g., a cube represents low confidence, a multifaceted fine crystal represents high confidence); 2) Color: Gradient from red (low confidence) to green (high confidence); 3) Transparency: The higher the confidence level, the less transparent the object. (Example: A rough cube appearing as a semi-transparent red indicates low confidence in the diagnosis of "acute otitis media"; a fine icosahedron appearing as an opaque green indicates high confidence in the diagnosis of "normal tympanic membrane").

[0093] Report generation and export: Integrates all visual information and text descriptions to generate structured diagnostic reports with one click (including patient information, images, diagnostic conclusions, detailed feature analysis, reasoning basis, and confidence level descriptions), supporting export in PDF and HTML formats or direct printing.

[0094] Knowledge base search (optional): Integrates a knowledge base search portal, allowing doctors to instantly search for terms or rules mentioned in diagnostic reports to obtain more detailed medical background knowledge.

[0095] In one embodiment, the front-end interface adopts a professional medical style with a blue-gray tone and a responsive layout. The desktop version uses a top-bottom split design, while the mobile version dynamically adjusts the spacing between elements to ensure readability. At the same time, a four-level interpretation system is adopted to improve the interpretability of the diagnosis.

[0096] Regional level: Lesion area heatmap. Suspected lesion areas are visually located using color overlays (e.g., red indicates areas identified as lesions by the model with a confidence level > 0.8, yellow indicates areas with medium confidence). Color depth is positively correlated with confidence level, and on / off comparison with the original image is supported.

[0097] Feature Level: Key Pathological Parameter Panel. Presented in structured data tables or information cards, this panel clearly displays the quantitative features and their values ​​identified in the model that play a decisive role in the diagnostic conclusion. Each feature can be linked to knowledge base definitions and diagnostic criteria (mouse hover tooltip or click to view details). For example, it clearly lists "Perforation diameter: 3.2mm (medium)", "Effusion turbidity: 78% (severely turbid)", and "Calcification density: 4.1 lesions / mm²". 2 "wait.

[0098] Decision-level: Diagnostic reasoning path tree. A dynamic, interactive tree diagram (e.g., implemented in D3.js) fully presents the logical reasoning chain from visual image evidence to the final diagnostic conclusion. The root node represents the diagnostic result, branch nodes represent key features and their matching with knowledge base rules (e.g., "perforation diameter > 2mm → meets the criteria for chronic otitis media"), and leaf nodes can link back to specific regions on the image. Expanding / collapsed branches facilitates doctors' tracing and understanding of the AI's decision-making process.

[0099] Risk Level: 3D Confidence Indicator. This indicator comprehensively utilizes triple visual encoding—shape complexity (simple geometry -> complex polyhedrons), color (red -> orange -> yellow -> green), and transparency (high transparency -> opaque)—to render a dynamic object in 3D space. It intuitively and unambiguously conveys the model's overall confidence level regarding the current diagnostic result. This design overcomes the limitations of single numbers or progress bars, providing a richer and more intuitive perception.

[0100] In one embodiment, step S104 specifically includes the following steps:

[0101] Step S1041: Extract features from the tympanic membrane image sample using an image encoder to generate image features;

[0102] Step S1042: Extract features from the medical description text sample using a text encoder to generate text features;

[0103] Step S1043: Calculate the cosine similarity between image features and text features;

[0104] Step S1044: Perform cross-modal alignment processing on image features and text features based on cosine similarity;

[0105] Step S1045: Perform joint inference on the aligned image features and text features to generate the predicted category of the tympanic membrane image sample.

[0106] In one embodiment, step S1041 specifically includes the following steps:

[0107] Step S2011: Extract features from the tympanic membrane image samples to obtain initial image features;

[0108] Step S2012: Perform a 1×1 convolution operation on the initial image features to obtain the first feature map;

[0109] Step S2013: Calculate the gradient of the first feature map;

[0110] Step S2014: Multiply the gradient element-wise with the initial image features to obtain the attention heatmap; wherein, the features in the attention heatmap are the generated image features.

[0111] Specifically, steps S2012-S2014 are implemented through the following expression:

[0112]

[0113] Among them, H att Represents an attention heatmap, F v Represents the initial image features, Conv 1×1 This represents a 1D convolution operation. It represents the degree of change (gradient) in image features.

[0114] In another embodiment, step S1041 specifically includes the following steps:

[0115] Step S2021: Extract features from the tympanic membrane image samples to obtain initial image features;

[0116] Step S2022: Concatenate the initial image features and the hidden vector of the previous layer to obtain the concatenated features;

[0117] Step S2023: Perform a linear transformation on the concatenated features using the first learnable parameter to obtain the linearly transformed features;

[0118] Step S2024: Activate the features after linear transformation to generate an interest mask;

[0119] Step S2025: Use the attention mask to weight the initial image features to obtain the weighted features;

[0120] Step S2026: Add the weighted features to the second learnable parameters to obtain the summed features;

[0121] Step S2027: Perform layer normalization on the summed features to obtain image features.

[0122] Specifically, steps S2022-S2027 are implemented through the following expression:

[0123]

[0124] Among them, F v For image features, H t-1 It is the hidden vector of the previous layer, W m ,b m For learnable parameters, LayerNorm is the layer normalization operation.

[0125] In yet another embodiment, step S1041 specifically includes the following steps:

[0126] Step S2031: Extract features from the tympanic membrane image samples to obtain initial image features;

[0127] Step S2032: Perform a 1×1 convolution operation on the initial image features to obtain the first feature map;

[0128] Step S2033: Calculate the gradient of the first feature map;

[0129] Step S2034: Multiply the gradient element-wise with the initial image features to obtain the attention heatmap;

[0130] Step S2035: Concatenate the features in the attention heatmap with the hidden vectors of the previous layer to obtain the concatenated features;

[0131] Step S2036: Perform a linear transformation on the concatenated features using the first learnable parameter to obtain the linearly transformed features;

[0132] Step S2037: Activate the features after linear transformation to generate an interest mask;

[0133] Step S2038: Use the attention mask to weight the initial image features to obtain the weighted features;

[0134] Step S2039: Add the weighted features to the second learnable parameter to obtain the summed features;

[0135] Step S2040: Perform layer normalization on the summed features to obtain image features.

[0136] In this embodiment, a pathological region self-localization module is innovatively introduced, which improves the focusing accuracy of the pars tensa of the tympanic membrane by 37.8% in the detection of inflammatory lesions of the tympanic membrane.

[0137] In one embodiment, step S1041 specifically includes the following steps:

[0138] Step S2051: Use a feature extraction network to extract features from the tympanic membrane image samples to obtain image feature maps at different levels; wherein, the image feature maps contain local detail information and preliminary semantic information of the image;

[0139] Step S2052: Obtain the fine-grained image features of the bottom layer from the feature extraction network, process the fine-grained image features of the bottom layer using the third learnable parameter, and calculate the offset.

[0140] Step S2053: Obtain multi-scale image features from different layers of the feature extraction network. For each scale of image features, transform the image features using deformable convolution operation based on the offset to obtain the transformed features.

[0141] Step S2054: Based on the preset weights for each scale, the transformed features corresponding to each scale are weighted to obtain the weighted features.

[0142] Step S2055: Sum the weighted features corresponding to all scales to obtain the image features.

[0143] Specifically, step S2052 is implemented through the following expression:

[0144]

[0145] in, W represents fine-grained image features. offset These are the parameters that need to be learned.

[0146] Steps S2053-S2055 are achieved through the following expressions:

[0147]

[0148] in, α represents the i-th layer feature of the image. i Represents the weights of the features in the i-th layer, and DeformConv is a deformable convolution operation.

[0149] In one embodiment, step S1042 specifically includes the following steps:

[0150] Medical descriptive text samples are input into a graph neural network to generate text features. The graph neural network uses medical terms as nodes and the anatomical correlation strength between medical terms as edges. The feature vectors corresponding to the nodes are updated through a dynamic propagation function. The expression of the dynamic propagation function is as follows:

[0151]

[0152] in, Let N(v) represent the feature vector of node v in the graph neural network after the k-th iteration, where v represents the current node in the graph neural network, k represents the iteration number in the graph neural network, u represents the neighboring nodes of node v, and N(v) represents the set of all neighboring nodes of node v. β represents the feature vector of node v and its neighbor node u in the previous round. uv The weights between nodes u and v are represented by CONCAT, which represents the concatenation operation used to combine the feature vectors of node v from the previous round. The feature vector is concatenated with the weighted feature vectors of all its neighboring nodes. W represents a learnable weight matrix used to perform a linear transformation on the concatenated vector, and ReLU represents the activation function.

[0153] In this embodiment, to address the challenge of ambiguity in medical terminology (such as "turbidity" potentially referring to exudate or calcification), the professional expressions are transformed into 256-dimensional computable vectors, achieving a disambiguation accuracy of 92.6%.

[0154] A dynamic medical knowledge base serves as the knowledge hub of a system, aiming to address the disconnect between medical knowledge and visual features. Traditional methods rely on static labels or simple rule bases, which cannot adapt to the fine-grained feature descriptions of tympanic membrane lesions (such as the quantitative standard of "blunt and rounded perforation edges") and the dynamic updates of clinical guidelines. This invention constructs a computable multimodal knowledge topology to achieve the dynamic evolution and precise retrieval of medical knowledge. In one embodiment, the steps for constructing a dynamic medical knowledge base are as follows:

[0155] Step S122, Multi-source medical data acquisition and processing

[0156] Data source and processing flow:

[0157] (1) Structured data (30%):

[0158] Clinical electronic medical records: Parsing XML / HL7 format medical records and extracting key fields (diagnosis, lesion location, surgical record).

[0159] Medical Knowledge Graph: Integrating SNOMED-CT and MeSH otology terminology to construct entity relationships (e.g., tympanic membrane perforation → complications → hearing loss).

[0160] Quantitative research databases: Import measurement indicators (perforation diameter, effusion CT value) from public datasets (such as the CANDLE-PED otitis media database).

[0161] (2) Semi-structured data (45%):

[0162] Clinical guidelines: PDF parsing NLP pipelines (such as PDFPlumber+BiLSTM-CRF or large models) to extract diagnostic criteria (e.g., for a confirmed diagnosis of acute otitis media, the following criteria must be met: tympanic membrane bulging + hyperemia area >50%).

[0163] Expert consensus: Use rule template matching to generate the logical expression (IF light reflex disappears AND vasodilation THEN effusion probability > 80%).

[0164] (3) Unstructured data (25%):

[0165] Medical Literature: Retrieve the latest papers from PubMed API; ViT-BERT model jointly extracts image-text association knowledge (image caption description → image region).

[0166] Surgical video: keyframe sampling (1 frame every 10 seconds), YOLOv7 for tympanic membrane localization, 3D ResNet for dynamic pathological feature extraction.

[0167] Quality control mechanism:

[0168] Terminology standardization: Mapping dialect terms (e.g., "eardrum") based on UMLS dictionary

[0169] "eardrum").

[0170] Conflict detection: Experts mark contradictory items (such as conflicting thresholds for "small perforations" in different guidelines), triggering a confidence-weighted vote (weight = literature impact factor × 0.6 + expert title level × 0.4).

[0171] Data anonymization: Noise injection (ε = 0.1) to protect patient privacy.

[0172] Step S124, tympanic membrane pathology knowledge topology, such as Figure 3 As shown.

[0173] Step S126, dynamic update mechanism, such as Figure 4 As shown.

[0174] Version control employs "semantic fingerprint" technology: a review is triggered when the similarity of knowledge changes is less than 0.85. The update process is: automatic data collection → difference detection → expert review → version release.

[0175] In one embodiment, the dynamic prompt generation algorithm includes the following:

[0176] (1) Template selection engine. For example... Figure 5 As shown, a clinical decision tree and deep reinforcement learning are used to dynamically call up a template library, and a template value evaluation function V(t) = Q is introduced. clin (t)×exp(-γ|F v -T t | 2 (Q) clin The template confidence level verified by the hospital's expert panel, F v Image features, T t Text features, γ preset hyperparameters):

[0177] (2) Feature injection mechanism. Unstructured medical knowledge is transformed into computable parameters, and knowledge triple encoding is combined to construct knowledge units (anatomical structure, pathological features, quantitative standards), which are then automatically embedded into the generated text to measure error ranges.

[0178] (3) Temperature adaptive adjustment. This addresses the similarity calculation drift problem in high-dimensional feature spaces, improving the model's recall rate by 24.3% in the otoscope image and text alignment task. It includes two core modules:

[0179] Dynamic temperature coefficient function:

[0180]

[0181] (k: learnable parameter, λ: gradient penalty factor, F) v Image features, rank(): feature ranking. Image feature gradient.

[0182] Multimodal contrastive loss function:

[0183]

[0184] Where, sim(F) v ,F t ) represents the similarity calculation between image features and text features, τ is the weighting factor, and N represents all samples involved in the calculation.

[0185] In one embodiment, the key technical implementation of the knowledge refinement model includes:

[0186] Knowledge base embedding layer: embedding structured knowledge base (JSON format) Encoded as a topology that a Graph Neural Network (GNN) can process:

[0187] Nodes: Disease entity (e.g., "chronic otitis media"), pathological features (e.g., "tympanic membrane perforation"), anatomical structure (e.g., "pars tensa of the tympanic membrane"), and quantitative parameters (e.g., "perforation diameter").

[0188] Edge: Medical Relationships (has_symptom, differential_diagnosis, quantified_by, location_in, contraindicates)

[0189] Graph-text joint reasoning: The model simultaneously processes the input image features (visual encodings from the alignment engine or Meta-Diagnoser) and a knowledge base subgraph relevant to the current diagnostic task (obtained by querying keywords or entities in the preliminary diagnostic results and retrieving them from the knowledge base).

[0190] Image-text alignment loss function: used to train the model to ensure that its knowledge-based inference output is consistent with image evidence and standard medical descriptions.

[0191]

[0192] Where: ε is the set of connected edges in the knowledge graph, h i ,h j KL(p) is the embedding vector of the connected nodes. GNN |p RoBERTa ) is the KL divergence, and p is the probability distribution of the diagnostic results of the constrained graphical neural network (GNN). GNN The distribution p of text-based semantic models (such as RoBERTa) RoBERTa To maintain consistency, sim(F) v ,h diagnosis ) is the visual feature F of the image v Embedded with refined post-diagnostic entities h diagnosis Similarity measures (such as cosine similarity) are used to ensure that diagnostic conclusions are supported by image evidence. λ and γ are weighting coefficients that balance the various losses.

[0193] Based on the same inventive concept, please refer to Figure 6 In one embodiment, a few-sample tympanic membrane image recognition method based on meta-cues and knowledge-driven methods is also provided, comprising the following steps:

[0194] Step S301: Obtain the training set, which consists of multiple training samples. Each training sample contains input data and corresponding labels. The input data includes tympanic membrane image samples and corresponding medical description text samples. The labels include the true category of the tympanic membrane image samples and the corresponding category description.

[0195] Step S302: Input the training set into the initial multimodal pre-trained model for training and generate classification results.

[0196] Specifically, step S302 is implemented as follows:

[0197] Step S3021: Extract features from the tympanic membrane image sample using an image encoder to generate an image feature vector.

[0198] In this step, the image encoder can adopt the Vision Transformer (ViT) structure to segment the input tympanic membrane image sample into a 16×16 pixel patch sequence, extract global context features through a 24-layer Transformer encoder, and output an image feature vector with a dimension of 768.

[0199] Step S3022: Extract features from the medical description text sample using a text encoder to generate a text feature vector.

[0200] In this step, the medical descriptive text sample is enhanced with semantic expression through a manually designed dynamic prompt template for medical context. The specific generation process is as follows: The system receives the actual category and corresponding category description of a tympanic membrane image sample provided by the user; the actual category and corresponding category description are then filled into a preset dynamic prompt template for medical context to generate the medical descriptive text sample.

[0201] The medical context dynamic cue template contains multiple templates, enhancing the distinctiveness of text representation through multi-perspective cues. The template generation process employs a context-aware strategy; for example, see [link to template]. Figure 7 For the category "chronic otitis media" and its corresponding feature description, the system dynamically generates the following prompt text: ["Otoscopic examination shows: chronic otitis media, typical features include marginal perforation of the pars tensa of the tympanic membrane...", "According to the otological diagnostic criteria, this image presents the characteristics of chronic otitis media:"

[0202] The tympanic cavity mucosa is congested and edematous...).

[0203] The text encoder can employ the RoBERTa model. After segmenting and positionally encoding the input medical description text samples, it generates text feature vectors through 12 Transformer layers. In one specific implementation, each medical description text sample generated from a dynamic medical context prompt template has its 768-dimensional feature vector extracted by the RoBERTa encoder, and then features are fused using mean pooling. The processing formula is as follows:

[0204]

[0205] in, Let K represent the text feature vector of category j, and K represent the number of dynamic prompt templates for medical context, i.e., there are K dynamic prompt templates for medical context for each category.

[0206] Step S3023: Calculate the cosine similarity between the image feature vector and the text feature vector.

[0207] In this step, the formula for calculating cosine similarity is:

[0208]

[0209] Wherein s(I,T) j ) represents cosine similarity, I represents tympanic membrane image sample, φ(I) represents image feature vector, and τ represents temperature coefficient, used to adjust similarity distribution.

[0210] Step S3024: Perform cross-modal alignment processing on image feature vectors and text feature vectors based on cosine similarity.

[0211] Step S3025: Perform joint inference on the aligned image feature vector and text feature vector to generate the predicted category of the tympanic membrane image sample.

[0212] In this step, the joint reasoning process uses the maximum similarity criterion, and the processing formula is as follows:

[0213]

[0214] in, Indicates the prediction category, argmax represents the average similarity between tympanic membrane image sample I and all K medical context dynamic cue templates of category j. j The parentheses (·) indicate that the value of j that maximizes the expression within the parentheses is selected, meaning the category with the highest average similarity is chosen as the prediction result. This represents the k-th medical context dynamic prompt template for category j.

[0215] By averaging multiple medical contextual prompt templates, the impact of single template bias is reduced, and the multi-angle semantic expression of the category is covered (such as "chronic otitis media" can correspond to different clinical manifestations such as "tympanic membrane perforation" and "ear canal effusion").

[0216] Step S3026: Compare the predicted category with the true category to determine the classification result.

[0217] Step S303: Based on the classification results, filter out misclassified samples, and generate structured error message information according to the classification results of the misclassified samples and the preset analysis task.

[0218] Specifically, for misclassified samples (such as misclassifying "chronic otitis media" as "normal tympanic membrane"), the system calls an automated script (generate_prompts.py) to parse the classification results, using a state machine to extract key information such as sample identifier, image path, true class, predicted class, and confidence scores for all classes line by line. The structured error message for misclassified samples is stored in "prompt_sampleID.txt" format, containing four parts: basic image information (e.g., resolution 420×380 pixels), a description of classification inconsistencies (e.g., semantic conflict between the true and predicted classes), a visualization of the confidence distribution (showing the model's probabilistic tendency towards candidate classes), and a list of analysis tasks (requiring feedback from four dimensions: feature validation, description accuracy, misclassification attribution, and improvement suggestions).

[0219] Step S304: Input the structured error message and the corresponding tympanic membrane image sample into the multimodal large model to generate an error analysis report.

[0220] Specifically, in the initial optimization stage, the general multimodal large model GPT-4 (which can be an example of a primary diagnostic model) receives structured error messages and corresponding tympanic membrane image samples, and achieves visual-text feature alignment through a cross-modal attention mechanism. The visual branch uses a deformable convolutional network to extract local pathological features (such as the ellipticity of the edge morphology of the tympanic membrane perforation and the density distribution of earwax impaction), while the text branch parses the medical logic chain in the prompt template (such as "tympanic membrane not visible" requires the external auditory canal occlusion area > 80%). For example, in a sample (ID: 002-41e), the light cone shape is distorted due to image overexposure. GPT-4 detects that the visible area in the anterior quadrant of the tympanic membrane is 35%, triggering a description correction to "tympanic membrane partially visible (visible area < 35%)", and labeling the feature confidence as 0.63. Error attribution analysis uses a decision tree decomposition strategy to classify the root causes of misjudgment as data quality defects (32%, such as image blurring and abnormal lighting) or insufficient semantic expression (68%, such as "tympanic membrane calcification" not clearly defining the distribution pattern of calcification foci).

[0221] Step S305: Input the error analysis report into the large language model, and use the large language model to call the tympanic membrane medical knowledge base to optimize the category description.

[0222] Specifically, the secondary optimization stage involves DeepSeek's large language model (which could be an example of a knowledge refinement model) combined with a structured knowledge base (medical_knowledge.json) to refine the tag descriptions.

[0223] Step S306: Update the medical description text samples according to the optimized category descriptions to obtain the updated training samples.

[0224] Step S307: Input the updated training samples into the initial multimodal pre-trained model for iterative training until the preset termination condition is met.

[0225] The aforementioned method for few-sample tympanic membrane image recognition based on meta-hints and knowledge-driven approaches constructs a training set containing tympanic membrane images and medical descriptive text. An initial multimodal pre-trained model is used for preliminary classification, and misclassified samples are filtered based on the classification results to generate structured error hints. Furthermore, a dual-model collaborative optimization mechanism combining the GPT-4 multimodal large model and the DeepSeek large language model is employed to perform in-depth analysis of error samples and optimize category descriptions using a tympanic membrane medical knowledge base, achieving dynamic updates of training samples and iterative model training. This method achieves efficient and accurate tympanic membrane disease recognition under limited sample conditions.

[0226] In one embodiment, the system adopts a phased iterative strategy, with an evolution path of "coarse-grained correction → fine-grained optimization → threshold termination": Initial stage: Each class inputs a small number of labeled samples, and uses GPT-4 to quickly locate high-frequency misclassification patterns (e.g., tympanic membrane calcification misclassification rate > 40%), generating preliminary description correction suggestions; Intermediate stage: Based on the corrected description, a certain number of new samples are added for fine-grained feature verification (e.g., perforation edge angle measurement error < 5°); Final stage: The remaining samples are introduced to complete semantic alignment optimization, until the classification accuracy improvement is lower than a preset threshold for three consecutive times, at which point the iteration terminates.

[0227] In one embodiment, the process of constructing a tympanic membrane medical knowledge base includes:

[0228] Obtain medical data on the tympanic membrane;

[0229] Identify key fields in the original tympanic membrane medical data and extract relevant pathological feature descriptions;

[0230] The extracted data was converted into JSON format, and a hierarchical data structure was constructed that includes disease categories, full morphological descriptions, histological states, functional indicators, and key identification points.

[0231] A medical knowledge base for tympanic membranes is constructed based on a hierarchical data structure.

[0232] Specifically, the construction of the tympanic membrane medical knowledge base is based on the systematic integration and structured representation of multimodal medical knowledge, aiming to provide calculable, fine-grained pathological feature priors for small-sample tympanic membrane image recognition. The knowledge base is constructed by systematically collecting authoritative medical literature, clinical practice guidelines, and expert-annotated data in the field of otolaryngology, covering common otological diseases such as chronic suppurative otitis media, earwax impaction, and tympanosclerosis. Data sources include monographs such as *Practical Otorhinolaryngology Head and Neck Surgery* and *Clinical Diagnosis and Treatment Guidelines for Otitis Media*, as well as ethically reviewed clinical imaging report texts, ensuring the professionalism and clinical consistency of the knowledge content.

[0233] In the structured processing stage, automated tools based on Natural Language Processing (NLP) are used to parse and extract features from unstructured text. Specifically, a Python script calls the python-docx library to parse the original Word document, identify key fields such as "morphological features" and "distinguishing features," and extract relevant pathological descriptions. For example, for chronic otitis media, the system extracts feature descriptions such as "marginal perforation of the pars tensa of the tympanic membrane (blunt and rounded edges)" and "tympanic mucosal congestion and edema with mucopurulent discharge," while also marking the clinical diagnostic rule "needs to be differentiated from earwax impaction (features of external auditory canal obstruction)." The extracted data is further converted into JSON format, constructing a hierarchical data structure that includes disease category, full morphological description, histological state, functional indicators, and distinguishing features.

[0234] The validation and quality control of the knowledge base were jointly completed by several senior otolaryngologists. Cross-validation was used to ensure the accuracy of feature descriptions and the rigor of diagnostic logic. Experts reviewed each feature entry for each disease category, focusing on validating the following: 1) visual consistency between morphological descriptions and clinical images (e.g., "translucent grayish-white tympanic membrane" must correspond to the fibrous layer structure of a healthy tympanic membrane); 2) complete coverage of key diagnostic points (e.g., earwax impaction must clearly define the "grading of external auditory canal obstruction"); 3) standardization of terminology (e.g., "mucopurulent discharge" should be standardized as "viscous purulent discharge"). The validated knowledge base is ultimately stored as a searchable matrix structure, supporting rapid alignment with image features through semantic similarity calculations, providing high-confidence medical prior knowledge support for subsequent multimodal fusion.

[0235] In summary, the key protection of this invention lies in a small-sample tympanic membrane image recognition method that integrates a structured medical knowledge base, dual-model collaborative Metaprompt optimization, and an interactive Web diagnostic system. This method can achieve high-precision recognition even with scarce data, possesses fine-grained feature classification capabilities, and exhibits good interpretability, bringing a new technological paradigm to the field of medical imaging AI.

[0236] The technical advantages of this invention are as follows:

[0237] Addressing the Data Scarcity Problem: Traditional deep learning heavily relies on large-scale labeled data, but labeling tympanic membrane images is costly and the samples are unevenly distributed. This invention constructs a structured medical knowledge base, encoding clinical guidelines and expert experience into computable morphological, histological, and functional features, providing the model with rich prior knowledge. Combined with a dynamic prompt template generation strategy (multiple medical context templates), the model can semantically align image features extracted from limited samples with pathological descriptions in the knowledge base, thereby significantly reducing its dependence on labeled data.

[0238] Enhanced Fine-Grained Classification Capabilities: The identification of tympanic membrane diseases relies on microscopic differences in features such as color gradation and perforation morphology (e.g., central / peripheral). This invention utilizes the cross-modal alignment mechanism of the CLIP framework to map image features (e.g., extracted by the ViT encoder) and dynamically aggregated text features (e.g., RoBERTa mean pooling) to a unified semantic space. Furthermore, it incorporates a temperature coefficient (τ) to optimize the similarity distribution, enabling the model to accurately capture subtle differences such as the distribution of tympanic membrane calcifications and the density of external auditory canal masses.

[0239] Enhanced interpretability and generalization ability: Traditional models struggle to gain clinical trust due to their "black box" decision-making. This invention generates structured diagnostic reports through collaborative optimization of two large models (GPT-4 multimodal analysis and DeepSeek knowledge-guided correction), clearly identifying the causes of misjudgments (such as "image overexposure leading to light cone distortion"), and combining this with a web system to enable doctors to intuitively understand the model's decision-making logic, thereby enhancing the model's interpretability and generalization ability.

[0240] In summary, this invention achieves high-precision and highly interpretable small-sample tympanic membrane image recognition through knowledge-guided feature enhancement, cross-modal semantic alignment, and dynamic optimization strategies, providing an innovative solution for the field of intelligent medical imaging diagnosis.

[0241] The above description is merely a specific implementation of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of the present invention should be included within the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention should be determined by the protection scope of the claims.

Claims

1. A small sample tympanic membrane image recognition method based on meta prompt and knowledge driving, characterized by, The method comprises: obtaining a small sample training set; wherein the training set is composed of multiple training samples, each training sample containing input data and its corresponding label, the input data containing a tympanic membrane image sample and its corresponding medical description text sample, and the label being the true class of the tympanic membrane image sample and the medical description text sample being the specific class description of the true class; inputting the training set into an initial multi-modal pre-training model, and performing class prediction on the tympanic membrane image sample by the model to output the predicted class of the tympanic membrane image sample; according to the comparison result of the predicted class and the true class, combining the confidence of the predicted class, filtering out misclassified samples from the training set, and constructing a meta-task set based on the filtered misclassified samples; inputting the meta-task set into a preset primary diagnosis model, processing the meta-task set by the model, and outputting a preliminary diagnosis report; wherein the preliminary diagnosis report contains a semantic conflict sample detected from the meta-task set, an image feature of the tympanic membrane image sample in the semantic conflict sample, a preliminary diagnosis class generated by the primary diagnosis model for the semantic conflict sample, a confidence distribution and a preliminary correction suggestion, the preliminary diagnosis class being the result of class prediction of the primary diagnosis model on the semantic conflict sample, and the confidence distribution being the probability of different class prediction of the primary diagnosis model on the semantic conflict sample; inputting the preliminary diagnosis report into a preset knowledge refining model, processing the related information in the preliminary diagnosis report by the model, and outputting an optimized medical description text sample; using the optimized medical description text sample to replace the original medical description text sample in the training set to obtain updated training samples; inputting the updated training samples into the initial multi-modal pre-training model again for iterative training until a preset termination condition is met; wherein the trained multi-modal pre-training model is used for actual tympanic membrane image recognition tasks; Further, the initial multi-modal pre-training model comprises an image encoder and a text encoder; inputting the training set into the initial multi-modal pre-training model, performing class prediction on the tympanic membrane image sample by the model, and outputting the predicted class of the tympanic membrane image sample, comprising: extracting features of the tympanic membrane image sample by the image encoder to generate corresponding image features; extracting features of the medical description text sample by the text encoder to generate corresponding text features; calculating the cosine similarity between the image features and the text features; performing cross-modal alignment processing on the image features and the text features according to the cosine similarity; performing joint inference on the aligned image features and text features to generate the predicted class of the tympanic membrane image sample.

2. The method of claim 1, wherein, extracting features of the tympanic membrane image sample by the image encoder to generate corresponding image features, comprising: extracting features of the tympanic membrane image sample to obtain initial image features; performing 1x1 convolution operation on the initial image features to obtain a first feature map; calculating gradients of the first feature map; element-wise multiplying the gradients and the initial image features to obtain an attention heat map; concatenating features in the attention heat map and hidden vectors of a previous layer to obtain concatenated features; linearly transforming the concatenated features by first learnable parameters to obtain linearly transformed features; activating the linearly transformed features to generate an attention mask; weighting features in the attention heat map by using the attention mask to obtain weighted features; adding the weighted features and second learnable parameters to obtain added features; performing layer normalization on the added features to obtain image features corresponding to the tympanic membrane image sample.

3. The method of claim 1, wherein, extracting features of the tympanic membrane image sample by the image encoder to generate corresponding image features, including: extracting features of the tympanic membrane image sample by a feature extraction network to obtain image feature maps at different levels; wherein the image feature maps contain local detail information and preliminary semantic information of the image; obtaining bottom-level fine-grained image features from the feature extraction network, processing the bottom-level fine-grained image features by third learnable parameters, and calculating an offset; obtaining multi-scale image features from different layers of the feature extraction network, for each scale of image features, based on the offset, performing a deformable convolution operation on the image features to obtain transformed features; weighting the transformed features corresponding to each scale by a pre-set weight of each scale to obtain weighted features; summing the weighted features corresponding to all scales to obtain image features corresponding to the tympanic membrane image sample.

4. The method of claim 1, wherein, extracting features of the medical description text sample by the text encoder to generate corresponding text features, including: input the medical description text sample into a graph neural network to generate text features; wherein the graph neural network takes medical terms as nodes and takes anatomical correlation strength between medical terms as edges, a text feature vector corresponding to the node is updated through a dynamic propagation function; the dynamic propagation function expression is as follows: wherein, denotes a node in a graph neural network In the first updated feature vector of the node denotes a current node in a graph neural network, denotes the iteration round number in a graph neural network, denotes the neighbor nodes of a node denotes the set of all neighbor nodes of a node , denotes the set of all neighbor nodes of a node , denotes the feature vector of a node and its neighbor nodes in the previous round, denotes the weight between a node and a node , denotes a concatenation operation for concatenating the feature vector of a node in the previous round and the weighted feature vectors of all its neighbor nodes, denotes a learnable weight matrix for linearly transforming the concatenated vector, denotes an activation function.

5. The method of claim 1, wherein, based on the comparison result of the predicted category and the true category, combining the confidence of the predicted category, filtering out misclassified samples from the training set, and constructing a meta-task set based on the filtered misclassified samples, including: if the predicted category does not match the true category and the confidence of the predicted category exceeds a pre-set confidence threshold, marking the corresponding training sample as a misclassified sample; constructing a meta-task set based on the marked misclassified samples.

6. The method of claim 1, wherein, inputting the meta-task set into a pre-set primary diagnosis model, processing the meta-task set by the model, and outputting a preliminary diagnosis report, including: for each misclassified sample in the meta-task set, generating a structured prompt file according to a pre-set prompt generation template; wherein the prompt file contains an image identifier and a true category of a tympanic membrane image sample in the misclassified sample, a predicted category generated by the initial multi-modal pre-training model for classifying the misclassified sample, a confidence distribution and image features of different category predictions, and related medical logical rules; for each misclassified sample in the meta-task set, calculating the cosine similarity between its corresponding image features and text features; The misclassified sample with the cosine similarity less than the preset similarity threshold is marked as a semantic conflict sample; A medical logic rule is extracted from a prompt file corresponding to the semantic conflict sample by a dynamic prompt analysis algorithm, and the medical logic rule is converted into executable verification instructions; According to the verification instructions, the semantic conflict sample is diagnosed to generate a preliminary diagnosis report.

7. The method of claim 1, wherein, The preliminary diagnosis report is input into a preset knowledge refinement model, and the model processes related information in the preliminary diagnosis report to output an optimized medical description text sample, including: According to the preliminary diagnosis category, the true category, and the keywords contained in the semantic conflict sample in the preliminary diagnosis report, relevant knowledge subgraphs are retrieved from a preset dynamic medical knowledge base; It is checked whether the preliminary correction suggestion in the preliminary diagnosis report is consistent with the rules in the knowledge subgraph; If yes, the medical description text sample in the semantic conflict sample is optimized based on the knowledge subgraph to obtain an optimized medical description text sample.

8. The method of claim 1, wherein, The updated training sample is input again into the initial multi-modal pre-training model for iterative training until a preset termination condition is met, including: The updated training sample is input again into the initial multi-modal pre-training model for iterative training; During the iterative training process, if the accuracy improvement amplitude of three consecutive iterations is less than a preset convergence threshold, it is determined that the optimization process has converged, and the iterative training is terminated at this time; In the formula, the initial multi-modal pre-training model adopts the following formula for parameter updating: wherein, denotes model parameters at time step denotes model parameters at time step denotes model parameters at time step denotes model parameters at time step denotes a meta learning rate, denotes a meta task set, denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters denotes a gradient of model parameters den 9. The method of claim 1, wherein, The method further includes: Receiving a user-uploaded image to be recognized; The multi-modal pre-training model deployed on the server is used to recognize the image to be recognized, and a predicted category, a corresponding specific category description, and a confidence distribution of each category are output; The predicted category, the corresponding specific category description, and the confidence distribution of each category are returned to the user terminal for visual display.

Images