Small sample tympanic membrane image recognition method based on meta prompt and knowledge driving

Abstract

The invention provides a small sample tympanic membrane image recognition method based on meta-prompt and knowledge driving, and the method comprises the steps: inputting a small sample training set into an initial multi-mode pre-training model, obtaining a prediction category, comparing the prediction category with a real category, and screening misclassification samples in combination with confidence to construct a meta-task set; and inputting the meta-task set into the primary diagnosis model, and outputting a primary diagnosis report. And then, optimizing the medical description text sample based on the preliminary diagnosis report by utilizing a knowledge refining model, and replacing the original text sample, so as to obtain an updated sample. And finally, iteratively training the initial multi-modal pre-training model by using the updated sample until a termination condition is met. According to the method, a closed-loop optimization system composed of a primary diagnosis model and a knowledge refining model is constructed. Under the condition of small samples, the system dynamically optimizes the visual-semantic understanding ability of the model by using error samples generated by the model, and the accuracy of small sample tympanic membrane image recognition is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a small sample tympanic membrane image recognition method based on meta-cues and knowledge-driven. Background Art

[0002] With the in-depth application of artificial intelligence in medicine, the need for accurate and efficient identification of eardrum diseases (such as acute otitis media, eardrum perforation, and tympanic sclerosis) is becoming increasingly prominent. However, current eardrum image recognition technology faces challenges with data scarcity and insufficient learning efficiency for small sample sizes. With limited sample resources, traditional methods struggle to maintain high performance, resulting in low accuracy in eardrum disease identification. Summary of the Invention

[0003] Based on this, it is necessary to provide a small sample tympanic membrane image recognition method based on meta-cues and knowledge-driven to address the above technical issues, including:

[0004] Obtain a small sample training set; wherein the training set is composed of multiple training samples, each training sample includes input data and its corresponding label, the input data includes a tympanic membrane image sample and its corresponding medical description text sample, the label is the true category of the tympanic membrane image sample, and the medical description text sample is a specific category description of the true category;

[0005] Inputting the training set into an initial multimodal pre-training model, allowing the model to perform category prediction on the tympanic membrane image sample and output the predicted category of the tympanic membrane image sample;

[0006] According to the comparison result between the predicted category and the true category, combined with the confidence level corresponding to the predicted category, misclassified samples are screened out from the training set, and a meta-task set is constructed based on the screened misclassified samples;

[0007] The meta-task set is input into a preset primary diagnostic model, which processes the meta-task set and outputs a preliminary diagnosis report; wherein the preliminary diagnosis report includes semantic conflict samples detected from the meta-task set, image features of tympanic membrane image samples in the semantic conflict samples, preliminary diagnosis categories, confidence distribution and preliminary correction suggestions generated by the primary diagnostic model for the semantic conflict samples, the preliminary diagnosis category is the result obtained by the primary diagnostic model performing category prediction on the semantic conflict samples, and the confidence distribution is the probability of the primary diagnostic model predicting different categories for the semantic conflict samples;

[0008] Inputting the preliminary diagnosis report into a preset knowledge refinement model, which processes the relevant information in the preliminary diagnosis report and outputs an optimized medical description text sample;

[0009] Using the optimized medical description text samples to replace the original medical description text samples in the training set to obtain updated training samples;

[0010] The updated training samples are inputted into the initial multimodal pre-training model again for iterative training until a preset termination condition is met; wherein the trained multimodal pre-training model is used for the actual tympanic membrane image recognition task.

[0011] This meta-cue- and knowledge-driven approach to small-sample tympanic membrane image recognition begins by feeding a small-sample training set consisting of tympanic membrane image samples, medical description text samples, and ground-truth class labels into an initial multimodal pre-trained model. The predicted categories are then compared with the ground-truth categories. Misclassified samples are filtered out using confidence scores to construct a meta-task set. This meta-task set is then fed into a primary diagnosis model, which outputs a preliminary diagnosis report containing semantically conflicting samples, image features, preliminary diagnosis categories, confidence distributions, and preliminary correction recommendations. A knowledge refinement model then optimizes the medical description text samples based on the preliminary diagnosis report, replacing the original text samples to obtain updated samples. Finally, the initial multimodal pre-trained model is iteratively trained using the updated samples until a termination condition is met. This method constructs a closed-loop optimization system consisting of a primary diagnosis model and a knowledge refinement model. Under small-sample conditions, this system leverages the model's own error samples (meta-tasks) to dynamically optimize the model's visual-semantic understanding capabilities through an iterative "diagnosis-verification-correction-rediagnosis" process. This improves the model's small-sample generalization and fine-grained recognition accuracy, thereby effectively enhancing the accuracy of small-sample tympanic membrane image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of a system architecture in one embodiment;

[0013] Figure 2 A schematic diagram of a Web interactive system in one embodiment;

[0014] Figure 3 A schematic diagram of the topology of tympanic membrane pathology knowledge in one embodiment;

[0015] Figure 4 A schematic diagram of a knowledge dynamic update process in one embodiment;

[0016] Figure 5 A schematic diagram of a template selection engine flow in one embodiment;

[0017] Figure 6 A schematic diagram of an overall framework in another embodiment;

[0018] Figure 7This is a medical context dynamic prompt template in another embodiment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0020] In one embodiment, see Figure 1 The system architecture shown in the figure includes four innovative modules: dynamic medical knowledge base, meta-feature alignment engine, meta-prompt dual-core optimization and explainable diagnosis system.

[0021] The dynamic medical knowledge base, as the core knowledge hub of the system, continuously integrates structured and updates multi-source medical knowledge (such as clinical guidelines, expert consensus, scientific research literature, and case reports), constructs a semantic network and quantitative standards for tympanic membrane pathology, and provides diagnostic logic, feature definitions, and rule constraints for subsequent modules.

[0022] The meta-feature alignment engine processes input tympanic membrane images. Its core task is to accurately cross-modally align image features (such as the shape, texture, and color distribution of the lesion) with semantic features in the knowledge base (such as "perforation edge blunt and rounded" and "effusion turbidity >70%). By leveraging advanced visual encoding (spatial attention, multi-scale feature fusion) and semantic encoding (medical context enhancement, dynamic term expansion) technologies, and utilizing a dynamic cue generation algorithm, it transforms complex visual information into structured feature representations rich in medical semantics, providing high-quality input for diagnostic models.

[0023] Metaprompt dual-core optimization is the core innovation of this invention. It constructs a closed-loop optimization system consisting of a primary diagnosis model (Meta-Diagnoser) and a knowledge refinement model (Knowledge-Refiner), coordinated by a dynamic iteration engine (Description Updater). Under small sample conditions, the system uses the knowledge base and the misclassified samples (meta-tasks) generated by the model itself, and through an iterative "diagnosis-verification-correction-rediagnosis" process, dynamically optimizes the model's visual-semantic understanding ability and diagnostic logic, significantly improving the small sample generalization ability and fine-grained recognition accuracy. The primary diagnosis model is used for preliminary diagnosis and locating feature conflicts; the knowledge refinement model performs logical verification and semantic refinement on the preliminary diagnosis based on the knowledge base to generate a more accurate and more medically compliant feature description.

[0024] The Interpretable Diagnosis System converts the analysis results of the aforementioned modules into a diagnostic report and visualization interface that clinicians can understand. Through a four-level interpretation system (region-level heat map, feature-level quantitative panel, decision-level inference path tree, and risk-level confidence indicator), it comprehensively displays the diagnostic basis, key pathological features, decision logic, and model confidence, addressing the "black box" problem of AI diagnosis and improving clinical acceptance. The web-based interactive system provides a user-friendly interface that supports image upload, analysis, result display, and report generation.

[0025] In one embodiment, a small sample eardrum image recognition method based on meta-cues and knowledge-driven is provided, the method comprising the following steps:

[0026] Step S102: Obtain a small sample training set.

[0027] The training set consists of multiple training samples, each of which contains input data and its corresponding label. The input data includes tympanic membrane image samples and their corresponding medical description text samples. The labels are the true categories of the tympanic membrane image samples, and the medical description text samples are the specific category descriptions of the true categories.

[0028] Specifically, we first need to build a meta-task module (Meta-Task Constructor)

[0029] When building this module, its input is the prepared small sample training set It can be expressed in mathematical form as:

[0030]

[0031] Among them, I i represents the i-th eardrum image sample, y i Indicates the label corresponding to the tympanic membrane image sample, and N represents the total number of samples in the training set.

[0032] Step S104: input the training set into the initial multimodal pre-training model, and the model outputs the predicted category of the eardrum image sample.

[0033] Optionally, the multimodal pre-training model is a Contrastive Language-Image Pre-training (CLIP) model.

[0034] Specifically, the initial CLIP model is run through the error sampler with the training set as input to generate prediction results.

[0035] Step S106 , based on the comparison results of the predicted category and the true category and the confidence level corresponding to the predicted category, misclassified samples are screened out from the training set to construct a meta-task set.

[0036] Specifically, if the predicted category does not match the true category, and the confidence corresponding to the predicted category exceeds the preset confidence threshold, the corresponding training sample is marked as a misclassified sample, and a meta-task set is constructed based on the marked misclassified sample.

[0037] In a specific example, the error sampler is used to filter out the prediction results from all the generated prediction results. and the true label y i Inconsistency, that is And confidence High-certainty error samples are marked as misclassified samples.

[0038] Step S108: input the meta-task set into a preset primary diagnosis model (Meta-Diagnoser), and the model outputs a preliminary diagnosis report.

[0039] Among them, the preliminary diagnosis report includes the semantic conflict samples detected from the meta-task set, the image features corresponding to the semantic conflict samples, the preliminary diagnosis categories and confidence distribution generated by the primary diagnosis model, and preliminary correction suggestions.

[0040] Specifically, step S108 is implemented by the following steps:

[0041] Step S1081 : For each misclassified sample in the meta-task set, a structured prompt file is generated according to a preset prompt generation template.

[0042] The prompt file contains the image identification, true category, predicted category, confidence distribution, image features and medical logic rules corresponding to the misclassified samples.

[0043] In step S1081, the multimodal prompt generator generates a Call the dynamic template engine to generate a structured prompt file. The file can be in JSON format.

[0044] The following is an example of a Prompt file:

[0045]

[0046] Medical logic rules (i.e., knowledge_constraints) can be generated by matching the true category corresponding to the tympanic membrane image sample in the misclassified sample, the predicted category generated by the initial multimodal pre-trained model for the misclassified sample, the confidence distribution of different category predictions, and keywords contained in at least one of the image features with a pre-built medical logic rule library to obtain the corresponding medical logic rules. Medical logic rules can also be pre-configured content in the prompt file.

[0047] Step S1082 : For each misclassified sample in the meta-task set, calculate the cosine similarity between the corresponding image feature and text feature.

[0048] Specifically, for each misclassified sample in the meta-task set, the cosine similarity between the image feature and the text feature is calculated.

[0049] Step S1083 : Mark misclassified samples whose cosine similarity is less than a preset similarity threshold as semantic conflict samples.

[0050] Specifically, when sim<δ (threshold δ=0.4), it is marked as a semantic conflict sample.

[0051] Step S1084 , extracting medical logic rules from the prompt file corresponding to the semantic conflict sample through a dynamic prompt parsing algorithm, and converting the medical logic rules into verification instructions.

[0052] Specifically, the following is the dynamic prompt parsing algorithm:

[0053]

[0054] Step S1085: Process the semantic conflict sample according to the verification instruction and generate a preliminary diagnosis report.

[0055] Specifically, the semantic conflict samples are processed according to the verification instructions, and correction suggestions containing three elements are output, as follows: First, the feature correction item, such as correcting "tympanic membrane is not visible" to "tympanic membrane is partially visible (visible area <35%)"; second, quantitative parameter supplementation, such as "adding perforation diameter grading indicators, specifically divided into small (<2mm), medium (2-4mm), large (>4mm)"; third, misjudgment attribution coding, classifying misjudgment situations as data quality (Data-Quality) or semantic gap (Semantic-Gap), and outputting the corresponding probability distribution. In addition, the semantic conflict samples are used as input, and the preliminary diagnosis model is run to generate preliminary diagnosis categories. A cross-modal attention enhancement layer is set in the preliminary diagnosis model, which inserts a learnable weight matrix To achieve pathological area positioning, the calculation formula is:

[0056]

[0057] Among them, Attention cross (Q, K, V) represents the output of the feature response heat map focused on the tympanic membrane perforation area, Q = φ (I i ) represents the image I i The features extracted from φ represent a feature extraction function used to map the image to a feature vector space. Representation and Image I i Corresponding text features, ψ represents the function that maps text to feature vector space, Represents a scaling factor to prevent the dot product result from being too large.

[0058] Finally, the image features corresponding to the semantic conflict samples, the preliminary diagnosis categories, the confidence distribution generated by the primary diagnosis model, the preliminary correction suggestions, and the semantic conflict samples detected from the meta-task set are integrated to generate a preliminary diagnosis report.

[0059] Step S110: input the preliminary diagnosis report into a preset knowledge refinement model, and the model outputs an optimized medical description text sample.

[0060] Specifically, the knowledge refinement model plays the role of “expert review”, which receives the preliminary diagnosis report (including image I i , preliminary diagnosis results (such as "normal tympanic membrane"), confidence distribution, detected visual-semantic conflict items (such as sim(...) < 0.4), and correction suggestions proposed by Meta-Diagnoser (such as feature correction items, quantitative parameter supplements), and based on the verified medical knowledge stored in the dynamic medical knowledge base (disease definitions, diagnostic criteria, differential diagnosis rules, quantitative parameter relationships, etc.), these preliminary results are deeply verified, logically reasoned, and semantically refined. As an embodiment, step S110 is specifically implemented by the following steps:

[0061] Step S1101 : Retrieve relevant knowledge subgraphs in a preset dynamic medical knowledge base based on the keywords in the preliminary diagnosis category, the real category, and the semantic conflict sample.

[0062] Specifically, the output of the preliminary diagnosis model is received, including semantic conflict samples detected from the meta-task set (such as sim(...) < 0.4), image features corresponding to the semantic conflict samples, preliminary diagnosis categories generated by the primary diagnosis model (such as "normal tympanic membrane"), confidence distribution, and preliminary correction suggestions (such as feature correction items, quantitative parameter supplementation). Based on the preliminary diagnosis results, the true labels from the meta-task, and the keywords in the semantic conflict samples (such as "chronic otitis media", "tympanic membrane perforation", "light cone integrity"), the relevant knowledge subgraph is retrieved in the dynamic medical knowledge base.

[0063] Step S1102 , checking whether the preliminary correction suggestion is consistent with the rules in the knowledge subgraph.

[0064] Specifically, when performing logic rule verification, we first need to check whether the preliminary correction suggestions output by the preliminary diagnosis model are consistent with the knowledge subgraph retrieved from the dynamic medical knowledge base. For example, if it is recommended to "revise 'tympanic membrane invisible' to 'tympanic membrane partially visible (visible area <35%)'", it is necessary to verify whether there is a rule in the knowledge base that supports "visible area <35%" as a key diagnostic indicator (such as the degree of external auditory canal obstruction); secondly, it is necessary to verify whether the preliminary diagnosis result meets the diagnostic criteria in the knowledge base. For example, the knowledge base defines that "chronic otitis media" must meet "tympanic membrane perforation lasting >3 months" or "granulation tissue is present". If the image features and description do not reflect these conditions, the diagnosis result is questioned; finally, differential diagnosis is to be performed. With the help of the differential diagnosis (differential_diagnosis) edge in the knowledge base, it is necessary to analyze whether the current features are more consistent with other diseases (such as tympanic sclerosis).

[0065] Step S1103: If yes, the medical description text sample corresponding to the semantic conflict sample is optimized based on the knowledge subgraph to obtain an optimized medical description text sample.

[0066] Specifically, based on the term definitions, quantitative standards and relationships in the knowledge base, the original or revised medical description text samples generated by the preliminary diagnostic model are normalized and refined. For example, the original description "a yellow-brown area can be seen on the tympanic membrane" is refined to "calcification foci are seen in the posterior superior quadrant of the tympanic membrane" based on the knowledge base (associated with the typical manifestations of 'tympanic sclerosis' in the knowledge base), and the supplementary description is: diameter 1.8±0.3mm (reference knowledge base quantitative standards: common size range of punctate calcification foci), average CT value>120HU (based on the threshold value of CT value in the knowledge base to distinguish calcification from other substances). Tympanic sclerosis needs to be ruled out (based on the knowledge base differential diagnosis rules: if the density of calcification foci is>5 / mm 2(The diagnosis suggests tympanic sclerosis and requires further measurement or examination). Simultaneously, quantitative parameters are automatically supplemented based on the quantitative relationships (quantified_by edges) in the knowledge base, adding necessary measurement indicators, classification criteria, and error ranges. For example, perforation diameter is classified as small (<2mm), medium (2-4mm), and large (>4mm±0.2mm). Furthermore, the knowledge graph path is used to generate interpretable reasoning steps from image features to the final diagnosis, forming a diagnostic reasoning chain. Finally, the knowledge-verified and refined medical description text sample, the precise diagnosis category (which may be revised or maintained), the updated confidence level, and the diagnostic reasoning path are passed to the dynamic iteration engine.

[0067] Step S112: Use the optimized medical description text sample to replace the original medical description text sample to obtain an updated training sample.

[0068] In step S114, the updated training samples are inputted into the initial multimodal pre-training model again for iterative training until a preset termination condition is met.

[0069] Among them, the trained multimodal pre-training model is used for the actual tympanic membrane image recognition task.

[0070] Specifically, the dynamic iteration engine receives the output from the knowledge refinement model (i.e., the refined diagnostic descriptions and conclusions that meet the knowledge base standards) and feeds these optimized descriptions back to the meta-feature alignment engine and / or the primary diagnostic model to update the model's internal representation or parameters, thereby improving the model's diagnostic accuracy and semantic alignment capabilities the next time it encounters a similar image. This process is carried out on the screened meta-task sample set. The above iteration is carried out. The iterative steps are as follows:

[0071] First, the knowledge refinement model receives optimized medical description text samples and precise diagnosis categories for the current batch of meta-task samples. Next, the parameters are updated, treating the optimized medical description text samples as more accurate and richer supervisory signals. Losses (such as contrastive loss or cross-entropy loss) are calculated, and the weights of relevant models (primarily the visual / semantic encoder in the meta-feature alignment engine, the dynamic prompt generator, and the parameters in the preliminary diagnosis model) are updated according to a meta-learning-based optimization strategy. The meta-learning-based optimization strategy is expressed as follows:

[0072]

[0073] Among them, θ t+1 represents the model parameters at time step t+1, θ t represents the model parameters at time step t, α represents the meta-learning rate, represents the meta-task set, represents the gradient, I i represents the i-th eardrum image sample, represents the corresponding optimized medical description text sample, Represents the alignment loss between image features and text features.

[0074] Finally, make an evaluation and terminate the judgment. Or evaluate the performance of the updated model (such as diagnostic accuracy Acc) on a small validation set, set the convergence threshold ∈ = 0.03, and when the accuracy of three consecutive iterations is improved by ΔAcc < ∈, determine that the optimization process has converged and terminate the iteration, otherwise return to step 102 for the next iteration (you can choose to use the updated model for resampling or construct a new meta-task set for subsequent iterations).

[0075] In one embodiment, the method further comprises the following steps:

[0076] Receive the eardrum image to be identified uploaded by the user;

[0077] Identify the eardrum image to be identified using a multimodal pre-trained model deployed on a server, and output a predicted category, a corresponding category description, and a confidence score for each category;

[0078] The predicted category, the corresponding specific category description, and the confidence distribution of each category are returned to the user terminal for visual display.

[0079] Specifically, if Figure 2 As shown, the present invention also provides a Web interactive system, which adopts a front-end and back-end separation architecture, builds a server-side RESTful API with the Flask framework as the core, and combines HTML5 / CSS3 / JavaScript (frameworks such as Vue.js / React can be used to implement a responsive front-end interface, aiming to provide clinicians with an intuitive and efficient tympanic membrane image intelligent diagnostic tool. Among them,

[0080] The backend (Flask Server) is deployed on a Linux server.

[0081] Core functions: Receive tympanic membrane images uploaded by the front-end; call the meta-feature alignment engine and Metaprompt dual-core optimization mechanism for processing; generate structured data including original images, diagnosis results, four-level interpretation information, and diagnosis reports (which can be exported as PDF); access the dynamic medical knowledge base to provide relevant medical knowledge query support.

[0082] Implement API interfaces: / upload (image upload), / analyze (start analysis), / result / <image_id> (Get results), / report / <image_id>(Generate report).

[0083] Front-end (Web UI): Adapts to mainstream browsers (Chrome / Firefox / Edge / Safari). The core interaction process includes:

[0084] User login / registration: Doctor identity authentication and operation history recording.

[0085] Image upload: Supports drag and drop or file selector to upload single or multiple tympanic membrane images (common formats: JPG, PNG, DICOM).

[0086] Analysis start and status: Automatically or manually trigger analysis after uploading, and display a processing progress bar or status prompt.

[0087] Diagnosis results display:

[0088] Main View: Displays the original uploaded image side-by-side with a visualized image overlaid with a regional heatmap. The heatmap color mapping (e.g., red-yellow-green) visually indicates the model confidence level of the lesion area (probability > 0.8 indicates a high-confidence lesion in red). Image zooming and panning are supported.

[0089] Feature-level panel: Key pathological features and their quantitative values ​​are clearly listed in a table or card format. For example: 1) Perforation diameter: 3.2 mm (medium) [according to the knowledge base definition: 2-4 mm]; 2) Fluid turbidity: 78% (severe turbidity) [threshold reference: >70% is severe]; 3) Calcification density: 4.1 / mm 2 [Reference: The critical value of tympanosclerosis is 5 / mm 2 ]; 4) Light cone integrity loss rate: 12% [Rule: <10% excludes acute otitis media]

[0090] Decision-level reasoning tree: This displays the complete logical chain of diagnostic reasoning in a collapsible / expandable tree diagram. The root node represents the final diagnosis conclusion, while the child nodes represent the key features supporting that conclusion and the knowledge base rules they satisfy. Leaf nodes can be associated with corresponding image evidence regions.

[0091] For example:

[0092] Risk-level confidence indicator: A 3D visual encoding is used to display the model's overall confidence in the final diagnosis. 1) Shape: A 3D prism (e.g., a cube indicates low confidence, while a fine-faceted crystal indicates high confidence); 2) Color: A gradient from red (low confidence) to green (high confidence); 3) Transparency: The higher the confidence level, the more opaque the object. (Example: A rough cube rendered in translucent red indicates low confidence in the diagnosis of "acute otitis media"; a fine-faceted icosahedron rendered in opaque green indicates high confidence in the diagnosis of "normal tympanic membrane").

[0093] Report generation and export: Integrate all visual information and text descriptions to generate a structured diagnostic report (including patient information, images, diagnostic conclusions, detailed feature analysis, reasoning basis, and confidence level description) with one click. Support PDF, HTML format export or direct printing.

[0094] Knowledge base query (optional): An integrated knowledge base search portal allows doctors to conduct instant queries on terms or rules mentioned in the diagnostic report to obtain more detailed medical background knowledge.

[0095] In one embodiment, the front-end interface uses blue-gray tones to create a professional medical style, adopts a responsive layout, uses an upper and lower column design on the desktop, and dynamically adjusts the element spacing on the mobile terminal to ensure readability. At the same time, it adopts a 4-level interpretation system to improve the interpretability of the diagnosis.

[0096] Regional level: Lesion area heatmap. Use color overlays to intuitively locate suspicious lesion areas (e.g., red indicates areas where the model has determined a lesion with a confidence level > 0.8, and yellow indicates areas with medium confidence). Color depth is positively correlated with confidence, and supports switching between original images.

[0097] Feature level: Key pathological parameter panel. In the form of structured data tables or information cards, clearly display the quantitative features and their values ​​identified in the model that play a decisive role in the diagnosis conclusion. Each feature item can be associated with the knowledge base definition and diagnostic criteria (hover the mouse to prompt or click to view details). For example, clearly list "perforation diameter: 3.2mm (medium)", "effusion turbidity: 78% (severe turbidity)", "calcification density: 4.1 / mm 2 "wait.

[0098] Decision-making level: Diagnostic reasoning path tree. This fully presents the logical reasoning chain from image visual evidence to the final diagnosis using a dynamic, interactive tree diagram (e.g., implemented using D3.js). The root node represents the diagnosis, while branch nodes represent key features and their matching with knowledge base rules (e.g., "perforation diameter > 2mm → meets criteria for chronic otitis media"). Leaf nodes link back to specific areas on the image. Expanding / collapsed branches facilitates physicians' tracing and understanding of the AI's decision-making process.

[0099] Risk Level: A 3D confidence indicator. This uses a combination of visual encodings—shape complexity (simple geometry -> complex polyhedron), color (red -> orange -> yellow -> green), and transparency (highly transparent -> opaque)—to render a dynamic object in 3D space. This intuitively and unambiguously conveys the model's overall confidence level in the current diagnosis result. This design overcomes the limitations of single numbers or progress bars, providing a richer and more intuitive experience.

[0100] In one embodiment, step S104 specifically includes the following steps:

[0101] Step S1041, performing feature extraction on the eardrum image sample through an image encoder to generate image features;

[0102] Step S1042, performing feature extraction on the medical description text sample through a text encoder to generate text features;

[0103] Step S1043, calculating the cosine similarity between the image feature and the text feature;

[0104] Step S1044: performing cross-modal alignment processing on the image features and the text features according to cosine similarity;

[0105] Step S1045 : performing joint inference on the aligned image features and text features to generate a predicted category of the eardrum image sample.

[0106] In one embodiment, step S1041 specifically includes the following steps:

[0107] Step S2011, extracting features from the eardrum image sample to obtain initial image features;

[0108] Step S2012, performing a 1×1 convolution operation on the initial image features to obtain a first feature map;

[0109] Step S2013, calculating the gradient of the first feature map;

[0110] In step S2014, the gradient is multiplied element-by-element by the initial image features to obtain an attention heat map; wherein the features in the attention heat map are the generated image features.

[0111] Specifically, steps S2012 to S2014 are implemented by the following expressions:

[0112]

[0113] Among them, H att represents the attention heat map, F v Represents the initial image features, Conv 1×1 represents a 1D convolution operation, Indicates the degree of change (gradient) of image features.

[0114] In another embodiment, step S1041 specifically includes the following steps:

[0115] Step S2021, performing feature extraction on the eardrum image sample to obtain initial image features;

[0116] Step S2022: concatenate the initial image features and the hidden vector of the previous layer to obtain concatenated features;

[0117] Step S2023, performing a linear transformation on the concatenated features using the first learnable parameter to obtain linearly transformed features;

[0118] Step S2024, activating the linearly transformed features to generate an attention mask;

[0119] Step S2025, weighting the initial image features using the attention mask to obtain weighted features;

[0120] Step S2026, adding the weighted feature to the second learnable parameter to obtain the added feature;

[0121] Step S2027: perform layer normalization on the added features to obtain image features.

[0122] Specifically, steps S2022 to S2027 are implemented by the following expressions:

[0123]

[0124] Among them, F v is the image feature, H t-1 is the hidden vector of the previous layer, W m ,b m is a learnable parameter, and LayerNorm is a layer normalization operation.

[0125] In another embodiment, step S1041 specifically includes the following steps:

[0126] Step S2031, extracting features from the eardrum image sample to obtain initial image features;

[0127] Step S2032: performing a 1×1 convolution operation on the initial image features to obtain a first feature map;

[0128] Step S2033, calculating the gradient of the first feature map;

[0129] Step S2034: multiply the gradient by the initial image feature element by element to obtain an attention heat map;

[0130] Step S2035: Concatenate the features in the attention heat map with the hidden vector of the previous layer to obtain concatenated features;

[0131] Step S2036, performing a linear transformation on the concatenated features using the first learnable parameter to obtain linearly transformed features;

[0132] Step S2037, activating the linearly transformed features to generate an attention mask;

[0133] Step S2038, weighting the initial image features using the attention mask to obtain weighted features;

[0134] Step S2039, adding the weighted feature to the second learnable parameter to obtain the added feature;

[0135] Step S2040: perform layer normalization on the added features to obtain image features.

[0136] In this embodiment, a pathological region self-positioning module is innovatively introduced, and the focusing accuracy on the tense part of the tympanic membrane is improved by 37.8% in the detection of inflammatory lesions of the tympanic membrane.

[0137] In one embodiment, step S1041 specifically includes the following steps:

[0138] Step S2051: extract features from the eardrum image sample using a feature extraction network to obtain image feature maps at different levels; wherein the image feature maps contain local detail information and preliminary semantic information of the image;

[0139] Step S2052: Obtain underlying fine-grained image features from the feature extraction network, process the underlying fine-grained image features using a third learnable parameter, and calculate an offset;

[0140] Step S2053: Obtain multi-scale image features from different layers of the feature extraction network. For each scale of the image features, transform the image features using a deformable convolution operation based on the offset to obtain transformed features.

[0141] Step S2054: weighting the transformed features corresponding to each scale based on the preset weights for each scale to obtain weighted features;

[0142] In step S2055 , the weighted features corresponding to all scales are summed to obtain image features.

[0143] Specifically, step S2052 is implemented by the following expression:

[0144]

[0145] in, represents fine-grained image features, W offset are the parameters that need to be learned.

[0146] Steps S2053 to S2055 are implemented by the following expressions:

[0147]

[0148] in, Represents the i-th layer feature of the image, α i Represents the weight of the i-th layer feature, and DeformConv is a deformable convolution operation.

[0149] In one embodiment, step S1042 specifically includes the following steps:

[0150] Medical description text samples are input into the graph neural network to generate text features. The graph neural network uses medical terms as nodes and the anatomical association strength between medical terms as edges. The feature vectors corresponding to the nodes are updated using a dynamic propagation function. The dynamic propagation function expression is as follows:

[0151]

[0152] in, represents the feature vector of node v in the graph neural network after the kth round of update, v represents the current node in the graph neural network, k represents the number of iterations in the graph neural network, u represents the neighbor node of node v, N(v) represents the set of all neighbor nodes of node v, represents the feature vector of node v and its neighbor node u in the previous round, β uv Represents the weight between node u and node v, CONCAT represents the concatenation operation, which is used to concatenate the feature vector of node v in the previous round. It is concatenated with the weighted feature vectors of all its neighboring nodes. W represents a learnable weight matrix used to perform linear transformation on the concatenated vector, and ReLU represents the activation function.

[0153] In this embodiment, to address the challenge of ambiguity in medical terms (e.g., “turbid” may refer to exudate or calcification), professional expressions are converted into 256-dimensional computable vectors, with a disambiguation accuracy rate of 92.6%.

[0154] The dynamic medical knowledge base is the knowledge center of the system, aiming to solve the problem of the gap between medical knowledge and visual features. Traditional methods rely on static labels or simple rule bases, which cannot adapt to the fine-grained feature description of tympanic membrane lesions (such as the quantitative standard of "perforation edge bluntness") and the dynamic update of clinical guidelines. The present invention constructs a computable multimodal knowledge topology to achieve the dynamic evolution and precise call of medical knowledge. In one embodiment, the steps of constructing a dynamic medical knowledge base are as follows:

[0155] Step S122: Multi-source medical data collection and processing

[0156] Data source and processing flow:

[0157] (1) Structured data (30%):

[0158] Clinical Electronic Medical Records: Analyze XML / HL7 format medical records and extract key fields (diagnostic conclusions, lesion sites, surgical records)

[0159] Medical Knowledge Graph: Integrate otolaryngology terms in SNOMED-CT and MeSH to construct entity relationships (such as tympanic membrane perforation → complications → hearing loss)

[0160] Quantitative Research Database: Import measurement indicators (perforation diameter, CT value of effusion) from public datasets (such as CANDLE-PED otitis media library)

[0161] (2) Semi-structured data (45%):

[0162] Clinical Guidelines: Parse PDF using NLP pipelines (such as PDFPlumber + BiLSTM-CRF or large models) to extract diagnostic criteria (for example, the diagnosis of acute otitis media requires: tympanic membrane bulging + congestion area > 50%)

[0163] Expert Consensus: Use rule template matching to generate logical expressions (IF the light cone disappears AND blood vessel dilation THEN the probability of effusion > 80%)

[0164] (3) Unstructured data (25%):

[0165] Medical Literature: Obtain the latest papers using the PubMed API and jointly extract image-text association knowledge with the ViT-BERT model (caption description → image region)

[0166] Surgical Videos: Sample key frames (1 frame per 10 seconds), localize the tympanic membrane area using YOLOv7, and extract dynamic pathological features using 3D ResNet

[0167] Quality Control Mechanism:

[0168] Term Standardization: Map dialect terms based on the UMLS dictionary (such as "eardrum" →

[0169] "tympanic membrane").

[0170] Conflict Detection: Experts label conflicting entries (such as the threshold conflict of "small perforation" in different guidelines), and trigger confidence-weighted voting (weight = literature impact factor × 0.6 + expert title level × 0.4).

[0171] Data Desensitization: Inject noise (ε = 0.1) to protect patient privacy.

[0172] Step S124, Topology of Tympanic Pathology Knowledge, as Figure 3 shown.

[0173] Step S126, Dynamic Update Mechanism, as Figure 4 shown.

[0174] Version control uses "semantic fingerprinting" technology: when the similarity of knowledge changes is less than 0.85, an audit is triggered. Update process: automatic collection → difference detection → expert review → version release.

[0175] In one embodiment, the dynamic prompt generation algorithm includes the following:

[0176] (1) Template selection engine. Figure 5 As shown, the clinical decision tree and deep reinforcement learning are used to dynamically call the template library, and the template value evaluation function V(t)=Q is introduced. clin (t)×exp(-γ|F v -T t | 2 )(Q clin : Template confidence verified by hospital expert group, F v : Image features, T t : text features, γ preset hyper parameters):

[0177] (2) Feature injection mechanism. Unstructured medical knowledge is converted into computable parameters, combined with knowledge triple encoding, to construct (anatomical structure, pathological characteristics, quantitative standards) knowledge units, and the measurement error range is automatically embedded in the generated text.

[0178] (3) Temperature Adaptive Adjustment. This solves the similarity calculation drift problem in high-dimensional feature spaces, improving the recall rate of the model by 24.3% in the task of otoscope image and text alignment. It includes two core modules:

[0179] Dynamic temperature coefficient function:

[0180]

[0181] (k: learnable parameter, λ: gradient penalty factor, F v : Image features, rank(): feature sorting, Image feature gradient,).

[0182] Multimodal contrast loss function:

[0183]

[0184] Among them, sim(F v ,F t ) represents the similarity calculation between image features and text features, τ is the weight factor, and N represents all samples involved in the calculation.

[0185] In one embodiment, the key technologies for implementing the knowledge refinement model include:

[0186] Knowledge base embedding layer: embedding structured knowledge base (JSON format) encoded as a topology that can be processed by a graph neural network (GNN):

[0187] Nodes: disease entities (such as "chronic otitis media"), pathological features (such as "tympanic membrane perforation"), anatomical structures (such as "pars tensa"), quantitative parameters (such as "perforation diameter")

[0188] Edge: Medical relationship (has_symptom, differential_diagnosis, quantified_by, location_in, contraindicates)

[0189] Image-text joint reasoning: The model simultaneously processes the input image features (from the alignment engine or Meta-Diagnoser's visual encoding) and the knowledge base subgraph related to the current diagnostic task (obtained by retrieving the knowledge base by querying keywords or entities in the preliminary diagnosis results).

[0190] Image-text alignment loss function: used to train the model to ensure that its knowledge-based reasoning output is consistent with image evidence and standard medical descriptions:

[0191]

[0192] Where: ε is the set of connected edges in the knowledge graph, h i ,h j is the embedding vector of the connected nodes, KL(p GNN |p RoBERTa ) is the KL divergence, constraining the probability distribution p of the diagnosis result of the graph neural network (GNN) GNN Compared with the distribution p of text semantics-based models (such as RoBERTa) RoBERTa Keep consistent, sim(F v ,h diagnosis ) is the image visual feature F v and the refined diagnostic entity embedding h diagnosis Similarity metrics (such as cosine similarity) are used to ensure that the diagnosis conclusion is supported by image evidence. λ and γ are weight coefficients that balance various losses.

[0193] Based on the same invention, please see Figure 6 In one embodiment, a small sample eardrum image recognition method based on meta-hints and knowledge-driven is also provided, comprising the following steps:

[0194] Step S301: Obtain a training set, which consists of multiple training samples. Each training sample contains input data and a corresponding label. The input data contains a tympanic membrane image sample and a corresponding medical description text sample, and the label contains the true category of the tympanic membrane image sample and the corresponding category description.

[0195] Step S302: Input the training set into the initial multimodal pre-training model for training to generate classification results.

[0196] Specifically, the specific implementation of step S302 is as follows:

[0197] Step S3021: extract features from the eardrum image sample through an image encoder to generate an image feature vector.

[0198] In this step, the image encoder can use the Vision Transformer (ViT) structure to segment the input eardrum image sample into a 16×16 pixel tile sequence, extract global context features through a 24-layer Transformer encoder, and output an image feature vector with a dimension of 768.

[0199] Step S3022: Perform feature extraction on the medical description text sample through a text encoder to generate a text feature vector.

[0200] In this step, the medical description text sample is semantically enhanced using a manually designed medical context dynamic prompt template. The specific generation process involves receiving the user-provided true category and corresponding category description of the eardrum image sample, and then populating the preset medical context dynamic prompt template with the true category and corresponding category description to generate the medical description text sample.

[0201] The medical context dynamic prompt template contains multiple,multi-perspective prompts to enhance the discrimination of text representation.,The template generation process adopts a context-aware strategy,,e.g., Figure 7 For the category "chronic otitis media" and the corresponding feature description, the system dynamically generates the following prompt text: ["Otoendoscopy shows: chronic otitis media, typical features include marginal perforation of the pars tensa of the tympanic membrane...", "According to the otological diagnostic criteria, this image shows the characteristics of chronic otitis media:

[0202] Tympanic mucosa congestion and edema..."].

[0203] The text encoder can use the RoBERTa model to perform word segmentation and position encoding on the input medical description text sample, and then generate a text feature vector through a 12-layer Transformer layer. In a specific implementation, the medical description text sample generated by each medical context dynamic prompt template is extracted by an encoder such as RoBERTa, and then feature fusion is performed using mean pooling. The processing formula is:

[0204]

[0205] in, represents the text feature vector of category j, K represents the number of medical context dynamic prompt templates, that is, there are K medical context dynamic prompt templates for each category.

[0206] Step S3023: Calculate the cosine similarity between the image feature vector and the text feature vector.

[0207] In this step, the cosine similarity calculation formula is:

[0208]

[0209] Among them, s(I,T j ) represents the cosine similarity, I represents the eardrum image sample, φ(I) represents the image feature vector, and τ represents the temperature coefficient, which is used to adjust the similarity distribution.

[0210] Step S3024: Perform cross-modal alignment processing on the image feature vector and the text feature vector based on cosine similarity.

[0211] Step S3025 : performing joint inference on the aligned image feature vector and text feature vector to generate a predicted category of the eardrum image sample.

[0212] In this step, the joint reasoning process adopts the maximum similarity criterion, and the processing formula is:

[0213]

[0214] in, represents the predicted category, represents the average similarity between the tympanic membrane image sample I and all K medical context dynamic prompt templates of category j, argmax j (·) means selecting the j value that maximizes the expression in the brackets, that is, selecting the category with the highest average similarity as the prediction result. Represents the kth medical context dynamic prompt template of category j.

[0215] By averaging multiple medical context dynamic prompt templates, the impact of single template deviation is reduced, and the multi-angle semantic expression of the category is covered (for example, "chronic otitis media" can correspond to different clinical manifestations such as "tympanic membrane perforation" and "ear canal effusion").

[0216] Step S3026: Compare the predicted category with the actual category to determine the classification result.

[0217] Step S303 , screening misclassified samples based on the classification results, and generating structured error prompt information according to the classification results of the misclassified samples and the preset analysis tasks.

[0218] Specifically, for misclassified samples (e.g., misclassifying "chronic otitis media" as "normal eardrum"), the system calls an automated script (generate_prompts.py) to parse the classification results. A state machine is used to extract key information, such as the sample ID, image path, true category, predicted category, and confidence scores for all categories, line by line. Structured error prompts for misclassified samples are stored in the "prompt_sampleID.txt" format and contain four parts: basic image information (e.g., resolution 420×380 pixels), a description of classification contradictions (e.g., semantic conflicts between the true and predicted categories), a visualization of the confidence distribution (showing the model's probabilistic bias for candidate categories), and an analysis task list (requiring feedback from four dimensions: feature verification, description accuracy, misclassification attribution, and improvement suggestions).

[0219] Step S304: input the structured error prompt information and the corresponding eardrum image sample into the multimodal large model to generate an error analysis report.

[0220] Specifically, during the primary optimization phase, the general multimodal large model GPT-4 (which can be an example of a primary diagnostic model) receives structured error information and corresponding tympanic membrane image samples, and achieves visual-text feature alignment through a cross-modal attention mechanism. The visual branch uses a deformable convolutional network to extract local pathological features (such as the ellipticity of the tympanic membrane perforation edge shape and the density distribution of earwax plug clumps). The text branch parses the medical logic chain in the prompt template (for example, "tympanic membrane invisible" requires the external auditory canal to be obscured by more than 80%). For example, in a sample (ID: 002-41e) with overexposure resulting in distorted light cone shape, GPT-4 detected that the anterior quadrant of the tympanic membrane was visible for 35%, triggering a correction to the description of "tympanic membrane partially visible (visible area <35%)" and annotating the feature with a confidence score of 0.63. Error attribution analysis uses a decision tree decomposition strategy, categorizing the root causes of misjudgment as data quality defects (32%, such as image blur and abnormal lighting) or insufficient semantic representation (68%, such as "tympanic membrane calcification" without a clear distribution pattern of calcification foci).

[0221] Step S305: Input the error analysis report into the large language model, and optimize the category description by calling the tympanic membrane medical knowledge base through the large language model.

[0222] Specifically, in the secondary optimization stage, the DeepSeek large language model (which can be an example of a knowledge refinement model) is combined with the structured knowledge base (medical_knowledge.json) to complete the fine-grained correction of the label description.

[0223] Step S306: Update the medical description text sample according to the optimized category description to obtain an updated training sample.

[0224] Step S307: input the updated training samples into the initial multimodal pre-training model for iterative training until a preset termination condition is met.

[0225] In this meta-cue-based, knowledge-driven small-sample tympanic membrane image recognition method, a training set consisting of tympanic membrane images and medical descriptions is constructed. Initial multimodal pre-trained models are used for preliminary classification, and based on the classification results, misclassified samples are screened to generate structured error prompts. Furthermore, a dual-model collaborative optimization mechanism, combining the GPT-4 multimodal large model and the DeepSeek large language model, is used to conduct in-depth analysis of erroneous samples and optimize category descriptions using a tympanic membrane medical knowledge base, enabling dynamic updating of training samples and iterative model training. This method achieves efficient and accurate tympanic membrane disease recognition with small sample sizes.

[0226] In one embodiment, the system adopts a phased iterative strategy with "coarse-grained correction → fine-grained optimization → threshold termination" as the evolution path: initial stage: fewer labeled samples are input for each category, and GPT-4 is used to quickly locate high-frequency misjudgment patterns (such as tympanic membrane calcification misjudgment rate >40%), and generate preliminary description correction suggestions; intermediate stage: based on the corrected description, a certain number of samples are added for fine-grained feature verification (such as perforation edge angle measurement error <5°); final stage: the remaining samples are introduced to complete semantic alignment optimization, and the iteration is terminated until the improvement in classification accuracy is lower than the preset threshold for three consecutive times.

[0227] In one embodiment, the process of constructing the tympanic membrane medical knowledge base includes:

[0228] Obtain tympanic membrane medical data;

[0229] Identify key fields in the original tympanic membrane medical data and extract relevant pathological feature descriptions;

[0230] Convert the extracted data into JSON format and construct a hierarchical data structure including disease category, full morphological description, histological status, functional indicators and identification points;

[0231] Based on the hierarchical data structure, a tympanic membrane medical knowledge base is constructed.

[0232] Specifically, the construction process of the tympanic membrane medical knowledge base is based on the systematic integration and structured representation of multimodal medical knowledge, aiming to provide computable fine-grained pathological feature priors for small sample tympanic membrane image recognition. The construction of the knowledge base begins with the systematic collection of authoritative medical literature, clinical practice guidelines and expert annotation data in the field of otolaryngology, covering common ear diseases such as chronic suppurative otitis media, cerumen impaction, and tympanosclerosis. Data sources include monographs such as "Practical Otorhinolaryngology Head and Neck Surgery" and "Clinical Diagnosis and Treatment Guidelines for Otitis Media", as well as ethically reviewed clinical imaging report texts to ensure the professionalism and clinical consistency of the knowledge content.

[0233] In the structured processing stage, an automated tool based on natural language processing (NLP) is used to parse and extract features from unstructured text. Specifically, the python-docx library is called through a Python script to parse the original Word document, identify key fields such as "morphological features" and "differentiation points", and extract relevant pathological descriptions. For example, for chronic otitis media, the system extracts feature descriptions such as "marginal perforation of the tense part of the tympanic membrane (blunt edges)" and "congestion and edema of the tympanic cavity mucosa with mucopurulent secretions", and annotates the clinical diagnostic rule of "need to be differentiated from cerumen plug (external auditory canal obstruction characteristics)". The extracted data is further converted into JSON format to construct a hierarchical data structure containing disease categories, full morphological descriptions, histological status, functional indicators and identification points.

[0234] The verification and quality control of the knowledge base are jointly completed by several senior ear experts, and cross-validation is used to ensure the accuracy of feature descriptions and the rigor of diagnostic logic. Experts review the feature entries of each type of disease one by one, focusing on verifying the following: 1) the visual consistency between the morphological description and the clinical image (such as "translucent grayish white tympanic membrane" must correspond to the fibrous layer structure of a healthy tympanic membrane); 2) the completeness of the coverage of key identification points (such as earwax plugs must clearly define the "degree of obstruction of the external auditory canal"); 3) the degree of standardization of terminology (such as "mucopurulent secretions" is standardized as "viscous purulent secretions"). The verified knowledge base is ultimately stored as a searchable matrix structure, which supports rapid alignment with image features through semantic similarity calculations, providing high-confidence medical prior knowledge support for subsequent multimodal fusion.

[0235] In summary, the key claim of this invention lies in a small-sample tympanic membrane image recognition method that integrates a structured medical knowledge base, dual-model collaborative Metaprompt optimization, and an interactive web diagnostic system. This method achieves high-precision recognition in data-scarce environments, possesses fine-grained feature classification capabilities, and exhibits good interpretability, bringing a new technological paradigm to the field of medical imaging AI.

[0236] The technical advantages of the present invention are as follows:

[0237] Solution to the data scarcity problem: Traditional deep learning is highly dependent on large-scale annotated data, but the annotation cost of tympanic membrane images is high and the sample distribution is uneven. This invention builds a structured medical knowledge base, encodes clinical guidelines and expert experience into computable morphological, histological and functional features, and provides rich prior knowledge for the model. Combined with a dynamic prompt template generation strategy (multiple medical context templates), the model can semantically align image features extracted from limited samples with pathological descriptions in the knowledge base, thereby significantly reducing dependence on annotated data.

[0238] Improved fine-grained classification capabilities: Identification of eardrum diseases relies on differences in microscopic features such as color gradients and perforation morphology (e.g., central vs. peripheral). This paper leverages the cross-modal alignment mechanism of the CLIP framework to map image features (e.g., ViT encoder extraction) and dynamically aggregated text features (e.g., RoBERTa mean pooling) into a unified semantic space. This, combined with the temperature coefficient (τ) to optimize similarity distribution, enables the model to accurately capture subtle differences in the distribution of eardrum calcifications and the density of external auditory canal masses.

[0239] Enhanced interpretability and generalization capabilities: Traditional models struggle to gain clinical trust due to their "black box" decision-making. This invention generates structured diagnostic reports through the collaborative optimization of two large models (GPT-4 multimodal analysis and DeepSeek knowledge-guided correction), clearly attributing misjudgments (such as "light cone distortion caused by image overexposure"). Combined with a web system, it enables doctors to intuitively understand the model's decision-making logic, thereby enhancing the model's interpretability and generalization capabilities.

[0240] In summary, the present invention achieves high-precision and highly interpretable small-sample tympanic membrane image recognition through knowledge-guided feature enhancement, cross-modal semantic alignment, and dynamic optimization strategies, providing an innovative solution for the field of intelligent diagnosis of medical images.

[0241] The above description is merely a specific implementation of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any variation or substitution that a person skilled in the art can easily conceive within the technical scope disclosed in the embodiments of the present invention should be included within the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention should be based on the protection scope of the claims.

Claims

1. A small sample tympanic membrane image recognition method based on meta-cues and knowledge-driven, characterized by: include: Obtain a small sample training set; wherein the training set is composed of multiple training samples, each training sample includes input data and its corresponding label, the input data includes a tympanic membrane image sample and its corresponding medical description text sample, the label is the true category of the tympanic membrane image sample, and the medical description text sample is a specific category description of the true category; Inputting the training set into an initial multimodal pre-training model, allowing the model to perform category prediction on the tympanic membrane image sample and output the predicted category of the tympanic membrane image sample; According to the comparison result between the predicted category and the true category, combined with the confidence level corresponding to the predicted category, misclassified samples are screened out from the training set, and a meta-task set is constructed based on the screened misclassified samples; The meta-task set is input into a preset primary diagnostic model, which processes the meta-task set and outputs a preliminary diagnosis report; wherein the preliminary diagnosis report includes semantic conflict samples detected from the meta-task set, image features of tympanic membrane image samples in the semantic conflict samples, preliminary diagnosis categories, confidence distribution and preliminary correction suggestions generated by the primary diagnostic model for the semantic conflict samples, the preliminary diagnosis category is the result obtained by the primary diagnostic model performing category prediction on the semantic conflict samples, and the confidence distribution is the probability of the primary diagnostic model predicting different categories for the semantic conflict samples; Inputting the preliminary diagnosis report into a preset knowledge refinement model, which processes the relevant information in the preliminary diagnosis report and outputs an optimized medical description text sample; Using the optimized medical description text samples to replace the original medical description text samples in the training set to obtain updated training samples; The updated training samples are inputted into the initial multimodal pre-training model again for iterative training until a preset termination condition is met; wherein the trained multimodal pre-training model is used for the actual tympanic membrane image recognition task.

2. The method according to claim 1, characterized in that The initial multimodal pre-training model includes an image encoder and a text encoder; the training set is input into the initial multimodal pre-training model, and the model performs category prediction on the eardrum image sample and outputs the predicted category of the eardrum image sample, including: Extracting features from the eardrum image sample using the image encoder to generate corresponding image features; Performing feature extraction on the medical description text sample by the text encoder to generate corresponding text features; Calculating the cosine similarity between the image feature and the text feature; Performing cross-modal alignment processing on the image features and the text features according to the cosine similarity; The aligned image features and text features are jointly inferred to generate a predicted category of the eardrum image sample.

3. The method according to claim 2, characterized in that Extracting features from the eardrum image sample by the image encoder to generate corresponding image features includes: Performing feature extraction on the eardrum image sample to obtain initial image features; Performing a 1×1 convolution operation on the initial image features to obtain a first feature map; Calculating the gradient of the first feature map; Multiplying the gradient by the initial image feature element-wise to obtain an attention heat map; Concatenating the features in the attention heat map with the hidden vector of the previous layer to obtain concatenated features; Performing a linear transformation on the concatenated features using a first learnable parameter to obtain linearly transformed features; Activating the linearly transformed features to generate an attention mask; Weighting the features in the attention heat map using the attention mask to obtain weighted features; Adding the weighted feature to a second learnable parameter to obtain an added feature; The added features are layer-normalized to obtain image features corresponding to the tympanic membrane image samples.

4. The method according to claim 2, characterized in that Extracting features from the eardrum image sample by the image encoder to generate corresponding image features includes: Using a feature extraction network to extract features from the eardrum image sample to obtain image feature maps at different levels; wherein the image feature maps contain local detail information and preliminary semantic information of the image; Obtaining underlying fine-grained image features from the feature extraction network, processing the underlying fine-grained image features using a third learnable parameter, and calculating an offset; Acquire multi-scale image features from different layers of the feature extraction network, and transform the image features at each scale using a deformable convolution operation based on the offset to obtain transformed features; Based on the preset weights of each scale, the transformed features corresponding to each scale are weighted to obtain weighted features; The weighted features corresponding to all scales are summed to obtain the image features corresponding to the eardrum image sample.

5. The method according to claim 2, characterized in that The text encoder is used to extract features from the medical description text sample to generate corresponding text features, including: The medical description text sample is input into a graph neural network to generate text features. The graph neural network uses medical terms as nodes and the anatomical association strength between medical terms as edges. The text feature vectors corresponding to the nodes are updated using a dynamic propagation function. The dynamic propagation function is expressed as follows: in, represents the feature vector of node v in the graph neural network after the kth round of update, v represents the current node in the graph neural network, k represents the number of iterations in the graph neural network, u represents the neighbor node of node v, N(v) represents the set of all neighbor nodes of node v, Represent the feature vectors of node v and its neighbor node u in the previous round, β uv Represents the weight between node u and node v, CONCAT represents the concatenation operation, which is used to concatenate the feature vector of node v in the previous round. It is concatenated with the weighted feature vectors of all its neighboring nodes. W represents a learnable weight matrix used to perform linear transformation on the concatenated vector, and ReLU represents the activation function.

6. The method according to claim 1, characterized in that According to the comparison result between the predicted category and the true category, combined with the confidence level corresponding to the predicted category, misclassified samples are screened out from the training set, and a meta-task set is constructed based on the screened misclassified samples, including: If the predicted category does not match the true category, and the confidence level of the predicted category exceeds a preset confidence threshold, the corresponding training sample is marked as a misclassified sample; Construct a meta-task set based on labeled misclassified samples.

7. The method according to claim 1, characterized in that The meta-task set is input into a preset primary diagnosis model, which processes the meta-task set and outputs a preliminary diagnosis report, including: For each misclassified sample in the meta-task set, a structured prompt file is generated according to a preset prompt generation template; wherein the prompt file includes the image identifier and true category corresponding to the tympanic membrane image sample in the misclassified sample, the predicted category generated by the initial multimodal pre-trained model for the category prediction of the misclassified sample, the confidence distribution and image features of different category predictions, and relevant medical logic rules; For each misclassified sample in the meta-task set, calculating the cosine similarity between the corresponding image feature and text feature; Marking misclassified samples whose cosine similarity is less than a preset similarity threshold as semantic conflict samples; Extracting medical logic rules from the prompt file corresponding to the semantic conflict sample through a dynamic prompt parsing algorithm, and converting the medical logic rules into executable verification instructions; According to the verification instruction, the semantic conflict sample is diagnosed to generate a preliminary diagnosis report.

8. The method according to claim 1, characterized in that The preliminary diagnosis report is input into a preset knowledge refinement model, which processes the relevant information in the preliminary diagnosis report and outputs an optimized medical description text sample, including: Retrieving relevant knowledge subgraphs in a preset dynamic medical knowledge base based on the keywords contained in the preliminary diagnosis category, the true category, and the semantic conflict sample in the preliminary diagnosis report; Checking whether the preliminary correction suggestions in the preliminary diagnosis report are consistent with the rules in the knowledge subgraph; If so, the medical description text sample in the semantic conflict sample is optimized based on the knowledge subgraph to obtain an optimized medical description text sample.

9. The method according to claim 1, characterized in that The updated training samples are inputted again into the initial multimodal pre-training model for iterative training until a preset termination condition is met, including: Inputting the updated training samples into the initial multimodal pre-training model again for iterative training; During the iterative training process, if the accuracy improvement of three consecutive iterations is less than the preset convergence threshold, the optimization process is considered to have converged and the iterative training is terminated. The initial multimodal pre-training model uses the following formula to update parameters: Among them, θ t+1 represents the model parameters at time step t+1, θ t represents the model parameters at time step t, α represents the meta-learning rate, represents the meta-task set, Represents the gradient of the model parameter θ, I i represents the i-th eardrum image sample, represents the corresponding optimized medical description text sample, Represents the alignment loss between image features and text features.

10. The method according to claim 1, characterized in that The method further comprises: Receive the eardrum image to be identified uploaded by the user; Identify the eardrum image to be identified using a multimodal pre-trained model deployed on a server, and output a predicted category, a corresponding specific category description, and a confidence distribution for each category; The predicted category, the corresponding specific category description, and the confidence distribution of each category are returned to the user terminal for visual display.

Images