Small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence

Abstract

The present invention provides an artificial intelligence-based small sample tympanic membrane image recognition and diagnosis system, which includes a category matching module, a category enhancement module, a knowledge enhancement module and a diagnosis generation module, wherein the category matching module is used to obtain an input image and an image category set, calculate the similarity between the input image and at least one image category, and determine the matching category of the input image based on the similarity; the category enhancement module is used to determine the disease category corresponding to the input image based on the input image, the matching category and the query; the knowledge enhancement module is used to determine the diagnosis and treatment plan for the disease category based on the structured knowledge corresponding to the matching category of the input image; and the diagnosis generation module is used to generate a diagnosis report corresponding to the input image.

Description

Technical Field

[0001] The present invention mainly relates to the field of image recognition technology, and in particular to an artificial intelligence-based small sample tympanic membrane image recognition and diagnosis system. Background Art

[0002] Otitis media is a common ear disease with high morbidity and multiple complications. Accurate endoscopic diagnosis is crucial for early detection, accurate treatment, and effective follow-up of otitis media. However, existing computer-assisted methods for diagnosing tympanic membrane lesions rely primarily on deep learning models that consume large amounts of annotated data. This presents challenges in the medical field, such as data acquisition difficulties and privacy protection. Summary of the Invention

[0003] Based on this, the present invention provides an artificial intelligence-based small sample tympanic membrane image recognition and diagnosis system, which can not only locate the disease category of the input image, but also determine the diagnosis and treatment plan corresponding to the input image, thereby generating a comprehensive diagnostic report and improving the accuracy and practicality of the diagnosis.

[0004] An artificial intelligence-based small sample tympanic membrane image recognition and diagnosis system, the system comprising:

[0005] a category matching module, configured to obtain an input image and a set of image categories, calculate a similarity between the input image and at least one image category, and determine a matching category for the input image based on the similarity;

[0006] a category enhancement module, configured to enhance the feature expression of the input image based on the input image, the matching category, and the first query, so as to determine the disease category corresponding to the input image;

[0007] a knowledge enhancement module, configured to enhance the feature expression of the input image and determine a diagnosis and treatment plan for the disease category based on the structured knowledge corresponding to the matching category of the input image;

[0008] The diagnosis generating module is used to generate a diagnosis report corresponding to the input image.

[0009] Wherein, the category matching module includes:

[0010] An image encoding module is used to obtain an image feature vector corresponding to an input image through a pre-trained image encoder;

[0011] A category encoding module, configured to obtain a category feature vector corresponding to at least one image category through a pre-trained category encoder;

[0012] The similarity matching module is used to map the image feature vector and the category feature vector to the same feature space, calculate the similarity between the image feature vector and the category feature vector, and determine the matching category of the input image according to the similarity.

[0013] The similarity matching module is specifically used to:

[0014] respectively calculating the similarity between the image feature vector and each image category in the category feature vector;

[0015] The image category with the highest similarity is selected as the matching category of the input image.

[0016] The pre-trained category encoder is used to obtain the category feature vector according to the received embedded text template and image category set; wherein the embedded text template is obtained through training.

[0017] Wherein, the category enhancement module includes:

[0018] a first self-attention layer, configured to perform a multi-head attention calculation based on the image feature vector of the input image and the query feature vector corresponding to the first query to obtain a first self-attention output;

[0019] a first cross-attention layer, configured to perform a cross-attention calculation based on the first self-attention output and a matching category of the input image to obtain a first cross-attention output;

[0020] A feed-forward layer is configured to determine a disease category of the input image based on the first cross-attention output.

[0021] Wherein, the knowledge enhancement module further includes:

[0022] A second self-attention layer is used to perform multi-head attention calculation according to the disease category of the input image to obtain a second self-attention output;

[0023] a second cross-attention layer, configured to perform a cross-attention calculation based on the second self-attention output and structured knowledge corresponding to the disease category of the input image to obtain a second cross-attention output;

[0024] The feedforward layer is used to determine the diagnosis and treatment plan corresponding to the input image according to the second cross-attention output.

[0025] The system further includes a knowledge screening and integration module, which includes:

[0026] a knowledge base, for storing at least one knowledge item;

[0027] A knowledge encoder, configured to encode at least one knowledge item in the knowledge base to obtain a knowledge feature vector corresponding to the at least one knowledge item;

[0028] A feature fusion device, configured to concatenate the image feature vector with the vector corresponding to the matching category to obtain a fused feature vector;

[0029] A knowledge filter is used to filter and obtain structured knowledge corresponding to the disease category of the input image from the knowledge base according to the fusion feature vector and the knowledge feature vector.

[0030] The knowledge filter is specifically used for:

[0031] respectively calculating a second similarity between each knowledge feature vector and the fusion feature vector;

[0032] sorting at least one of the second similarities, and selecting a preset number of knowledge items corresponding to the second similarities as selected knowledge;

[0033] The selected knowledge is processed by spacy library functions to generate the structured knowledge.

[0034] The diagnosis generation module is configured to obtain an output of the knowledge enhancement module, extract a second query from the output of the knowledge enhancement module, and generate a diagnosis report based on the second query.

[0035] Among them, the category enhancement module and the knowledge enhancement module are trained Transformer models; and the diagnosis generation module is a trained language model.

[0036] The artificial intelligence-based small sample tympanic membrane image recognition and diagnosis system of the present invention first determines the matching category of the input image through the category matching module, then enhances the feature expression of the input image through the category enhancement module to determine the disease category corresponding to the input image, then further enhances the feature expression of the input image through the knowledge enhancement module, and determines the diagnosis and treatment plan corresponding to the disease category, and finally generates a comprehensive and complete diagnosis report through the diagnosis generation module. This system can improve the accuracy and practicality of diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic structural diagram of a small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence according to an embodiment of the present invention;

[0038] Figure 2 Schematic diagram of the structure of a small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence according to another embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0040] Otitis media is a common ear disease with a high incidence and multiple complications. Accurate endoscopic diagnosis is crucial for early detection, accurate treatment, and effective follow-up of otitis media. However, existing computer-assisted tympanic membrane lesion diagnosis methods rely primarily on deep learning models that use large amounts of annotated data, which presents challenges in the medical field, such as data acquisition difficulties and privacy protection. Currently, the implementation solutions most similar to the present invention mainly include the following categories:

[0041] The first category involves image classification methods based on convolutional neural networks (CNNs). For example, Wu Zebin et al. used the Xception and MobileNetV2 models to classify normal, acute otitis media (AOM), and secretory otitis media (OME) categories. This method used a dataset of 12,203 otoscope images and achieved overall classification accuracies of 97.45% and 95.72%, respectively. However, this method requires a large amount of labeled data, and performance degrades significantly with small sample sizes.

[0042] The second category involves methods that combine object detection algorithms. For example, Lu Shuochen et al. used the Faster RCNN object detection algorithm to perform computer-assisted tympanic membrane diagnosis. They used image enhancement techniques to increase the number of otoscope images from 1,297 to 5,188, and employed ten-fold cross-validation, achieving an overall recognition accuracy of 91.43%. However, this method requires labeling the specific location of the lesion, which is often unnecessary and time-consuming in primary care.

[0043] The third category is CNN models that incorporate an attention mechanism. Alhudhif et al. proposed a CNN-based computer-aided decision-making model composed of three basic modules: an attention module, a residual block, and a hypercolumn. Using image enhancement techniques in preprocessing, they achieved an accuracy of 98.26% in classifying normal, AOM, chronic suppurative otitis media (CSOM), and earwax. While this approach improves model performance, it still requires a large amount of labeled data.

[0044] The fourth category is a hybrid architecture that combines CNN and long short-term memory network (LSTM), which is mainly used to process otoscope videos. Viscaino et al. proposed a functional hybrid architecture that combines CNN (such as ResNet-50, EfficientNet-B4, Inception-V3 and DenseNet) and LSTM. They use the ability of CNN to extract features and the ability of LSTM network to learn long-term dependencies in spatial sequence data to realize the diagnosis function of otoscope videos, achieving an average accuracy of 91.65%. However, this method is mainly used for video data, and the diagnostic effect of a single image may be limited.

[0045] The fifth category combines traditional machine learning methods with feature extraction techniques. Viscaino et al. used support vector machines (SVM), K-nearest neighbor (KNN), and decision tree algorithms in conjunction with three feature extraction methods: color coherence vectors, discrete cosine transforms, and filter banks to classify normal, calcified tympanic membranes, CSOM, and earwax. Using a dataset of 720 otoscope images, they achieved an accuracy of 93.9%. This method uses a smaller dataset than deep learning models, but its feature extraction capabilities and model complexity are limited, making it difficult to handle more complex diagnostic tasks.

[0046] These methods typically employ techniques such as data augmentation and transfer learning to improve model performance on small datasets. For example, Basaran et al. proposed a deep convolutional neural network (DCNN) model called AlexNet, which was trained using transfer learning and a ten-fold cross-validation approach. Using 598 tympanic membrane images, they ultimately achieved an accuracy of 98.77%. Camalan Seda et al. also employed data augmentation and transfer learning to retrain the Inception-ResNet-V2 convolutional neural network using 300 otoscope image samples, achieving a peak accuracy of 85.8%.

[0047] This demonstrates the strong reliance of existing technologies on large amounts of labeled data. This is because traditional deep learning models require a large number of samples to learn feature representations and decision boundaries. However, in the field of medical imaging, obtaining high-quality labeled data faces numerous obstacles, such as limited professional time and patient privacy protection. This results in significant performance degradation in small sample sizes, making it difficult to meet real-world clinical needs. Furthermore, existing methods primarily rely on a single image input, neglecting other important diagnostic information, such as patient history and symptom descriptions. This single-modality approach limits the model's comprehensive understanding of the disease and its ability to accurately diagnose it.

[0048] Based on this, the present invention provides an artificial intelligence-based small sample tympanic membrane image recognition and diagnosis system. The system can classify and diagnose ear diseases through a small number of tympanic membrane image samples, enhancing the ability to distinguish different types of otitis media under small sample conditions; moreover, the system can provide comprehensive diagnosis and treatment plans related to the disease category, and the diagnosis report can include rich medical knowledge, thereby improving the practicality of the diagnosis results. Figure 1 As shown, the system 100 of the present invention may include a category matching module, a category enhancement module, a knowledge enhancement module and a diagnosis generation module.

[0049] The category matching module 110 is used to obtain an input image and an image category set, calculate the similarity between the input image and at least one image category, and determine the matching category of the input image based on the similarity. In an embodiment of the present invention, the input image may be a medical image such as an eardrum image, and the input image may be used for the diagnosis and treatment of a disease. The embodiment of the present invention only uses the eardrum image as an example and is not intended to limit the scope of implementation of the present invention. The image category set may refer to a case category set, and the image category set may include multiple category items for identifying disease categories. For example, in the diagnosis of ear diseases, the image category set may include multiple image categories such as acute otitis media AOM, otitis media with effusion OME, etc.

[0050] The category matching module of an embodiment of the present invention can roughly determine the matching category corresponding to the input image through similarity calculation, thereby identifying and diagnosing diseases based on the input image. For example, the category matching module can compare the input image with each image category and determine the matching category of the input image based on the image category with the highest similarity. For example, the category matching module determines the matching category of the input image as acute otitis media.

[0051] The category enhancement module 120 is used to enhance the feature expression of the input image based on the input image, the matching category and the first query, so as to determine the disease category corresponding to the input image. The first query token can be a random number used to extract the features of the input image. In an embodiment of the present invention, the category enhancement module can further enhance the feature expression of the input image based on the input image, the first query and the determined matching category, thereby improving the accuracy of subsequent image recognition. The type of the input image is roughly located based on the matching category of the input image, thereby determining the disease category of the input image, to provide a basis for subsequent fine diagnosis. In addition, the system of the embodiment of the present invention can improve the accuracy of disease diagnosis by determining the disease category corresponding to the input image through the category matching module and the category enhancement module.

[0052] The knowledge enhancement module 130 is used to enhance the feature expression of the input image and determine the diagnosis and treatment plan for the disease category based on the structured knowledge corresponding to the matching category of the input image. Among them, the knowledge enhancement module can retrieve the structured knowledge related to the disease category obtained in the knowledge base through the matching category of the input image, thereby further enhancing the feature expression of the input image, and further using this structured medical knowledge to perform accurate diagnosis. This structured knowledge can be medical knowledge associated with the disease category of the input image. For example, this structured medical knowledge can be the specific pathological characteristics of otitis media, diagnosis and treatment guidelines and prescription recommendations, etc. In this way, the system of the embodiment of the present invention can not only identify the disease category of the input image, but also give corresponding diagnostic suggestions and treatment plans, thereby improving the accuracy and practicality of the diagnosis.

[0053] Diagnosis generation module 140 is used to generate a diagnostic report corresponding to the input image. The diagnosis may include the disease category of the input image, its definition, and the corresponding treatment plan. Compared to existing diagnostic methods that rely on a single image input, the system of this embodiment of the present invention can fully understand and accurately diagnose the disease based on the input image, and generate a comprehensive diagnostic report.

[0054] The image recognition and diagnosis system of the present invention first determines the matching category of the input image through the category matching module, then enhances the feature expression of the input image through the category enhancement module to determine the disease category corresponding to the input image, then further enhances the feature expression of the input image through the knowledge enhancement module, and determines the diagnosis and treatment plan corresponding to the disease category, and finally generates a comprehensive and complete diagnosis report through the diagnosis generation module. The system can improve the accuracy and practicality of diagnosis.

[0055] Figure 2 FIG. 1 shows an image recognition and diagnosis system according to an embodiment of the present invention. Figure 2 As shown, the system of the present invention may include a category matching module 210, a category enhancement module 220, a knowledge enhancement module 230, a diagnosis generation module 240, and a knowledge screening and integration module 250.

[0056] The category matching module 210 may include an image encoding module, a category encoding module and a similarity matching module. The image encoding module is used to obtain the image feature vector corresponding to the input image through a pre-trained image encoder. Figure 2 As shown, the image encoder obtains an input image and outputs an image feature vector I corresponding to the input image. In an embodiment of the present invention, the image encoder may be a pre-trained CLIP (Contrastive Language-Image Pre-training) image encoder.

[0057] The category encoding module is used to obtain a category feature vector corresponding to at least one image category through a pre-trained category encoder. Figure 2 As shown, the category encoder can obtain an image category set (such as a case category set, including multiple category items such as AOM, OME, etc.), and output the category feature vector ({C1, C2, ..., C n In an embodiment of the present invention, the category encoder may be a pre-trained CLIP (Contrastive Language-Image Pre-training) text encoder.

[0058] Further optionally, the pre-trained category encoder is used to obtain a category feature vector according to the received embedded text template and image category set; wherein the embedded text template (V = {V1, V2, ..., V M}) is obtained through training. In the embodiment of the present invention, the category matching module can train the embedded text for each category ci. The following takes one category as an example to illustrate the training process of the embedded text module as follows:

[0059] First, obtain a small number of labeled samples ({(x1, y1), (x2, y2), ..., (x m ,y m )}), where x1~x m Used to identify different sample images, y1~y m Used to identify the true category label corresponding to the corresponding sample image.

[0060] Secondly, the labeled samples are input into the category matching module in batches. At this time, the image encoder of the category matching module encodes the input sample image to obtain the image feature vector corresponding to the sample image. The category encoder of the category matching module encodes the input image category set and embedded text template to obtain the corresponding category feature vector. In the initial state, V i =C i During the training of embedded text templates, the parameters of the image encoder and category encoder remain unchanged.

[0061] Again, the category matching module can perform similarity calculation based on the image feature vector and the category feature vector to determine the matching category corresponding to the sample image; and calculate the contrast loss based on the matching category and the true category label corresponding to the sample image; wherein, the calculation formula of the contrast loss is as follows:

[0062]

[0063] Finally, the category matching module updates the embedded text V through gradient descent i .

[0064] The above process is repeated until a preset condition is met to obtain a trained embedded text template. In an embodiment of the present invention, the preset condition may refer to a preset number of training rounds, such as 50 rounds.

[0065] In an embodiment of the present invention, the embedded text templates are initialized by pre-trained image encoders and category encoders, and these trainable embeddings can be optimized using a very small number of sample images. This method can quickly adapt to new ear disease classifications using only a few eardrum sample images, achieving better disease recognition results. This feature reduces the reliance on large amounts of labeled data, greatly solving the problems of difficulty in acquiring ear clinical data and time-consuming labeling. It is particularly suitable for medical institutions at all levels to quickly deploy and update eardrum disease diagnosis models, and improves applicability in more than a dozen clinical settings. In addition, the embodiment of the present invention, by introducing a contrastive learning loss function for eardrum features, further improves the recognition of various ear disease embeddings and enhances the model's ability to distinguish different types of otitis media under small sample conditions.

[0066] Continue to see Figure 2 , the category matching module may also include a similarity matching module. Among them, the similarity matching module is used to map the image feature vector and the category feature vector to the same feature space respectively, calculate the similarity between the image feature vector and the category feature vector, and determine the matching category of the input image based on the similarity. Specifically, the similarity matching module calculates the similarity between the image feature vector and each image category in the category feature vector respectively; and sorts these calculated similarities, and selects the image category with the highest similarity as the matching category of the input image. In an embodiment of the present invention, the similarity calculation can be based on the following similarity function:

[0067]

[0068] Where I represents the image feature vector of the input image, and Ci represents the image category in the category feature vector.

[0069] Afterwards, the similarity matching module can select the image category with the highest similarity as the matching category of the input image. The matching category is calculated as follows:

[0070]

[0071] like Figure 2 As shown, the category enhancement module 220 can be a trained Transformer model, which is constructed based on the attention mechanism. Specifically, the category enhancement module can include a first self-attention layer, a first cross-attention layer and a feedforward layer.

[0072] The first self-attention layer is used to perform multi-head attention calculation based on the image feature vector of the input image and the query feature vector corresponding to the first query to obtain a first self-attention output. The first query can be a random number used to extract the feature expression of the input image. In an embodiment of the present invention, the first self-attention layer can concatenate the feature vector of the input image and the first query to obtain an input sequence, and perform multi-head self-attention calculation on the input sequence to obtain a first self-attention output. The multi-head self-attention calculation can be referred to the following formula:

[0073] MultiHead(S)=Concat(head1,...,head h )W O

[0074] in, W O is a trainable parameter;

[0075] head i Used to capture information of a certain subspace, where S represents an input sequence, which is obtained by concatenating the feature vector of the input image and the first query, S = [Q; F]); are trainable parameters respectively;

[0076] The first cross attention layer is used to perform a cross attention calculation based on the matching category of the first self-attention output and the input image to obtain a first cross attention output. In this embodiment of the present invention, the first cross attention calculation can refer to the following formula:

[0077]

[0078] Where S′ represents the first self-attention output of the first self-attention layer; Indicates the matching category corresponding to the introduced input image; W Q 、W K 、W V They represent the trainable parameters respectively.

[0079] The feedforward layer is used to determine the disease category of the input image based on the first cross-attention output. In an embodiment of the present invention, the feedforward layer can adopt a two-layer fully connected network. The feedforward layer is used to perform feature conversion so that the output of the category enhancement module can adapt to its subsequent module. The feedforward layer can be processed according to the following formula:

[0080] FFN(x)=max(0,xW1+b1)W2+b2

[0081] Where x represents the input image, W1 represents the weight of the first fully connected network layer, W2 represents the weight of the second fully connected network layer, b1 and b2 represent bias values, which can be hyperparameters.

[0082] The category enhancement module of the embodiment of the present invention can repeatedly perform the above-mentioned attention calculation and feature conversion operations to enhance the feature representation capability. By adjusting the attention based on the matching category of the input image, the input image and the first query through the above-mentioned category enhancement module, the coarse positioning of the type of the input image can be achieved, thereby determining the disease category of the input image, providing a basis for subsequent fine diagnosis. In addition, the system of the embodiment of the present invention enhances the feature expression of the input image through the category enhancement module to determine the disease category corresponding to the input image, thereby improving the accuracy of subsequent image recognition, and further improving the accuracy of disease diagnosis. In the prior art, most methods can only complete coarse-grained classification tasks, such as distinguishing normal from lesions, or simple lesion type classification, and lack fine-grained recognition capabilities. However, clinical diagnosis often requires a more detailed description of the lesion, including information such as location, degree, and characteristics, which is difficult to achieve with existing methods.

[0083] Continue to see Figure 2 , the knowledge enhancement module 230 can be a trained Transformer model, which is built based on the attention mechanism. This multimodal adapter integrates the features of the ear endoscope image and the ear disease category embedding, and introduces learnable medical knowledge tokens to enhance the model's ability to capture key lesion information, thereby further enhancing the feature expression of the input image. Specifically, the knowledge enhancement module may include a second self-attention layer, a second cross-attention layer, and a feedforward layer. Through the self-attention and cross-attention architecture, deep feature interaction between the ear endoscope image and the disease description is achieved, and fine-tuning is performed through a small number of ear endoscope images and diagnostic reports. Among them,

[0084] The second self-attention layer is used to perform multi-head attention calculation based on the disease category of the input image to obtain the second self-attention output. The multi-head self-attention calculation can be referred to the following formula:

[0085] MultiHead(S)=Concat(head1,...,head h )W O

[0086] in,

[0087] head i Used to capture information of a certain subspace, where S represents an input sequence, which is obtained by concatenating the feature vector of the input image and the first query, S = [Q; F]); are trainable parameters respectively;

[0088] The second cross attention layer is used to perform cross attention calculation based on the second self-attention output and the structured knowledge corresponding to the disease category of the input image to obtain a second cross attention output. In this embodiment of the present invention, the second cross attention calculation can be referred to the following formula:

[0089] CrossAttn(S′1,S know )=Attention(S′1W Q , S know W K , S know W V )

[0090] Where S′ represents the output of the feedforward layer in the category enhancement module; S know Represents the structured knowledge corresponding to the disease category of the input image; W Q 、W K 、W V They represent the trainable parameters respectively.

[0091] The feedforward layer is used to determine the diagnosis and treatment plan corresponding to the input image based on the second cross-attention output. In an embodiment of the present invention, the feedforward layer can adopt a two-layer fully connected network. The feedforward layer is used to perform feature conversion so that the output of the category enhancement module can adapt to its subsequent module. The feedforward layer can be processed according to the following formula:

[0092] FFN(x)=max(0,xW1+b1)W2+b2

[0093] Where x represents the input image, W1 represents the weight of the first layer of the fully connected network, W2 represents the weight of the second layer of the fully connected network, b1 and b2 represent bias values, which can be hyperparameters.

[0094] The knowledge enhancement module of this embodiment of the present invention can repeatedly perform the aforementioned attention calculation and feature conversion operations to enhance the feature representation capabilities of the input image. This system, through the knowledge enhancement module, introduces structured knowledge related to disease categories, further enhancing the feature representation of the input image. This structured medical knowledge is then used to perform precise diagnoses, enabling the system of this embodiment of the present invention to not only identify the disease category of the input image but also provide corresponding diagnostic recommendations and treatment plans, thereby improving the accuracy and practicality of the diagnosis.

[0095] like Figure 2 As shown, the system also includes a knowledge screening and integration module 250, which includes a knowledge base, a knowledge encoder, a feature fusion device and a knowledge filter.

[0096] A knowledge base can be a database that stores at least one knowledge item; this knowledge item can be various medical knowledge items, including the pathological characteristics, diagnosis and treatment guidelines, and prescription recommendations of at least one disease. For example, the knowledge base may contain pathological knowledge related to various types of otitis media, such as the symptoms, signs, causes, standardized diagnosis and treatment processes, treatment plans, and commonly used prescription recommendations, including medication and surgical indications, for acute otitis media, chronic otitis media, and secretory otitis media.

[0097] The knowledge encoder is used to encode at least one knowledge item in the knowledge base to obtain a knowledge feature vector corresponding to the at least one knowledge item; the knowledge encoder can be a pre-trained CLIP encoder. The knowledge items in the knowledge base (K = {k1, k2, ..., k m}) can be encoded by the knowledge encoder to obtain a knowledge vector set ({K1, K2, ..., K m}), where (K j =f know (k j )).

[0098] The feature fusion device is used to concatenate the image feature vector with the vector corresponding to the matching category to obtain the fused feature vector. Figure 2 As shown in the figure, the “+” is used to concatenate the image feature vector of the input image with the matching category to obtain a fused feature vector. This fused feature vector is then used to filter knowledge items related to the matching category from the knowledge base.

[0099] The knowledge filter is used to filter the structured knowledge corresponding to the disease category of the input image from the knowledge base based on the fusion feature vector and the knowledge feature vector. Figure 2 As shown, “.” is used to filter relevant knowledge items from the knowledge base based on the fused feature vector and the knowledge feature vector output by the knowledge encoder, and organize the knowledge items according to certain rules to obtain structured knowledge.

[0100] Specifically, the knowledge filter is used to calculate the second similarity between each knowledge feature vector and the fusion feature vector respectively; the calculation of the second similarity can refer to the following formula:

[0101]

[0102] Among them, I1 represents the fusion feature vector, K j Represents the knowledge feature vector.

[0103] Afterwards, the knowledge filter is used to sort the at least one second similarity and select a preset number of knowledge items corresponding to the second similarity as selected knowledge. In an embodiment of the present invention, the knowledge filter can select the knowledge items corresponding to the first N knowledge feature vectors with the highest similarity as selected knowledge.

[0104] Finally, the knowledge filter processes the selected knowledge using spacy library functions to generate structured knowledge. In an embodiment of the present invention, the knowledge filter can extract keywords from the selected knowledge using spacy functions and filter out duplicate content to obtain structured knowledge. The structured knowledge can be represented as follows:

[0105]

[0106] Continue to see Figure 2 The system also includes a diagnosis generation module 240. In an embodiment of the present invention, the diagnosis generation module 240 is constructed based on a pre-trained large-scale language model. The language model can receive the output of the knowledge enhancement module, allowing the language model to obtain external medical knowledge through the Transformer model. This medical knowledge can be used to freeze the language model. By introducing rich medical knowledge, the language model can better cope with different cases and complex clinical scenarios, improving generalization capabilities and reducing dependence on specific datasets. In this way, after integrating medical knowledge and image features, the generated diagnostic report is not limited to simple classification results, but also includes professional medical descriptions and treatment recommendations, such as features such as tympanic membrane color, light cones, and perforations, providing clinicians with a comprehensive diagnostic reference. However, in the existing technology, due to the complexity and variability of medical images, relying solely on limited training data is difficult to cover all possible pathological conditions, and the generalization ability is limited. This makes the model prone to misjudgment when faced with unseen samples or complex cases, affecting the reliability of diagnosis.

[0107] Specifically, the language model can receive the output of the knowledge enhancement module and extract the feature representation (H) corresponding to the second query token from the output of the knowledge enhancement module. The second query may include information related to the disease category and its diagnosis and treatment plan. Then, based on the feature representation of the second query, a detailed diagnosis report is generated. The diagnosis report may include the disease category, the definition of the disease category, and the recommended treatment. The generation process of the diagnosis report can be expressed by the following formula:

[0108] Output = LM(H).

[0109] The system of the embodiment of the present invention generates detailed diagnostic reports through a diagnostic report generation mechanism based on a large model. The present invention utilizes a pre-trained large-scale language model and an introduced medical knowledge module. In this way, after integrating medical knowledge and image features, the generated diagnostic report is not limited to simple classification results, but also includes professional medical descriptions and treatment recommendations, such as tympanic membrane color, light cones, perforations and other features, providing clinicians with a comprehensive diagnostic reference. The system of the embodiment of the present invention, through the introduction of external medical knowledge, enables the language model to recognize and understand more complex disease characteristics, especially for rare or complex ear diseases, and provide more accurate diagnostic results.

[0110] Optionally, the above-mentioned category enhancement module and knowledge enhancement module are trained Transformer models. Before the system of the embodiment of the present invention uses the above-mentioned category enhancement module and knowledge enhancement module to identify and diagnose the input image, the system first trains the corresponding Transformer model to improve the feature fusion and representation capabilities of the Transformer model, thereby improving the accuracy of the model output. The above-mentioned diagnosis generation module is a trained language model. Before the system of the embodiment of the present invention uses the above-mentioned diagnosis generation module to obtain a diagnosis report, the system first trains the language model to improve the recognition and expression capabilities of the language model.

[0111] Optionally, the system of the embodiment of the present invention can train the category enhancement module and the knowledge enhancement module as a whole. During the training process of the Transformer model, the parameters of the language model and the CLIP encoder remain unchanged. This design allows the language model to introduce and apply external medical knowledge by fine-tuning the Transformer model while keeping the parameters of the large language model frozen, thereby achieving control of the language model and further achieving the effect of intelligent diagnosis. In this way, the general language model can adapt to specific tympanic membrane disease diagnosis tasks, not only accurately classify the type of otitis media, but also generate diagnostic reports that meet medical professional standards, describe in detail the color, light cone, perforation and other characteristics of the tympanic membrane, and provide clinicians with a comprehensive diagnostic reference. The specific training process of the Transformer model is as follows:

[0112] First, the sample data is input into the category enhancement module in batches, according to Figure 2 The embodiment shown processes the sample data to generate a diagnosis report, that is, the category enhancement module, the knowledge enhancement module and the diagnosis generation module sequentially process the sample data to obtain the diagnosis report.

[0113] Secondly, the loss value is calculated according to the following loss function:

[0114] L total =αLce +(1-α)L gen

[0115] Among them, (L ce ) is the cross entropy loss for category prediction, (L gen ) is the language model loss for diagnostic report generation (usually negative log likelihood), and (α) is the weight hyperparameter, set to 0.5.

[0116] Finally, all parameters of the Transformer model, including the query token, are updated through mini-batch gradient descent.

[0117] Repeat the above process until the preset condition is met to obtain a trained Transformer model. In the embodiment of the present invention, the preset condition may refer to a preset number of training rounds, such as 100 rounds.

[0118] Furthermore, for the above-mentioned preset training rounds, the system of the embodiment of the present invention can implement the training of the entire Transformer model in steps. The details are as follows:

[0119] First, the sample data is input into the category enhancement module in batches, according to Figure 2 The embodiment shown processes the sample data to generate a diagnosis report, that is, the category enhancement module, the knowledge enhancement module and the diagnosis generation module sequentially process the sample data to obtain the diagnosis report.

[0120] Secondly, in the first 50 training rounds, only the category prediction task is optimized, that is, only the first loss value (L ce ), and according to the first loss value (L ce ) Update the parameters of the Transformer model corresponding to the category enhancement module. During this training process, the performance of each Transformer model is evaluated every five training rounds. If the performance of the Transformer model does not improve significantly after 10 consecutive training rounds, training is terminated early.

[0121] Again, in the next 50 training rounds, we jointly optimize the category prediction and report generation tasks, that is, we calculate L according to the above loss function. total , and according to the loss value L total Update the parameters of the Transformer model corresponding to the category enhancement module and the Transformer model corresponding to the knowledge enhancement module. During this training process, evaluate the performance of each Transformer model after every five training rounds. If there is no significant improvement in the Transformer model's performance after 10 consecutive training rounds, stop training early.

[0122] The system in this embodiment of the present invention, by employing a phased training strategy combining embedded text templates with a Transformer model, effectively learns category representation and feature fusion even with a small sample size, thereby enabling accurate recognition of tympanic membrane images and generation of diagnostic reports. Furthermore, by employing a phased training and multi-task learning strategy, a balance is achieved between classification accuracy and diagnostic report quality.

[0123] In summary, the system in this embodiment of the present invention can accurately identify and diagnose tympanic membrane images even with a small sample size, while providing interpretable results and offering clinicians a powerful auxiliary diagnostic tool. This system not only improves diagnostic accuracy but also provides detailed diagnostic recommendations, significantly enhancing the practical value of AI-assisted diagnosis systems in the field of otology.

[0124] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence, characterized by: The system comprises: a category matching module, configured to obtain an input image and an image category set, calculate a similarity between the input image and at least one image category, and determine a matching category of the input image based on the similarity to roughly determine the category of the input image; A category enhancement module is configured to enhance the feature expression of the input image based on the input image, the matching category, and the first query to determine the disease category corresponding to the input image and locate the category of the input image; wherein the category enhancement module includes: a first self-attention layer, configured to perform a multi-head attention calculation based on the image feature vector of the input image and the query feature vector corresponding to the first query to obtain a first self-attention output; a first cross-attention layer, configured to perform a cross-attention calculation based on the first self-attention output and a matching category of the input image to obtain a first cross-attention output; a feed-forward layer, configured to determine a disease category of the input image based on the first cross-attention output; A knowledge screening and integration module, comprising: a knowledge base, for storing at least one knowledge item; A knowledge encoder, configured to encode at least one knowledge item in the knowledge base to obtain a knowledge feature vector corresponding to the at least one knowledge item; A feature fusion device, configured to concatenate the image feature vector with the vector corresponding to the matching category to obtain a fused feature vector; a knowledge filter, configured to filter and obtain structured knowledge corresponding to the disease category of the input image from the knowledge base based on the fused feature vector and the knowledge feature vector; A knowledge enhancement module is configured to enhance the feature expression of the input image based on the structured knowledge corresponding to the disease category of the input image, and determine a diagnosis and treatment plan for the disease category; wherein the knowledge enhancement module includes: A second self-attention layer is used to perform multi-head attention calculation according to the disease category of the input image to obtain a second self-attention output; a second cross-attention layer, configured to perform a cross-attention calculation based on the second self-attention output and structured knowledge corresponding to the disease category of the input image to obtain a second cross-attention output; a feedforward layer, configured to determine a diagnosis and treatment plan corresponding to the input image based on the second cross-attention output; The diagnosis generating module is used to generate a diagnosis report corresponding to the input image.

2. The system according to claim 1, wherein: The category matching module includes: An image encoding module is used to obtain an image feature vector corresponding to an input image through a pre-trained image encoder; A category encoding module, configured to obtain a category feature vector corresponding to at least one image category through a pre-trained category encoder; The similarity matching module is used to map the image feature vector and the category feature vector to the same feature space, calculate the similarity between the image feature vector and the category feature vector, and determine the matching category of the input image according to the similarity.

3. The system according to claim 2, characterized in that The similarity matching module is specifically used for: respectively calculating the similarity between the image feature vector and each image category in the category feature vector; The image category with the highest similarity is selected as the matching category of the input image.

4. The system according to claim 2, wherein: The pre-trained category encoder is used to obtain the category feature vector according to the received embedded text template and image category set; wherein the embedded text template is obtained through training.

5. The system according to claim 1, wherein: The knowledge filter is specifically used to: respectively calculating a second similarity between each knowledge feature vector and the fusion feature vector; sorting at least one of the second similarities, and selecting a preset number of knowledge items corresponding to the second similarities as selected knowledge; The selected knowledge is processed by spacy library functions to generate the structured knowledge.

6. The system according to claim 1, wherein: The diagnosis generation module is configured to obtain an output of the knowledge enhancement module, extract a second query from the output of the knowledge enhancement module, and generate a diagnosis report based on the second query.

7. The system according to claim 1, wherein: The category enhancement module and the knowledge enhancement module are trained Transformer models; the diagnosis generation module is a trained language model.

Images