Small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence

Abstract

The invention provides a small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence, and the system comprises a category matching module, a category enhancement module, a knowledge enhancement module and a diagnosis generation module, calculating the similarity between the input image and at least one image category, and determining the matching category of the input image according to the similarity; the category enhancement module is used for determining a disease category corresponding to the input image according to the input image, the matching category and the inquiry; the knowledge enhancement module is used for determining a diagnosis and treatment scheme of the disease category according to the structured knowledge corresponding to the matching category of the input image; the diagnosis generation module is used for generating a diagnosis report corresponding to the input image.

Description

Technical Field

[0001] The present invention mainly relates to the field of image recognition technology, and in particular to a small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence. Background Art

[0002] Otitis media is a common ear disease with high morbidity and multiple complications. Accurate otendoscopic diagnosis is essential for early detection, accurate treatment, and effective follow-up of otitis media. However, existing computer-aided diagnosis methods for tympanic membrane lesions mainly rely on deep learning models with large amounts of annotated data, which faces challenges in the medical field, such as difficulty in data acquisition and privacy protection. Summary of the invention

[0003] Based on this, the present invention provides a small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence, which can not only locate the disease category of the input image, but also determine the diagnosis and treatment plan corresponding to the input image, thereby generating a comprehensive diagnosis report and improving the accuracy and practicality of the diagnosis.

[0004] A small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence, the system comprising:

[0005] A category matching module, used to obtain an input image and an image category set, calculate the similarity between the input image and at least one image category, and determine a matching category of the input image according to the similarity;

[0006] a category enhancement module, configured to enhance the feature expression of the input image according to the input image, the matching category and the first query, so as to determine the disease category corresponding to the input image;

[0007] A knowledge enhancement module, used to enhance the feature expression of the input image and determine the diagnosis and treatment plan of the disease category according to the structured knowledge corresponding to the matching category of the input image;

[0008] The diagnosis generation module is used to generate a diagnosis report corresponding to the input image.

[0009] Wherein, the category matching module includes:

[0010] An image encoding module, used to obtain an image feature vector corresponding to an input image through a pre-trained image encoder;

[0011] A category encoding module, used to obtain a category feature vector corresponding to at least one image category through a pre-trained category encoder;

[0012] The similarity matching module is used to map the image feature vector and the category feature vector to the same feature space respectively, calculate the similarity between the image feature vector and the category feature vector, and determine the matching category of the input image according to the similarity.

[0013] Wherein, the similarity matching module is specifically used for:

[0014] Calculating the similarity between the image feature vector and each image category in the category feature vector respectively;

[0015] The image category with the highest similarity is selected as the matching category of the input image.

[0016] The pre-trained category encoder is used to obtain the category feature vector according to the received embedded text template and image category set; wherein the embedded text template is obtained through training.

[0017] Wherein, the category enhancement module includes:

[0018] A first self-attention layer, configured to perform multi-head attention calculation according to an image feature vector of the input image and a query feature vector corresponding to the first query to obtain a first self-attention output;

[0019] A first cross-attention layer, configured to perform a cross-attention calculation according to the first self-attention output and a matching category of the input image to obtain a first cross-attention output;

[0020] A feed-forward layer is used to determine the disease category of the input image according to the first cross-attention output.

[0021] Wherein, the knowledge enhancement module further includes:

[0022] A second self-attention layer, used for performing multi-head attention calculation according to the disease category of the input image to obtain a second self-attention output;

[0023] a second cross attention layer, configured to perform a cross attention calculation according to the second self-attention output and the structured knowledge corresponding to the disease category of the input image, to obtain a second cross attention output;

[0024] The feed-forward layer is used to determine the diagnosis and treatment plan corresponding to the input image according to the second cross-attention output.

[0025] The system further includes a knowledge screening and integration module, which includes:

[0026] A knowledge base, used to store at least one knowledge item;

[0027] A knowledge encoder, used to encode at least one knowledge item in the knowledge base to obtain a knowledge feature vector corresponding to the at least one knowledge item;

[0028] A feature fusion device, used for concatenating the image feature vector with the vector corresponding to the matching category to obtain a fused feature vector;

[0029] The knowledge filter is used to filter and obtain structured knowledge corresponding to the disease category of the input image from the knowledge base according to the fused feature vector and the knowledge feature vector.

[0030] The knowledge filter is specifically used for:

[0031] Calculating the second similarity between each knowledge feature vector and the fusion feature vector respectively;

[0032] sorting at least one of the second similarities, and selecting a preset number of knowledge items corresponding to the second similarities as selected knowledge;

[0033] The selected knowledge is processed by spacy library functions to generate the structured knowledge.

[0034] The diagnosis generation module is used to obtain the output of the knowledge enhancement module, extract the second query from the output of the knowledge enhancement module, and generate a diagnosis report according to the second query.

[0035] Among them, the category enhancement module and the knowledge enhancement module are trained Transformer models; and the diagnosis generation module is a trained language model.

[0036] The artificial intelligence-based small sample tympanic membrane image recognition and diagnosis system of the present invention first determines the matching category of the input image through the category matching module, then enhances the feature expression of the input image through the category enhancement module to determine the disease category corresponding to the input image, then further enhances the feature expression of the input image through the knowledge enhancement module, and determines the diagnosis and treatment plan corresponding to the disease category, and finally generates a comprehensive and complete diagnosis report through the diagnosis generation module. The system can improve the accuracy and practicality of diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a structural schematic diagram of a small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence according to an embodiment of the present invention;

[0038] Figure 2 It is a schematic structural diagram of a small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence according to another embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] Otitis media is a common ear disease with a high incidence and multiple complications. Accurate otendoscopic diagnosis is essential for early detection, accurate treatment and effective follow-up of otitis media. However, existing computer-aided tympanic membrane lesion diagnosis methods mainly rely on deep learning models with a large amount of annotated data, which faces challenges in the medical field such as difficulty in data acquisition and privacy protection. At present, the implementation schemes closest to the present invention mainly include the following categories:

[0041] The first category is the image classification method based on convolutional neural network (CNN). For example, Wu Zebin et al. used Xception and MobileNetV2 models to classify normal, acute otitis media (AOM), and secretory otitis media (OME) categories. This method used 12,203 ear endoscopy images as a dataset, and the overall classification accuracy reached 97.45% and 95.72%, respectively. However, this method requires a large amount of labeled data, and the performance will drop significantly in the case of small samples.

[0042] The second category is the method that combines the target detection algorithm. For example, Lu Shuochen et al. used the Faster RCNN target detection algorithm to complete computer-assisted tympanic membrane diagnosis. They used image enhancement technology to increase 1297 otoscope images to 5188, and adopted ten-fold cross validation, with an overall recognition accuracy of 91.43%. However, this method requires marking the specific location of the lesion, which is often unnecessary and time-consuming in primary care.

[0043] The third category is the CNN model that integrates the attention mechanism. Alhudhif et al. proposed a computer-aided decision-making model based on CNN, which consists of three basic modules: attention module, residual block and hypercolumn. Image enhancement technology was used in preprocessing, and finally an accuracy of 98.26% was achieved in the classification of normal, AOM, chronic suppurative otitis media (CSOM) and earwax tasks. Although this method improves the performance of the model, it still requires a large amount of labeled data.

[0044] The fourth category is a hybrid architecture that combines CNN and long short-term memory network (LSTM), which is mainly used to process otoscope videos. Viscaino et al. proposed a functional hybrid architecture that combines CNN (such as ResNet-50, EfficientNet-B4, Inception-V3 and DenseNet) and LSTM, using the ability of CNN to extract features and the ability of LSTM network to learn long-term dependencies in spatial sequence data to achieve the diagnosis function of otoscope videos, achieving an average accuracy of 91.65%. However, this method is mainly aimed at video data, and the diagnostic effect for a single image may be limited.

[0045] The fifth category is the combination of traditional machine learning methods and feature extraction techniques. Viscaino et al. used support vector machines (SVM), K-nearest neighbors (KNN), and decision tree algorithms combined with three feature extraction methods: color coherence vectors, discrete cosine transforms, and filter banks to complete the classification tasks of normal, tympanic membrane calcification, CSOM, and earwax. An accuracy of 93.9% was achieved using a dataset of 720 otoscope images. This method uses fewer datasets than deep learning models, but its feature extraction capabilities and model complexity are limited, making it difficult to handle more complex diagnostic tasks.

[0046] The above methods usually use data enhancement and transfer learning techniques to improve model performance on small sample data sets. For example, Basaran et al. proposed a deep convolutional neural network (DCNN) model called AlexNet, which was trained using transfer learning methods and a ten-fold cross-validation method. A total of 598 tympanic membrane images were used, and the final accuracy rate was 98.77%. Camalan Seda et al. also used data enhancement and transfer learning methods to retrain the Inception-ResNet-V2 convolutional neural network, using a total of 300 otoscope image samples, and the highest accuracy of the model was 85.8%.

[0047] It can be seen that existing technologies are highly dependent on large amounts of labeled data. This is because traditional deep learning models require a large number of samples to learn feature representations and decision boundaries. However, in the field of medical imaging, there are many obstacles to obtaining high-quality labeled data, such as limited time for professionals and patient privacy protection. This causes the model's performance to drop significantly in small sample scenarios, making it difficult to meet actual clinical needs. In addition, existing methods mainly rely on a single image input, ignoring other important diagnostic information, such as patient medical history, symptom descriptions, etc. This single-modality processing method limits the model's comprehensive understanding and accurate diagnosis of the disease.

[0048] Based on this, the present invention provides a small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence. The system can classify and diagnose ear diseases through a small number of tympanic membrane image samples, enhancing the ability to distinguish different types of otitis media under small sample conditions; and the system can provide comprehensive diagnosis and treatment plans related to the disease category, and the diagnosis report can include rich medical knowledge, thereby improving the practicality of the diagnosis results. Figure 1 As shown, the system 100 of the present invention may include a category matching module, a category enhancement module, a knowledge enhancement module and a diagnosis generation module.

[0049] The category matching module 110 is used to obtain an input image and an image category set, calculate the similarity between the input image and at least one image category, and determine the matching category of the input image based on the similarity. In an embodiment of the present invention, the input image may be a medical image such as a tympanic membrane image, and the input image may be used for the diagnosis and treatment of a disease. The embodiment of the present invention only takes the tympanic membrane image as an example and is not intended to limit the scope of implementation of the present invention. The image category set may refer to a case category set, and the image category set may include multiple category items for identifying disease categories. For example, in the diagnosis of ear diseases, the image category set may include multiple image categories such as acute otitis media AOM, otitis media with exudate OME, etc.

[0050] The category matching module of the embodiment of the present invention can roughly determine the matching category corresponding to the input image by similarity calculation, so as to identify and diagnose the disease according to the input image. For example, the category matching module can compare the input image with each image category, and determine the matching category of the input image with the image category with the highest similarity. For example, the category matching module determines the matching category of the input image as acute otitis media.

[0051] The category enhancement module 120 is used to enhance the feature expression of the input image according to the input image, the matching category and the first query, so as to determine the disease category corresponding to the input image. Among them, the first query token can be a random number, which is used to extract the features of the input image. In the embodiment of the present invention, the category enhancement module can further enhance the feature expression of the input image according to the input image, the first query and the determined matching category, so as to improve the accuracy of subsequent image recognition. The type of the input image is roughly located based on the matching category of the input image, so as to determine the disease category of the input image, so as to provide a basis for subsequent fine diagnosis. In addition, the system of the embodiment of the present invention can improve the accuracy of disease diagnosis by determining the disease category corresponding to the input image through the category matching module and the category enhancement module.

[0052] The knowledge enhancement module 130 is used to enhance the feature expression of the input image and determine the diagnosis and treatment plan of the disease category according to the structured knowledge corresponding to the matching category of the input image. Among them, the knowledge enhancement module can retrieve the structured knowledge related to the disease category obtained in the knowledge base through the matching category of the input image, thereby further enhancing the feature expression of the input image, and further using these structured medical knowledge for accurate diagnosis. These structured knowledge can be medical knowledge associated with the disease category of the input image, for example, these structured medical knowledge can be specific pathological characteristics of otitis media, diagnosis and treatment guidelines and prescription suggestions, etc. In this way, the system of the embodiment of the present invention can not only identify the disease category of the input image, but also give corresponding diagnosis suggestions and treatment plans, thereby improving the accuracy and practicality of the diagnosis.

[0053] The diagnosis generation module 140 is used to generate a diagnosis report corresponding to the input image. The diagnosis may include the disease category of the input image, the definition of the disease category, and the diagnosis and treatment plan corresponding to the disease category. Compared with the prior art method of relying on a single image input for diagnosis, the system of the embodiment of the present invention can fully understand and accurately diagnose the disease based on the input image, and generate a comprehensive diagnosis report.

[0054] The image recognition and diagnosis system of the present invention first determines the matching category of the input image through the category matching module, and then enhances the feature expression of the input image through the category enhancement module to determine the disease category corresponding to the input image, and then further enhances the feature expression of the input image through the knowledge enhancement module, and determines the diagnosis and treatment plan corresponding to the disease category, and finally generates a comprehensive and complete diagnosis report through the diagnosis generation module. The system can improve the accuracy and practicality of diagnosis.

[0055] Figure 2 FIG. 2 shows an image recognition and diagnosis system according to an embodiment of the present invention. Figure 2 As shown, the system of the present invention may include a category matching module 210, a category enhancement module 220, a knowledge enhancement module 230, a diagnosis generation module 240, and a knowledge screening and integration module 250.

[0056] The category matching module 210 may include an image encoding module, a category encoding module and a similarity matching module. The image encoding module is used to obtain an image feature vector corresponding to the input image through a pre-trained image encoder. Figure 2 As shown, the image encoder obtains an input image and outputs an image feature vector I corresponding to the input image. In the embodiment of the present invention, the image encoder may be a pre-trained CLIP (Contrastive Language-Image Pre-training) image encoder.

[0057] The category encoding module is used to obtain a category feature vector corresponding to at least one image category through a pre-trained category encoder. Figure 2 As shown in FIG. 1 , the category encoder can obtain an image category set (e.g., a case category set, including multiple category items such as AOM and OME), and output a category feature vector ({C1, C2, ..., C n In the embodiment of the present invention, the category encoder may be a pre-trained CLIP (Contrastive Language-Image Pre-training) text encoder.

[0058] Further optionally, the pre-trained category encoder is used to obtain a category feature vector according to the received embedded text template and the image category set; wherein the embedded text template (V = {V1, V2, ..., V M}) is obtained through training. In the embodiment of the present invention, the category matching module can train the embedded text for each category ci. The training process of the embedded text module is described below by taking one of the categories as an example:

[0059] First, obtain a small number of labeled samples ({(x1, y1), (x2, y2), ..., (x m ,y m )}), where x1~x m Used to identify different sample images, y1~y m Used to identify the true category label corresponding to the corresponding sample image.

[0060] Secondly, the labeled samples are input into the category matching module in batches. At this time, the image encoder of the category matching module encodes the input sample image to obtain the image feature vector corresponding to the sample image. The category encoder of the category matching module encodes the input image category set and the embedded text template to obtain the corresponding category feature vector. In the initial state, V i =C i During the training of embedded text templates, the parameters of the image encoder and category encoder remain unchanged.

[0061] Again, the category matching module can perform similarity calculation based on the image feature vector and the category feature vector to determine the matching category corresponding to the sample image; and calculate the contrast loss based on the matching category and the true category label corresponding to the sample image; wherein the calculation formula of the contrast loss is as follows:

[0062]

[0063] Finally, the category matching module updates the embedded text V through gradient descent i .

[0064] The above process is repeated until the preset condition is met to obtain a trained embedded text template. In the embodiment of the present invention, the preset condition may refer to a preset number of training rounds, such as 50 rounds.

[0065] In an embodiment of the present invention, the embedded text templates are initialized by pre-trained image encoders and category encoders, and these trainable embeddings can be optimized through a very small number of sample images. This method can quickly adapt to new ear disease classifications using only a few eardrum sample images and achieve better disease recognition results. This feature reduces the reliance on a large amount of labeled data, greatly solving the problems of difficulty in acquiring ear clinical data and time-consuming labeling. It is particularly suitable for medical institutions at all levels to quickly deploy and update eardrum disease diagnosis models, and improves the applicability in more than a dozen clinical settings. In addition, in an embodiment of the present invention, by introducing a contrastive learning loss function for eardrum features, the recognition of various ear disease embeddings is further improved, and the model's ability to distinguish different types of otitis media under small sample conditions is enhanced.

[0066] Continue to see Figure 2 , the category matching module may also include a similarity matching module. Among them, the similarity matching module is used to map the image feature vector and the category feature vector to the same feature space respectively, calculate the similarity between the image feature vector and the category feature vector, and determine the matching category of the input image according to the similarity. Specifically, the similarity matching module calculates the similarity between the image feature vector and each image category in the category feature vector respectively; and sorts these calculated similarities, and selects the image category with the highest similarity as the matching category of the input image. In an embodiment of the present invention, the similarity calculation can be based on the following similarity function:

[0067]

[0068] Among them, I represents the image feature vector of the input image, and Ci represents the image category in the category feature vector.

[0069] Afterwards, the similarity matching module can select the image category with the highest similarity as the matching category of the input image. The calculation of the matching category can be seen in the following formula:

[0070]

[0071] like Figure 2 As shown, the category enhancement module 220 may be a trained Transformer model, which is constructed based on an attention mechanism. Specifically, the category enhancement module may include a first self-attention layer, a first cross-attention layer, and a feedforward layer.

[0072] The first self-attention layer is used to perform multi-head attention calculation based on the image feature vector of the input image and the query feature vector corresponding to the first query to obtain a first self-attention output. The first query can be a random number used to extract the feature expression of the input image. In an embodiment of the present invention, the first self-attention layer can concatenate the feature vector of the input image and the first query to obtain an input sequence, and perform multi-head self-attention calculation on the input sequence to obtain a first self-attention output. The multi-head self-attention calculation can be referred to the following formula:

[0073] MultiHead(S)=Concat(head1,...,head h )W O

[0074] in, W O is a trainable parameter;

[0075] head i Used to capture information of a certain subspace, where S represents an input sequence, which is obtained by concatenating the feature vector of the input image and the first query, S = [Q; F]); They are trainable parameters respectively;

[0076] The first cross attention layer is used to perform cross attention calculation according to the matching category of the first self-attention output and the input image to obtain a first cross attention output. In the embodiment of the present invention, the first cross attention calculation can refer to the following formula:

[0077]

[0078] Where S′ represents the first self-attention output of the first self-attention layer; Indicates the matching category corresponding to the introduced input image; W Q , W K , W V They represent trainable parameters respectively.

[0079] The feedforward layer is used to determine the disease category of the input image according to the first cross-attention output. In the embodiment of the present invention, the feedforward layer can adopt a two-layer fully connected network, and the feedforward layer is used to perform feature conversion so that the output of the category enhancement module can adapt to its subsequent module. The feedforward layer can be processed according to the following formula:

[0080] FFN(x)=max(0,xW1+b1)W2+b2

[0081] Among them, x represents the input image, W1 represents the weight of the first fully connected network layer, W2 represents the weight of the second fully connected network layer, b1 and b2 represent bias values, which can be hyperparameters.

[0082] The category enhancement module of the embodiment of the present invention can repeatedly perform the above-mentioned attention calculation and feature conversion operations to enhance the feature representation capability. Through the above-mentioned category enhancement module, attention adjustment is performed based on the matching category of the input image, the input image and the first query, so that the coarse positioning of the type of the input image can be achieved, thereby determining the disease category of the input image to provide a basis for subsequent fine diagnosis. In addition, the system of the embodiment of the present invention enhances the feature expression of the input image through the category enhancement module to determine the disease category corresponding to the input image, thereby improving the accuracy of subsequent image recognition, and thus improving the accuracy of disease diagnosis. In the prior art, most methods can only complete coarse-grained classification tasks, such as distinguishing normal from lesions, or simple lesion type classification, and the fine-grained recognition capability is insufficient. However, clinical diagnosis often requires a more detailed description of the lesion, including information such as location, degree, and characteristics, which is difficult to achieve with existing methods.

[0083] Continue to see Figure 2 , the knowledge enhancement module 230 can be a trained Transformer model, which is built based on the attention mechanism. This multimodal adapter integrates the features of the otoscope image and the embedding of the ear disease category, and introduces learnable medical knowledge tokens to enhance the model's ability to capture key lesion information, thereby further enhancing the feature expression of the input image. Specifically, the knowledge enhancement module may include a second self-attention layer, a second cross-attention layer, and a feedforward layer. Through the self-attention and cross-attention architecture, deep feature interaction between the otoscope image and the disease description is achieved, and fine-tuning is performed through a small number of otoscope images and diagnostic reports. Among them,

[0084] The second self-attention layer is used to perform multi-head attention calculation according to the disease category of the input image to obtain the second self-attention output. The multi-head self-attention calculation can be referred to the following formula:

[0085] MultiHead(S)=Concat(head1,...,head h )W O

[0086] in,

[0087] head i Used to capture information of a certain subspace, where S represents an input sequence, which is obtained by concatenating the feature vector of the input image and the first query, S = [Q; F]); They are trainable parameters respectively;

[0088] The second cross attention layer is used to perform cross attention calculation according to the second self-attention output and the structured knowledge corresponding to the disease category of the input image to obtain a second cross attention output. In the embodiment of the present invention, the second cross attention calculation can refer to the following formula:

[0089] CrossAttn(S′1,S know ) = Attention(S′1W Q , S know W K , S know W V )

[0090] Where S′ represents the output of the feed-forward layer in the category enhancement module; S know Represents the structural knowledge corresponding to the disease category of the input image; W Q , W K , W V They represent trainable parameters respectively.

[0091] The feedforward layer is used to determine the diagnosis and treatment plan corresponding to the input image according to the second cross-attention output. In an embodiment of the present invention, the feedforward layer can adopt a two-layer fully connected network, and the feedforward layer is used to perform feature conversion so that the output of the category enhancement module can adapt to its subsequent module. Among them, the feedforward layer can be processed according to the following formula:

[0092] FFN(x)=max(0,xW1+b1)W2+b2

[0093] Among them, x represents the input image, W1 represents the weight of the first layer of the fully connected network, W2 represents the weight of the second layer of the fully connected network, b1 and b2 represent bias values, which can be hyperparameters.

[0094] The knowledge enhancement module of the embodiment of the present invention can repeatedly perform the above-mentioned attention calculation and feature conversion operations to enhance the feature representation capability of the input image. The system introduces structured knowledge related to the disease category through the knowledge enhancement module, further enhances the feature expression of the input image, and uses this structured medical knowledge for accurate diagnosis, so that the system of the embodiment of the present invention can not only identify the disease category of the input image, but also give corresponding diagnostic suggestions and treatment plans, thereby improving the accuracy and practicality of the diagnosis.

[0095] like Figure 2 As shown, the system also includes a knowledge screening and integration module 250, which includes a knowledge base, a knowledge encoder, a feature fusion device and a knowledge filter.

[0096] The knowledge base may be a database for storing at least one knowledge item; the knowledge item may be various medical knowledge, including the pathological characteristics, diagnosis and treatment guidelines, and prescription recommendations of at least one disease. For example, the knowledge base contains pathological knowledge related to various types of otitis media, such as symptoms, signs, causes, standardized diagnosis and treatment processes, treatment plans, and commonly used prescription recommendations for acute otitis media, chronic otitis media, and secretory otitis media, including drug treatment and surgical indications.

[0097] The knowledge encoder is used to encode at least one knowledge item in the knowledge base to obtain a knowledge feature vector corresponding to the at least one knowledge item; the knowledge encoder may be a pre-trained CLIP encoder. m}) can be encoded by the knowledge encoder to obtain a knowledge vector set ({K1, K2, ..., K m}), where (K j =f know (k j )).

[0098] The feature fusion device is used to concatenate the image feature vector with the vector corresponding to the matching category to obtain a fused feature vector. Figure 2 As shown in FIG. 1 , “+” is used to realize the concatenation of the image feature vector of the input image and the matching category to obtain a fused feature vector. Afterwards, the fused feature vector is used to filter knowledge items related to the matching category from the knowledge base.

[0099] The knowledge filter is used to filter and obtain the structured knowledge corresponding to the disease category of the input image from the knowledge base according to the fusion feature vector and the knowledge feature vector. Figure 2 As shown, “.” is used to filter relevant knowledge items from the knowledge base according to the fused feature vector and the knowledge feature vector output by the knowledge encoder, and organize the knowledge items according to certain rules to obtain structured knowledge.

[0100] Specifically, the knowledge filter is used to calculate the second similarity between each knowledge feature vector and the fusion feature vector respectively; the calculation of the second similarity can refer to the following formula:

[0101]

[0102] Among them, I1 represents the fusion feature vector, K j Represents the knowledge feature vector.

[0103] Afterwards, the knowledge filter is used to sort at least one second similarity and select a preset number of knowledge items corresponding to the second similarity as selected knowledge. In the embodiment of the present invention, the knowledge filter can select the knowledge items corresponding to the first N knowledge feature vectors with the highest similarity as selected knowledge.

[0104] Finally, the knowledge filter processes the selected knowledge through the spacy library function to generate structured knowledge. In an embodiment of the present invention, the knowledge filter can extract keywords from the selected knowledge through the spacy function and filter out duplicate content to obtain structured knowledge. The structured knowledge can be expressed as follows:

[0105]

[0106] Continue to see Figure 2 , the system also includes a diagnosis generation module 240. In an embodiment of the present invention, the diagnosis generation module 240 is constructed based on a pre-trained large-scale language model. The language model can receive the output of the knowledge enhancement module, so that the language model can obtain external medical knowledge through the Transformer model, and this medical knowledge can be used to freeze the language model. By introducing rich medical knowledge, the language model can better cope with different cases and complex clinical scenarios, improve the generalization ability, and reduce the dependence on specific data sets. In this way, after integrating medical knowledge and image features, the generated diagnosis report is not limited to simple classification results, but also includes professional medical descriptions and treatment recommendations, such as tympanic membrane color, light cone, perforation and other features, providing clinicians with a comprehensive diagnostic reference. In the prior art, due to the complexity and variability of medical images, it is difficult to cover all possible pathological conditions by relying only on limited training data, and the generalization ability is limited. This causes the model to be prone to misjudgment when facing unseen samples or complex cases, affecting the reliability of diagnosis.

[0107] Specifically, the language model can receive the output of the knowledge enhancement module, and extract the feature representation (H) corresponding to the second query token from the output of the knowledge enhancement module. The second query may include information related to the disease category and its diagnosis and treatment plan, and then generate a detailed diagnosis report based on the feature representation of the second query. The diagnosis report may include the disease category, the definition of the disease category, and the recommended treatment. The generation process of the diagnosis report can be expressed by the following formula:

[0108] Output = LM (H).

[0109] The system of the embodiment of the present invention, through a diagnostic report generation mechanism based on a large model, uses a pre-trained large-scale language model and an introduced medical knowledge module to generate a detailed diagnostic report. In this way, after integrating medical knowledge and image features, the generated diagnostic report is not limited to simple classification results, but also includes professional medical descriptions and treatment recommendations, such as tympanic membrane color, light cones, perforations and other features, to provide clinicians with a comprehensive diagnostic reference. The system of the embodiment of the present invention, through the introduction of external medical knowledge, enables the language model to recognize and understand more complex symptom characteristics, especially for rare or complex ear diseases, to provide more accurate diagnostic results.

[0110] Optionally, the above-mentioned category enhancement module and knowledge enhancement module are trained Transformer models. Before using the above-mentioned category enhancement module and knowledge enhancement module to identify and diagnose the input image, the system of the embodiment of the present invention first trains the corresponding Transformer model to improve the feature fusion and representation capabilities of the Transformer model, thereby improving the accuracy of the model output. The above-mentioned diagnosis generation module is a language model obtained through training. Before using the above-mentioned diagnosis generation module to obtain a diagnosis report, the system of the embodiment of the present invention first trains the language model to improve the recognition and expression capabilities of the language model.

[0111] Optionally, the system of the embodiment of the present invention can train the category enhancement module and the knowledge enhancement module as a whole. During the training process of the Transformer model, the parameters of the language model and the CLIP encoder remain unchanged. This design allows the language model to introduce and apply external medical knowledge by fine-tuning the Transformer model while keeping the parameters of the large language model frozen, thereby achieving control of the language model and achieving the effect of intelligent diagnosis. In this way, the general language model can adapt to specific tympanic membrane disease diagnosis tasks, not only can it accurately classify the type of otitis media, but also can generate a diagnostic report that meets medical professional standards, and describe in detail the color, light cone, perforation and other characteristics of the tympanic membrane, providing a comprehensive diagnostic reference for clinicians. Among them, the specific training process of the Transformer model is as follows:

[0112] First, the sample data is input into the category enhancement module in batches. Figure 2 The illustrated embodiment processes the sample data to generate a diagnosis report, that is, the category enhancement module, the knowledge enhancement module and the diagnosis generation module sequentially process the sample data to obtain a diagnosis report.

[0113] Secondly, the loss value is calculated according to the following loss function:

[0114] L total =αLce +(1-α)L gen

[0115] Among them, (L ce ) is the cross entropy loss for category prediction, (L gen ) is the language model loss for diagnostic report generation (usually negative log likelihood), and (α) is the weight hyperparameter, set to 0.5.

[0116] Finally, all parameters of the Transformer model, including the query token, are updated via mini-batch gradient descent.

[0117] The above process is repeated until the preset condition is met to obtain a trained Transformer model. In the embodiment of the present invention, the preset condition may refer to a preset number of training rounds, such as 100 rounds.

[0118] Furthermore, for the above-mentioned preset training rounds, the system of the embodiment of the present invention can implement the training of the entire Transformer model in steps. Specifically as follows:

[0119] First, the sample data is input into the category enhancement module in batches. Figure 2 The illustrated embodiment processes the sample data to generate a diagnosis report, that is, the category enhancement module, the knowledge enhancement module and the diagnosis generation module sequentially process the sample data to obtain a diagnosis report.

[0120] Secondly, in the first 50 training rounds, only the category prediction task is optimized, that is, only the first loss value (L ce ), and according to the first loss value (L ce ) Update the parameters of the Transformer model corresponding to the category enhancement module. During the training process, the performance of the Transformer model is evaluated every 5 training rounds. If the performance of the Transformer model does not improve significantly after 10 consecutive training rounds, the training is stopped early.

[0121] Again, in the next 50 training rounds, we jointly optimize the category prediction and report generation tasks, that is, we calculate L according to the above loss function. total , and according to the loss value L total Update the parameters of the Transformer model corresponding to the category enhancement module and the Transformer model corresponding to the knowledge enhancement module. During the training process, the performance of the Transformer model is evaluated every 5 training rounds. If the performance of the Transformer model does not improve significantly after 10 consecutive training rounds, stop the training early.

[0122] The system of the embodiment of the present invention can effectively learn category representation and feature fusion under small sample conditions by adopting a strategy of training the embedded text template and the Transformer model in stages, thereby achieving accurate recognition of tympanic membrane images and generation of diagnostic reports. At the same time, by adopting a strategy of step-by-step training and multi-task learning, it is possible to balance classification accuracy and diagnostic report generation quality.

[0123] In summary, the system of the embodiment of the present invention can achieve accurate recognition and diagnosis of tympanic membrane images under small sample conditions, while providing interpretable results, providing clinicians with a powerful auxiliary diagnosis tool. The system not only improves the diagnostic accuracy, but also provides detailed diagnostic suggestions, greatly enhancing the practical value of AI-assisted diagnosis systems in the field of otology.

[0124] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A small sample tympanic membrane image recognition and diagnosis system based on artificial intelligence, characterized in that: The system comprises: A category matching module, used to obtain an input image and an image category set, calculate the similarity between the input image and at least one image category, and determine a matching category of the input image according to the similarity; a category enhancement module, configured to enhance the feature expression of the input image according to the input image, the matching category and the first query, so as to determine the disease category corresponding to the input image; A knowledge enhancement module, used to enhance the feature expression of the input image and determine the diagnosis and treatment plan of the disease category according to the structured knowledge corresponding to the matching category of the input image; The diagnosis generation module is used to generate a diagnosis report corresponding to the input image.

2. The system according to claim 1, characterized in that The category matching module comprises: An image encoding module, used to obtain an image feature vector corresponding to an input image through a pre-trained image encoder; A category encoding module, used to obtain a category feature vector corresponding to at least one image category through a pre-trained category encoder; The similarity matching module is used to map the image feature vector and the category feature vector to the same feature space respectively, calculate the similarity between the image feature vector and the category feature vector, and determine the matching category of the input image according to the similarity.

3. The system according to claim 2, characterized in that The similarity matching module is specifically used for: Calculating the similarity between the image feature vector and each image category in the category feature vector respectively; The image category with the highest similarity is selected as the matching category of the input image.

4. The system according to claim 2, characterized in that The pre-trained category encoder is used to obtain the category feature vector according to the received embedded text template and image category set; wherein the embedded text template is obtained through training.

5. The system according to any one of claims 2 to 4, characterized in that: The category enhancement module includes: A first self-attention layer, configured to perform multi-head attention calculation according to an image feature vector of the input image and a query feature vector corresponding to the first query to obtain a first self-attention output; A first cross-attention layer, configured to perform a cross-attention calculation according to the first self-attention output and a matching category of the input image to obtain a first cross-attention output; A feed-forward layer is used to determine the disease category of the input image according to the first cross-attention output.

6. The system according to claim 5, characterized in that The knowledge enhancement module further includes: A second self-attention layer, used for performing multi-head attention calculation according to the disease category of the input image to obtain a second self-attention output; a second cross attention layer, configured to perform a cross attention calculation according to the second self-attention output and the structured knowledge corresponding to the disease category of the input image, to obtain a second cross attention output; The feed-forward layer is used to determine the diagnosis and treatment plan corresponding to the input image according to the second cross-attention output.

7. The system according to claim 6, characterized in that The system also includes a knowledge screening and integration module, which includes: A knowledge base, used to store at least one knowledge item; A knowledge encoder, used to encode at least one knowledge item in the knowledge base to obtain a knowledge feature vector corresponding to the at least one knowledge item; A feature fusion device, used for concatenating the image feature vector with the vector corresponding to the matching category to obtain a fused feature vector; The knowledge filter is used to filter and obtain structured knowledge corresponding to the disease category of the input image from the knowledge base according to the fused feature vector and the knowledge feature vector.

8. The system according to claim 7, characterized in that The knowledge filter is specifically used for: Calculating the second similarity between each knowledge feature vector and the fusion feature vector respectively; sorting at least one of the second similarities, and selecting a preset number of knowledge items corresponding to the second similarities as selected knowledge; The selected knowledge is processed by spacy library functions to generate the structured knowledge.

9. The system according to claim 1, characterized in that The diagnosis generation module is used to obtain the output of the knowledge enhancement module, extract a second query from the output of the knowledge enhancement module, and generate a diagnosis report according to the second query.

10. The system according to claim 1, characterized in that The category enhancement module and the knowledge enhancement module are trained Transformer models; the diagnosis generation module is a trained language model.

Images