Evaluation device and method for identifying ear status classification of people based on local features

Abstract

The present invention discloses an evaluation device and method for classifying the ear status of a population based on local feature recognition, comprising: collecting images of the patient's ear canal and tympanic membrane area, cleaning and statistically classifying the images to obtain a preprocessed image; extracting features from the preprocessed image to obtain ear surface features; pre-labeling and automatically labeling the ear surface features to obtain a labeled image; further cleaning the labeled image to obtain a cleaned image; constructing an image feature recognition model based on the YOLO framework, training the image feature recognition model based on the YOLO framework using the cleaned image to obtain a classification evaluation probability model for ear typing, and performing ear typing classification evaluation based on the ear typing classification evaluation probability model to obtain an evaluation result. The present invention can provide ordinary users with better ear information management and, to a certain extent, deepen people's understanding of ear status.

Description

Technical Field

[0001] The present invention belongs to the field of ear recognition and evaluation, and in particular relates to an evaluation device and method for identifying ear status typing of a group of people based on local features. Background Art

[0002] Currently, ear problems are facing a serious global challenge. According to relevant data, approximately 1.1 billion young people worldwide are at risk of irreversible hearing loss. Hearing loss not only affects individual quality of life but also imposes a socioeconomic burden. Although awareness and care for ear health are gradually increasing, the shortage of specialized equipment and resources remains a challenge. The lack of professional ear specialists and advanced equipment, particularly in primary care hospitals and community settings, limits early screening and treatment of ear problems. Fortunately, the rise of deep learning and artificial intelligence technologies has brought unprecedented changes to the field of ear health monitoring. Leveraging deep learning algorithms, we can efficiently analyze otoscope images, construct accurate ear health status prediction and clustering models, and achieve personalized early feature identification. This innovation not only enhances public awareness of ear health issues but also encourages people to take proactive measures in their daily lives to improve their ear health. Furthermore, the application of artificial intelligence (AI) technology in otolaryngology clinical diagnosis and treatment is also progressing. This includes image analysis, speech signal processing, and the construction of condition prediction models, all of which play a crucial role in supporting decision-making. For example, CN114494816A discloses an efficient ear feature recognition system based on integrated deep learning. This system can identify features from ear images, thereby building a classifier to achieve the goal of clustering ear conditions. However, ear conditions often exhibit multiple concurrent features, which poses challenges for prediction and prevention through feature classification. A single feature classification model often fails to accurately predict existing problems because it cannot reasonably parse and distinguish multiple concurrent ear features. For example, otitis media may coexist with other ear conditions, such as chronic otitis externa. The presence of these concurrent features further complicates the classification of ear conditions. Furthermore, there is currently a lack of convenient ear feature recognition systems available on the market to support the development of ear health status management systems. An easy-to-use, comprehensive, and accurate ear condition management system that helps identify ear features is crucial for improving the efficiency and quality of health management. Such a system can provide real-time feedback on ear data and ear characteristics, as well as ear status classification, providing the general public with fast data analysis and health knowledge services, helping people understand ear health issues more easily and directly. Summary of the Invention

[0003] To solve the above technical problems, the present invention provides an evaluation device and method for identifying the ear status of a group of people based on local features. Among them, an evaluation device for identifying the ear status of a group of people based on local features includes:

[0004] An image acquisition module, including a wired device and a wireless device, is used to acquire images of the patient's ear canal and tympanic membrane area using a camera equipped with a cold light source or a warm light source;

[0005] An image preprocessing module, connected to the image acquisition module, for cleaning and statistically classifying the image to obtain a preprocessed image;

[0006] A feature extraction module, connected to the image preprocessing module, for extracting features from the preprocessed image to obtain ear appearance features;

[0007] A feature annotation module, connected to the feature extraction module, is used to perform feature pre-annotation and automatic annotation on the ear appearance features to obtain an annotated image;

[0008] An image cleaning module, connected to the feature annotation module, for further cleaning the annotated image to obtain a cleaned image;

[0009] a model building module, connected to the image cleaning module, for building a picture feature recognition model based on the YOLO framework, training the picture feature recognition model based on the YOLO framework using the cleaned image, and obtaining a classification evaluation probability model for ear typing;

[0010] The classification evaluation module is connected to the model building module and is used to perform ear typing classification evaluation based on the ear typing classification evaluation probability model to obtain an evaluation result.

[0011] Preferably, the image preprocessing module includes a first filtering unit, a second filtering unit, a third filtering unit, and a state statistics unit;

[0012] The first filtering unit is used to repeatedly sample each ear of the patient from the ear canal to the eardrum area at different angles to obtain initial sampling samples, and then use Wiener filtering and constrained least squares filtering to filter out blurred images;

[0013] The second filtering unit is used to filter similar images by using an image similarity comparison method to calculate the grayscale histograms of the two images and compare and evaluate the similarity between the images. When the similarity is greater than a preset threshold, the images are considered similar and only one of them is retained;

[0014] The third filtering unit is configured to filter pictures that lack ear status descriptions based on the patient's medical history information;

[0015] The state statistics unit is used to collect statistics on the normal or abnormal states in the patient medical records corresponding to the retained samples, and obtain different types of ear typing state images.

[0016] Preferably, the different types of ear classification status images include: ear NE classification representing normal people, ear MEC classification representing people with middle ear cholelithiasis, ear CSOM classification representing people with chronic suppurative otitis media, ear EABC classification representing people with external auditory canal bleeding, ear OE classification representing people with external otitis, ear IC classification representing people with external auditory canal cerumen embolism, ear OM classification representing people with external auditory canal fungal infection, ear OME classification representing people with secretory otitis media, ear AOM classification representing people with acute otitis media, ear TMC classification representing people with tympanic membrane calcification and ear BM classification representing people with bullous myringitis.

[0017] Preferably, the ear appearance characteristics include physiological structure characteristics and clinical manifestation characteristics;

[0018] The physiological structural features include tympanic membrane, pars flaccida, and hammertos;

[0019] The clinical features include calcifications, amber color, cerumen, bleeding, discharge, redness, perforation, and fungus;

[0020] The redness characteristics are further divided into deep red and reddish according to the redness area and appearance degree;

[0021] The deep red refers to redness of the handle of the malleus or swelling of the tympanic membrane, which is not bloodshot, and all visible areas of the ear canal are red.

[0022] The reddish state means that the tympanic membrane is not red or swollen, the handle of the malleus is not red, but there are blood streaks, and there are obvious blood streaks or redness in the flaccid part of the ear canal.

[0023] The present invention also provides an evaluation method for identifying ear status types of a group of people based on local features, comprising:

[0024] collecting images of the patient's ear canal and tympanic membrane area, and performing cleaning and statistical classification on the images to obtain preprocessed images;

[0025] performing feature extraction on the preprocessed image to obtain ear appearance features;

[0026] Performing feature pre-labeling and automatic labeling on the ear surface features to obtain a labeled image;

[0027] Further cleaning the annotated image to obtain a cleaned image;

[0028] A picture feature recognition model based on the YOLO framework is constructed, and the picture feature recognition model based on the YOLO framework is trained using the cleaned image to obtain a classification evaluation probability model for ear typing. Ear typing classification evaluation is performed based on the classification evaluation probability model for ear typing to obtain an evaluation result.

[0029] Preferably, the process of cleaning and statistically classifying the image includes:

[0030] First, the area from the ear canal to the eardrum of each patient is repeatedly sampled from different angles to obtain initial sampling samples. Then, Wiener filtering and constrained least squares filtering are used to filter out blurred images.

[0031] Secondly, for filtering similar images, the grayscale histogram of the two images is calculated using the image similarity comparison method and the similarity between the images is compared and evaluated. When the similarity is greater than a preset threshold, the images are considered similar and only one of them is retained.

[0032] Finally, based on the patient's medical records, images with missing ear status descriptions were filtered out, and the normal or abnormal status in the patient's medical records corresponding to the retained samples were counted to obtain different types of ear classification status images.

[0033] Preferably, the basis for further cleaning the annotated image includes:

[0034] There are feature representations in the image;

[0035] The feature representation pixel ratio is greater than 0.0001;

[0036] When the pixel ratio of a feature in the image cannot meet the above two conditions, the feature area information is recorded through the cleaning program, and then the information of the feature area is deleted, and the action of modifying the JSON file is performed at the same time.

[0037] Preferably, the process of training the YOLO framework-based image feature recognition model using the cleaned image to obtain a classification evaluation probability model for ear typing includes:

[0038] Unsupervised learning training phase and supervised learning training phase;

[0039] The unsupervised learning training phase includes:

[0040] Adopting the contrastive learning method in the self-supervised learning method, the encoder network of the deep convolutional neural network ResNet50 model is used to extract image features;

[0041] Perform data augmentation on each unlabeled image, feed the augmented image into an encoder to obtain a feature representation, and map the feature representation into a contrast space through a projection head;

[0042] Among them, the loss function uses contrast loss, specifically InfoNCE loss, and the formula is:

[0043]

[0044] where sim(z i ,zj) represents the cosine similarity τ is the temperature parameter, N is the batch size; T represents the transpose operation of the matrix z; 1[k=i] is the indicator function, which takes 1 when k≠i and 0 otherwise.

[0045] Preferably, the data enhancement includes randomly selecting Gaussian blur, brightness, saturation, random color jitter, random grayscale, and random horizontal flip.

[0046] Preferably, the supervised learning training phase includes:

[0047] Load the weights of the encoder obtained from unsupervised pre-training into the backbone network of the YOLO model, and use labeled data to perform supervised training on the model based on pre-training;

[0048] Wherein, the YOLO model includes a YOLOv8 model;

[0049] The YOLOv8 model includes a backbone network, a neck network, and a detection head;

[0050] The neck network is used to fuse features from different levels of the backbone network;

[0051] The detection head is responsible for predicting bounding boxes, confidence scores, and category probabilities;

[0052] The labeled data includes images, bounding boxes, and category labels;

[0053] The total loss function of the YOLO model includes positioning loss, confidence loss and classification loss;

[0054] L total =L loc +L conf +L cls

[0055] Among them, the positioning loss L loc : Use GIoU, DIoU or CIoU loss to measure the overlap between the predicted box and the true box;

[0056] Confidence loss Lconf : Use binary cross entropy loss to measure the confidence that the predicted box contains the target;

[0057] Classification loss L cls : Use multi-class cross entropy loss to measure the difference between the predicted category and the true category.

[0058] Compared with the prior art, the present invention has the following advantages and technical effects:

[0059] The present invention solves the limitations and low accuracy of the recognition of classification models based on single features through a target recognition model based on multiple superficial features related to ear typing, realizes the diversification of model input features, and greatly enhances the parseability and universality of the model. From the perspective of the entire system, the convenient edge operation and rapid feedback of results can, to a certain extent, bring better ear information management to ordinary users, and deepen people's understanding of the state of the ear to a certain extent, and can also understand some precautions and suggestions for daily life when the ear has certain superficial features. For the current severe ear health problems, this solution can also alleviate problems such as resource shortages, and can also help doctors with insufficient clinical experience to evaluate the patient's ear typing and the superficial features before diagnosis and treatment, thereby assisting in diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0061] Figure 1 Schematic diagram of the system structure of an embodiment of the present invention;

[0062] Figure 2 Characteristic statistical diagram annotated for the embodiment of the present invention;

[0063] Figure 3 : is a normalized confusion matrix diagram of an embodiment of the present invention;

[0064] Figure 4 This is a first example diagram of the automatic annotation result according to an embodiment of the present invention;

[0065] Figure 5 This is a second example diagram of the automatic annotation result according to an embodiment of the present invention;

[0066] Figure 6 This is a diagram showing the correlation analysis and visualization between the characteristic features and ear typing of an embodiment of the present invention;

[0067] Figure 7 Schematic diagram showing the similarity between the characteristic features of an embodiment of the present invention and ear typing;

[0068] Figure 8 This is a logic diagram of the implementation framework of the image feature recognition model based on the YOLO framework in an embodiment of the present invention;

[0069] Figure 9 This is a partial picture display of the validation set model annotation results of an embodiment of the present invention. DETAILED DESCRIPTION

[0070] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0071] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0072] Example 1

[0073] like Figure 1 As shown, this embodiment provides an evaluation device for identifying ear status types of a group of people based on local features, including:

[0074] An image acquisition module, including a wired device and a wireless device, is used to acquire images of the patient's ear canal and tympanic membrane area using a camera equipped with a cold light source or a warm light source;

[0075] The image preprocessing module is connected to the image acquisition module and is used to clean and statistically classify the image to obtain the preprocessed image;

[0076] A feature extraction module is connected to the image preprocessing module and is used to extract features from the preprocessed image to obtain ear appearance features;

[0077] A feature annotation module, connected to the feature extraction module, is used to pre-annotate and automatically annotate the ear's surface features to obtain an annotated image;

[0078] The image cleaning module is connected to the feature annotation module and is used to further clean the annotated image to obtain a cleaned image;

[0079] The model building module is connected to the image cleaning module and is used to build an image feature recognition model based on the YOLO framework. The model is trained on the image feature recognition model based on the YOLO framework by cleaning the image to obtain a classification evaluation probability model for ear typing.

[0080] The classification evaluation module is connected to the model building module and is used to perform ear typing classification evaluation based on the ear typing classification evaluation probability model to obtain evaluation results.

[0081] Furthermore, the image preprocessing module includes a first filtering unit, a second filtering unit, a third filtering unit, and a state statistics unit;

[0082] The first filtering unit is used to repeatedly sample the area from the ear canal to the eardrum of each ear of the patient from different angles to obtain initial sampling samples, and then use Wiener filtering and constrained least squares filtering to filter out blurred images;

[0083] The second filtering unit is used to filter similar images by using an image similarity comparison method to calculate the grayscale histogram of the two images and compare and evaluate the similarity between the images. When the similarity is greater than a preset threshold, the images are considered similar and only one of them is retained;

[0084] The third filtering unit is used to filter out pictures that lack ear status descriptions based on the patient's medical record information;

[0085] The state statistics unit is used to collect statistics on the normal or abnormal states in the patient medical records corresponding to the retained samples to obtain different types of ear typing state images.

[0086] Furthermore, different types of ear classification status images include: ear NE classification representing normal people, ear MEC classification representing people with middle ear cholelithiasis, ear CSOM classification representing people with chronic suppurative otitis media, ear EABC classification representing people with external auditory canal bleeding, ear OE classification representing people with external otitis, ear IC classification representing people with external auditory canal cerumen embolism, ear OM classification representing people with external auditory canal fungal infection, ear OME classification representing people with secretory otitis media, ear AOM classification representing people with acute otitis media, ear TMC classification representing people with tympanic membrane calcification and ear BM classification representing people with bullous myringitis.

[0087] Furthermore, the ear appearance characteristics include physiological structure characteristics and clinical manifestation characteristics;

[0088] Physiological structural features include the tympanic membrane, pars flaccida, and hammertos;

[0089] Clinical features include calcifications, amber color, cerumen, bleeding, discharge, redness, perforation, and fungus;

[0090] Among them, the characteristics of redness are further divided into deep red and reddish according to the redness area and the degree of appearance;

[0091] Deep red is non-bloodshot redness of the malleus handle or redness of the tympanic membrane, and all visible areas of the ear canal are red;

[0092] The reddish state indicates that the tympanic membrane is not red or swollen, the handle of the malleus is not red, but there are blood streaks, and there are obvious blood streaks or redness in the flaccid part of the ear canal.

[0093] In the device of this embodiment, it can be a PC (Personal Computer), or it can be a terminal device such as a smartphone, tablet computer, portable computer, or other electronic products. These terminal devices collect images of the ear and then transmit them to the intelligent AI model system.

[0094] The intelligent AI model mentioned here refers to the ear appearance feature recognition model (EFA) described above. This model can be deployed in its entirety on a local server or on a cloud server. The model receives ear images collected from the edge, performs model operations, identifies which of the 12 feature categories the image possesses, and calculates the pixel ratios of each of these features. These results are then passed as an information flow to the appearance feature and ear typing relationship model.

[0095] The model of the relationship between appearance features and ear typing mainly refers to a method that can combine the ear typing correlation coefficient matrix M mentioned above with the above-mentioned information flow to calculate the weighted probability value of the correlation between the image and the ear typing of the population. Through this step, the input information flow required by the probability evaluation model of an ear typing can be obtained. The information flow is input into the probability evaluation model, and the ear typing evaluation rules are used to determine the possible level of the ear typing to which the currently collected ear image belongs. The levels are divided into no, low possibility, medium possibility and high possibility. The results are combined with the detection time as the result of the individual's current test and stored in the personal file folder of the system, and some daily life precautions and suggestions are given for the currently identified appearance features. The above results and suggestions will be further fed back to the edge device for visual display, so that the user can obtain information such as the current test results.

[0096] Example 2

[0097] Based on the same inventive concept, this embodiment further provides an evaluation method for identifying ear status types of a population based on local features, including:

[0098] Collect images of the patient's ear canal and tympanic membrane area, clean and statistically classify the images, and obtain preprocessed images;

[0099] Perform feature extraction on the preprocessed image to obtain ear appearance features;

[0100] Perform feature pre-labeling and automatic labeling of ear surface features to obtain labeled images;

[0101] Further cleaning the labeled image to obtain a cleaned image;

[0102] A picture feature recognition model based on the YOLO framework was constructed. The picture feature recognition model based on the YOLO framework was trained by cleaning the image to obtain a classification evaluation probability model of ear typing. Based on the classification evaluation probability model of ear typing, ear typing classification evaluation was performed to obtain the evaluation results.

[0103] Furthermore, the process of cleaning and statistically classifying the image includes:

[0104] First, the area from the ear canal to the eardrum of each patient is repeatedly sampled from different angles to obtain initial sampling samples. Then, Wiener filtering and constrained least squares filtering are used to filter out blurred images.

[0105] Secondly, for filtering similar images, the image similarity comparison method is used to calculate the grayscale histogram of the two images and compare and evaluate the similarity between the images. When the similarity is greater than the preset threshold, the images are considered similar and only one of them is retained;

[0106] Finally, based on the patient's medical records, images with missing ear status descriptions were filtered out, and the normal or abnormal status in the patient's medical records corresponding to the retained samples were counted to obtain different types of ear classification status images.

[0107] Specifically, data collection, cleaning and statistics:

[0108] The data collection mainly includes wired devices and wireless devices, both of which are equipped with cameras with cold light sources or warm light sources. These cameras are used to collect images of the patient's ear canal and tympanic membrane area.

[0109] The acquisition and cleaning method usually involves first performing repeated sampling from the ear canal to the tympanic membrane area of ​​each patient at different angles to obtain an initial sampling sample, and then performing an initial screening of the image, using Wiener filtering and constrained least squares filtering to filter out blurred images.

[0110] A blurred image is defined as an image that is still not clear after deblurring (i.e., less than a known blur kernel. In this embodiment, the blur kernel is set to PSF = np.ones((5,5)) / 25, where np is the abbreviation of the Python library numpy). This image is considered pixel-blurred and is then filtered out.

[0111] For filtering very similar images, the image similarity comparison method is used. That is, the grayscale histograms of the two images are calculated and compared to evaluate the similarity between the images. When the similarity is greater than a given threshold (the threshold is set to 0.9 in this embodiment), the two images are considered similar. In this case, only one of them can be retained.

[0112] Finally, images without descriptions of ear status (normal / abnormal) were filtered out based on the patient's medical records.

[0113] Data statistics mainly refer to the statistics of normal or abnormal conditions in the patient medical records corresponding to the retained samples, so as to obtain common ear condition types.

[0114] In this embodiment, about 100,000 ear image samples (including ear canal images and tympanic membrane images, which are obtained from about 30,000 patients in total) of a large number of patients obtained through cold light source ear endoscopes from different hospitals, and according to the diagnosis report and doctor's advice of each patient, the normal or abnormal ear status information corresponding to the image sample is obtained, and the summary statistics are performed to preliminarily obtain ear classifications of people with 11 different ear conditions, namely: NE classification of the ear representing normal people, MEC classification of the ear representing people with middle ear cholelithiasis, and chronic suppurative ear classification. The ear types include CSOM classification for people with otitis media with serous otitis, EABC classification for people with external auditory canal bleeding, OE classification for people with external otitis, IC classification for people with external auditory canal cerumen obstruction, OM classification for people with external auditory canal fungal infection, OME classification for people with secretory otitis media, AOM classification for people with acute otitis media, TMC classification for people with tympanic membrane calcification, and BM classification for people with bullous myringitis.

[0115] Furthermore, the process of obtaining the ear appearance feature type includes:

[0116] First, we used the Octopus tool to crawl descriptive texts related to ear diseases from public medical forums and otolaryngology websites on the domestic and international Internet. Then, we used the Python language NLTK library and the Jieba library to segment the text data and count the vocabulary and word frequency. Then, we downloaded the available medical guidelines and used the text big model to analyze and refine the feature descriptions related to ear diseases in the text to form feature description text. Then, we continued to use the Jieba library to segment all text fields related to symptom descriptions or diagnosis descriptions in the patient's medical records, and count the vocabulary and word frequency. Finally, combined with domestic and foreign literature and based on medical records, the above vocabulary was compared and summarized sentence by sentence, and combined with the suggestions of otolaryngologists, 11 characteristics related to the ear canal and eardrum were identified, including 3 physiological structural characteristics: tympanic membrane (GM in Chinese or TM in English), pars flaccida (SCB in Chinese or PF in English), and the shortprocess of the malleus (CT in Chinese or SPM in English); and 8 clinical manifestations: calcification Tympanosclerosis / calcification (GHB in Chinese or TS in English), amber (HPS in Chinese or AM in English), cerumen (DN in Chinese or CE in English), bleeding (CX in Chinese or BE in English), secretion (FMW in Chinese or SE in English), redness (SH in Chinese or RE in English), and perforated tympanic membrane. perforation (CK in Chinese or TMP in English), fungus (MJ in Chinese or FG in English).

[0117] It is important to note that for features such as redness, we further categorize them into deep red (SH) and partial red (PH) based on the area and degree of redness. The distinction is made as follows: i) Deep red: The handle of the malleus is red (not bloodshot) or the tympanic membrane is red and swollen, and the entire visible area of ​​the ear canal is red; ii) Partial red: The tympanic membrane is not red or swollen, the handle of the malleus is not red, but bloodshot is present, and there is obvious bloodshot or localized redness in the flaccid part of the ear canal. Thus, we obtained 12 categories of ear appearance features.

[0118] Furthermore, the process of feature pre-annotation and automatic annotation includes:

[0119] First, based on the image library of 11 categories of ear classification, about 1,000 pictures corresponding to each category of status were randomly selected independently, and finally a total of 11,838 pictures were obtained. Otolaryngologists labeled the images using ENTLABEL software, and then professional doctors in the ENT department used labelme software to annotate the 12 categories of features in each picture. The annotation method is to use a square to mark a smallest square area on the image for this type of feature. This square area contains all the pixel information of the feature and is labeled. The label name is the English abbreviation of this type of feature, such as TMP.

[0120] Then, if Figure 2-3 As shown in the figure, for these 11,838 images that have completed information annotation, a pre-annotated target recognition model preEFA1.0 (pre Ear Feature Analysis of version 1.0, the present invention adopts the YOLO V8 model, parameters: batch = 32, epochs = 200; 9:1 ratio division balanced data set into training set and test set, verification accuracy rate reaches more than 98%) is constructed based on the YOLO framework in the deep learning model.

[0121] Next, with the target recognition model preEFA1.0 as the core, a target detection and custom annotation framework was built. This framework can perform target recognition calculations on any input ear sample image, automatically outline the feature area, and finally output a JSON file containing information such as the pixel size and coordinates of the target area outline. This file can be read and visualized by the labelme software together with the image file. All sample images are iteratively input into this framework to achieve automatic annotation of about 100,000 ear image samples. The automatic annotation result example is shown in the figure below. Figure 4-5 shown.

[0122] Finally, the automatically annotated images are reviewed and corrected. This step primarily involves otolaryngologists reviewing and correcting the results of approximately 100,000 automatically annotated images. Images with JSON files are imported into the labelme software, and the visually displayed areas and their labels are corrected. Corrections include whether the area size is correct (displacement, oversize, and undersize are corrected), whether the area labeling is misjudged (if misjudged, the label is corrected), and whether certain features are missing (if missing, they are supplemented).

[0123] Furthermore, the basis for further cleaning the annotated image includes:

[0124] There are feature representations in the image;

[0125] The feature representation pixel ratio is greater than 0.0001;

[0126] When the pixel ratio of a feature in the image cannot meet the above two conditions, the feature area information is recorded through the cleaning program, and then the feature area information is deleted, and the JSON file is modified at the same time.

[0127] More specifically, we first conducted further cleaning on the approximately 100,000 images reviewed and corrected. This cleaning process was based on the following criteria: a) the image contained the characteristic representation; b) the pixel ratio of the characteristic representation was greater than 0.0001. If the pixel ratio of a feature in an image failed to meet these two conditions, the cleaning program would record and delete the information about the feature region, and simultaneously modify the JSON file.

[0128] Then, for the features of the ear images of the people under each classification, a Correlation Score is defined as the score corresponding to each feature under each classification. The calculation formula is as follows:

[0129]

[0130] Where i represents a type, j represents a feature, and Freq ij It represents the frequency of appearance feature j under classification i, that is, the proportion of pictures with appearance feature j in all pictures corresponding to classification i, and Mean ij =(\frac{\text{pixel ratio}}\left({\text{pixel ratio}}}) represents the mean pixel ratio of feature j for subtype i, i.e., the mean pixel ratio of feature j across all images corresponding to subtype i. The pixel ratio here refers to the ratio of the pixel size of feature j to the pixel size of the entire image, and n is the total number of features (12 in this paper). A score is calculated for each subtype corresponding to each feature, yielding a scoring basis coefficient matrix M with a dimension of 11×12.

[0131] Then, if Figure 6-7As shown in the figure, the correlation analysis and visualization of epigenetic features and ear types were performed using the R language pheatmap and ggalluvial packages. The results showed that different types correspond to one or more relatively obvious epigenetic features, and these epigenetic features are correlated with ear types to a certain extent. For example, some epigenetic features also appear in ear NE types, such as DN, PH, and FMW, but these epigenetic features are not strongly correlated with ear NE types. For ear IC types, its main feature is DN, which is very consistent with the epigenetic features. For example, the obvious epigenetic feature of ear EACB types is CX, the epigenetic features of ear OME types are clearly strongly correlated with HPS, and ear OM types are mainly caused by MJ infection. At the same time, some features are common in different ear types, such as FMW, which is common in ear MEC and OE types, DN, which appears in different types, GHB, which is common in ear TMC and CSOM types, and more importantly, CK, which is common in ear CSOM, TMC, and MEC types. Based on these findings, ear typing has one or more phenotypic characteristics that can be used to distinguish them. In other words, the characterization characteristics help to further understand and study the ear status of different populations and deepen the understanding of ear problems.

[0132] Furthermore, the process of building an image feature recognition model based on the YOLO framework includes:

[0133] like Figure 8 As shown in the figure, for about 100,000 images, the balanced data set is divided into training set and test set in a ratio of 9:1, and the ear canal feature recognition model EFA v1.5 (Ear Feature Analysis of version 1.5) is constructed using the YOLO framework. The framework logic of the model is as follows Figure 8 shown.

[0134] Furthermore, the image cleaning process is used to train the image feature recognition model based on the YOLO framework to obtain the classification evaluation probability model for ear typing. The process includes:

[0135] Unsupervised learning training phase and supervised learning training phase;

[0136] Among them, the unsupervised learning training stage includes:

[0137] Adopting the contrastive learning method in the self-supervised learning method, the encoder network of the deep convolutional neural network ResNet50 model is used to extract image features;

[0138] Perform data augmentation on each unlabeled image, feed the augmented image into the encoder to obtain feature representation, and then map the feature representation to the contrast space through the projection head;

[0139] Among them, the loss function uses contrast loss, specifically InfoNCE loss, and the formula is:

[0140]

[0141] where sim(z i ,zj) represents the cosine similarity τ is the temperature parameter, N is the batch size; T represents the transpose operation of the matrix z; 1[k=i] is the indicator function, which takes 1 when k≠i and 0 otherwise.

[0142] Furthermore, data augmentation includes random selection of Gaussian blur, brightness, saturation, random color jitter, random grayscale, and random horizontal flip.

[0143] Furthermore, the supervised learning training phase includes:

[0144] Load the weights of the encoder obtained from unsupervised pre-training into the backbone network of the YOLO model, and use labeled data to perform supervised training on the model based on pre-training;

[0145] Among them, the YOLO model includes the YOLOv8 model;

[0146] The YOLOv8 model includes a backbone network, a neck network, and a detection head;

[0147] Among them, the neck network is used to fuse features from different levels of the backbone network;

[0148] The detection head is responsible for predicting bounding boxes, confidence scores, and category probabilities;

[0149] Labeled data includes images, bounding boxes, and category labels;

[0150] The total loss function of the YOLO model includes positioning loss, confidence loss, and classification loss;

[0151] L total =L loc +L conf +L cls

[0152] Among them, the positioning loss L loc : Use GIoU, DIoU or CIoU loss to measure the overlap between the predicted box and the true box;

[0153] Confidence loss L conf : Use binary cross entropy loss to measure the confidence that the predicted box contains the target;

[0154] Classification loss L cls: Use multi-class cross entropy loss to measure the difference between the predicted category and the true category.

[0155] More specifically, the unsupervised learning phase:

[0156] Method selection: Use contrastive learning in the self-supervised learning method.

[0157] Model architecture: Using an encoder network f θ ,Here we use the deep convolutional neural network ResNet50 model to extract image features.

[0158] Unsupervised training process:

[0159] Data augmentation: Apply two random data augmentations (Gaussian blur, brightness, saturation, random color jitter, random grayscale, random horizontal flip, etc.) to each unlabeled image to generate two views x i and x j .

[0160] Feature extraction: Input the enhanced image into the encoder to obtain the feature representation h i =f θ (x i ) and h j =f θ (x j ).

[0161] Projection head: Through a projection head g φ Map the features to the contrast space and get z i =g φ (h i ), z i =g φ (h i ).

[0162] Loss function: Use contrast loss (Contrastive Loss), here we use InfoNCE loss:

[0163]

[0164] where sim(z i ,zj) represents the cosine similarity τ is the temperature parameter, N is the batch size, T represents the transpose of the matrix z, and 1[k=i] is the indicator function, which takes the value 1 when k≠i and 0 otherwise.

[0165] Supervised learning stage (YOLO target detection):

[0166] Model initialization: The encoder f obtained by unsupervised pre-training θ The weights are loaded into the backbone network of the YOLO model.

[0167] Model Architecture: This model uses the YOLO (You Only Look Once) family of object detection models, specifically the YOLOv8 model. It consists of a backbone network, a neck network, and a detection head. The neck network (Neck) fuses features from different layers of the backbone network. Common architectures include FPN and PANet. The detection head (Head) is responsible for predicting bounding boxes, confidence scores, and class probabilities.

[0168] Supervised training process:

[0169] Data preparation: Use labeled data, including images, bounding boxes, and category labels.

[0170] Fine-tuning training: Based on pre-training, the model is supervised trained, and you can choose to freeze some layers of training.

[0171] Loss function: YOLO's total loss function consists of positioning loss, confidence loss, and classification loss:

[0172] L total =L loc +L conf +L cls

[0173] Among them, the positioning loss L loc : Use GIoU, DIoU or CIoU loss to measure the overlap between the predicted box and the true box;

[0174] Confidence loss L conf : Use binary cross entropy loss to measure the confidence that the predicted box contains the target;

[0175] Classification loss L cls : Use multi-class cross entropy loss to measure the difference between the predicted category and the true category.

[0176] Some pictures of the validation set model annotation results are shown as follows Figure 9 shown.

[0177] Furthermore, the classification evaluation probability model of ear typing is constructed including:

[0178] Based on the relationship between appearance features and ear typing and the scoring basis coefficient matrix M, a classification evaluation probability model for ear typing based on ear images is defined. The steps are as follows:

[0179] Image acquisition: Capture high-definition images of the ear canal or eardrum using a camera with a cold or warm light source;

[0180] With the target recognition model EFA1.5 as the core, an inference framework for automatic target recognition and detection is constructed. This inference framework can perform inference on any input ear sample image, thereby automatically detecting and identifying the target and outputting a set of pixel ratio feature vectors S consisting of 9 components. The header of the pixel ratio feature vector S is c("GHB","DN","CX","CK","HPS","PH","SH","MJ","FMW"), where c() represents a vector whose individual components correspond to the pixel ratios of the feature.

[0181] Classification evaluation probability model of ear typing: Based on the score basis coefficient matrix M (also called disease correlation coefficient matrix M) and the pixel ratio feature vector S, it can be calculated. The calculation formula is as follows: R = M * S;

[0182] Here, R is a probability evaluation vector consisting of 9 components. Its header is the same as that of S, and the value of each component represents the probability weight of the test subject belonging to that type. By performing probability evaluation calculations on approximately 100,000 images, based on the data evaluation probability values ​​of the ear NE type, and using the quartile method to calculate the values ​​of the other 10 types of groups, we can finally obtain the following ear type probability evaluation system for the following populations, as shown in Table 1:

[0183] Table 1

[0184]

[0185] The low probability described above means that based on the detected ear image, the probability that this person belongs to this ear type is very low, but it does not mean that it does not exist. In daily life, you should pay a little attention to the condition of your ears. The medium probability means that the probability of having this ear type is relatively high. You need to evaluate your own ear condition and keep an eye on changes in the ear state. If you feel uncomfortable, you can consult a professional. The high probability means that it is very likely to belong to this ear type. It is recommended to go to a professional institution for evaluation. Therefore, based on this classification probability assessment model for ear typing, people can always understand the condition of their own ears, what type they are currently in, what characteristics this type has, and what to pay attention to based on these apparent characteristics. This is fundamentally different from the single-feature classification model and is more advantageous and universal.

[0186] Case 1:

[0187] The system analysis and evaluation results of the ear images collected from people with ear BM typing are shown in Table 2:

[0188] Table 2

[0189] Ear typing Probability value Evaluate MEC 0 no CSOM 0 no EABC 0 no OE 0 no IC 0.00000003 Low probability OM 0 no OME 0 no AOM 0 no TMC 0 no BM 0.00700203 High probability

[0190] Case 2:

[0191] Table 3 shows the system analysis and evaluation results of the ear images collected from people with ear OME classification.

[0192] Table 3

[0193]

[0194]

[0195] This example uses a large amount of ear data statistics to discover 11 ear types distributed among different populations. Each type has its own relevant appearance characteristics. In combination with the YOLO framework, a feature recognition model based on 12 ear features is developed. Based on this model, a probabilistic system for ear type judgment is further constructed. At the same time, based on the implementation of the system, a set of convenient, fast, and practical smart application products for auxiliary management of people's ear health knowledge are developed. This not only improves the self-management ability of individuals' ear health, but also reduces the pressure on resources and makes high-quality resources widely available, which is of great significance to the realization of the "Healthy China" strategy.

[0196] The evaluation method for identifying ear status types of a group of people based on local features provided in this embodiment has all the advantages of the evaluation device for identifying ear status types of a group of people based on local features provided in the first embodiment.

[0197] Example 3

[0198] This embodiment further discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first embodiment.

[0199] Example 4

[0200] This embodiment further discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first embodiment are implemented.

[0201] Example 5

[0202] This embodiment further discloses a computer program product, including a computer program, which implements the steps of the method described in the first embodiment when executed by a processor.

[0203] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An evaluation device for identifying ear status of a group of people based on local features, characterized in that: include: An image acquisition module, including a wired device and a wireless device, is used to acquire images of the patient's ear canal and tympanic membrane area using a camera equipped with a cold light source or a warm light source; An image preprocessing module, connected to the image acquisition module, for cleaning and statistically classifying the image to obtain a preprocessed image; A feature extraction module, connected to the image preprocessing module, for extracting features from the preprocessed image to obtain ear appearance features; A feature annotation module, connected to the feature extraction module, is used to perform feature pre-annotation and automatic annotation on the ear appearance features to obtain an annotated image; An image cleaning module, connected to the feature annotation module, for further cleaning the annotated image to obtain a cleaned image; a model building module, connected to the image cleaning module, for building a picture feature recognition model based on the YOLO framework, training the picture feature recognition model based on the YOLO framework using the cleaned image, and obtaining a classification evaluation probability model for ear typing; a classification evaluation module, connected to the model building module, for performing classification evaluation of ear typing based on the classification evaluation probability model of ear typing to obtain an evaluation result; The process of training the image feature recognition model based on the YOLO framework using the cleaned image to obtain a classification evaluation probability model for ear typing includes: Unsupervised learning training phase and supervised learning training phase; The unsupervised learning training phase includes: Adopting the contrastive learning method in the self-supervised learning method, the encoder network of the deep convolutional neural network ResNet50 model is used to extract image features; Perform data augmentation on each unlabeled image, feed the augmented image into an encoder to obtain a feature representation, and map the feature representation into a contrast space through a projection head; Among them, the loss function uses InfoNCE loss, and the formula is: where sim(z i ,z j ) represents cosine similarity ;τ is the temperature parameter, N is the batch size; T represents the transpose operation of the matrix z; 1 [k=i] is the indicator function, which takes 1 when k≠i and 0 otherwise; The supervised learning training phase includes: The weights of the encoder obtained by unsupervised pre-training are loaded into the backbone network of the YOLO model, and the model is supervised trained based on the pre-training using labeled data.

2. The device according to claim 1, characterized in that The image preprocessing module includes a first filtering unit, a second filtering unit, a third filtering unit, and a state statistics unit; The first filtering unit is used to repeatedly sample each ear of the patient from the ear canal to the eardrum area at different angles to obtain initial sampling samples, and then use Wiener filtering and constrained least squares filtering to filter out blurred images; The second filtering unit is used to filter similar images by using an image similarity comparison method to calculate the grayscale histograms of the two images and compare and evaluate the similarity between the images. When the similarity is greater than a preset threshold, the images are considered similar and only one of them is retained; The third filtering unit is configured to filter pictures that lack ear status descriptions based on the patient's medical history information; The state statistics unit is used to collect statistics on the normal or abnormal states in the patient medical records corresponding to the retained samples, and obtain different types of ear typing state images.

3. The device according to claim 2, characterized in that The different types of ear classification status images include: ear NE classification representing normal people, ear MEC classification representing people with middle ear cholelithiasis, ear CSOM classification representing people with chronic suppurative otitis media, ear EABC classification representing people with external auditory canal bleeding, ear OE classification representing people with external otitis, ear IC classification representing people with external auditory canal cerumen embolism, ear OM classification representing people with external auditory canal fungal infection, ear OME classification representing people with secretory otitis media, ear AOM classification representing people with acute otitis media, ear TMC classification representing people with tympanic membrane calcification and ear BM classification representing people with bullous myringitis.

4. The device according to claim 1, characterized in that The ear appearance characteristics include physiological structure characteristics and clinical manifestation characteristics; The physiological structural features include tympanic membrane, pars flaccida, and hammertos; The clinical features include calcifications, amber color, cerumen, bleeding, discharge, redness, perforation, and fungus; The redness characteristics are further divided into deep red and reddish according to the redness area and appearance degree; The deep red refers to redness of the handle of the malleus or swelling of the tympanic membrane, which is not bloodshot, and all visible areas of the ear canal are red. The reddish state means that the tympanic membrane is not red or swollen, the handle of the malleus is not red, but there are blood streaks, and there are obvious blood streaks or redness in the flaccid part of the ear canal.

5. A method for evaluating ear status of a population based on local feature recognition, characterized in that: include: collecting images of the patient's ear canal and tympanic membrane area, and performing cleaning and statistical classification on the images to obtain preprocessed images; performing feature extraction on the preprocessed image to obtain ear appearance features; Performing feature pre-labeling and automatic labeling on the ear surface features to obtain a labeled image; Further cleaning the annotated image to obtain a cleaned image; Constructing an image feature recognition model based on the YOLO framework, training the image feature recognition model based on the YOLO framework using the cleaned image to obtain a classification evaluation probability model for ear typing, and performing ear typing classification evaluation based on the ear typing classification evaluation probability model to obtain an evaluation result; The process of training the image feature recognition model based on the YOLO framework using the cleaned image to obtain a classification evaluation probability model for ear typing includes: Unsupervised learning training phase and supervised learning training phase; The unsupervised learning training phase includes: Adopting the contrastive learning method in the self-supervised learning method, the encoder network of the deep convolutional neural network ResNet50 model is used to extract image features; Perform data augmentation on each unlabeled image, feed the augmented image into an encoder to obtain a feature representation, and map the feature representation into a contrast space through a projection head; Among them, the loss function uses InfoNCE loss, and the formula is: where sim(z i ,z j ) represents cosine similarity ;τ is the temperature parameter, N is the batch size; T represents the transpose operation of the matrix z; 1 [k=i] is the indicator function, which takes 1 when k≠i and 0 otherwise; The supervised learning training phase includes: The weights of the encoder obtained by unsupervised pre-training are loaded into the backbone network of the YOLO model, and the model is supervised trained based on the pre-training using labeled data.

6. The method according to claim 5, characterized in that The process of cleaning and statistically classifying the image includes: First, the area from the ear canal to the eardrum of each patient is repeatedly sampled from different angles to obtain initial sampling samples. Then, Wiener filtering and constrained least squares filtering are used to filter out blurred images. Secondly, for filtering similar images, the grayscale histogram of the two images is calculated using the image similarity comparison method and the similarity between the images is compared and evaluated. When the similarity is greater than a preset threshold, the images are considered similar and only one of them is retained. Finally, based on the patient's medical records, images with missing ear status descriptions were filtered out, and the normal or abnormal status in the patient's medical records corresponding to the retained samples were counted to obtain different types of ear classification status images.

7. The method according to claim 5, characterized in that The basis for further cleaning the annotated image includes: There are feature representations in the image; The feature representation pixel ratio is greater than 0.0001; When the pixel ratio of a feature in the image cannot meet the above two conditions, the feature area information is recorded through the cleaning program, and then the information of the feature area is deleted, and the action of modifying the JSON file is performed at the same time.

8. The method according to claim 5, characterized in that The data enhancement includes random selection of Gaussian blur, brightness, saturation, random color jitter, random grayscale, and random horizontal flip.

9. The method according to claim 5, characterized in that The YOLO model includes a YOLOv8 model; The YOLOv8 model includes a backbone network, a neck network, and a detection head; The neck network is used to fuse features from different levels of the backbone network; The detection head is responsible for predicting bounding boxes, confidence scores, and category probabilities; The labeled data includes images, bounding boxes, and category labels; The total loss function of the YOLO model includes positioning loss, confidence loss and classification loss; L total =L loc +L conf +L cls Among them, the positioning loss L loc : Use GIoU, DIoU or CIoU loss to measure the overlap between the predicted box and the true box; Confidence loss L conf : Use binary cross entropy loss to measure the confidence that the predicted box contains the target; Classification loss L cls : Use multi-class cross entropy loss to measure the difference between the predicted category and the true category.

Images