Eardrum classification apparatus, eardrum classification system, and eardrum classification program

Abstract

An eardrum classification apparatus 30 includes: a communication unit that receives an input of image data including an eardrum; a first classification unit 31b that extracts features from the input image data by using an algorithm and classifies the eardrum as having acute otitis media or something other than acute otitis media; and a second classification unit 31c that extracts features, by using an algorithm, from the image data classified as having something other than acute otitis media, and classifies the eardrum as being normal or having exudative otitis media. On the basis of the classification results given by the first classification unit 31b and the second classification unit 31c, the communication unit provides an output indicating acute otitis media, normal eardrum, or exudative otitis media.

Description

Tympanic membrane classification device, tympanic membrane classification system, and tympanic membrane classification program

[0001] The present invention relates to a device for classifying the presence or absence of a pathology of a tympanic membrane from an image of the tympanic membrane.

[0002] A method for classifying the condition of a eardrum based on an image of the eardrum is known, for example, as disclosed in Japanese Patent Application Laid-Open No. 2019-534723 (Patent Document 1). This conventional method determines whether the eardrum is normal or abnormal based on an image of the eardrum.

[0003] Special table 2019-534723 publication

[0004] However, the inventors have found that there are areas that need further improvement in the above-mentioned conventional methods: namely, it is difficult to make an accurate diagnosis, such as incorrectly determining that a normal eardrum is abnormal, or that an abnormal eardrum is normal, or incorrectly determining whether otitis media is acute or exudative, and there is room for improvement in the accuracy rate.

[0005] There is also a high demand for eardrum diagnosis late at night, on Sundays, and on holidays when doctors' offices and clinics are closed. Parents, who are particularly busy balancing work and child-rearing, often fail to notice abnormalities deep inside the ear canal of their infants and children, worsening the inflammation around the eardrum and leading to them requesting a consultation late at night, on Sundays, or on holidays.

[0006] Furthermore, medical services are inadequate in remote areas, and it is difficult to receive medical treatment at hospitals or clinics, as there are often no pediatricians or otolaryngologists.

[0007] On the other hand, early detection of otitis media is essential for appropriate treatment, and delayed diagnosis leads to delayed appropriate treatment.

[0008] In view of the above-mentioned circumstances, the present invention aims to provide a technology that can easily and correctly determine the condition of an area including the eardrum at an appropriate time, even in cases where medical services such as hospitals and clinics are limited.

[0009] For this purpose, the eardrum classification device of the present invention comprises a communication unit to which image data including the eardrum is input, a first classification unit that extracts characteristics from the image data using an algorithm and classifies the eardrum as either acute otitis media or other than acute otitis media, and a second classification unit that extracts characteristics from image data classified as other than acute otitis media using an algorithm and classifies the eardrum as either normal otitis media or serous otitis media, and outputs acute otitis media, normal otitis media, or serous otitis media from the communication unit based on the classification results of the first and second classification units.

[0010] According to the present invention, by performing two-stage, two-classification, classification results with a higher accuracy rate than conventional methods can be obtained. Therefore, even if a patient is unable to visit a medical institution, they can receive support equivalent to a medical examination and be informed of their symptoms at the appropriate time. The tympanic membrane classification device of the present invention is not particularly limited in its hardware or software configuration. The tympanic membrane classification device may be, for example, a server on a network or a cloud.

[0011] Iridescence and foreign matter reflected in the image have a negative impact on classification. When photographing an eardrum, it is difficult for an average person unfamiliar with photography to adjust the amount of light needed for photography or photograph the eardrum without capturing iris or foreign matter. Therefore, one aspect of the present invention includes a trimming unit that trims the eardrum from image data input to a communication unit and provides the trimmed image to a first classification unit. This aspect makes it possible to remove iris and foreign matter from images to be classified, thereby improving the accuracy rate of classification. Furthermore, machine learning can be performed effectively. In another aspect of the present invention, the eardrum may be trimmed before inputting the image data to the communication unit, such as when photographing the eardrum or transmitting image data to the communication unit, and the trimmed image data may be input to the communication unit of the eardrum classification device.

[0012] In a preferred aspect of the present invention, the system further includes a trimming unit that trims a predetermined portion of the eardrum from image data input to the communication unit and provides the trimmed portion to the first classification unit. According to this aspect, at least one of the first classification unit and the second classification unit focuses on the predetermined portion of the eardrum rather than the entire eardrum, thereby improving the accuracy rate of classification and enabling effective machine learning. For example, trimming the lower half of the eardrum allows for accurate classification of cases with significant lesions in the lower half, such as serous otitis media, and also enables effective machine learning. Alternatively, for example, trimming the center of the eardrum.

[0013] The eardrum classification device of the present invention includes artificial intelligence (AI). The AI ​​model of the AI ​​algorithm is not particularly limited. In a preferred aspect of the present invention, at least one of the first classification unit and the second classification unit incorporates labeled training data and performs supervised machine learning. According to this aspect, the accuracy rate of the eardrum classification device increases as the eardrum classification device undergoes machine learning.

[0014] The tympanic membrane classification system of the present invention includes the above-described tympanic membrane classification device, an earscope that photographs the tympanic membrane, and a terminal that receives image data of the tympanic membrane from the earscope and outputs it to the communication unit of the tympanic membrane classification device. According to this aspect, even in an ordinary household where medical technology is not yet advanced, tympanic membrane symptoms can be identified by operating the earscope and terminal in the home. The terminal can, for example, simply display the classification results or optionally display information related to the classification results. The earscope is not limited to medical equipment as long as it can photograph the interior of the ear canal. It is sufficient that the imaging unit and the image can handle a reasonable resolution (DPI). Preferably, the earscope has an imaging unit capable of capturing images with 4 million pixels or more. More preferably, the earscope has an imaging unit capable of capturing images with 8 million pixels or more. The device and hardware configuration of the terminal are not particularly limited.

[0015] In one aspect of the present invention, the terminal has a screen that displays an image captured by the earscope and a guidance means that guides the user to display the eardrum in the center of the screen, and outputs image data with the eardrum in the center to the eardrum classification device. According to this aspect, since the terminal has the guidance means, even a user who is not skilled in medical technology can easily capture the eardrum image D'. The guidance means is not particularly limited, but could be, for example, a circular frame displayed in the center of the terminal screen.

[0016] In one aspect of the present invention, the earscope is a digital camera attached to an ear canal cleaning device, and according to this aspect, the ear canal cleaning device equipped with a digital camera can be used to identify eardrum conditions.

[0017] In a preferred aspect of the present invention, the tympanic membrane classification system includes a medical institution terminal capable of communicating with the tympanic membrane classification device and the terminal. According to this aspect, the medical institution can provide support that is substantially equivalent to a medical examination.

[0018] Iridescence and foreign matter reflected in the image have a negative effect on classification. When photographing an eardrum, it is difficult for an average person unfamiliar with photography to adjust the amount of light needed for photography or photograph the eardrum without capturing iris or foreign matter. Therefore, in a preferred aspect of the present invention, the earscope or terminal has a trimming unit that trims the eardrum from an image including the eardrum to create image data. According to this aspect, it is possible to remove iris and foreign matter reflected from the image to be classified, thereby improving the accuracy rate of classification. Furthermore, machine learning can be performed effectively. From the perspective of the eardrum classification device, trimming is performed on the earscope side. Alternatively, in another aspect, trimming may be performed on the eardrum classification device side.

[0019] In a preferred aspect of the present invention, the guidance unit displays on the screen whether the image displayed on the screen is suitable for classification. According to this aspect, even a user who is unfamiliar with photography can photograph an eardrum image.

[0020] In a preferred aspect of the present invention, the terminal displays on a screen the probabilities associated with the classification results of the first and second classification units. This aspect can alert the user. In another aspect, a threshold value may be set for the probability, and if the probability is equal to or exceeds the threshold value, a notification urging the user to visit a doctor's office or clinic may be displayed on the screen.

[0021] The operations (processes) in the classification method of the present invention can also be realized in the form of a computer-executable program stored in various machine-readable storage media. The tympanic membrane classification program of the present invention is a program that causes a computer to classify image data including the tympanic membrane, and causes the computer to execute the following steps: a first classification step of extracting characteristics from the image data using an algorithm and classifying the tympanic membrane as acute otitis media or other than acute otitis media; a second classification step of extracting characteristics from the image data classified as other than acute otitis media using an algorithm and classifying the tympanic membrane as normal tympanic membrane or serous otitis media; and a step of outputting acute otitis media, normal tympanic membrane, or serous otitis media based on the classification results of the first and second classification steps.

[0022] The present invention includes program code for causing a computer to perform classification, a computer-readable program product, and a storage medium for storing the program code. The storage medium may include, but is not limited to, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. The storage medium on which the program code is stored is provided directly or indirectly to a system or device, and the computer, ROM, RAM, or central processing unit (CPU) of the system or device reads and executes the program code. The embodiment of the present invention is not limited to a program, and can be realized by other software or firmware. In addition to an apparatus, the present invention may also be a method.

[0023] As described above, according to the present invention, the condition of the eardrum can be known with a high accuracy rate without going to a medical institution, so the condition of the area including the eardrum can be grasped at the appropriate time regardless of time or place, preventing late diagnosis. It also eliminates regional disparities in medical services. Furthermore, when used in medical settings, it improves diagnostic efficiency and contributes to labor savings in the field.

[0024] FIG. 1 is a schematic overall diagram showing a tympanic membrane classification system according to one embodiment of the present invention; FIG. 2 is a diagram illustrating classification results performed by the tympanic membrane classification device of the same embodiment; FIG. 3 is a photograph showing an image including a tympanic membrane and a cropped image; FIG. 4 is a schematic diagram showing a classification algorithm performed by the tympanic membrane classification device of the same embodiment; FIG. 5 is a schematic diagram showing classification performed by the tympanic membrane classification device of Comparative Example 1; FIG. 6 is a screen displayed on a terminal of the same embodiment, the terminal screen displaying a guidance means; FIG. 7 is a schematic diagram showing the classification algorithm performed by the tympanic membrane classification device of the same embodiment together with a probability [%]; FIG. 8 is a screen displayed on a terminal of the same embodiment, the classification results displayed on the terminal screen being indexed.

[0025] An embodiment of the present invention will now be described in detail with reference to the drawings. Fig. 1 is a schematic overall view of an eardrum classification system according to one embodiment of the present invention. The eardrum classification system includes an earscope 10, a terminal 20, and an eardrum classification device 30. The earscope 10 includes a rod-shaped main body 11, an ear canal cleaning structure 12 provided at the tip of the main body, and a light-emitting unit 14 and a photographing unit 15 attached to the ear canal cleaning structure 12. The photographing unit is, for example, a digital camera.

[0026] The terminal 20 has a screen 21 and is capable of communicating with the earscope 10. An image B captured by the image capturing unit 15 is projected onto the screen 21. The image B has a circular frame C, for example.

[0027] A user of the earscope 10 holds the main body 11 in his / her hand and inserts the ear canal cleaning structure 12 into the ear canal H of a subject, for example, a child, to remove earwax from the inner periphery of the ear canal H. Alternatively, the user can activate the light emitting unit 14, take an image of the inside of the ear canal H with the imaging unit 15, and project the image of the inside of the ear canal H onto the screen 21.

[0028] The user can move the earscope 10 while checking the screen 21 so that the image of the eardrum D (hereinafter referred to as eardrum image D') is centered within frame C. Frame C serves as a guide to guide the user to capture eardrum image D'. Image B displayed by the imaging unit 15 is projected moment by moment onto the screen 21. The user can move the earscope 10 while checking the eardrum image D' included in image B, and when eardrum image D' is projected large in the center of frame C, the user can select and capture an eardrum image D' that minimizes light reflection, glare, and color washout (iridescence F), while avoiding image out-of-focus or camera shake. The user is not particularly limited, and does not need to be an expert in photography or a medical professional; they can be an ordinary person.

[0029] The terminal 20 can communicate with an external tympanic membrane classification device 30 (specifically, a communication unit 33) via wireless or wired communication, for example, via a network NW such as the Internet. The terminal 20 can be replaced by a widely used mobile phone terminal, such as a smartphone or tablet terminal, or a terminal equipped with other communication functions such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). By submitting the captured image B (strictly speaking, image data of image B) to the tympanic membrane classification device 30, the user can have the tympanic membrane classification device 30 classify the tympanic membrane D as a normal tympanic membrane or as having acute or serous otitis media. Such classification can also be referred to as a judgment or diagnosis.

[0030] The eardrum classification device 30 includes a classification unit 31 with artificial intelligence (AI) that drives a unique algorithm, a memory unit 32 that stores a large number of image data (synonymous with images), and a communication unit 33 that communicates with external devices. The eardrum classification device 30 of this embodiment is a server connected to the Internet. The memory unit 32 stores a large number of image data of normal eardrums labeled as normal eardrums. It also stores a large number of image data of acute otitis media labeled as acute otitis media. It also stores a large number of image data of serous otitis media labeled as serous otitis media. These data are used as labeled training data for machine learning by the classification unit 31. The classification unit 31 performs machine learning using multiple training data as training materials to improve the accuracy rate of classification. It should be understood that the accuracy rate is synonymous with the accumulated scores of the classification results of the classification unit 31 scored by an external organization separate from the eardrum classification device 30, such as a human doctor.

[0031] When an image (synonymous with image data, hereinafter the same) including an eardrum image D' is input to the classification unit 31, the classification unit 31 extracts characteristics for classification from the eardrum image D', processes the characteristics using an algorithm, classifies the eardrum image D' as a normal eardrum, acute otitis media, or serous otitis media, and outputs the classification result. The characteristics refer to the color distribution of the eardrum image D', the size of the shape of the distribution, etc., and are defined within the classification unit 31 in accordance with an algorithm.

[0032] The communication unit 33 receives image data from an external device via wireless or wired communication such as a network means NW, and outputs the image data to the classification unit 31. The communication unit 33 can also communicate with a computer terminal at a medical institution 40 via the same communication means. Alternatively, the eardrum classification device 30 may be a computer installed in a medical institution. In either case, labeled training data is gradually accumulated in the memory unit 32. This increases the amount of machine learning learning in the classification unit 31 and improves the accuracy rate of classification. Furthermore, new eardrum images D' input from the terminal 20 to the eardrum classification device 30 are later labeled by a doctor at the medical institution 40. This gradually accumulates training data in the memory unit 32.

[0033] The classification unit 31 includes a trimming unit, a first classification unit, and a second classification unit. In this embodiment, as shown in FIG. 3 , the trimming unit in the classification unit 31 uses an algorithm to trim an eardrum image D' from image B (a). This prevents iris F, which is reflected in image B and is separated from the eardrum image D', the color of the inner surface of the ear canal H, and foreign matter G, such as body hair or earwax, from affecting the classification. (b) shows image data related to the eardrum image D' after trimming.

[0034] 4 is a schematic diagram showing algorithms sequentially executed by the first and second classification units in the classification unit 31. Such algorithms are executed by ResNet (Residual Network), VGG16 (Visual Geometry Group 16-layer), DenseNet (Densely Connected Convolutional Network), or other AI models.

[0035] First, the first classification unit 31b in the classification unit 31 classifies the eardrum image D' into two categories. Specifically, the eardrum image D' is classified as either acute otitis media or other than acute otitis media. These two categories are determined based on the ratio of reddish pixels to the total pixels of the eardrum image D', the size of the brownish pixel (also called cloudy) area, and other characteristics.

[0036] Second, the second classification unit 31c in the classification unit 31 classifies the eardrum image D' other than acute otitis media into two categories. Specifically, the eardrum image D' other than acute otitis media is classified as either a normal eardrum or serous otitis media. This classification is determined based on the ratio of brown pixels to the total pixels and other characteristics. By this two-stage classification, the eardrum image D' is classified into three categories with a higher accuracy rate than conventional methods.

[0037] 2 is a diagram illustrating an example of the classification results performed by the tympanic membrane classification device of this embodiment. The classification results output by the tympanic membrane classification device 30 are transmitted to the terminal 20 via the network means NW and displayed on the screen 21. The probability [%] calculated by the tympanic membrane classification device 30 for each classification result is also displayed. Note that the probability [%] here is a numerical value calculated by the classification unit 31.

[0038] The terminal 20 can communicate with the medical institution 40 via wireless or wired communication such as a network means NW. The user of the terminal 20 can have a question-and-answer session with the staff of the medical institution 40 about the classification results of the eardrum image D', the validity of the classification, future diagnoses and measures, etc.

[0039] If the classification result is a normal eardrum, the probability X [%] that the eardrum is normal is displayed together with the classification result. If the classification result is acute otitis media, the probability Y [%] that the eardrum is acute otitis media is displayed together with the classification result. If the classification result is serous otitis media, the probability Z [%] that the eardrum is serous otitis media is displayed together with the classification result. The probabilities X, Y, and Z [%] are based on the comparison of the classification result previously performed by the classification unit 31 with the labeling performed by the doctor on that classification result, in other words, the accuracy rate.

[0040] It should be noted that in FIG. 2 , a warning may be issued by changing the color of the border of image B or the background surrounding image B, by changing the thickness or color of the text, by displaying a scale or other indicator to indicate a numerical value or degree, or by displaying a symbol or marker. For example, for image B classified as normal in FIG. 2( a), image B is outlined in white. For image B classified as acute otitis media in FIG. 2( b), instead of displaying the probability Y [%], image B is outlined in a light color such as pink if acute otitis media is suspected (probability Y [%] < 50 [%]). Furthermore, if acute otitis media is almost certain (probability Y [%] ≥ 50 [%]), image B is outlined in a dark color such as red. The same applies to serous otitis media in FIG. 2( c). This ensures that the viewer of the screen 21 is informed of the impending onset of symptoms of the eardrum D. Probabilities X, Y, and Z [%] may also be displayed together with the indicator or marker.

[0041] The accuracy rates of the eardrum classification device 30 of this embodiment and a comparative eardrum classification device were investigated. The accuracy rates of these eardrum classification devices are shown in Table 1.

[0042]

[0043] First, multiple sets of training data (for multiple people and the same number of ears) were subjected to machine learning. Next, the eardrum classification device 30 was made to classify multiple sets of eardrum images D' (this time). Furthermore, multiple sets of training data (for multiple people and the same number of ears) were subjected to machine learning. Next, the eardrum classification device 30 was made to classify multiple sets of eardrum images D' (second time). If the second classification is not performed and only the first classification (this time) is performed, the accuracy rate for the second time will not be displayed.

[0044] The accuracy rates of the classification results of this embodiment are shown in No. 3 and No. 4 in Table 1. It was found that the two-stage, two-classification method of this embodiment achieved higher accuracy rates than conventional methods, with accuracy rates of 99.98% for Stage 1 and 99.50% for Stage 2. The reason for this is thought to be that the eardrum D of an acute otitis media patient is redder than the eardrum D of a non-acute otitis media patient, and the eardrum D of an otitis media with effusion has a more pronounced cloudy area than the eardrum D of a non-acute otitis media patient.

[0045] The correct answer rate for the comparative ratio was also investigated.,The classification process for the comparative ratio is shown in,Figure 5.

[0046] The eardrum classification device of the comparative example is equipped with a one-stage multi-classification (three-classification) algorithm that classifies input eardrum images into normal eardrum, acute otitis media, or serous otitis media in one stage. According to the one-stage three-classification algorithm of the comparative example, the accuracy rate was 68.26% in the first test (this time), and 91.12% in the second test, as shown in No. 1 of Table 1. Thus, the accuracy rate of the eardrum classification device of the comparative example (No. 1) was lower than the accuracy rates of the eardrum classification device of this embodiment (No. 3 and No. 4). This demonstrates that the method of performing two-stage classification in multiple stages, as in this embodiment, improves the accuracy rate compared to conventional methods.

[0047] Next, a modified example of this embodiment will be described. The eardrum classification device 30 of this modified example trims approximately the lower half of the eardrum D as shown in FIG. 3 . The trimmed semicircular image D″ is then classified as described above. Serous otitis media exhibits a noticeable cloudy area in the lower half of the eardrum D. Therefore, trimming approximately the lower half of the eardrum D (e.g., the lower half of the central region of the eardrum D) improves the accuracy rate of the classification results. Furthermore, labeling the image of the lower half of the eardrum D and using it as training data for machine learning in the classification unit 31 can improve the accuracy rate of the classification unit 31 itself.

[0048] The correct answer rate for the modified example of this embodiment was 96.07% in the first test (this time), as shown in No. 5, which is better than the case of eardrum image D' (No. 4). The correct answer rate was also improved to 100% in the second test.

[0049] The correct answer rate for the comparative example modification, as shown in No. 2, was 62.41% in the first test (this time), which was actually inferior to that of eardrum image D' (No. 1). Furthermore, the correct answer rate for the second test was 65.74%, which was an improvement, but did not exceed the correct answer rate for this embodiment (No. 5).

[0050] 1, the eardrum classification device 30 of this embodiment includes a communication unit 33 to which an eardrum image D' including an eardrum D is externally input, a first classification unit 31b (FIG. 3) that extracts characteristics from the eardrum image D' using an algorithm and classifies the eardrum D as acute otitis media or other than acute otitis media, and a second classification unit 31c that extracts characteristics from the eardrum image D' classified as other than acute otitis media by the first classification unit 31b using an algorithm and classifies the eardrum D as normal or serous otitis media, and outputs acute otitis media, normal otitis media, or serous otitis media to the outside from the communication unit 33 based on the classification results of the first classification unit 31b and the second classification unit 31c. According to the eardrum classification device 30, the accuracy rate of classification is improved compared to conventional methods by using a two-stage, two-classification algorithm shown in FIG.

[0051] The effects of the eardrum classification device 30 will be explained for each of the following phases.

[0052] In the first phase, the eardrum classification device 30 classifies the eardrum D using the unique AI algorithm described above. This allows the symptoms of the eardrum D to be diagnosed without the patient having to go directly to a medical institution 40 to be examined by an otolaryngologist. For example, in an ordinary household where medical technology is not well-versed, a family member can take a photo of their child's eardrum D and use it to determine whether the eardrum D has any diseases.

[0053] In the second phase, the medical institution 40 is equipped with an earscope 10 and a terminal 20 as tools used in examinations, and is connected for mutual communication with the eardrum classification device 30. As a result, the eardrum classification device 30 is used by an otolaryngologist at the medical institution 40 as a diagnostic support tool for diagnosing eardrum image D' captured by the earscope 10. The otolaryngologist also uses image B. Alternatively, for example, in an ordinary household where medical technology is not highly skilled, a family member can photograph their child's eardrum D and receive support substantially equivalent to a medical examination, such as advice on when to seek medical advice, from an otolaryngologist at the medical institution 40, allowing the family member to learn of symptoms at an appropriate time.

[0054] In the third phase, the eardrum classification device 30 from the first and second phases described above will be used as a diagnostic support tool or as a medical device. By incorporating AI machine learning, the eardrum classification device 30 will be able to perform classification at the same level as an otolaryngologist. For example, a drug prescription program can be incorporated into an AI algorithm, enabling prescriptions to be made based on the diagnostic support tool of this embodiment. For example, the eardrum classification device 30 will be connected to a pharmacy (not shown) for mutual communication.

[0055] 3(a) and 3(b), the eardrum classification device 30 of this embodiment includes a trimming unit that trims an image portion of the eardrum D from an image B input from the outside to the communication unit 33 and provides the trimmed image to the first classification unit 31b. When the light-emitting unit 14 (FIG. 1) illuminates the ear canal H, iris F and foreign matter G are likely to be reflected on the exit side of the ear canal H. According to this embodiment, the image on the exit side of the ear canal H is removed, and therefore the iris F and foreign matter G are also removed from the image. This allows for effective machine learning.

[0056] 3(a) and 3(c), the eardrum classification device 30 according to a modified example of this embodiment includes a cropping unit that crops the lower half of the image of the eardrum D from the image B input to the communication unit 33 and provides the cropped image to the first classification unit 31b. This allows for correct classification of cases in which the lesion in the lower half is prominent, such as serous otitis media, and also enables effective machine learning.

[0057] Furthermore, the eardrum classification device 30 of this embodiment stores the classification target image B in the memory unit 32, and the classification is later correctly labeled by the medical institution 40. At least one of the first classification unit 31b and the second classification unit 31c then imports the labeled training data and performs supervised machine learning. As a result, the eardrum classification device 30 accumulates machine learning and further improves its accuracy rate.

[0058] The tympanic membrane classification system of this embodiment also includes a tympanic membrane classification device 30, an earscope 10 that photographs the tympanic membrane D, and a terminal 20 that receives an image B of the tympanic membrane D from the earscope 10 and outputs it to a communication unit 33 of the tympanic membrane classification device 30. This allows ordinary households that are not skilled in medical technology to operate the earscope 10 and terminal 20 to learn about the symptoms of the tympanic membrane D.

[0059] The terminal 20 of this embodiment also includes a screen 21 that displays an image captured by the earscope 10, and displays a frame C in the center of the screen 21 to guide the user to view the eardrum D. An image B of the eardrum D captured within the central frame C is input to the communication unit 33. This allows even a user who is not skilled in medical technology to properly capture an eardrum image D'.

[0060] The photographing unit 15 in this embodiment is a digital camera attached to the ear canal cleaning structure 12. This allows the symptoms of the eardrum to be known using a digital camera mounted on an ear canal cleaning structure that is a commonly used everyday item.

[0061] Furthermore, since the communication unit 33 and the terminal 20 of this embodiment can communicate with the medical institution 40, even if a patient is unable to visit a medical institution, the patient can receive support that is almost equivalent to a medical examination, and can learn about the patient's symptoms at the appropriate time.

[0062] Next, a supplementary explanation will be given regarding the screen 21 provided on the user's terminal 20. The terminal 20 communicates with the earscope 10 and projects the image B captured by the earscope 10 onto the screen 21 in real time.

[0063] FIG. 6 is a diagram illustrating a terminal screen according to the embodiment, showing a screen 21 as a guidance device. An image B of the ear canal H, captured by a user inserting the earscope 10 into the subject's ear, is output in real time from the terminal 20 to the tympanic membrane classification device 30 via the network NW ( FIG. 1 ). The classification unit 31 of the tympanic membrane classification device 30 includes a program (photography guidance program) as a guidance device that determines in real time whether image B includes the entire tympanic membrane D and is usable for tympanic membrane classification. Specifically, it determines whether image B includes the tympanic membrane image D' shown in FIG. 3 . When the earscope 10 begins to be inserted into the subject's ear, image B does not include tympanic membrane image D', and therefore image B is deemed unusable for tympanic membrane classification. A blue ring 22 is displayed on the screen 21. The ring 22 has a diameter half the size of the frame C of image B. Furthermore, the area J outside the frame C on the screen 21 is a solid color, such as light gray.

[0064] When the photography guidance program inputs image B and determines that the area surrounded by the blue circular ring 22 represents the eardrum D, the photography guidance program determines that image B can be used for eardrum classification and changes the color of the ring 22 from blue to green. Then, a judgment operation button 23 is displayed in the outer region J. As a result, an appropriately photographed eardrum image is outlined by the ring 22.

[0065] The photography guidance program of this embodiment can assist a user who is unfamiliar with photographing the eardrum D in photographing the eardrum D. Note that the size, shape, and color of the ring 22 described above are merely examples, and any visual change can be used to indicate whether or not the ring can be used for eardrum classification.

[0066] When the user touches the judgment operation button 23 with a finger, the terminal 20 outputs a classification command to (the communication unit 33 of) the eardrum classification device 30. The eardrum classification device 30 classifies the image data input from the terminal 20 into two categories and displays the classification results on the screen 21.

[0067] FIG. 7 is a diagram illustrating the probability [%] of each classification result in the two-stage two-classification in the classification unit 31.

[0068] FIG. 8 is a diagram showing an example of the classification results displayed on the screen 21. As shown in FIG.

[0069] More specifically, the first classification unit 31b shown in Fig. 4 calculates the probability (numerical value) [%] of each of the two classifications using an algorithm, as shown in Fig. 7. The second classification unit 31c performs the same process.

[0070] For example, if the classification result of the first classification unit 31b is 10% for acute otitis media and 90% for no acute otitis media, the second classification unit 31c outputs a result indicating 70% for normal and 30% for serous otitis media. As a result, as shown in Table 2, the 90% for no acute otitis media classification result of the first classification unit 31b is proportionally divided by the classification result of the first classification unit 31b to calculate the following probabilities for the final classification result: normal probability X = 63%; acute otitis media probability Y = 10%; and serous otitis media probability Z = 27%. The screen 21, which serves as the classification result display unit, displays these probabilities using indicators so that the probabilities X, Y, and Z for each of the three classifications total 100%.

[0071]

[0072] Furthermore, as shown in FIG. 8, the screen 21 as a classification result display section displays the probability [%] of each classification in a circular frame C surrounding the image B using indicators of three different colors.

[0073] 8 , in the eardrum image D′ of a normal eardrum, the blue indicator Cx (63%]) representing normal is displayed most frequently in the circle of the frame C that displays the classification result of the eardrum image D′, the orange indicator Cz (27%]) representing serous otitis media is displayed less frequently, and the red indicator Cy (10%]) representing acute otitis media is displayed in small amounts. The indicators Cx, Cy, and Cz are rearranged appropriately in descending order of probability.

[0074] According to this embodiment, the classification result is displayed as an indicator in frame C on the screen 21, and image B is framed with indicators Cx, Cy, and Cz, thereby attracting the user's attention.

[0075] The three different colors described above are merely examples, and each color can be changed or selected as appropriate. Optionally, the date and time of the image used for classification and the identification of the left or right ear may be displayed.

[0076] Although the embodiments of the present invention have been described above with reference to the drawings, the present invention is not limited to the illustrated embodiments. Various modifications and variations can be made to the illustrated embodiments within the same scope as the present invention or within an equivalent scope.

[0077] As a modified example (not shown), instead of the configuration in which the eardrum classification device 30 and the terminal 20 are communicatively connected via the network means NW shown in Fig. 1, the eardrum classification device 30 shown in Fig. 1 may be a program installed on the terminal 20. The program is, for example, application software stored in the terminal 20, and executes the above-mentioned algorithm processing within the terminal 20.

[0078] As a modification example not shown, various options can be added to the screen 21 of the terminal 20 as a classification result display unit, and these options can be substituted. The various displays such as the color coding and shape described above may be audio or light displays.

[0079] The present invention is advantageously used in the medical industry.

[0080] DESCRIPTION OF SYMBOLS 10 Earscope 11 Main body 12 Ear canal cleaning structure 14 Light emitting unit 15 Photography unit 20 Terminal 21 Screen 22 Ring (guiding means) 30 Tympanic membrane classification device 31 (First and second) classification unit 32 Memory unit 33 Communication unit 40 Medical institution B Image C Frame (guiding means) Cx, Cy, Cz Classification result indicator D Tympanic membrane D' Tympanic membrane image H Ear canal

Claims

1. A tympanic membrane classification device comprising: a communication unit to which image data including a tympanic membrane is input; a first classification unit which extracts characteristics from the image data using an algorithm and classifies the tympanic membrane as acute otitis media or other than acute otitis media; and a second classification unit which extracts characteristics from the image data classified as other than acute otitis media using an algorithm and classifies the tympanic membrane as normal tympanic membrane or serous otitis media, and outputs acute otitis media, normal tympanic membrane, or serous otitis media from the communication unit based on the classification results of the first and second classification units.

2. The tympanic membrane classification device according to claim 1, further comprising a trimming unit that trims the tympanic membrane from the image data input to the communication unit and provides the trimmed image to the first classification unit.

3. The eardrum classification device according to claim 1, further comprising a trimming unit that trims a predetermined portion of the eardrum from the image data input to the communication unit and provides the trimmed portion to the first classification unit.

4. The eardrum classification device according to claim 1, wherein at least one of the first classification unit and the second classification unit takes in labeled training data and performs supervised machine learning.

5. A tympanic membrane classification system comprising: the tympanic membrane classification device according to claim 1; an earscope for photographing the tympanic membrane; and a terminal for receiving image data of the tympanic membrane from the earscope and outputting the image data to the tympanic membrane classification device.

6. The tympanic membrane classification system of claim 5, wherein the terminal has a screen for displaying an image captured by the earscope and a guidance means for guiding the user so that the tympanic membrane is displayed in the center of the screen, and outputs image data in which the tympanic membrane is captured in the center to the communication unit.

7. The eardrum classification system of claim 5, wherein the ear scope is a digital camera mounted on an ear canal cleaning structure.

8. The tympanic membrane classification system according to claim 5, further comprising a medical institution terminal capable of communicating with said tympanic membrane classification device and said terminal.

9. The tympanic membrane classification system according to claim 5, wherein the earscope or the terminal has a cropping unit that creates the image data by cropping the tympanic membrane from an image including the tympanic membrane.

10. The eardrum classification system according to claim 6, wherein the guidance means displays on the screen whether or not the image displayed on the screen is suitable for the classification.

11. The eardrum classification system according to claim 5, wherein the terminal displays probabilities regarding the classification results of the first and second classification units on the screen.

12. A tympanic membrane classification program that causes a computer to classify image data including a tympanic membrane, comprising: a first classification step of extracting characteristics from the image data using an algorithm and classifying the tympanic membrane as acute otitis media or other than acute otitis media; a second classification step of extracting characteristics from the image data classified as other than acute otitis media using an algorithm and classifying the tympanic membrane as normal tympanic membrane or serous otitis media; and a step of outputting acute otitis media, normal tympanic membrane, or serous otitis media based on the classification results of the first and second classification steps.

Images