Tympanic membrane classification device, tympanic membrane classification system, and tympanic membrane classification program

Abstract

A tympanic membrane classification device (30) is provided with: a communication unit into which image data including a tympanic membrane is input; a first classification unit (31b) that uses an algorithm to extract characteristics from the inputted image data and classifies the tympanic membrane as acute otitis media or as other than acute otitis media; and a second classification unit (31c) that uses an algorithm to extract characteristics from the image data classified as other than acute otitis media and classifies the tympanic membrane as normal or exudative otitis media, and the tympanic membrane classification device (30) outputs information on acute otitis media or normal tympanic membrane or exudative otitis media from the communication unit on the basis of the classification results of the first classification unit (31b) and the second classification unit (31c).

Description

TECHNICAL FIELD

[0001] The present application relates to a device that classifies the presence or absence of a condition of a tympanic membrane from an image of the tympanic membrane. BACKGROUND

[0002] As a method of classifying a condition of a tympanic membrane from an image of the tympanic membrane, for example, Japanese Patent Application Publication No. 2019-534723 (Patent Literature 1) is known. The conventional method is to determine whether the tympanic membrane is normal or abnormal from an image obtained by photographing the state of the tympanic membrane.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2019-534723 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] However, the present inventors have found that, in the conventional method described above, there is room for further improvement. That is, it is sometimes difficult to make a correct diagnosis because a normal tympanic membrane is erroneously determined to be abnormal, or an abnormal tympanic membrane is erroneously determined to be normal, or a determination of whether otitis media is acute or exudative is erroneous, and so on.

[0008] In addition, at night, on Sundays, and during holidays, when medical services are not provided at hospitals and clinics, there are many cases in which a tympanic membrane diagnosis is desired. In particular, parents who are very busy with work and child-rearing often do not notice abnormalities in the deep part of the ear of infants and children, and inflammation around the tympanic membrane worsens, and cases in which a diagnosis is desired occur at night, on Sundays, and during holidays.

[0009] In addition, in remote areas where medical services are not comprehensive, there are no pediatricians or otolaryngologists on duty, and it is difficult to receive a diagnosis at a hospital or a clinic.

[0010] On the other hand, early detection of otitis media is indispensable for proper treatment. Delay in diagnosis can delay proper treatment.

[0011] The present application aims to provide a technique that enables a simple and correct determination of a condition of a region including a tympanic membrane at an appropriate time even in a situation in which medical services are limited like a hospital or a clinic, in view of the above-described actual circumstances.

[0012] MEANS FOR SOLVING THE PROBLEM

[0013] To achieve the object, the tympanic membrane classification device according to the present application is equipped with: a communication section into which image data containing a tympanic membrane is input; a first classification section that extracts features from the image data using an algorithm to classify the tympanic membrane as acute otitis media or other than acute otitis media; and a second classification section that extracts features from the image data classified as other than acute otitis media using an algorithm to classify the tympanic membrane as a normal tympanic membrane or otitis media with effusion, and the tympanic membrane classification device outputs information of acute otitis media or a normal tympanic membrane or otitis media with effusion from the communication section based on the classification results of the first and second classification sections.

[0014] According to the present application, since two-stage two-classification is performed, a classification result with higher accuracy than ever before can be obtained. Therefore, even if it is not possible to visit a medical institution, assistance almost equivalent to that of a medical examination can be obtained, and the symptoms can be known at an appropriate time. The hardware configuration and software configuration of the tympanic membrane classification device of the present application are not particularly limited. The tympanic membrane classification device can be, for example, a server on a network or a cloud server.

[0015] Glare and foreign matter that are reflected in the image can adversely affect classification. When the tympanic membrane is photographed, it is difficult for an ordinary person who is not familiar with the photographing operation to adjust the amount of light required for photographing or to avoid glare or foreign matter from being reflected when the tympanic membrane is photographed. Therefore, as one aspect of the present application, the tympanic membrane classification device is equipped with a trimming section that trims the tympanic membrane from the image data input to the communication section and provides it to the first classification section. According to this aspect, the reflected glare and foreign matter can be removed from the image of the classification target, and thus the accuracy of the classification is improved. In addition, machine learning can be efficiently performed. As another aspect of the present application, the tympanic membrane can be trimmed before being input to the communication section at the time of photographing the tympanic membrane, or at the time of transmitting the image data to the communication section, and the like, and the trimmed image data can be input to the communication section of the tympanic membrane classification device.

[0016] As a preferred aspect of the present application, the tympanic membrane classification device is equipped with a trimming section that trims a prescribed portion of the tympanic membrane from the image data input to the communication section and provides it to the first classification section. According to this aspect, at least one of the first classification section and the second classification section focuses on the prescribed portion of the tympanic membrane, rather than the entire tympanic membrane, and thus the accuracy of the classification can be improved, and machine learning can be efficiently performed. For example, by trimming the lower half of the tympanic membrane, cases in which the lower half is significantly diseased, such as otitis media with effusion, can be correctly classified, and machine learning can be efficiently performed. Alternatively, for example, the center portion of the tympanic membrane is trimmed.

[0017] The tympanic membrane classification device of the present invention is equipped with artificial intelligence (AI). The AI ​​model of the AI ​​algorithm is not particularly limited. As a preferred aspect of the invention, at least one of the first classification unit and the second classification unit performs supervised machine learning after importing labeled training data. According to this aspect, the accuracy is further improved because the tympanic membrane classification device continuously performs machine learning.

[0018] The tympanic membrane classification system of the present invention includes the aforementioned tympanic membrane classification device, an otoscope for imaging the tympanic membrane, and a terminal that receives image data of the tympanic membrane from the otoscope and outputs it to the communication unit of the tympanic membrane classification device. According to this aspect, even in ordinary households without medical expertise, symptoms of the tympanic membrane can be determined by operating a home otoscope and terminal. For example, the terminal can simply display the classification results or information related to the classification results as an option. The otoscope is not limited to medical devices as long as it can image deep within the ear canal, and the imaging unit and image only need to provide a suitable resolution (DPI). Preferably, the otoscope has an imaging unit capable of capturing images with 4 megapixels or more. More preferably, the otoscope has an imaging unit capable of capturing images with 8 megapixels or more. The device and hardware configuration of the terminal are not particularly limited.

[0019] As one aspect of the invention, the terminal includes: a screen for displaying an image captured by an otoscope; and a guide unit for guiding the tympanic membrane to be displayed on the screen at its center, and the terminal outputs image data obtained by capturing the tympanic membrane at its center to a tympanic membrane classification device. According to this aspect, because the terminal has the guide unit, even users unfamiliar with medical technology can easily capture an image D' of the tympanic membrane. The guide unit is not particularly limited, and may be, for example, a circular frame displayed at the center of the terminal's screen.

[0020] As one aspect of the invention, the otoscope is a digital camera mounted in an ear canal cleaning structure. According to this aspect, symptoms of the tympanic membrane can be detected by an ear canal cleaning device equipped with a digital camera.

[0021] As a preferred aspect of the invention, the tympanic membrane classification system includes a medical institution terminal capable of communicating with the tympanic membrane classification device and the terminal. According to this aspect, assistance nearly equivalent to that obtained from a medical institution can be obtained.

[0022] Glare and foreign objects reflected in an image can negatively impact classification. When photographing the tympanic membrane, it is difficult for ordinary people unfamiliar with the photographing operation to adjust the required light intensity or avoid glare or foreign objects during the photographing process. Therefore, as a preferred aspect of the invention, the otoscope or terminal has a trimming unit that trims the tympanic membrane from an image containing the tympanic membrane to create image data. According to this aspect, reflected glare and foreign objects can be removed from the image of the object to be classified, thus improving classification accuracy. Furthermore, machine learning can be performed effectively. From the perspective of the tympanic membrane classification device, trimming is performed on the otoscope side. Alternatively, in other aspects, trimming can be performed on the tympanic membrane classification device side.

[0023] As a preferred aspect of the invention, the guiding unit displays information on the screen indicating whether the image shown on the screen is suitable for classification. According to this aspect, even users unfamiliar with the shooting operation can capture images of the tympanic membrane.

[0024] As a preferred aspect of the invention, the terminal displays the probabilities associated with the classification results of the first and second classification sections on the screen. This can attract the user's attention. Alternatively, a threshold can be preset for the probabilities; if the probability is greater than or equal to the threshold, a notification is displayed on the screen suggesting a visit to a hospital or clinic.

[0025] Each operation (process) in the classification method of the present invention can also be implemented by a computer-executable program stored in various machine-readable storage media. The tympanic membrane classification program of the present invention is a program that enables a computer to classify image data containing the tympanic membrane, by causing the computer to perform the following steps: a first classification step, namely, using an algorithm to extract characteristics from the image data to classify the tympanic membrane as acute otitis media or something other than acute otitis media; a second classification step, namely, using an algorithm to extract characteristics from the image data classified as something other than acute otitis media to classify the tympanic membrane as a normal tympanic membrane or effusion-related otitis media; and an output step, namely, based on the classification results of the first and second classification steps, outputting information about acute otitis media, a normal tympanic membrane, or effusion-related otitis media.

[0026] This invention includes program code that enables a computer to execute a classification, a computer-readable program product, and a storage medium for storing the program code. The storage medium may include, for example, a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., but is not limited thereto. The storage medium storing the program code is provided directly or indirectly to a system or device, and the system or device's computer, ROM, RAM, or central processing unit (CPU) reads and executes the program code. Embodiments of this invention are not limited to a program; they can be implemented through other software or firmware. This invention can be either an apparatus or a method.

[0027] Invention Effects

[0028] Thus, according to the present invention, the condition of the tympanic membrane can be known with high accuracy without going to a medical institution. Therefore, it is not limited by time and location, and the condition of the area containing the tympanic membrane can be grasped at the appropriate time, preventing missed diagnostic opportunities. In addition, regional differences in medical services are eliminated. Furthermore, its application in the medical field improves diagnostic efficiency and helps to achieve labor-saving on-site procedures. Attached Figure Description

[0029] Figure 1 This is a schematic diagram showing an overall tympanic membrane classification system as an embodiment of the present invention.

[0030] Figure 2 This is a diagram showing the classification results performed by the tympanic membrane classification device of this embodiment.

[0031] Figure 3 It represents a photograph containing an image of the tympanic membrane and a cropped image.

[0032] Figure 4 This is a schematic diagram illustrating the classification algorithm executed by the tympanic membrane classification device of this embodiment.

[0033] Figure 5 This is a schematic diagram illustrating the classification performed by the tympanic membrane classification device in Comparative Example 1.

[0034] Figure 6 This is a screen displayed on the terminal in this embodiment, and the terminal screen displays the guidance unit.

[0035] Figure 7 This is a schematic diagram showing the classification algorithm executed by the tympanic membrane classification device of this embodiment, along with the probability [%].

[0036] Figure 8 The screen displayed on the terminal in this embodiment shows the classification results displayed on the terminal screen in an indexed manner. Detailed Implementation

[0037] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Figure 1 This is a schematic diagram showing an overall tympanic membrane sorting system according to one embodiment of the present invention. The tympanic membrane sorting system includes an otoscope 10, a terminal 20, and a tympanic membrane sorting device 30. The otoscope 10 has a rod-shaped body 11, an ear canal cleaning structure 12 disposed at the front end of the body, and a light-emitting part 14 and a camera part 15 attached to the ear canal cleaning structure 12. The camera part is, for example, a digital camera.

[0038] Terminal 20 has a screen 21 and is capable of communicating with otoscope 10. Image B captured by imaging unit 15 is projected onto screen 21. As an example, image B is equipped with a circular frame C.

[0039] The user of the otoscope 10 holds the main body 11 with their hand and inserts the ear canal cleaning structure 12 into the subject's, such as a child's, external auditory canal H to remove earwax from the inner circumference of the external auditory canal H. Alternatively, the user can turn on the light-emitting part 14 and use the imaging part 15 to take a picture of the inside of the external auditory canal H, projecting the inside of the external auditory canal H onto the screen 21.

[0040] The user can move the otoscope 10 while checking the screen 21 to center the image of the tympanic membrane D (hereinafter referred to as tympanic membrane image D') in the frame C. The frame C serves as a guide for the user to capture the tympanic membrane image D'. The image B projected by the imaging unit 15 is constantly projected onto the screen 21. While checking the tympanic membrane image D' contained in the image B, the user moves the otoscope 10. When the tympanic membrane image D' is projected more prominently in the center of the frame C, the user can select the tympanic membrane image D' with less light reflection, glare, and color distortion that causes localized whitening (flare F) to crop. When selecting and cropping, the user should avoid image defocus and hand tremors. Furthermore, there are no special restrictions on the user; skilled shooting techniques and medical personnel are not required, but rather any ordinary person.

[0041] Terminal 20 can communicate with an external tympanic membrane classification device 30 (specifically, the communication unit 33) via wireless or wired communication, such as a network unit NW like the Internet. Terminal 20 can be replaced by a widely available mobile phone terminal, such as a smartphone, tablet computer, or a terminal equipped with other communication functions such as Wi-Fi (registered trademark), Bluetooth (registered trademark). The user can submit a captured image B (strictly speaking, the image data of image B) to the tympanic membrane classification device 30, which classifies the tympanic membrane D into a normal tympanic membrane or acute or effusion-related otitis media. This classification can be referred to as judgment or diagnosis.

[0042] The tympanic membrane classification device 30 includes a classification unit 31 with artificial intelligence (AI) that drives a proprietary algorithm, a storage unit 32 that stores a large amount of image data (synonymous with images), and a communication unit 33 that communicates with external systems. The tympanic membrane classification device 30 of this embodiment is a server connected to the Internet. The storage unit 32 stores image data of normal tympanic membranes labeled as normal. It also stores image data of acute otitis media labeled as acute otitis media. Furthermore, it stores image data of effusion otitis media labeled as effusion otitis media. This data is used as labeled training data for machine learning in the classification unit 31. The classification unit 31 uses multiple training data as teaching materials for machine learning to improve the accuracy of classification. It should also be understood that the term "accuracy" is synonymous with the accumulated result of external institutions other than the tympanic membrane classification device 30, such as human doctors, scoring the correctness of the classification results of the classification unit 31.

[0043] When an image containing a tympanic membrane image D' (synonymous with image data, hereinafter the same) is input into the classification unit 31, the classification unit 31 extracts features for classification from the tympanic membrane image D', processes these features using an algorithm, and classifies the tympanic membrane image D' into one of the following: normal tympanic membrane, acute otitis media, or effusion otitis media, and outputs the classification result. The features refer to the tonal distribution, shape, and size of the tympanic membrane image D', etc., and are defined within the classification unit 31 according to the algorithm.

[0044] The communication unit 33 receives image data from the outside via wireless or wired communication, such as a network unit NW, and outputs the image data to the classification unit 31. Alternatively, the communication unit 33 can communicate with the computer terminal of the medical institution 40 via the same communication unit. Alternatively, the tympanic membrane classification device 30 can be a computer configured within the medical institution. In either case, the labeled training data gradually accumulates in the storage unit 32. Therefore, the learning load of the machine learning in the classification unit 31 increases, and the classification accuracy improves. Furthermore, the tympanic membrane image D' newly input from the terminal 20 to the tympanic membrane classification device 30 will subsequently be labeled by the physicians of the medical institution 40. Thus, the training data gradually accumulates in the storage unit 32.

[0045] The classification section 31 includes a trimming section, a first classification section, and a second classification section. In this embodiment, as... Figure 3 As shown, the trimming unit within the classification unit 31 uses an algorithm to trim the tympanic membrane image D' from image B in (a). Thus, the glare F reflected in image B, the hue of the inner circumference of the external auditory canal H, and foreign matter such as body hair and earwax G are separated from the tympanic membrane image D' without affecting the classification. (b) represents the associated image data of the trimmed tympanic membrane image D'.

[0046] Figure 4 This is a schematic diagram illustrating the algorithms executed sequentially by the first and second classification departments in classification unit 31. These algorithms utilize ResNet (Residual Network), VGG16 (VisualGeometry Group 16-layer), DenseNet (Densely Connected Convolutional Network), and other AI models.

[0047] In the first step, the first classification section 31b in classification section 31 performs binary classification of the tympanic membrane image D'. Specifically, it classifies the tympanic membrane image D' as either acute otitis media or something else. This binary classification is based on the proportion of red-toned pixels in the overall pixels of the tympanic membrane image D', the size of the brown-toned pixel (also known as cloudy) area, and other characteristics.

[0048] In the second step, the second classification section 31c in classification section 31 performs binary classification of the tympanic membrane image D' other than acute otitis media. Specifically, it classifies the tympanic membrane image D' other than acute otitis media into either a normal tympanic membrane or effusion-related otitis media. This binary classification is based on the proportion of brown-toned pixels in the overall pixel count and other characteristics. Through this two-stage classification, the tympanic membrane image D' is classified into three categories with a higher accuracy than before.

[0049] Figure 2 This is a diagram illustrating the classification results performed by the tympanic membrane classification device of this embodiment. The classification results output by the tympanic membrane classification device 30 are sent to the terminal 20 via the network unit NW and displayed on the screen 21. The probability [%] calculated by the tympanic membrane classification device 30 for each classification result is also displayed. It should be noted that the probability [%] mentioned here is a value calculated by the classification unit 31.

[0050] Terminal 20 can communicate with medical institution 40 via wireless or wired communication, such as through a network unit NW. Users of terminal 20 can ask and answer questions with staff of medical institution 40 regarding the classification results of tympanic membrane image D', the appropriateness of the classification, and future diagnosis and treatment.

[0051] When the classification result is a normal tympanic membrane, the probability X [%] of having a normal tympanic membrane is displayed along with the classification result. When the classification result is acute otitis media, the probability Y [%] of having acute otitis media is displayed along with the classification result. When the classification result is effusion otitis media, the probability Z [%] of having effusion otitis media is displayed along with the classification result. Probabilities X, Y, and Z [%] are based on the classification results performed by the previous classification department 31 and compared with the annotations made by the doctor for their classification results; in other words, they are based on accuracy.

[0052] Here's a supplementary explanation: Figure 2 In this context, warnings can be issued by changing the color of the periphery or surrounding background of image B, changing the thickness or color of text, displaying numerical or degree indicators such as size scales, or displaying symbols or markings. For example, for... Figure 2 Image B, which is classified as normal in (a), has a white border added to it. Additionally, for... Figure 2 Image B, classified as acute otitis media in (b), does not display the probability Y[%]. Instead, when acute otitis media is suspected (probability Y[%] < 50[%]), a light-colored border (such as pink) is added to image B. When it is almost certain to be acute otitis media (probability Y[%] ≥ 50[%]), a dark-colored border (such as dark red) is added to image B. Figure 2 The same applies to effusion-related otitis media (c). Thus, the viewer on screen 21 can be reliably informed that the symptoms of tympanic membrane D are imminent. Furthermore, probabilities X, Y, and Z [%] can be displayed along with indicators and markers.

[0053] The accuracy of the tympanic membrane sorting device 30 of this embodiment and the tympanic membrane sorting device of the comparative example was investigated. The results of the accuracy of these tympanic membrane sorting devices are shown in Table 1.

[0054] [Table 1]

[0055]

[0056] First, machine learning was performed on multiple sets of training data (multiple individuals, multiple ears). Next, the tympanic membrane classification device 30 classified multiple tympanic membrane images D' (this time). Machine learning was then performed again on multiple sets of training data (multiple individuals, multiple ears). Next, the tympanic membrane classification device 30 classified multiple tympanic membrane images D' (second time). When only the first (this time) classification is performed without performing the second classification, the accuracy of the second classification is not displayed.

[0057] The accuracy rates of the classification results in this embodiment are shown in Tables 1, No. 3 and No. 4. It can be seen that, based on the two-stage binary classification of this embodiment, the accuracy rates are 99.98% in the first stage and 99.50% in the second stage, which is higher than previously achieved. The reason for this is that the tympanic membrane D in acute otitis media has a stronger red hue than the tympanic membrane D in other types of otitis media, and the tympanic membrane D in effusion otitis media shows a more pronounced cloudy area than the tympanic membrane D in other types of otitis media.

[0058] The accuracy of the comparative examples was also investigated. The classification process for the comparative examples is shown below. Figure 5 .

[0059] The comparative tympanic membrane classification device is equipped with a one-stage multi-classification (three-classification) algorithm, that is, classifying the input tympanic membrane image into a normal tympanic membrane, acute otitis media, or effusion otitis media in one stage. According to the comparative one-stage three-classification, as shown in Table 1 No. 1, the accuracy was 68.26% in the first (this) test and 91.12% in the second. Thus, the accuracy of the comparative tympanic membrane classification device (No. 1) is worse than the accuracy of the tympanic membrane classification device of this embodiment (No. 3 and No. 4). Therefore, it can be seen that performing a two-classification method like that of this embodiment in multiple stages improves the accuracy compared to previous methods.

[0060] Next, a modified example of this embodiment will be described. The modified tympanic membrane sorting device 30 is as follows: Figure 3 As shown, approximately the lower half of the tympanic membrane D is trimmed. Then, the trimmed semicircular image D′′ is classified as described above. Exudative otitis media presents as a distinct cloudy area in the lower half of the tympanic membrane D. Therefore, trimming approximately the lower half of the tympanic membrane D (e.g., the lower half of the central region of the tympanic membrane D) improves the accuracy of the classification results. Furthermore, by annotating the image of the lower half of the tympanic membrane D and using it as training data for machine learning by the classification unit 31, the accuracy of the classification unit 31 itself can be improved.

[0061] The accuracy of the modified example of this embodiment, as shown in No. 5, is 96.07% in the first (this) instance, which is better than the case of tympanic membrane image D' (No. 4). Furthermore, it is 100% in the second instance, indicating an improvement in accuracy.

[0062] As shown in No. 2, the accuracy of the modified example in the first instance (this time) was 62.41%, which was worse than that of the tympanic membrane image D' (No. 1). In the second instance, the accuracy was 65.74%, which was an improvement, but did not exceed the accuracy of this embodiment (No. 5).

[0063] Reference Figure 1 The tympanic membrane classification device 30 of this embodiment is equipped with: a communication unit 33, which receives an externally input tympanic membrane image D' containing the tympanic membrane D; a first classification unit 31b ( Figure 3The tympanic membrane classification device 30 uses an algorithm to extract features from the tympanic membrane image D' and classifies the tympanic membrane D as either acute otitis media or something other than acute otitis media; and the second classification unit 31c uses an algorithm to extract features from the tympanic membrane image D', which is classified as something other than acute otitis media in the first classification unit 31b, and classifies the tympanic membrane D as either normal or effusion-related otitis media. Based on the classification results of these first classification units 31b and second classification units 31c, the tympanic membrane classification device 30 outputs information about acute otitis media, normal, or effusion-related otitis media to the outside from the communication unit 33. According to this tympanic membrane classification device 30, using Figure 4 The two-stage binary classification algorithm shown has improved classification accuracy compared to previous methods.

[0064] The function of the tympanic membrane sorting device 30 is explained from the following aspects.

[0065] On the first side, the tympanic membrane classification device 30 uses the aforementioned proprietary AI algorithm to classify the tympanic membrane D. Therefore, it is possible to have a doctor assess the symptoms of tympanic membrane D without having to go directly to a medical institution 40 for examination by an ENT specialist. For example, in ordinary families without medical expertise, a family member can take a picture of the child's tympanic membrane D and use it for tympanic membrane D disease assessment.

[0066] On the second side, the medical facility 40 is equipped with an otoscope 10 and a terminal 20 for examination, and is communicatively connected to a tympanic membrane classification device 30. Thus, the tympanic membrane classification device 30 is used by the ENT specialist of the medical facility 40 as a diagnostic support tool to diagnose the tympanic membrane image D' taken through the otoscope 10. Additionally, image B is used by the ENT specialist. Alternatively, for example, in a typical family member without medical expertise, taking a picture of the child's tympanic membrane D can provide almost equivalent assistance to the ENT specialist of the medical facility 40, such as consultation advice, allowing the symptoms to be recognized at an appropriate time.

[0067] On the third side, the tympanic membrane classification device 30 on the first and second sides described above is used as a diagnostic support tool or as a medical device. The tympanic membrane classification device 30, through accumulated AI machine learning, becomes capable of performing classification at the same level as an ENT specialist. For example, by embedding a drug prescription process into the AI ​​algorithm, it is possible to prescribe medications based on the diagnostic support tool of this embodiment. For example, the tympanic membrane classification device 30 can be communicatively connected to a pharmacy (computer) (not shown).

[0068] Additionally, refer to Figure 3 (a) and (b), the tympanic membrane sorting device 30 of this embodiment is equipped with a trimming unit. This trimming unit trims an image portion of the tympanic membrane D from the image B input from the outside to the communication unit 33 and provides it to the first sorting unit 31b. Light-emitting unit 14 ( Figure 1When the external auditory canal H is illuminated, the glare F and foreign object G are easily reflected on the exit side of the external auditory canal H. According to this embodiment, since the image of the exit side of the external auditory canal H is removed, the glare F and foreign object G are also removed from the image. Therefore, machine learning can be performed effectively.

[0069] Additionally, refer to Figure 3 (a) and (c), in the modified example of this embodiment, the tympanic membrane classification device 30 is equipped with a trimming unit that trims the lower half of the image of the tympanic membrane D from the image B input to the communication unit 33 and provides it to the first classification unit 31b. This allows for the correct classification of cases where the lesion in the lower half is obvious, such as effusion otitis media, and enables effective machine learning.

[0070] Furthermore, the tympanic membrane classification device 30 of this embodiment first stores the image B of the classification object in the storage unit 32, and then the medical institution 40 correctly labels the classification. Then, at least one of the first classification unit 31b and the second classification unit 31c imports the labeled training data and performs supervised machine learning. As a result, the tympanic membrane classification device 30 continuously accumulates machine learning data, and its accuracy improves day by day.

[0071] Furthermore, the tympanic membrane classification system of this embodiment includes a tympanic membrane classification device 30, an otoscope 10 for photographing the tympanic membrane D, and a terminal 20. The terminal 20 receives an image B of the tympanic membrane D from the otoscope 10 and outputs it to the communication unit 33 of the tympanic membrane classification device 30. Thus, even ordinary households without medical expertise can know the symptoms of the tympanic membrane D by operating the otoscope 10 and the terminal 20.

[0072] Furthermore, the terminal 20 of this embodiment is equipped with a screen 21 that displays images captured by the otoscope 10. A frame C is displayed on the screen 21, and the tympanic membrane D is displayed in the center of the screen. The terminal 20 inputs the image B of the tympanic membrane D captured within the frame C in the center to the communication unit 33. Thus, even users who are not skilled in medical technology can accurately capture images of the tympanic membrane D'.

[0073] Furthermore, the imaging unit 15 in this embodiment is a digital camera installed in the ear canal cleaning structure 12. Therefore, by using the digital camera mounted in an ear canal cleaning structure that is normally used as an everyday item, symptoms of tympanic membrane disease can be detected.

[0074] In addition, the communication unit 33 and the terminal 20 of this embodiment can communicate with the medical institution 40, so that even if it is not possible to go to the medical institution, it can obtain assistance that is almost equivalent to a consultation and be aware of the symptoms at the appropriate time.

[0075] Next, we will provide a supplementary description of the screen 21 installed on the user's terminal 20. The terminal 20 communicates with the otoscope 10 and projects the image B captured by the otoscope 10 onto the screen 21 in real time.

[0076] Figure 6 This is a diagram of a terminal screen illustrating this embodiment, showing screen 21 as a guide unit. The user inserts the otoscope 10 into the ear of the person being examined, taking an image B of the external auditory canal H, which is then transmitted via network NW (…). Figure 1 The image is output in real time from terminal 20 to tympanic membrane classification device 30. The classification unit 31 of tympanic membrane classification device 30 has a program (image guidance program) as a guidance unit, which determines in real time whether image B contains the entire tympanic membrane D and is suitable for tympanic membrane classification. Specifically, it determines whether it contains the aforementioned... Figure 3 The image D' represents the tympanic membrane. When the otoscope 10 is initially inserted into the subject's ear, image B does not contain the tympanic membrane image D', and image B is unusable for tympanic membrane classification. A blue ring 22 is displayed on screen 21. The diameter of ring 22 is half the diameter of frame C in image B. Furthermore, in screen 21, the outer region J of frame C is a plain, monochrome color, such as light gray.

[0077] In the image capture guidance procedure for inputting image B, when it is determined that the area surrounded by the blue ring 22 displays the tympanic membrane D, the procedure determines that image B is suitable for tympanic membrane classification and changes the color of the ring 22 from blue to green. Then, the judgment operation button 23 is displayed in the outer area J. Thus, a properly captured tympanic membrane image is framed by the ring 22.

[0078] The shooting guidance procedure according to this embodiment can assist users unfamiliar with shooting the tympanic membrane D to perform tympanic membrane D shooting. It should be noted that the size, shape and color of the above-mentioned ring 22 are only one example, as long as the visual changes indicate whether it can be used for tympanic membrane classification.

[0079] When the user touches the judgment operation button 23 with their finger, the terminal 20 outputs a classification command to the tympanic membrane classification device 30 (communication unit 33). The tympanic membrane classification device 30 performs two-stage binary classification on the image data input from the terminal 20 and displays the classification results on the screen 21.

[0080] Figure 7 This is a graph showing the probability [%] of each classification result in the two-stage binary classification in example classification section 31.

[0081] Figure 8 This is a graph of the classification results displayed on example screen 21.

[0082] The foregoing Figure 4The binary classification performed by the first classification unit 31b shown is more specifically achieved using an algorithm, such as... Figure 7 The probabilities (numerical values) [%] of each of the two categories are calculated as shown. The same applies to the second category 31c.

[0083] For example, when the classification result of the first classification section 31b is 10% for acute otitis media and 90% for non-acute otitis media, the process proceeds to the second classification section 31c, where the classification result is 70% for normal and 30% for effusion otitis media, and the result is output. The relevant results are shown in Table 2. The 90% for non-acute otitis media from the classification result of the first classification section 31b is proportionally allocated to the classification result of the second classification section 31c, and the probabilities of normal (X=63%), acute otitis media (Y=10%), and effusion otitis media (Z=27%) are calculated as the final classification result probabilities. The screen 21, which serves as the classification result display section, displays these probabilities as an index where the sum of the probabilities X, Y, and Z for the three categories is 100%.

[0084] [Table 2]

[0085]

[0086] In addition, screen 21, which serves as the classification result display unit, such as Figure 8 As shown, the probability [%] of each category is indexed using three different colored indicators and displayed on the box C of the ring surrounding image B.

[0087] exist Figure 8 In the example image D' of a normal tympanic membrane, on the ring of the box C displaying the classification results of image D', the blue indicator Cx (63%) indicating normal tympanic membrane is displayed at the highest proportion, the orange indicator Cz (27%) indicating effusion otitis media is displayed at a lower proportion, and the red indicator Cy (10%) indicating acute otitis media is displayed only in small quantities. The indicators Cx, Cy, and Cz are appropriately rearranged in order of probability.

[0088] According to this embodiment, the classification results are displayed as indicators on frame C of screen 21, and image B is bordered with indicators Cx, Cy, and Cz, thus attracting the user's attention.

[0089] It should be noted that the three different colors mentioned above are examples, and each color can be changed or selected as appropriate. As an optional feature, the date and time associated with the image used for classification, as well as the left and right ear identifiers, can be displayed.

[0090] The embodiments of the present invention have been described above with reference to the accompanying drawings; however, the present invention is not limited to the embodiments shown in the drawings. Various modifications and variations can be made to the embodiments shown in the drawings within the same or equivalent scope as the present invention.

[0091] As a variation not shown, instead Figure 1 The structure shown uses a network unit NW to communicate between the tympanic membrane sorting device 30 and the terminal 20. Figure 1 The tympanic membrane sorting device 30 shown can also be a program installed on the terminal 20. The program is, for example, application software stored in the terminal 20, and the above-described algorithm processing is executed within the terminal 20.

[0092] As a variation not shown, the screen 21 of terminal 20 can be added to or replace various options as a classification result display unit. Various displays, such as color differentiation and shape, can also be implemented using sound or light.

[0093] [Industrial Availability]

[0094] This invention can be advantageously applied to the medical industry.

[0095] Explanation of reference numerals in the attached figures

[0096] 10 otoscopes

[0097] 11 main bodies

[0098] 12-hole cleaning structure

[0099] 14 Light-emitting parts

[0100] 15th Filming Department

[0101] 20 terminals

[0102] 21 screens

[0103] 22-ring (guide unit)

[0104] 30 Tympanic membrane sorting device

[0105] 31 (First and Second) Classification Sections

[0106] 32 Storage Unit

[0107] 33 Ministry of Communications

[0108] 40 medical institutions

[0109] Image B

[0110] C-frame (Guide Unit)

[0111] Cx, Cy, Cz classification result indicators

[0112] D. Tympanic membrane

[0113] D'Tympanic membrane image

[0114] H external auditory canal

Claims

1. A tympanic membrane sorting device, equipped with: The communication unit that receives image data containing the tympanic membrane; The first classification section uses an algorithm to extract characteristics from the image data to classify the tympanic membrane as either acute otitis media or other types of acute otitis media; and The second classification section uses an algorithm to extract characteristics from the image data other than those classified as acute otitis media, and classifies the tympanic membrane as a normal tympanic membrane or effusion otitis media. Based on the classification results of the first and second classification units, the tympanic membrane classification device outputs information about acute otitis media, normal tympanic membrane, or effusion otitis media from the communication unit.

2. The tympanic membrane sorting device according to claim 1, wherein, The tympanic membrane sorting device is equipped with a trimming unit, which trims the tympanic membrane from the image data input to the communication unit and provides it to the first sorting unit.

3. The tympanic membrane sorting device according to claim 1, wherein, The tympanic membrane sorting device is equipped with a trimming unit, which trims a predetermined portion of the tympanic membrane from the image data input to the communication unit and provides it to the first sorting unit.

4. The tympanic membrane sorting device according to claim 1, wherein, At least one of the first classification division and the second classification division imports the labeled training data and performs supervised machine learning.

5. A tympanic membrane classification system, comprising: the tympanic membrane classification device according to claim 1; An otoscope used for imaging the tympanic membrane; and The terminal receives image data of the tympanic membrane from the otoscope and outputs the image data of the tympanic membrane to the tympanic membrane classification device.

6. The tympanic membrane classification system according to claim 5, wherein, The terminal includes: a screen for displaying images captured by the otoscope; and a guide unit for guiding the tympanic membrane to be displayed on the screen at the center of the screen, and the terminal outputs image data obtained by capturing the tympanic membrane at the center of the screen to the communication unit.

7. The tympanic membrane classification system according to claim 5, wherein, The otoscope is a digital camera installed in the ear canal cleaning structure.

8. The tympanic membrane classification system according to claim 5, wherein, The tympanic membrane classification system includes a medical institution terminal capable of communicating with the tympanic membrane classification device and the terminal.

9. The tympanic membrane classification system according to claim 5, wherein, The otoscope or the terminal has a trimming section that trims the tympanic membrane from an image containing the tympanic membrane to create the image data.

10. The tympanic membrane classification system according to claim 6, wherein, The guidance unit displays information on the screen indicating whether the image displayed on the screen is suitable for the classification.

11. The tympanic membrane classification system according to claim 5, wherein, The terminal displays the probabilities related to the classification results of the first and second classification departments on the screen.

12. A program that enables a computer to classify image data containing a tympanic membrane, wherein the computer performs the following steps: First classification step: Use an algorithm to extract features from the image data and classify the tympanic membrane as either acute otitis media or other types of acute otitis media; The second classification step involves using an algorithm to extract characteristics from the image data other than those classified as acute otitis media, and classifying the tympanic membrane as a normal tympanic membrane or effusion-related otitis media; and Output step: Based on the classification results of the first and second classification steps, output information on acute otitis media, normal tympanic membrane, or effusion otitis media.

Images