Oral condition evaluation support system

Abstract

Provided is a novel technology capable of providing more appropriate evaluation results regarding the condition of the oral cavity. This oral cavity condition evaluation support system comprises: an oral cavity image acquisition unit that acquires an oral cavity image of a subject; an oral cavity finding information storage unit that stores oral cavity finding information of the subject; and an evaluation unit that, on the basis of the oral cavity image acquired by the oral cavity image acquisition unit and the oral cavity finding information, generates information on an evaluation result regarding the oral cavity condition of the subject, using an oral cavity condition evaluation model obtained by performing machine learning on the basis of the oral cavity conditions of the subject and the results of evaluation thereof.

Description

Oral condition evaluation support system

[0001] The present invention relates to a system for assisting in the evaluation of the condition of the oral cavity.

[0002] In order to provide oral care, etc., an oral condition is first evaluated (for example, Patent Document 1). In Patent Document 1, an assessment is first conducted on a person requiring oral care to create assessment information, and an oral care assessment, an oral care plan, and a denture creation plan are created based on this assessment information.

[0003] Japanese Patent Application Laid-Open No. 2001-167215

[0004] An object of the present invention is to provide a novel technique that can obtain more appropriate evaluation results for the condition of the oral cavity.

[0005] The above-mentioned oral cavity evaluations are performed manually, which tends to result in variations in the evaluation results, and it is not always possible to obtain appropriate evaluation results.

[0006] The inventor discovered that more appropriate evaluation results can be obtained by using an oral condition evaluation model obtained by machine learning based on the oral condition and its evaluation results, and generating information about the evaluation results based on an image of the subject's oral cavity and sensor information, which is information about the subject's oral cavity obtained by a sensor.

[0007] The gist of the present invention is as follows: [1] An oral condition evaluation support system comprising: an oral image acquisition unit that acquires oral images of a subject; an oral finding information storage unit that stores oral finding information of the subject; and an evaluation unit that generates information on an evaluation result of the oral condition of the subject using an oral condition evaluation model obtained by machine learning based on the oral images acquired by the oral image acquisition unit and the oral finding information, using an oral condition evaluation model obtained by machine learning based on the oral condition and its evaluation results. Here, machine learning based on the oral condition and its evaluation results includes machine learning of teacher data indicating the relationship between the oral condition and its evaluation results. [2] The oral condition evaluation support system according to [1], further comprising an odor information acquisition unit that acquires odor information of the subject's oral cavity obtained by a sensor, wherein the oral finding information storage unit stores the odor information as the oral finding information, and the oral condition evaluation model is obtained by machine learning based on the oral images and their evaluation results, as well as machine learning based on the oral odor and its evaluation results. Here, machine learning based on the oral images and their evaluation results includes machine learning of teacher data indicating the relationship between the oral images and their evaluation results. Further, performing machine learning based on the oral odor and the evaluation results includes performing machine learning to learn teacher data indicating the relationship between the oral odor and the evaluation results. [3] The oral condition evaluation support system according to [1], further comprising a denture image acquisition unit that acquires denture images obtained by imaging the denture of the subject, wherein the oral finding information storage unit stores information about the denture condition based on the denture images as the oral finding information, and the oral condition evaluation model is further obtained by machine learning based on the denture condition and the evaluation results. Here, performing machine learning based on the denture condition and the evaluation results includes machine learning to learn teacher data indicating the relationship between the denture condition and the evaluation results. [4] The oral condition evaluation support system according to [1], further comprising a medical interview information acquisition unit that acquires medical interview information related to the medical interview results of the subject, wherein the oral finding information storage unit stores the medical interview information as the oral finding information, and the oral condition evaluation model is further obtained by machine learning based on the medical interview results and the evaluation results.Here, performing machine learning based on the medical interview results and their evaluation results includes machine learning training data showing the relationship between the medical interview results and their evaluation results. [5] The oral condition evaluation support system according to [1], further comprising: a denture image acquisition unit that acquires denture images obtained by imaging the denture of the subject; and a medical interview information acquisition unit that acquires medical interview information related to the medical interview results. The oral finding information storage unit stores, as the oral finding information, information on the denture condition based on the denture images and the medical interview information. The oral condition evaluation model is further obtained by machine learning based on the denture condition and its evaluation results, as well as machine learning based on the medical interview results and its evaluation results. Here, performing machine learning based on the denture condition and its evaluation results includes machine learning training data showing the relationship between the denture condition and its evaluation results. Furthermore, performing machine learning based on the medical interview results and its evaluation results includes machine learning training data showing the relationship between the medical interview results and its evaluation results. [6] The oral condition evaluation support system according to any one of [1] to [5], further comprising an auxiliary information storage unit that stores physical information about the subject's body and / or insurance information about the subject's insurance. [7] The oral condition evaluation support system of [6] further comprises an information output unit that outputs information about the evaluation result generated by the evaluation unit in association with the physical information and / or the insurance information. [8] The oral condition evaluation support system of [7] further comprises an evaluation result registration unit that stores information about the evaluation result generated by the evaluation unit in association with the physical information and / or the insurance information in the attached information storage unit. The present invention also encompasses the following method. That is, the present invention encompasses an information processing method executed by a computer, comprising: acquiring oral images of a subject; storing oral finding information of the subject; and generating information about the evaluation result of the oral condition of the subject using an oral condition evaluation model obtained by machine learning based on the oral condition and the evaluation result, based on the oral images and the oral finding information. Here, machine learning based on the oral condition and the evaluation result includes machine learning training of training data indicating the relationship between the oral condition and the evaluation result.The present invention also encompasses the following program. That is, the present invention encompasses a program for causing a computer to function as an oral image acquisition unit that acquires oral images of a subject, an oral finding information storage unit that stores oral finding information of the subject, and an evaluation unit that generates information on an evaluation result of the oral condition of the subject using an oral condition evaluation model obtained by machine learning based on the oral condition and its evaluation results, based on the oral images acquired by the oral image acquisition unit and the oral finding information. Here, machine learning based on the oral condition and its evaluation results includes machine learning training of training data showing the relationship between the oral condition and its evaluation result.

[0008] According to the present invention, a novel technique can be provided that can obtain more appropriate evaluation results for the state of the oral cavity.

[0009] 1 is a diagram showing an overview of the device configuration of the oral state evaluation support system of this embodiment. FIG. 2 is a diagram showing an overview of an evaluation device included in the oral state evaluation support system of this embodiment. FIG. 3 is a block diagram of an evaluation device included in the oral state evaluation support system of this embodiment. FIG. 4 is a diagram showing the procedure for generating a judgment algorithm related to oral cavity images of the oral state evaluation model of this embodiment. FIG. 5 is a table showing the relationship between evaluation items and evaluation values ​​(scores) of the Oral Health Assessment Tool (OHAT). FIG. 6 is a diagram showing the procedure for generating a judgment algorithm related to odor information of the oral state evaluation model of this embodiment. FIG. 7 is a diagram showing the procedure for generating a judgment algorithm related to medical interview information of the oral state evaluation model of this embodiment. FIG. 8 is a diagram showing a processing flow related to evaluation value generation of this embodiment. FIG. 9 is a diagram showing a processing flow related to evaluation value registration of this embodiment. FIG. 10 is a table showing the relationship between evaluation items and evaluation values ​​of other embodiments.

[0010] An embodiment of the present invention will be described in detail below. This embodiment relates to an oral cavity condition evaluation support system, which includes an oral cavity image acquisition unit, a sensor information acquisition unit, and an evaluation unit.

[0011] The oral cavity image acquisition unit acquires an image of the oral cavity of a subject.

[0012] The sensor information acquisition unit acquires sensor information about the oral cavity of the subject obtained by the sensor.

[0013] The evaluation unit generates information on the evaluation result of the target oral condition using an oral condition evaluation model obtained by machine learning based on the oral condition and its evaluation result, based on the oral images acquired by the oral image acquisition unit and the sensor information acquired by the sensor information acquisition unit. Here, machine learning based on the oral condition and its evaluation result includes machine learning of training data indicating the relationship between the oral condition and its evaluation result.

[0014] In this specification, an oral cavity image refers to an intraoral image, or an intraoral image and an image of the oral cavity periphery. In the following description of this embodiment, an example will be given in which an intraoral image and an image of the oral cavity periphery are used as the oral cavity image.

[0015] In this embodiment, the sensor information acquisition unit acquires odor information about the odor of the subject's oral cavity as sensor information (in this embodiment, the sensor information acquisition unit corresponds to the odor information acquisition unit). Note that the sensor information acquisition unit may also acquire information obtained by other sensors as sensor information. For example, the sensor information may include information about swallowing sounds obtained by a sound sensor, information about clenching strength obtained by a pressure sensor, and information about humidity in the oral cavity obtained by a humidity sensor.

[0016] In the following description of this embodiment, an example will be given in which an evaluation value (score) is generated as information about the evaluation result.

[0017] In addition, in this embodiment, the oral cavity finding information includes the subject's medical history information, the subject's oral cavity odor, swallowing sounds, bite pressure, oral humidity, and evaluations related to dentures.

[0018] FIG. 1 is a diagram showing an outline of the device configuration of an oral cavity condition evaluation support system 100 according to this embodiment.

[0019] The oral cavity condition evaluation support system 100 includes a terminal device 1, an odor information acquisition device 3, an evaluation device 5, a prescription device 7, a database DB1, and a database DB2, all of which can communicate with each other via a network NW.

[0020] The evaluation device 5 and the prescription device 7 may be, for example, a personal computer.

[0021] The terminal device 1 is a device capable of capturing an image of the oral cavity of a subject, and is, for example, a portable information terminal such as a smartphone or a tablet PC.

[0022] The odor information acquisition device 3 can use various information detection sensors capable of acquiring odor information. Examples of such sensors include breath sensors, and commercially available breath sensors capable of communicating via a network NW may be used. When using a breath sensor, a sensor device capable of detecting specific gas components, such as ammonia and acetone, contained in breath can be used. Here, there is a correlation between the amount of specific gas components, such as ammonia and acetone, contained in breath that cause bad breath and the tendency toward deterioration of human diseases and health conditions. Examples of diseases and health conditions include periodontal disease, tooth decay, dental plaque, tartar, tongue coating, decreased saliva, oral cancer, nose and throat diseases, respiratory diseases, digestive diseases, and stress. Other examples include a sweet smell associated with infectious diseases, acetone with diabetes, a gangrene odor of ammonia and protein with certain tumors and liver disease, isoprene with hypoglycemia and sleep disorders, methyl mercaptan with oral bacteria and liver disease, carbon monoxide with stress, ethanol with alcohol consumption, trimethylamine with kidney disease, and nitric oxide with asthma. Therefore, by detecting the gas components that cause these odors in the breath, it is possible to infer the disease. In other words, the gas component data measured by the breath sensor and the disease information inferred from it are used as biological detection information.

[0023] Furthermore, the databases DB1 and DB2 may be cloud-based, on-premise, or other suitable forms.

[0024] 2 is a schematic diagram of the evaluation device 5 of this embodiment. The prescription device 7 may also have the same configuration as the evaluation device 5.

[0025] The evaluation device 5 includes a processor 11, which is an arithmetic processing device, a memory 12 (such as a RAM (Random Access Memory)) which is a main storage device, and a solid state drive (SSD) 13 which is an auxiliary storage device. The oral condition evaluation support system 100 also includes a network interface (IF) 14, which controls communication with external units, a monitor 15, an input device 16 (such as a keyboard or mouse), and a media reading device 17.

[0026] Programs for realizing aspects of this embodiment are pre-stored in the SSD 13. These programs can be installed by reading the programs stored on external installation media (such as a CD-ROM or DVD) using the media reading device 17 and storing them in the SSD 13, or by downloading the programs via the network IF 14 and storing them in the SSD 13.

[0027] Furthermore, although the evaluation device 5 of this embodiment is a standalone device such as a PC (Personal Computer) that includes each component within the device, one or more of the components may be located outside the device and connected via a network NW. For example, the programs that can be stored in the SSD 13 may be stored on the cloud.

[0028] FIG. 3 is a block diagram of the evaluation device 5 of this embodiment.

[0029] 3, the evaluation device 5 is composed of a display unit 20 that displays various information, an operation reception unit 22 that receives operations from a user such as a doctor or nurse, a memory area 24 that stores various information, a communication unit 26 connected to the network NW, and a control unit 30 that controls the operations of these components. The memory area 24 includes an oral observation information storage unit 25 that stores oral observation information of the subject.

[0030] Specifically, the display unit 20 is configured by, for example, a monitor, and has the function of displaying various information in response to a control signal from the control unit 30 .

[0031] In addition, the operation reception unit 22 is composed of an input device 16, such as a keyboard, a mouse, or a touch panel that also serves as the display unit 20, and has the function of accepting input operations by the user and the function of outputting information indicating the received input content to the control unit 30.

[0032] The storage area 24 is composed of the memory 12, the SSD 13, etc., and has the functions of writing and storing various information and reading various information. This storage area 24 stores a program for implementing an oral condition evaluation model, a program for outputting physical information, a program for registering evaluation values, generated evaluation values ​​for each subject, and physical information for each subject. The oral observation information storage unit 25 stores odor information obtained by the odor information acquisition device 3, interview information related to the results of an interview conducted by a medical professional or the like to evaluate the oral condition of a subject, information about the condition of the dentures based on denture images, etc. The information about the condition of the dentures may be image data of the dentures, or may be information about the condition of the dentures obtained from the image data of the dentures. The odor information, interview information, and information about the condition of the dentures may be used for machine learning, as described below.

[0033] Furthermore, the oral cavity finding information storage unit 25 may store information obtained by other sensors in addition to or instead of the odor information, such as information about swallowing sounds obtained by the above-mentioned audio sensor, information about clenching strength obtained by a pressure sensor, and information about humidity in the oral cavity obtained by a humidity sensor.

[0034] The communication unit 26 is configured by the network IF 14 .

[0035] The control unit 30 is composed of hardware such as a processor 11 and a memory 12, and software such as a control program.

[0036] The control unit 30 has processing functions related to the exchange of various signals with the display unit 20, operation reception unit 22, memory area 24, communication unit 26, etc., and the function of controlling the operation of each unit connected via a specified bus.

[0037] In the present invention, the oral cavity image acquisition unit is composed of the control unit 30 and the operation acceptance unit 22, the sensor information acquisition unit is composed of the control unit 30 and the operation acceptance unit 22, the evaluation unit is composed of the control unit 30, the oral cavity finding information storage unit is composed of the storage area 24, the auxiliary information storage unit is composed of the storage area 24, the information output unit is composed of the control unit 30 and the display unit 20, the denture image acquisition unit is composed of the control unit 30 and the operation acceptance unit 22, the medical interview information acquisition unit is composed of the control unit 30 and the operation acceptance unit 22, and the evaluation result registration unit is composed of the control unit 30. The auxiliary information storage unit can also be configured to include databases DB1 and DB2 connected via the communication unit 26 in addition to the storage area 24.

[0038] As described above, in this embodiment, an oral condition evaluation model obtained by machine learning based on the oral condition and its evaluation results is used to generate an evaluation value. Here, machine learning based on the oral condition and its evaluation results includes machine learning of training data that indicates the relationship between the oral condition and its evaluation results.

[0039] The determination algorithm of the oral cavity condition evaluation model may be generated, for example, according to the procedures shown in FIGS.

[0040] The evaluation item index can be generated using the oral cavity image and the denture image, for example, as follows.

[0041] For example, as shown in Fig. 4, a medical institution first obtains images of the patient's oral cavity and the perioral area, and associated correct labels indicating a deterioration in the oral condition (step S001). The correct labels are assigned based on the results of, for example, dental examinations, periodontal disease tests, saliva tests, bacterial tests, PCR tests, and virus isolation and culture tests that the patient has undergone, as well as the findings of a doctor or dentist. Although PCR tests and virus isolation and culture tests take several weeks to produce results, they are highly accurate and therefore suitable as correct labels.

[0042] Furthermore, the correct answer label used in this learning may include, for example, image data of a person's oral cavity and data representing an evaluation value of the state of the oral cavity indicated by the image data.

[0043] The evaluation value can be assigned according to a predetermined evaluation standard.

[0044] For example, the Oral Health Assessment Tool (OHAT) can be used as the evaluation criterion.

[0045] As shown in Figure 5, the evaluation items of the OHAT consist of eight items: lips, tongue, gums / mucosa, saliva, remaining teeth, dentures (if the subject has them), oral hygiene, and toothache. The OHAT evaluation is based on three levels, each of which is assigned an evaluation value of "0 points," "1 point," or "2 points," with the condition increasing from "0 points" to "2 points." For example, each image data item is assigned an evaluation value of "0 points," "1 point," or "2 points."

[0046] In step S002, training data (image data) is generated in which the above test results are labeled as correct answers. Machine learning is performed based on this training data to generate image-based assessment algorithms A1, A2, A3, A4, A5, A6, A7, and A8 corresponding to the OHAT evaluation items (eight items) (step S003).

[0047] For example, a convolutional neural network (CNN) can be used for the machine learning. Alternatively, a method using a neural network such as a multi-layer neural network (MLP), a long short-term memory (LSTM), a gated recurrent unit (GRU), a graph neural network (GNN), or a transformer may be used for the machine learning.

[0048] The image-based assessment algorithms A1 to A8 are algorithms for assessing the degree of deterioration of the oral cavity condition for each evaluation item of the OHAT when image data is provided.

[0049] This makes it possible to quantify the degree of deterioration of oral condition for each OHAT evaluation item (eight items) and display the image-based OHAT evaluation item indices Ii1, Ii2, Ii3, Ii4, Ii5, Ii6, Ii7, and Ii8. The control unit 30 stores the image-based OHAT evaluation item indices Ii1, Ii2, Ii3, Ii4, Ii5, Ii6, Ii7, and Ii8 in the memory area 24.

[0050] It should be noted that instead of outputting the evaluation item indices I1 to I8 using the image-based judgment algorithms A1 to A8 described above, the evaluation item indices I1 to I8 may be output using a single judgment model (multi-class classification model).

[0051] In addition to this determination, the reliability of the determination may be quantified, or the degree of certainty that the disease is present may be indicated by a graded evaluation such as high, medium, or low.

[0052] Alternatively, a positive or negative judgment may be made, for example, as "Judgment: very poor oral condition," "Judgment: poor oral condition," "Judgment: normal oral condition," "Judgment: good oral condition," or "Judgment: very good oral condition." In addition to or in addition to this judgment, the reliability of the judgment may be quantified or indicated by a scale such as high, medium, or low. Alternatively, the degree of certainty that the oral condition is a disease may be indicated by a scale such as high, medium, or low.

[0053] The images used to generate the training data may be either still images or moving images. Furthermore, the training data may be generated using images captured at multiple positions or multiple directions, or may be images captured at the same position or direction. For example, multiple types of images capturing one side and the other side of the oral cavity may be used to generate the training data.

[0054] Furthermore, for the denture image, for example, an image of only the denture taken in a state where it has been removed from the oral cavity can be used.

[0055] Furthermore, when evaluating dentures, an AI algorithm for detecting abnormalities may be used as an oral condition evaluation model. For example, an autoencoder that has been trained on various images of normal dentures can be used to detect denture abnormalities. Since dentures have a fixed shape, deviations from this shape (such as chips or cracks) can be detected as abnormalities.

[0056] In addition to the above-mentioned Auto Encoder, other anomaly detection methods include methods that measure the degree of anomaly based on the distance from the distribution of features obtained from normal images, such as SPADE, Padim, and PatchCore; methods that use restoration errors using dictionaries obtained from normal images, such as Sparse Modeling; and Efficient-GAN, a method of anomaly detection based on restoration errors using GANs (Generative Adversarial Networks). For example, any of these may be used.

[0057] In addition, in this embodiment, an evaluation item index for oral odor is also generated.

[0058] In this case, for example, as shown in Figure 6, a medical institution acquires odor information using an odor information acquisition device 3, and also acquires associated correct labels indicating a deterioration in oral condition for each evaluation item of the OHAT (step S001a). Correct labels are assigned based on, for example, the results of dental examinations, periodontal disease tests, saliva tests, bacterial tests, PCR tests, and virus isolation and culture tests that the patient undergoes, as well as the findings of a doctor or dentist. Although PCR tests and virus isolation and culture tests take several weeks to produce results, they are highly accurate and therefore suitable as correct labels.

[0059] In step S002a, training data (smell data) is generated in which the test results are labeled as correct answers. Machine learning is performed based on this training data to generate an odor-based determination algorithm B7 (step S003a).

[0060] Machine learning of odor information can be performed using neural networks or gradient boosting methods. Alternatively, the machine learning can be performed using a gradient boosting decision tree (GBDT) such as LightGBM (Light Gradient Boosting Machine), XGBoost, or CatBoost.

[0061] The odor-based assessment algorithm B7 outputs an odor-based OHAT evaluation item index Is7 for the evaluation item "(7) Oral Cleaning" of the OHAT in response to input of actual patient odor information. The control unit 30 stores the odor-based OHAT evaluation item index Is7 in the memory area 24.

[0062] In addition to this determination, the reliability of the determination may be quantified, or the degree of certainty that the disease is present may be indicated by a graded evaluation such as high, medium, or low.

[0063] Alternatively, a positive or negative judgment may be made, for example, as "Judgment: very poor oral condition," "Judgment: poor oral condition," "Judgment: normal oral condition," "Judgment: good oral condition," or "Judgment: very good oral condition." In addition to or in addition to this judgment, the reliability of the judgment may be quantified or indicated by a scale such as high, medium, or low. Alternatively, the degree of certainty that the oral condition is a disease may be indicated by a scale such as high, medium, or low.

[0064] In addition, in this embodiment, evaluation item indices are also generated for medical interviews obtained from medical professionals, including doctors and dentists.

[0065] In this case, for example, as shown in Fig. 7, the medical interview information and the associated correct answer labels indicating the deterioration of the oral condition for each item of the OHAT are obtained (step S001b). The correct answer labels are assigned based on the medical interview information, findings, and patient attribute information obtained by a medical professional or the like checking the patient's oral cavity. The correct answer labels may be acquired by multiple medical professionals or the like assigning correct answer labels to one patient.

[0066] A method for acquiring medical interview information will be described using an example.

[0067] For example, regarding toothache, a medical professional or the like may ask a patient, care recipient, or other subject, "Does your tooth hurt?" to confirm the presence or absence of toothache and the location of the toothache. If the medical professional or the like touches the subject's teeth and the subject shows no particular signs, the evaluation score is determined to be 0, indicating that there are no verbal or physical signs of pain. On the other hand, if the medical professional or the like touches the subject's teeth and the subject shows verbal or behavioral signs such as a facial twitch, lip biting, or aggressive behavior, or if the subject shows signs such as not eating in daily life at a nursing home or day care center, the evaluation score is determined to be 1 (change). Furthermore, if the subject shows swelling of the cheeks or gums, broken teeth, ulcers, subgingival abscess, etc. in addition to verbal and behavioral signs, the evaluation score is determined to be 2 (pathological). Note that swelling of the cheeks or gums, broken teeth, ulcers, subgingival abscess, etc. may also be obtained as interview information from image data.

[0068] Regarding saliva, the medical professional will instruct the subject to "open their mouth slightly" to check the condition of the saliva in the oral cavity. If it is found to be moist and serous, a score of 0 will be assigned. On the other hand, if there is a small amount of saliva and the mucous membrane is sticky or the saliva is foamy, a score of 1 (change) will be assigned. If the saliva is dried up or the care recipient claims to have a very dry mouth, a score of 2 (pathological) will be assigned. Note that the stickiness of the mucous membrane can also be measured using a viscometer, for example.

[0069] Regarding the lips, the healthcare professional carefully examines the subject's lips. The healthcare professional instructs the subject to gently open the corners of their mouth, saying, "Please open your mouth slightly." If the result is a normal, moist, pink appearance, the evaluation score is 0 points. If dryness or cracks are observed at the corners of the mouth, the evaluation score is 1 point (change). If ulcerative lesions or associated bleeding are observed, the evaluation score is 2 points (pathological).

[0070] For the tongue, a healthcare professional or other professional instructs the subject to "open their mouth and stick out their tongue." They carefully compare the visible condition of the tongue with a reference photograph displaying the tongue condition and evaluation score, and determine an evaluation score for the "tongue" evaluation item. If necessary, the tongue may be touched. If the result is a normal, moist, pink appearance, the evaluation score is 0 points. If tongue coating is observed, the evaluation score is 1 point (change) regardless of the amount, nature, color, etc. If ulcerative lesions or associated bleeding are observed, the evaluation score is 2 points (pathological).

[0071] Next, training data is generated in which the test results are labeled as correct answers (step S002b). Machine learning is performed based on this training data to generate interview-based assessment algorithms C1, C2, C3, C4, C5, C6, C7, and C8 corresponding to the OHAT evaluation items (eight items) (step S003b).

[0072] The machine learning of the medical interview information can be performed using a neural network or a gradient boosting method. Alternatively, the machine learning can be performed using a gradient boosting decision tree (GBDT) such as LightGBM (Light Gradient Boosting Machine), XGBoost, or CatBoost.

[0073] In addition to this determination, the reliability of the determination may be quantified, or the degree of certainty that the disease is present may be indicated by a graded evaluation such as high, medium, or low.

[0074] Alternatively, a positive or negative judgment may be made, for example, as "Judgment: very poor oral condition," "Judgment: poor oral condition," "Judgment: normal oral condition," "Judgment: good oral condition," or "Judgment: very good oral condition." In addition to or in addition to this judgment, the reliability of the judgment may be quantified or indicated by a scale such as high, medium, or low. Alternatively, the degree of certainty that the oral condition is a disease may be indicated by a scale such as high, medium, or low.

[0075] The interview-based determination algorithms C1, C2, C3, C4, C5, C6, C7, and C8 output interview-based OHAT evaluation item indices Im1, Im2, Im3, Im4, Im5, Im6, Im7, and Im8 for each of the eight OHAT evaluation items in response to input of interview information, findings, and attribute information about the lips, tongue, gums / mucosa, saliva, remaining teeth, dentures, oral hygiene, and toothache of an actual medical professional, etc. The control unit 30 stores the interview-based OHAT evaluation item indices Im1, Im2, Im3, Im4, Im5, Im6, Im7, and Im8 in the memory area 24.

[0076] In addition, instead of outputting the evaluation item indices Im1 to Im8 using the above-mentioned interview-based judgment algorithms C1 to C8, the evaluation item indices Im1 to Im8 may be output using a single judgment model (multi-class classification model).

[0077] In addition to this determination, the reliability of the determination may be quantified, or the degree of certainty that the disease is present may be indicated by a graded evaluation such as high, medium, or low.

[0078] Furthermore, when the image-based OHAT evaluation item indices Ii1, Ii2, Ii3, Ii4, Ii5, Ii6, Ii7, and Ii8, the odor-based OHAT evaluation item index Is7, and the interview-based OHAT evaluation item indexes Im1, Im2, Im3, Im4, Im5, Im6, Im7, and Im8 are input, the control unit 30 may read each evaluation item index from the storage area 24 and perform ensemble processing. As an example of the ensemble processing, the obtained image-based OHAT evaluation item indexes, odor-based OHAT evaluation item indexes, and interview-based OHAT evaluation item indexes are input to a ridge regression model, and a final OHAT evaluation value in which each evaluation item index is ensembled is obtained as an evaluation result for the oral condition.

[0079] In addition to the ridge regression model described above, the ensemble method may also be, for example, a process for obtaining an average value, a process for obtaining a maximum value, a process for obtaining a minimum value, a weighted summation process, or a process using other machine learning methods such as bagging, boosting, stacking, lasso regression, or linear regression.

[0080] The ensemble processing is performed in multiple stages. The ensemble processing may combine all three types of judgment algorithms with different inputs (image-based judgment algorithm, odor-based judgment algorithm, and interview-based judgment algorithm), or any two of them. When combining multiple judgment algorithms, first, judgment algorithms with different inputs for each evaluation item are ensembled to create an algorithm for judging each evaluation item. Next, the algorithms for judging each of the ensembled evaluation items are further ensembled to obtain a comprehensive OHAT evaluation value that takes all evaluation items into consideration. By ensembling an image-based judgment algorithm corresponding to each OHAT evaluation item with an odor-based judgment algorithm corresponding to each OHAT evaluation item and / or an interview-based judgment algorithm corresponding to each OHAT evaluation item, each OHAT evaluation item can be judged more accurately than by each judgment algorithm alone.

[0081] As described above, the terminal device 1 may be a mobile information terminal such as a smartphone or tablet PC, and includes a touch panel display and a camera. The touch panel display is a device that combines an input function for accepting user input by touching the screen with a finger or the like and an output function for displaying an image on the screen. The housing of the terminal device 1 is formed with a shooting window that directs light from a subject into the housing. The camera includes a photoelectric conversion element that receives light entering the housing through the shooting window via an optical system, and an A / D conversion circuit that converts an analog signal generated by the photoelectric conversion element through photoelectric conversion into a digital signal. The terminal device 1 generates image data representing the subject from the digital signal output by the A / D conversion circuit.

[0082] Dedicated application software (hereinafter referred to as "app") for the oral cavity condition evaluation support system 100 is installed in the terminal device 1. When the app is launched, the terminal device 1 captures an image of the oral cavity of a target in accordance with the launched application.

[0083] The terminal device 1 transmits the captured oral cavity image data to the evaluation device 5.

[0084] When a breath sensor is used as the information detection sensor, the smell information acquisition device 3 detects and acquires smell information by inserting the breath sensor into the oral cavity. Alternatively, a breath sensor may be attached to a dental instrument such as a penlight that illuminates the oral cavity, a dental drill, a probe, or a sickle scaler, and smell information may be detected when the instrument is inserted into the oral cavity.

[0085] The odor information acquisition device 3 transmits the acquired odor information to the evaluation device 5.

[0086] Next, the processing flow in this embodiment will be described with reference to Figures 8 and 9. Figure 8 shows the processing flow relating to the generation of an evaluation value, which is information about evaluations, and Figure 9 shows the processing flow relating to the registration of an evaluation value.

[0087] In the following explanation, an example will be given in which denture images are also used in addition to oral cavity images to generate evaluation values, but this is not limiting, and only oral cavity images may be used as images. Oral cavity images and denture images are collectively referred to as oral cavity images, etc.

[0088] Similarly to the images used to generate the training data, the target images used to generate the evaluation value may be either still images or moving images. Furthermore, the evaluation value may be generated using images captured at multiple positions or multiple directions, or may be images captured at the same position or direction. For example, multiple types of images capturing one side and the other side of the oral cavity may be used to generate the evaluation value.

[0089] Furthermore, for the image of the dentures, for example, an image of the dentures taken in a state where they have been removed from the oral cavity can be used.

[0090] First, the process flow for generating an evaluation value shown in FIG. 8 will be described.

[0091] First, in step S101, when a user operation (instruction to generate an evaluation value) to execute a program related to evaluation value generation is acquired, the control unit 30 executes the program to enable acceptance of medical interview information, oral cavity images, etc., and odor information.

[0092] In step S102 , the control unit 30 acquires the medical interview information input via the operation reception unit 22 and stores it in the oral cavity finding information storage unit 25 of the storage area 24 .

[0093] In step S103, the control unit 30 acquires data such as oral cavity images transmitted from the terminal device 1 via the communication unit 26 and stores them in the memory area 24. The control unit 30 stores the denture images in the oral cavity finding information memory unit 25 of the memory area 24.

[0094] In step S104 , the control unit 30 acquires the odor information transmitted from the odor information acquisition device 3 via the communication unit 26 and stores it in the oral cavity finding information storage unit 25 of the storage area 24 .

[0095] In step S105, the control unit 30 generates an evaluation value using an oral cavity condition evaluation model based on the oral cavity image, the medical interview information, and the odor information.

[0096] The evaluation values ​​generated may be for each evaluation item (lips, tongue, gums / mucosa, saliva, remaining teeth, dentures, oral hygiene, and toothache), or an overall evaluation value (for example, an average value) may be generated.

[0097] In step S106, the control unit 30 determines whether or not the physical information of the subject is stored in the database DB 1. Note that this determination can be made, for example, based on information (such as name) for identifying the subject (individual) included in the medical interview information.

[0098] If the physical information is stored in database DB1, in step S107, control unit 30 acquires the physical information from database DB1 via communication unit 26.

[0099] In step S108, the control unit 30 causes the display unit 20 to display the obtained evaluation value and physical information.

[0100] On the other hand, if it is determined in step S106 that the physical information is not stored in database DB1, the process proceeds to step S109, where control unit 30 causes display unit 20 to display only the obtained evaluation value.

[0101] Next, the process flow for registering evaluation values ​​shown in FIG. 9 will be described.

[0102] First, in step S201, when a user operation (registration instruction) to execute a program for registering an evaluation value is received, the control unit 30 executes the program. In step S202, the control unit 30 determines whether physical information is stored in the database DB1. Note that this determination can be made based on information (such as name) for identifying the subject (individual) included in the medical interview information.

[0103] If it is determined that the physical information is not stored in the database DB1, the control unit 30 ends the process. In this case, the control unit 30 may cause the display unit 20 to display a message notifying the user that the physical information is not stored.

[0104] On the other hand, if it is determined that the physical information is stored in database DB1, control unit 30 associates the evaluation value with the physical information and stores the association in database DB1.

[0105] As described above, according to this embodiment, it is possible to obtain more appropriate evaluation results for the state of the oral cavity.

[0106] The present invention is not limited to the above-described embodiment, and other aspects are of course possible.

[0107] For example, in the above-described embodiment, an oral condition evaluation model is obtained by machine learning using oral cavity images, medical interview information, and odor information, and the control unit 30 generates an evaluation value, which is information about the evaluation result, using the oral cavity images, medical interview information, and odor information of the subject. However, without being limited to this, for example, an oral condition evaluation model may be constructed using oral cavity images and odor information, and information about the evaluation result may be generated using this oral condition evaluation model.

[0108] In the above-described embodiment, odor-based evaluation item indices are output for oral hygiene in the OHAT. In another embodiment, odor-based evaluation item indices may be output for other evaluation items in addition to or instead of oral hygiene in the OHAT. For example, odor-based evaluation item indices may be output for one or more of the other evaluation items in the OHAT (lips, sublingual, gums / mucosa, saliva, remaining teeth, dentures, toothache).

[0109] Also, for example, in the above-described embodiment, it is determined whether or not physical information is stored in database DB1, and if physical information is stored, information about the evaluation results and the physical information are displayed on display unit 20, and the information about the evaluation results and the physical information are associated and stored in database DB1.

[0110] In another aspect, it may be configured to determine whether or not the target's insurance information is stored in database DB1 or database DB2 (hereinafter also referred to as database DB1 / DB2) instead of or together with the physical information, and if the target's insurance information is stored, to associate information about the evaluation result with the insurance information and display it on display unit 20, and also to associate the information about the evaluation result with the insurance information and store it in database DB1 / DB2. In this aspect, the auxiliary information storage unit is configured to include the above-mentioned database DB1 / DB2.

[0111] In this specification, insurance information refers to information about a patient's enrollment in medical insurance (a system that reduces the burden of medical expenses) that applies to the costs of examinations and treatments for illnesses, injuries, etc. at medical institutions, and includes information indicating whether the patient has enrolled in medical insurance, the type of insurance, etc.

[0112] This embodiment or other embodiments may be used to assess each OHAT evaluation item based on an image and generate, for example, a notification document based on the assessment results. Such a processing system generates an oral condition evaluation result (including an evaluation value) based on, for example, an image of the oral cavity captured by a photographing device. Then, based on the information about the evaluation result, it generates information indicating whether the disease is eligible for insurance coverage, or a notification document required for various institutions such as the national government, local government, school, or workplace. Therefore, users (e.g., medical professionals) who need to prepare such notification documents can more efficiently prepare notification documents. In other words, it is expected that documents related to the oral health coordination enhancement surcharge can be prepared more efficiently.

[0113] Furthermore, other evaluation items may be included in addition to the evaluation items of the OHAT.

[0114] For example, the degree of mouth opening may be measured as one of the evaluation items. Specifically, the upper and lower teeth or lips are detected using object detection AI, and the distance between the upper and lower teeth is measured. The magnification of the image is calculated based on the size of the lips and teeth, and the degree of mouth opening is measured.

[0115] Examples of object detection AI algorithms include YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), DETR (End-to-End Object Detection with Transformers), and segmentation models such as UNet and Mask R-CNN.

[0116] In addition, other evaluation items may be used instead of the evaluation items of the OHAT, and are not particularly limited. For example, an oral condition evaluation model for the evaluation items as shown in Figure 10 may be generated, and information on the evaluation results regarding the oral condition of the subject may be generated using the oral condition evaluation model.

[0117] Furthermore, evaluation items may include swallowing function, chewing function, tongue movement, lip movement, and the moist state of the oral cavity.

Claims

1. An oral condition evaluation support system comprising: an oral image acquisition unit that acquires oral images of a subject; an oral finding information storage unit that stores oral finding information of the subject; and an evaluation unit that generates information on the evaluation results regarding the oral condition of the subject using an oral condition evaluation model obtained by machine learning based on the oral condition and its evaluation results, based on the oral images acquired by the oral image acquisition unit and the oral finding information.

2. An oral condition evaluation support system as described in claim 1, further comprising an odor information acquisition unit that acquires odor information of the subject's oral cavity obtained by a sensor, wherein the oral finding information storage unit stores the odor information as the oral finding information, and wherein the oral condition evaluation model is obtained by machine learning based on oral images and their evaluation results, as well as machine learning based on oral odors and their evaluation results.

3. An oral condition evaluation support system as described in claim 1, further comprising a denture image acquisition unit that acquires denture images obtained by photographing the target denture, wherein the oral finding information storage unit stores information about the condition of the denture based on the denture images as the oral finding information, and the oral condition evaluation model is further obtained by machine learning based on the condition of the denture and the evaluation results thereof.

4. An oral condition evaluation support system as described in claim 1, further comprising a medical interview information acquisition unit that acquires medical interview information regarding the medical interview results of the subject, wherein the oral finding information storage unit stores the medical interview information as the oral finding information, and wherein the oral condition evaluation model is further obtained by machine learning based on the medical interview results and their evaluation results.

5. An oral condition evaluation support system as described in claim 1, further comprising a denture image acquisition unit that acquires denture images obtained by photographing the denture of the subject, and a medical interview information acquisition unit that acquires medical interview information regarding the results of the medical interview of the subject, wherein the oral finding information storage unit stores information about the denture condition based on the denture images and the medical interview information as the oral finding information, and the oral condition evaluation model is further obtained by machine learning based on the denture condition and its evaluation results, and by machine learning based on the medical interview results and its evaluation results.

6. An oral condition evaluation support system as described in any one of claims 1 to 5, further comprising an auxiliary information storage unit that stores physical information about the subject's body and / or insurance information about the subject's insurance.

7. An oral cavity condition evaluation support system as described in claim 6, further comprising an information output unit that outputs information about the evaluation results generated by the evaluation unit in association with the physical information and / or the insurance information.

8. An oral condition evaluation support system as described in claim 7, further comprising an evaluation result registration unit that associates information about the evaluation results generated by the evaluation unit with the physical information and / or the insurance information and stores it in the auxiliary information storage unit.

Images