Tongue state estimation device, tongue state estimation method and program

Abstract

To provide a tongue state estimation device, tongue state estimation method and program which can easily evaluate the state of a tongue.SOLUTION: A tongue state estimation device 100 comprises an identification unit 31 and a tongue coat estimation unit 32. The identification unit 31 acquires an oral cavity image obtained by imaging an oral cavity by an imaging unit 10. The identification unit 31 identifies a tongue range image being an image of a range including the tongue from the oral cavity image. The tongue coat estimation unit 32 calculates a tongue coat feature amount for finding out the degree of adhesion of the tongue coat for each tongue section expressing the tongue in the plurality of sections obtained by dividing the tongue range image. The tongue coat estimation unit 32 calculates an estimation value of the degree of adhesion of the tongue coat for each tongue section on the basis of the calculated tongue coat feature amount.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a tongue state estimating device, a tongue state estimating method, and a program. [Background technology]

[0002] In recent years, the importance of evaluating the condition of the tongue has increased for the prevention or diagnosis of oral frailty, oral hypofunction, etc. For example, Patent Document 1 describes a tongue coating amount estimation system that includes a light-emitting element that emits light of a specific wavelength toward the tongue of a subject, a light-receiving element that receives fluorescence from the tongue, and an estimation device that estimates the amount of tongue coating based on the amount of fluorescence received by the light-receiving element. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-11736 Summary of the Invention [Problem to be solved by the invention]

[0004] The invention described in Patent Document 1 requires the preparation of a dedicated light-emitting element and light-receiving element for estimating the amount of tongue coating in addition to the estimation device, and there is room for improvement in terms of simply evaluating the condition of the tongue. Also, there are methods in which the evaluator visually evaluates the condition of the tongue without using a device, but relying on manual work every time the tongue condition is evaluated is cumbersome.

[0005] The present invention has been made in consideration of the above-described circumstances, and aims to provide a tongue state estimation device, a tongue state estimation method, and a program that can easily evaluate the state of the tongue. [Means for solving the problem]

[0006] In order to achieve the above object, a tongue state estimating device according to a first aspect of the present invention comprises: an identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured; a tongue coating estimation means for calculating tongue coating feature amounts for examining the degree of tongue coating adhesion for each tongue section representing the tongue out of a plurality of sections obtained by dividing the tongue area image, and for calculating an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature amounts. 、 The tongue coating estimation means has a trained neural network that has been subjected to machine learning using training data so that when the tongue area image is input, it calculates the tongue coating feature amount and calculates an estimated value of the degree of tongue coating adhesion.

[0008] The trained neural network of the tongue coating estimation means may, when the tongue area image is input, select the tongue section from a plurality of sections obtained by dividing the tongue area image.

[0009] In order to achieve the above object, a tongue state estimating device according to a second aspect of the present invention comprises: an identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured; a tongue coating estimation means for calculating tongue coating feature values for examining the degree of adhesion of tongue coating for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and for calculating an estimated value of the degree of adhesion of tongue coating for each tongue section based on the calculated tongue coating feature values, The degree of tongue coating is expressed in a plurality of predetermined stages, The tongue coating estimation means outputs an evaluation value expressed as a percentage of the sum of the estimated values of the degree of tongue coating calculated for each tongue section. do.

[0010] The tongue state estimating device includes: The tongue area image may further include a wetness estimation means for selecting a specific section from a plurality of sections obtained by dividing the image, calculating wetness features for examining the wetness of the tongue mucosa in the specific section, and calculating an estimated value of the wetness of the tongue mucosa based on the calculated wetness features.

[0011] In order to achieve the above object, a tongue state estimating device according to a third aspect of the present invention comprises: an identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured; a tongue coating estimation means for calculating tongue coating feature values for examining the degree of tongue coating adhesion for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and for calculating an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature values; a wetness estimating means for selecting a specific section from a plurality of sections obtained by dividing the tongue area image, calculating a wetness feature for examining the wetness of the tongue mucosa in the specific section, and calculating an estimated value of the wetness of the tongue mucosa based on the calculated wetness feature, the tongue area image identified by the identification means is rectangular, Each of the tongue coating estimation means and the wetness estimation means obtains a plurality of sections by dividing the tongue area image in a matrix. do.

[0012] The wetness estimation means may have a trained neural network that has been subjected to machine learning using training data so as to calculate the wetness feature and calculate an estimated value of the wetness of the tongue mucosa when the tongue area image is input.

[0013] The trained neural network included in the wetness estimation means may select a section corresponding to the vicinity of the tip of the tongue as the specific section from a plurality of sections obtained by dividing the tongue area image.

[0014] In order to achieve the above object, the present invention 4 The tongue state estimation method according to the aspect of A step of identifying a tongue area image that is an image of an area including the tongue from an oral cavity image that captures the oral cavity; Calculating tongue coating feature amounts for examining the degree of tongue coating adhesion for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and calculating an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature amounts. Tongue coating estimation Steps and 、 In the tongue coating estimation step, when the tongue area image is input, the tongue coating feature is calculated and an estimated value of the degree of adhesion of the tongue coating is calculated using a trained neural network that has been subjected to machine learning using training data. In order to achieve the above object, a tongue state estimating method according to a fifth aspect of the present invention comprises: A step of identifying a tongue area image that is an image of an area including the tongue from an oral cavity image that captures the oral cavity; a tongue coating estimation step of calculating tongue coating feature amounts for examining the degree of tongue coating adhesion for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and calculating an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature amounts, The degree of tongue coating is expressed in a plurality of predetermined stages, In the tongue coating estimation step, an evaluation value is output, which is the sum of the estimated values of the degree of tongue coating adhesion calculated for each tongue section, expressed as a percentage. In order to achieve the above object, a tongue state estimating method according to a sixth aspect of the present invention comprises: an identification step of identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image showing the oral cavity; a tongue coating estimation step of calculating tongue coating feature values for examining the degree of tongue coating adhesion for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and calculating an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature values; a wetness estimating step of selecting a specific section from a plurality of sections obtained by dividing the tongue area image, calculating a wetness feature for examining the wetness of the tongue mucosa in the specific section, and calculating an estimated value of the wetness of the tongue mucosa based on the calculated wetness feature, the tongue area image identified in the identifying step is rectangular, In each of the tongue coating estimation step and the wetness estimation step, the tongue area image is divided into a matrix to obtain a plurality of sections.

[0015] In order to achieve the above object, the present invention 7 The program related to the above points is Computer, an identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured; and a tongue coating estimation unit that calculates tongue coating feature amounts for examining the degree of tongue coating adhesion for each tongue section representing the tongue out of a plurality of sections obtained by dividing the tongue area image, and calculates an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature amounts. 、 The tongue coating estimation means has a trained neural network that has been subjected to machine learning using training data so that when the tongue area image is input, it calculates the tongue coating feature amount and calculates an estimated value of the degree of tongue coating adhesion. In order to achieve the above object, a program according to an eighth aspect of the present invention comprises: Computer, an identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured; a tongue coating feature amount for examining the degree of adhesion of tongue coating for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and a tongue coating estimation means for calculating an estimated value of the degree of adhesion of tongue coating for each tongue section based on the calculated tongue coating feature amount; The degree of tongue coating is expressed in a plurality of predetermined stages, The tongue coating estimation means outputs an evaluation value expressed as a percentage of the sum of the estimated values of the degree of tongue coating adhesion calculated for each tongue section. In order to achieve the above object, a program according to a ninth aspect of the present invention comprises: Computer, an identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured; a tongue coating estimation means for calculating tongue coating feature values for examining the degree of tongue coating adhesion for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and calculating an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature values; a wetness estimation unit that selects a specific section from a plurality of sections obtained by dividing the image of the tongue area, calculates a wetness feature for examining the wetness of the tongue mucosa in the specific section, and calculates an estimated value of the wetness of the tongue mucosa based on the calculated wetness feature; the tongue area image identified by the identification means is rectangular, Each of the tongue coating estimation means and the wetness estimation means obtains a plurality of sections by dividing the tongue area image in a matrix. [Effects of the Invention]

[0016] According to the present invention, the condition of the tongue can be easily evaluated. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram showing the configuration of a tongue state estimating device according to an embodiment of the present invention. [Figure 2] 10 is a flowchart showing a tongue state estimation process according to the embodiment. [Figure 3] Schematic diagram of an oral cavity image. [Figure 4] FIG. 10 is a schematic diagram of a tongue area image showing multiple compartments including a tongue compartment. [Figure 5] FIG. 10 is a schematic diagram of a tongue area image showing multiple sections including a specific section. [Figure 6] FIG. 10 is a diagram showing an image of the estimated tongue coating degree. [Figure 7]Figures showing an example of creating a tongue coating estimation unit using machine learning, where (a) shows the breakdown of labels assigned to each section in the training data, and (b) shows the breakdown of labels assigned to each section in the evaluation data. [Figure 8] A graph showing the correlation between the TCI estimated by the tongue coating estimation area and the TCI evaluated by the panelists. DETAILED DESCRIPTION OF THE INVENTION

[0018] An embodiment of the present invention will be described with reference to the drawings.

[0019] As shown in FIG. 1, the tongue state estimating device 100 includes an imaging unit 10, a display unit 20, and a control unit 30.

[0020] The photographing unit 10 is composed of, for example, a digital camera, and includes a photographing lens 11, an image sensor 12, and an image processing unit 13. The image sensor 12 is composed of a CCD (Charge Coupled Device), a CMOS (Complementary Metal-Oxide Semiconductor), or the like, and outputs data obtained by capturing an image formed by the photographing lens 11 to the image processing unit 13. The image processing unit 13 processes the data output from the image sensor 12 to convert it into image data and sends it to the control unit 30.

[0021] In this embodiment, the imaging unit 10 captures an image of the oral cavity of the subject, and sends data of an oral cavity image P1 capturing the oral cavity 4 to the control unit 30, as shown in Fig. 3. The oral cavity image P1 is an image captured by having the subject open their mouth so that the tongue 5 is visible.

[0022] The display unit 20 is composed of a liquid crystal display, an organic EL display, or the like, and displays images under the control of the control unit 30. For example, the display unit 20 displays the progress and estimation results of the tongue state estimation process described below.

[0023] The control unit 30 controls the operations of the photographing unit 10 and the display unit 20. A microcontroller implemented in an information terminal such as a personal computer, a smartphone, or a tablet terminal can be used as the control unit 30. At least one of the photographing unit 10 and the display unit 20 may be configured to be included in the information terminal. In other words, part or all of the tongue state estimating device 100 may be configured from the information terminal.

[0024] The control unit 30 executes a program stored in its built-in memory to perform the tongue state estimation process shown in Fig. 2. As functions for executing the tongue state estimation process, the control unit 30 includes a recognition unit 31, a tongue coating estimation unit 32, and a wetness estimation unit 33, as shown in Fig. 1. These functional units will be described below together with the tongue state estimation process. The control unit 30 starts the tongue state estimation process in response to instructions from an input device (not shown) comprising a keyboard, touch panel, etc.

[0025] (Identification unit 31) When the tongue state estimation process is started, the identification unit 31 identifies a tongue area image P2, which is an image of an area including the tongue 5, from the oral cavity image P1 (image data) acquired from the imaging unit 10, as shown in Fig. 3 (step S1). The tongue area image P2 in this embodiment is rectangular.

[0026] For example, the classification unit 31 has a trained neural network that has been subjected to machine learning, and classifies the tongue area image P2 based on the neural network. This trained neural network has been subjected to machine learning using training data so that, when the oral cavity image P1 is input, the trained neural network calculates tongue area features for examining the area including the tongue 5 and outputs the tongue area image P2.

[0027] As one example, the inventors of the present application used YOLO (You Only Look Once) v2 as an object detection algorithm for detecting the tongue 5, and performed machine learning on 208 photographs of an area including the tongue 5 from the oral cavity image P1 as training data to create a classification unit 31. The created classification unit 31 output 59 tongue area images P2 as evaluation data, and the classification accuracy was high at 58 / 59 images.

[0028] (Tongue coating estimated area 32) 4, the tongue coating estimation unit 32 calculates tongue coating feature amounts for examining the degree of tongue coating adhesion for each tongue section 6 representing the tongue 5 among the multiple sections obtained by dividing the tongue area image P2 (step S2). Then, the tongue coating estimation unit 32 calculates an estimate of the degree of tongue coating adhesion for each tongue section 6 based on the calculated tongue coating feature amounts (step S3).

[0029] For example, tongue coating estimation unit 32 calculates tongue coating features and calculates an estimated value of the degree of tongue coating adhesion as described above based on a trained neural network that has been subjected to machine learning. This trained neural network has been subjected to machine learning using training data so that when tongue area image P2 is input, it calculates tongue coating features for each tongue section 6 and calculates an estimated value of the degree of tongue coating adhesion for each tongue section 6 based on the calculated tongue coating features.

[0030] As an example, the inventors performed transfer learning on a trained neural network using training data obtained by dividing a rectangular tongue area image P2 into a 7 × 7 matrix and evaluating 49 sections obtained by five panelists. The convolutional neural network GoogleNet was used for transfer learning. Based on the panelists' evaluations, the sections of the tongue area image P2 were separated into tongue section 6, which was the subject of scoring, and non-target sections (colored sections in Figure 4), which were not the subject of scoring. The non-target sections consisted of margin section 6m, which indicated the tongue edge, and other sections 6o, which did not include the tongue 5. The tongue section 6 was then scored on a three-level scale: 0, 1, or 2. A higher score indicates a higher degree of tongue coating. Furthermore, the average score M of the scores given by the five panelists was calculated for each tongue section 6, and five-level labels were assigned based on the average score M. The label value L (0 to 4) is an estimate of the degree of tongue coating, and the larger the value, the higher the degree of tongue coating. The relationship between the label value L and the average value M is as follows: If 0≦M<0.4, L=0 If 0.4≦M<0.8, L=1 If 0.8≦M<1.2, L=2 If 1.2≦M<1.6, L=3 If 1.6≦M, L=4

[0031] The inventors created a tongue coating estimation unit 32 through machine learning using tongue area images P2 (157 images in total) in which each of the multiple sections was labeled with "0," "1," "2," "3," or "4" to indicate the tongue section 6, "margin" to indicate the margin section 6m, and "other" to indicate the other section 6o as training data (learning data). A total of 58 evaluation data were then output from the created tongue coating estimation unit 32. The breakdown of the labels assigned to each section in the training data and evaluation data is shown in Figures 7(a) and 7(b). The vertical axis in Figure 7(a) represents the number of sections in the 157 tongue area images P2 used as training data. Therefore, the total number of sections corresponding to each label in Figure 7(a) is 157 × 49 = 7693. Similarly, the vertical axis in Figure 7(b) represents the number of sections in the 58 tongue area images P2 used as evaluation data. Therefore, in FIG. 7(b), the total number of partitions corresponding to each label is 58×49=2842.

[0032] To investigate the accuracy of the evaluation data, the inventors calculated an evaluation value, expressed as a percentage, of the sum of estimated values of tongue coating adhesion calculated for each tongue section 6 of the tongue area image P2 used as evaluation data, using a method based on the TCI (Tongue Coating Index). The TCI is an index used for visually evaluating tongue coating, and the evaluation value is expressed as (sum of label values attached to each tongue section 6) / (maximum label value × number of tongue sections 6) × 100 [%].

[0033] The inventors calculated an evaluation value (hereinafter, "estimated TCI") for each of the 58 tongue area images P2 used as evaluation data, and also calculated an evaluation value (hereinafter, "panel-assessed TCI") based on the estimated tongue coating degree determined visually by panelists for each of the same 58 tongue area images P2. The correlation between the estimated TCI and the panelist-assessed TCI was then examined using intraclass correlation coefficients (ICC). The results are shown in Figure 8. As shown in the figure, the ICC(2,1) value, which represents inter-examiner reliability, was 0.798, indicating high reliability of the TCI between the two. Furthermore, the correlation coefficient r value was 0.814, indicating a strong correlation between the two. From the above, it can be seen that the tongue coating estimation unit 32 created by machine learning has high accuracy in estimating the tongue coating degree.

[0034] More specifically, the tongue coating estimation unit 32 of this embodiment constructed as described above performs the following process.

[0035] When tongue area image P5 is input, the trained neural network included in tongue coating estimation unit 32 calculates tongue coating features for each of the multiple sections obtained by dividing tongue area image P2. Based on the calculated tongue coating features, the neural network then labels each of the multiple sections with "0," "1," "2," "3," or "4" as an evaluation of tongue section 6, "margin" indicating margin section 6m, or "other" indicating other sections 6o. This series of processes by tongue coating estimation unit 32 includes at least the following steps. When the tongue area image P2 is input, a process of selecting the tongue area 6 from a plurality of areas obtained by dividing the tongue area image P2. A process of calculating tongue coating features for examining the degree of tongue coating adhesion for at least each of the tongue compartments 6 among the multiple compartments (step S2). A process of calculating an estimated value (label value) of the degree of tongue coating adhesion for each tongue section 6 based on the calculated tongue coating feature amount (step S3).

[0036] Furthermore, the tongue coating estimation unit 32 of this embodiment calculates a TCI evaluation value based on the estimated value (label value) of the tongue coating degree calculated for each tongue section 6. The estimated tongue coating degree is then displayed on the display unit 20 in the form of an image as shown in FIG. 6 (step S6). In the image shown in FIG. 6, each of the multiple labeled sections is assigned a color corresponding to the type of label. For example, sections labeled "margin" and "other" are displayed in black. Furthermore, sections labeled 0 to 4 as the evaluation of the tongue section 6 are displayed in a different color depending on the label value, so that the degree of tongue coating (i.e., the magnitude of the label value) can be intuitively grasped. Examples of colors displayed according to the label value are: 0 = green, 1 = yellow, 2 = orange, 3 = red, and 4 = purple. In the image shown in FIG. 6, TCI = 36.6% is written above the multiple sections as the TCI evaluation value.

[0037] (Wetness estimation unit 33) 5, the wetness estimation unit 33 selects a specific section 7 from a plurality of sections obtained by dividing the tongue area image P2, and calculates wetness features for examining the wetness of the tongue mucosa in the specific section 7 (step S4). Then, the wetness estimation unit 33 calculates an estimated value of the wetness of the tongue mucosa based on the calculated wetness features (step S5).

[0038] For example, the wetness estimation unit 33 calculates the wetness feature of the specific section 7 as described above based on a trained neural network that has been subjected to machine learning, and calculates an estimated value of the wetness of the tongue mucosa. This trained neural network has been subjected to machine learning using training data so that, when the tongue area image P2 is input, the trained neural network selects a section corresponding to the vicinity of the tongue tip as the specific section 7 from multiple sections obtained by dividing the tongue area image P2, calculates the wetness feature of the specific section 7, and calculates an estimated value of the wetness of the tongue mucosa. Note that the section corresponding to the vicinity of the tongue tip may be any section suitable for estimating the wetness of the tongue mucosa, and may be a section that includes the tongue tip itself, or may be a section that is one section or a predetermined number of sections away from the section.

[0039] For example, the training data can be data obtained by dividing a rectangular tongue area image P2 into a 7x7 matrix to obtain 49 sections, each of which is a section corresponding to 10 mm from the tip of the tongue toward the base of the tongue, which is the measurement site for an oral moisture meter, and selecting this section as a specific section 7. This training data includes the tongue area image P2 divided into multiple sections, a numerical value W of the oral moisture meter measured at an actual site corresponding to the specific section 7 selected from the multiple sections, and a label value Lw corresponding to the numerical value W as an evaluation method for dry mouth. The label value Lw (0 to 4) is an estimate of the wetness of the tongue mucosa, with a larger value indicating a higher wetness. The relationship between the label value Lw and the numerical value W of the oral moisture meter is, for example, as follows: If 0≦W<25, Lw=0 If 25≦W<27, Lw=1 If 27≦W<29, Lw=2 If 29≦W<30, Lw=3 If 30≦W, Lw=4

[0040] Using the data configured as above as training data, the wetness estimation unit 33 can be created by performing transfer learning to a trained neural network, similar to the tongue coating estimation unit 32. For transfer learning, for example, GoogLeNet, a convolutional neural network, can be used.

[0041] The wetness estimating unit 33 displays the estimated value of the wetness of the tongue mucosa calculated as above on the display unit 20 (step S6). At this time, the estimated value may be at least one of the label value Lw and the oral moisture meter value W. As in FIG. 6, the wetness estimating unit 33 may display an image on the display unit 20 in which the specific section 7 is colored in a stepwise manner according to the label value Lw or the oral moisture meter value W, so that the wetness of the tongue mucosa can be intuitively grasped.

[0042] If the number of sections obtained by dividing the tongue area image P2 in step S4 is the same as that in step S2, the wetness estimation unit 33 may perform the processes of steps S4 and S5 using the multiple sections divided in step S2. Also, the wetness estimation unit 33 may be configured to select a specific section 7 from the tongue section 6 selected by the tongue coating estimation unit 32 from the multiple sections of the tongue area image P2.

[0043] 2 shows an example in which the control unit 30 performs the process of estimating the degree of adhesion of tongue coating (steps S2 and S3) and the process of estimating the wetness of the tongue mucosa (steps S4 and S5) in parallel, but the order of the processes is arbitrary, and these processes may be performed sequentially. This concludes the description of the tongue state estimation device 100 and the tongue state estimation process of this embodiment.

[0044] The present invention is not limited to the above-described embodiments and drawings, and modifications (including the omission of components) can be made as appropriate within the scope of the present invention.

[0045] (Variation) The machine learning and deep learning algorithms used to create the identification unit 31, tongue coating estimation unit 32, and wetness estimation unit 33 are not limited to the examples in the above embodiments, and may be any algorithm.

[0046] The classification unit 31 may classify the tongue area image P2 from the oral cavity image P1 using known image analysis processing such as an edge extraction filter or pattern matching, without relying on a trained neural network. That is, the tongue area feature calculated by the classification unit 31 may be a difference value between a pixel of interest and an adjacent pixel adjacent to the pixel of interest, calculated in a calculation process using an edge extraction filter or the like. Furthermore, the tongue area image P2 classified by the classification unit 31 is not limited to a rectangular image, and may have any shape as long as it is an image of an area including the tongue 5. For example, the classification unit 31 may extract and classify the tongue area image P2 corresponding to the outline of the tongue 5 from the oral cavity image P1.

[0047] The wetness estimation unit 33 may select, as the specific section 7, a section corresponding to the vicinity of the tongue tip, determined by experiment or simulation, from the multiple sections of the tongue area image P2 identified by the identification unit 31. Furthermore, the specific section 7 is not limited to one section corresponding to the vicinity of the tongue tip among the multiple sections of the tongue area image P2, but may select multiple sections from the multiple sections. For example, the wetness estimation unit 33 may calculate wetness features for each tongue section 6, similar to the tongue coating estimation unit 32, and calculate an estimated value of the wetness of the tongue mucosa. In this case, all of the tongue sections 6 are the specific sections 7. Then, like the TCI, the wetness estimation unit 33 may calculate an evaluation value expressed as a percentage of the total wetness of the tongue mucosa calculated for each tongue section 6 of the tongue area image P2.

[0048] The number of compartments used by the tongue coating estimation unit 32 when selecting the tongue compartment 6 and the number of compartments used by the wetness estimation unit 33 when selecting the specific compartment 7 may be the same or different.

[0049] The tongue coating estimation unit 32 and the wetness estimation unit 33 may be configured without using a trained neural network. The tongue coating feature and the wetness feature calculated by the control unit 30 functioning as the tongue coating estimation unit 32 and the wetness estimation unit 33 may include the feature described below, regardless of whether a trained neural network is used.

[0050] The control unit 30 can calculate the following feature amounts 1 to 3 for the entire oral cavity image P1 and tongue area image P2 or a part of the image (region of interest). The region of interest is determined by 1) a region specified by a person, 2) a bounding box obtained by machine learning (region specified by machine learning), or 3) a segmentation obtained by a person or machine learning (region specified in pixel units). 1. Quantify color tone and use it as a feature (Color Tone A): Quantify color tone using RGB, Lab color space, HSV color space, or HSL color space, which are expressed as color systems. 2. Statistical analysis of the quantified color tones is performed and the features are used (Color Tone B-1): Calculate the color tone variation, the histogram of identical (equivalent, highly similar) color tones, and the area of identical (equivalent, highly similar) color tones. 3. The quantified color tone is analyzed on a pixel-by-pixel basis using surrounding pixels to determine the feature values (color tone B-1): the color contrast, gradient, and contour (edge) are calculated.

[0051] When tongue coating occurs, the following phenomena are observed: The surface of the tongue is pink, but the tongue coating can be white, yellowish white, or black. The surface of the tongue has small bumps called papillae. When covered with tongue coating, the papillae become invisible. Furthermore, when the moistness of the tongue mucosa decreases, the following phenomena are observed, for example. Dry mouth causes the tongue to turn red. The tongue surface may crack or become slippery (smooth). Mild dry mouth is characterized by stringy saliva, while moderate dry mouth is characterized by foamy saliva. In severe cases, saliva is lost from the tongue surface, causing it to lose its luster.

[0052] The color tone of the tongue surface can be expressed by color tone A. Color tone changes in parts of the tongue depending on the morphology of the tongue surface (papillae, cracks, smoothness) and the presence of substances other than the tongue on the tongue surface (tongue coating, saliva). Color tone B-1 can be perceived as a color change across the entire image, and color tone B-2 serves as an indicator for capturing the presence of the change itself (e.g., recognizing the position and number of papillae). It is considered that the color tones A, B-1, and B-2 of parts other than the tongue (skin and lips) are all significantly different from the tongue surface. Taking the above characteristics into consideration, the control unit 30 calculates tongue coating features and wetness features based on a program created through machine learning, experiments, or simulations, and can calculate an estimate of the degree of tongue coating adhesion and the wetness of the tongue mucosa.

[0053] The subject of tongue state estimation is not limited to a person (subject), but may be an animal.

[0054] The programs for executing the processes described above do not need to be stored in advance in the memory of the control unit 30, but may be distributed or provided on a removable recording medium. The programs may also be downloaded from another device connected to the tongue state estimation device 100. The tongue state estimation device 100 may also execute the processes according to the programs by exchanging various types of data with other devices via a telecommunications network or the like.

[0055] The effects of the tongue state estimating device 100 described above will be described below.

[0056] (effect) The tongue condition estimation device 100 includes an identification means (identification unit 31) for identifying a tongue area image P2 from an oral cavity image P1, and a tongue coating estimation means (tongue coating estimation unit 32) for calculating tongue coating features for each tongue section 6 out of multiple sections obtained by dividing the tongue area image P2, and calculating an estimate of the degree of tongue coating adhesion for each tongue section 6 based on the calculated tongue coating features. Furthermore, the tongue state estimation method using the tongue state estimation device 100 includes the steps of identifying a tongue area image P2 from an oral cavity image P1, calculating tongue coating features for each tongue section 6 among multiple sections obtained by dividing the tongue area image P2, and calculating an estimate of the degree of tongue coating adhesion for each tongue section 6 based on the calculated tongue coating features. The program executed by the control unit 30 causes the computer to function as a recognition unit and a tongue coating estimation unit. These configurations allow the tongue condition to be easily evaluated, since the degree of tongue coating can be estimated simply by image analysis of the captured oral cavity image P1. Furthermore, the tongue condition estimation device 100 can be configured simply by installing the above program in an information terminal such as a personal computer, smartphone, or tablet device, making it easy to provide the device. Furthermore, since the degree of tongue coating can be estimated for each tongue compartment 6, a more detailed distribution of the degree of tongue coating can be easily obtained, unlike TCI, which is based on the evaluator's visual inspection. Furthermore, since tongue condition evaluation can be automated, there is no variation in evaluation results depending on the evaluator, as occurs in visual evaluation.

[0057] The tongue coating estimation means may have a trained neural network that has undergone machine learning using training data so as to calculate tongue coating features and estimate the degree of tongue coating adhesion when the tongue area image P2 is input. This configuration can improve the accuracy of estimating the degree of tongue coating, as shown in Figure 8. Furthermore, by allowing the tongue coating estimation means having a neural network to perform additional learning as needed, the estimation accuracy can be further improved.

[0058] Specifically, when the tongue area image P2 is input, the trained neural network of the tongue coating estimation means may select tongue area 6 from a plurality of areas obtained by dividing the tongue area image P2.

[0059] Furthermore, the degree of tongue coating adhesion may be expressed in multiple predetermined stages, and the tongue coating estimation means may output an evaluation value expressed as a percentage of the sum of the estimated values of the degree of tongue coating adhesion calculated for each tongue section 6. This configuration allows evaluation of tongue coating on the entire tongue.

[0060] The tongue state estimation device 100 may further include a wetness estimation means (wetness estimation unit 33) that selects a specific section 7 from a plurality of sections obtained by dividing the tongue area image P2, calculates wetness features in the specific section 7, and calculates an estimate of the wetness of the tongue mucosa based on the calculated wetness features. This configuration allows for more detailed and simple evaluation of the tongue condition, since not only the degree of tongue coating but also the degree of moistness of the tongue mucosa can be estimated at once by simply analyzing the captured oral cavity image P1. Furthermore, the degree of moistness of the tongue mucosa can be easily evaluated without relying on a dedicated device such as an oral moisture meter.

[0061] The tongue area image P2 identified by the identification means may be rectangular, and each of the tongue coating estimation means and the wetness estimation means may obtain a plurality of sections by dividing the tongue area image P2 in a matrix. With this configuration, the area of the tongue with a margin added can be identified as the tongue area image P2, thereby preventing the loss of part of the tongue, which is the subject of evaluation, that can occur when extracting and identifying the outer shape of the tongue itself from the oral cavity image P1.

[0062] The wetness estimation means has a trained neural network that has undergone machine learning using training data so that when the tongue area image P2 is input, it calculates wetness features and calculates an estimate of the wetness of the tongue mucosa. This configuration can improve the accuracy of estimating the wetness of the tongue mucosa. In addition, by having the wetness estimation means having a neural network perform additional learning as needed, the estimation accuracy can be further improved.

[0063] Specifically, the trained neural network of the wetness estimation means selects a section corresponding to the vicinity of the tongue tip as the specific section 7 from a plurality of sections obtained by dividing the tongue area image P2.

[0064] In the above description, in order to facilitate understanding of the present invention, descriptions of well-known technical matters have been omitted as appropriate.

[0065] This invention allows various embodiments and modifications without departing from the broad spirit and scope of this invention. Furthermore, the above-described embodiments are intended to explain this invention and do not limit the scope of this invention. That is, the scope of this invention is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of the invention equivalent thereto are considered to be within the scope of this invention. [Explanation of symbols]

[0066] 100...Tongue state estimation device 10...imaging unit, 20...display unit, 30...control unit 31...identification section, 32...tongue coating estimation section, 33...moisture estimation section P1: Oral cavity image, P2: Tongue area image 4...oral cavity, 5...tongue 6...Lingual compartment, 6m...Margin compartment, 6o...Other compartment 7...Specific area

Claims

1. an identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured; a tongue coating estimation means for calculating tongue coating feature values for examining the degree of adhesion of tongue coating for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and for calculating an estimated value of the degree of adhesion of tongue coating for each tongue section based on the calculated tongue coating feature values, the tongue coating estimation means includes a trained neural network that has been subjected to machine learning using training data so as to calculate the tongue coating feature amount and calculate an estimated value of the degree of adhesion of the tongue coating when the tongue area image is input. Tongue state estimation device.

2. the trained neural network included in the tongue coating estimation means, when the tongue area image is input, selects the tongue section from a plurality of sections obtained by dividing the tongue area image; The tongue state estimating device according to claim 1 .

3. An identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image that captures the oral cavity; a tongue coating estimation means for calculating tongue coating feature values for examining the degree of adhesion of tongue coating for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and for calculating an estimated value of the degree of adhesion of tongue coating for each tongue section based on the calculated tongue coating feature values, The degree of tongue coating is expressed in a plurality of predetermined stages, the tongue coating estimation means outputs an evaluation value expressed as a percentage of the sum of the estimated values of the degree of tongue coating calculated for each tongue section. Tongue state estimation device.

4. a wetness estimating means for selecting a specific section from a plurality of sections obtained by dividing the image of the tongue area, calculating a wetness feature for examining the wetness of the tongue mucosa in the specific section, and calculating an estimated value of the wetness of the tongue mucosa based on the calculated wetness feature; The tongue state estimating device according to any one of claims 1 to 3.

5. An identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image that captures the oral cavity; a tongue coating estimation means for calculating tongue coating feature values for examining the degree of tongue coating adhesion for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and for calculating an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature values; a wetness estimating means for selecting a specific section from a plurality of sections obtained by dividing the tongue area image, calculating a wetness feature for examining the wetness of the tongue mucosa in the specific section, and calculating an estimated value of the wetness of the tongue mucosa based on the calculated wetness feature, the tongue area image identified by the identification means is rectangular, each of the tongue coating estimation means and the wetness estimation means obtains a plurality of sections by dividing the tongue area image in a matrix; Tongue state estimation device.

6. the wetness estimation means includes a trained neural network that has been subjected to machine learning using training data so as to calculate the wetness feature amount and calculate an estimated value of the wetness of the tongue mucosa when the tongue area image is input. The tongue state estimating device according to claim 4 or 5.

7. the trained neural network included in the wetness estimation means selects, as the specific section, a section corresponding to the vicinity of the tongue tip from a plurality of sections obtained by dividing the tongue area image; The tongue state estimating device according to claim 6.

8. A step of identifying a tongue area image that is an image of an area including the tongue from an oral cavity image that captures the oral cavity; a tongue coating estimation step of calculating tongue coating feature amounts for examining the degree of tongue coating adhesion for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and calculating an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature amounts, In the tongue coating estimation step, when the tongue area image is input, the tongue coating feature amount is calculated and an estimated value of the degree of adhesion of the tongue coating is calculated using a trained neural network that has been subjected to machine learning using training data. Tongue state estimation method.

9. A step of identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image that captures the oral cavity; a tongue coating estimation step of calculating tongue coating feature amounts for examining the degree of tongue coating adhesion for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and calculating an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature amounts, The degree of tongue coating is expressed in a plurality of predetermined stages, In the tongue coating estimation step, an evaluation value is output, which is a percentage of the sum of the estimated values of the degree of tongue coating calculated for each tongue section. Tongue state estimation method.

10. An identification step of identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured; a tongue coating estimation step of calculating tongue coating feature values for examining the degree of tongue coating adhesion for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and calculating an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature values; a wetness estimating step of selecting a specific section from a plurality of sections obtained by dividing the tongue area image, calculating a wetness feature for examining the wetness of the tongue mucosa in the specific section, and calculating an estimated value of the wetness of the tongue mucosa based on the calculated wetness feature, the tongue area image identified in the identifying step is rectangular, In each of the tongue coating estimation step and the wetness estimation step, the tongue area image is divided into a matrix to obtain a plurality of sections. Tongue state estimation method.

11. Computer, an identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured; a tongue coating feature amount for examining the degree of adhesion of tongue coating for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and a tongue coating estimation means for calculating an estimated value of the degree of adhesion of tongue coating for each tongue section based on the calculated tongue coating feature amount; the tongue coating estimation means includes a trained neural network that has been subjected to machine learning using training data so as to calculate the tongue coating feature amount and calculate an estimated value of the degree of adhesion of the tongue coating when the tongue area image is input. program.

12. A computer, an identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured; a tongue coating feature amount for examining the degree of adhesion of tongue coating for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and a tongue coating estimation means for calculating an estimated value of the degree of adhesion of tongue coating for each tongue section based on the calculated tongue coating feature amount; The degree of tongue coating is expressed in a plurality of predetermined stages, the tongue coating estimation means outputs an evaluation value expressed as a percentage of the sum of the estimated values of the degree of tongue coating calculated for each tongue section. program.

13. A computer, an identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured; a tongue coating estimation means for calculating tongue coating feature values for examining the degree of tongue coating adhesion for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and calculating an estimated value of the degree of tongue coating adhesion for each tongue section based on the calculated tongue coating feature values; a wetness estimation unit that selects a specific section from a plurality of sections obtained by dividing the image of the tongue area, calculates a wetness feature for examining the wetness of the tongue mucosa in the specific section, and calculates an estimated value of the wetness of the tongue mucosa based on the calculated wetness feature; the tongue area image identified by the identification means is rectangular, each of the tongue coating estimation means and the wetness estimation means obtains a plurality of sections by dividing the tongue area image in a matrix; program.

Images