Tongue evaluation method, tongue evaluation system and tongue evaluation program

Abstract

To accurately evaluate a state of a tongue.SOLUTION: A tongue evaluation method in one embodiment is a tongue evaluation method executed by a tongue evaluation system, and comprises: an acquisition step of acquiring a face image obtained by imaging a face of a user with his / her tongue out; a tongue detection step of detecting the tongue from the face image, using a tongue detection model; a trimming step of trimming a partial image including the tongue from the face image on the basis of a detection result of the tongue; and an evaluation step of using the tongue evaluation model to evaluate a degree of swelling of the tongue from the partial image.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to a tongue evaluation method, a tongue evaluation system, and a tongue evaluation program. [Background technology]

[0002] Conventionally, the state of the tongue has been estimated from the amount of tongue coating attached to the tongue. For example, cited document 1 describes a tongue state estimation device including an identification means for identifying a tongue area image, which is an image of an area including the tongue, from an oral cavity image in which the oral cavity is captured, and a tongue coating estimation means for calculating a tongue coating feature amount for examining the degree of adhesion of tongue coating for each tongue section representing the tongue among a plurality of sections obtained by dividing the tongue area image, and calculating an estimate of the degree of adhesion of the tongue coating for each tongue section based on the calculated tongue coating feature amount. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-49604 A Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional technology, only the amount of tongue coating is evaluated, which poses a problem in terms of improving the accuracy of evaluating the condition of the tongue.

[0005] The present invention has been made in consideration of the above problems, and has an object to accurately evaluate the condition of the tongue. [Means for solving the problem]

[0006] A tongue evaluation method according to one embodiment is a tongue evaluation method executed by a tongue evaluation system, and includes an acquisition step of acquiring a facial image of a face with the tongue out, a tongue detection step of detecting the tongue from the facial image using a tongue detection model, a trimming step of trimming a partial image including the tongue from the facial image based on the tongue detection result, and an evaluation step of evaluating the degree of tongue swelling from the partial image using the tongue evaluation model. Effect of the Invention

[0007] According to one embodiment, it is possible to suppress accurate evaluation of the tongue state. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a tongue evaluation system 1000. [Diagram 2] 1 is a diagram illustrating an example of a hardware configuration of a tongue evaluation device 1. FIG. [Diagram 3] 2 is a diagram illustrating an example of a hardware configuration of a user terminal 2. FIG. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of the tongue evaluation device 1. [Diagram 5] FIG. 2 is a diagram showing an example of a face image 121. [Figure 6] FIG. 13 is a diagram showing an example of a partial image 123. [Figure 7] FIG. 13 is a diagram showing an example of a tongue detection result. [Figure 8] FIG. 13 is a diagram showing an example of a detection result of tongue coating. [Figure 9] FIG. 2 is a diagram illustrating an example of a functional configuration of a user terminal 2. [Figure 10] 13 is a flowchart showing an example of processing executed by the tongue evaluation system 1000. [Figure 11] FIG. 13 is a diagram showing an example of an evaluation result screen SC. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, each embodiment of the present invention will be described with reference to the accompanying drawings. In the description of the specification and drawings of each embodiment, components having substantially the same functional configuration are designated by the same reference numerals, and redundant description will be omitted.

[0010] <System configuration> First, an overview of the tongue evaluation system 1000 according to this embodiment will be described. The tongue evaluation system 1000 is an information processing system that evaluates the state of the tongue of a subject based on a facial image of the subject sticking out his / her tongue. The subject is a person whose tongue state is evaluated. The subject is, for example, an elderly person, but is not limited to this.

[0011] Fig. 1 is a diagram showing an example of the configuration of a tongue evaluation system 1000. As shown in Fig. 1, the tongue evaluation system 1000 includes a tongue evaluation device 1 and a user terminal 2, which are communicatively connected to each other via a network N. The network N is, for example, a wired LAN (Local Area Network), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination of these. In the example of Fig. 1, the tongue evaluation system 1000 includes one tongue evaluation device 1 and one user terminal 2, but may include multiple of each.

[0012] The tongue evaluation device 1 is an information processing device that evaluates the state of the tongue of a subject based on a face image. The tongue evaluation device 1 is, for example, a personal computer (PC), a smartphone, a tablet terminal, a server device, or a microcomputer, but is not limited to these.

[0013] The user terminal 2 is an information processing device used by a user of the tongue evaluation system 1000. The user terminal 2 transmits a face image to the tongue evaluation device 1 and displays the evaluation result of the tongue state of the subject evaluated by the tongue evaluation device 1. The user terminal 2 is, for example, a PC, a smartphone, or a tablet terminal, but is not limited to these.

[0014] In the following, the user is a photographer who takes a face image, and a case will be described as an example in which the face image is taken by the user terminal 2. However, the user does not have to be the photographer. In this case, the user terminal 2 may obtain a face image taken in advance from an external device via the network N.

[0015] <Hardware configuration of tongue evaluation device 1> Next, a description will be given of a hardware configuration of the tongue evaluation device 1. Fig. 2 is a diagram showing an example of a hardware configuration of the tongue evaluation device 1. As shown in Fig. 2, the tongue evaluation device 1 includes a processor 101, a memory 102, a storage 103, a communication I / F 104, an input device 105, an output device 106, and a drive device 107, which are connected to each other via a bus B1.

[0016] The processor 101 loads into the memory 102 various programs including an OS (Operating System) and a tongue evaluation program stored in the storage 103 and executes them, thereby controlling each component of the tongue evaluation device 1 and realizing the functions of the tongue evaluation device 1. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or a DSP (Digital Signal Processor), but is not limited thereto.

[0017] The memory 102 is, for example, a Read Only Memory (ROM), a Random Access Memory (RAM), or a combination thereof. The ROM is, for example, a Programmable ROM (PROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), or a combination thereof. The RAM is, for example, but is not limited to, a Dynamic RAM (DRAM) or a Static RAM (SRAM).

[0018] The storage 103 stores various programs and data including an OS and a tongue evaluation program. The storage 103 is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a storage class memory (SCM), but is not limited to these.

[0019] The communication I / F 104 is an interface for connecting the tongue evaluation device 1 to an external device via the network N and controlling communication. The communication I / F 104 is, for example, Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), or Ethernet (registered trademark), but is not limited thereto.

[0020] The input device 105 is a device for inputting information to the tongue evaluation device 1. The input device 105 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, an imaging device (camera), various sensors, or an operation button, but is not limited to these.

[0021] The output device 106 is a device for outputting information from the tongue evaluation device 1. The output device 106 is, for example, a display device, a projector, a printer, a speaker, or a vibrator, but is not limited to these.

[0022] The drive device 107 is a device that reads and writes data to the recording medium 108. The drive device 107 is, for example, but not limited to, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or an SD card reader. The recording medium 108 is, for example, but not limited to, a CD (Compact Disc), a DVD (Digital Versatile Disc), a FD (Floppy Disk), an MO (Magneto-Optical disk), a BD (Blu-ray (registered trademark) Disc), a USB (registered trademark) memory, or an SD card.

[0023] In this embodiment, the tongue evaluation program may be written into the memory 102 or the storage 103 during the manufacturing stage of the tongue evaluation device 1, or may be provided to the tongue evaluation device 1 via the network N, or may be provided to the tongue evaluation device 1 via a non-transitory computer-readable recording medium such as a recording medium 108.

[0024] <Hardware configuration of user terminal 2> Next, a description will be given of the hardware configuration of the user terminal 2. Fig. 3 is a diagram showing an example of the hardware configuration of the user terminal 2. As shown in Fig. 3, the user terminal 2 includes a processor 201, a memory 202, a storage 203, a communication I / F 204, an input device 205, and an output device 206, which are connected to each other via a bus B2.

[0025] The processor 201 controls each component of the user terminal 2 and realizes the functions of the user terminal 2 by expanding into the memory 202 and executing various programs including the OS and the tongue evaluation program stored in the storage 203. The processor 201 is, for example, a CPU, an MPU, a GPU, an ASIC, or a DSP, but is not limited to these.

[0026] The memory 202 may be, for example, a ROM, a RAM, or a combination thereof. The ROM may be, for example, a PROM, an EPROM, an EEPROM, or a combination thereof. The RAM may be, for example, but is not limited to, a DRAM or an SRAM.

[0027] The storage 203 stores various programs and data including the OS and the tongue evaluation program. The storage 203 is, for example, a flash memory, a HDD, an SSD, or an SCM, but is not limited to these.

[0028] The communication I / F 204 is an interface for connecting the user terminal 2 to an external device via the network N and controlling communication. The communication I / F 204 is, for example, Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), or Ethernet (registered trademark), but is not limited to these.

[0029] The input device 205 is a device for inputting information to the user terminal 2. The input device 205 is, for example, but not limited to, a mouse, a keyboard, a touch panel, a microphone, a scanner, an image capturing device (camera), various sensors, or an operation button. The user terminal 2 includes an image capturing device 205C as the input device 205.

[0030] The output device 206 is a device for outputting information from the user terminal 2. The output device 206 is, for example, but is not limited to, a display device (display), a projector, a printer, a speaker, or a vibrator. The user terminal 2 includes a display device 206D as the output device 206.

[0031] The drive device 207 is a device that reads and writes data to the recording medium 208. The drive device 207 is, for example, but not limited to, a magnetic disk drive, an optical disk drive, a magneto-optical disk drive, or an SD card reader. The recording medium 208 is, for example, but not limited to, a CD, a DVD, a FD, an MO, a BD, a USB (registered trademark) memory, or an SD card.

[0032] In this embodiment, the tongue evaluation program may be written into memory 202 or storage 203 during the manufacturing stage of the user terminal 2, or may be provided to the user terminal 2 via the network N, or may be provided to the user terminal 2 via a non-transitory computer-readable recording medium such as a recording medium 208.

[0033] <Functional configuration of tongue evaluation device 1> Next, a description will be given of the functional configuration of the tongue evaluation device 1. Fig. 4 is a diagram showing an example of the functional configuration of the tongue evaluation device 1. As shown in Fig. 4, the tongue evaluation device 1 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0034] The communication unit 11 is realized by the communication I / F 104. The communication unit 11 transmits and receives information to and from the user terminal 2 via the network N. The communication unit 11 receives a face image from the user terminal 2, and transmits an evaluation result of the tongue state of the subject to the user terminal 2.

[0035] The storage unit 12 is realized by the memory 102 and the storage 103. The storage unit 12 stores a face image 121, a tongue detection model 122, a partial image 123, a tongue evaluation model 124, a tongue coating detection model 125, and an evaluation result 126.

[0036] The facial image 121 is an image of the subject's face with his / her tongue out. The facial image 121 includes an image of the subject's face and tongue. The facial image 121 may also include the subject's mouth and teeth.

[0037] Fig. 5 is a diagram showing an example of a face image 121. In the example of Fig. 5, the dotted parts are tongue coating, and the tongue coating is attached to the back of the subject's tongue, with a particularly large amount of tongue coating attached to the center of the tongue.

[0038] The tongue detection model 122 is a machine learning model that detects (segments) the tongue from the face image 121. When the face image 121 is input, the tongue detection model 122 outputs the position (coordinates) of the tongue included in the face image 121. The tongue detection model 122 is, for example, a neural network that has learned face images with tongues annotated as teacher data, but is not limited to this. As the tongue detection model 122, any machine learning model that can detect the tongue from the face image 121 can be used.

[0039] The partial image 123 is an image obtained by trimming (cutting out) a part including the tongue from the face image 121.

[0040] Fig. 6 is a diagram showing an example of partial image 123. Partial image 123 in Fig. 6 is cropped from face image 121 in Fig. 5, and is a rectangular image including the tongue of the subject. The shape of partial image 123 can be designed arbitrarily.

[0041] The tongue evaluation model 124 is a machine learning model that evaluates at least one of the degree of tongue swelling, the degree of dryness, and the color of the tongue coating from the partial image 123. When the partial image 123 is input, the tongue evaluation model 124 outputs at least one of the degree of tongue swelling, the degree of dryness, and the color of the tongue coating. The tongue evaluation model 124 is, for example, a neural network that has learned a face image or a partial image annotated with at least one of the degree of tongue swelling, the degree of dryness, and the color of the tongue coating as teacher data, but is not limited to this. As the tongue evaluation model 124, any machine learning model that can evaluate at least one of the degree of tongue swelling, the degree of dryness, and the color of the tongue coating from the partial image 124 can be used. Note that the tongue evaluation model 124 may be capable of evaluating two or more of the degree of tongue swelling, the degree of dryness, and the color of the tongue coating with one model. In addition, three tongue evaluation models 124 that evaluate the degree of tongue swelling, the degree of dryness, and the color of the tongue coating, respectively, may be prepared.

[0042] The tongue coating detection model 125 is a machine learning model that detects (segments) tongue coating from the partial image 123. When the partial image 123 is input, the tongue coating detection model 125 outputs the position (coordinates) and amount of tongue coating attached to the tongue included in the partial image 123. The position and amount of tongue coating may be detected for each pixel of the partial image 123, or may be detected for each area previously set on the tongue (for example, a square of a grid of A rows and B columns set on the tongue). The tongue coating detection model 125 is, for example, a neural network that has learned a face image or a partial image annotated with the position and amount of tongue coating as training data, but is not limited to this. Any machine learning model capable of detecting tongue coating from the partial image 123 can be used as the tongue coating detection model 125.

[0043] The evaluation result 126 is an evaluation result of the condition of the tongue of the subject. The evaluation result 126 includes at least one of the degree of swelling of the tongue of the subject, the degree of dryness, the color of the tongue coating, the position and amount of the tongue coating, and an overall evaluation.

[0044] The control unit 13 is realized by the processor 101 reading and executing a program from the memory 102 and cooperating with other hardware configurations. The control unit 13 controls the overall operation of the tongue evaluation device 1. The control unit 13 includes an acquisition unit 131, a tongue detection unit 132, a trimming unit 133, a tongue coating detection unit 134, and an evaluation unit 135.

[0045] The acquisition unit 131 acquires the face image 121 from the user terminal 2 and stores it in the storage unit 12 .

[0046] The tongue detection unit 132 detects the tongue of the subject from the face image 121 by using the tongue detection model 122. Specifically, the tongue detection unit 132 inputs the face image 121 to the tongue detection model 122 and obtains the detection result output by the tongue detection model 122.

[0047] Fig. 7 is a diagram showing an example of a tongue detection result. Fig. 7 shows the tongue detection result for the face image 121 of Fig. 5, and the tongue part is indicated by diagonal lines. As shown in Fig. 7, the tongue is segmented from the face image 121 by the tongue detection unit 132.

[0048] The trimming unit 133 trims the partial image 123 from the face image 121. Specifically, the trimming unit 133 trims a part including the tongue from the face image 121 based on the tongue detection result by the tongue detection unit 132, and stores the trimmed part as the partial image 123 in the storage unit 12. For example, the trimming unit 133 may trim the partial image 123 so as to leave a predetermined margin from the tongue, or may trim the partial image 123 so that the partial image 123 has a predetermined shape. The method of trimming the partial image 123 can be designed arbitrarily.

[0049] Tongue coating detection unit 134 detects tongue coating attached to the subject's tongue from partial image 123, using tongue coating detection model 125. Specifically, tongue coating detection model 134 inputs partial image 123 to tongue coating detection model 125, and obtains the position and amount of tongue coating output by tongue coating detection model 125 as the detection result.

[0050] Fig. 8 is a diagram showing an example of the detection result of tongue coating. Fig. 8 shows the detection result of tongue coating for partial image 123 of Fig. 6, and three levels of tongue coating amount are shown for each square of an 8-row, 8-column grid set on the tongue. A colorless square indicates no tongue coating, a square with light vertical lines indicates little tongue coating, and a square with dark vertical lines indicates much tongue coating. As shown in Fig. 8, in partial image 123 of Fig. 7, little tongue coating is detected at the back of the tongue and much tongue coating is detected in the center.

[0051] The evaluation unit 135 evaluates the state of the tongue of the subject by using the tongue evaluation model 124. Specifically, the evaluation unit 135 inputs the partial image 123 to the tongue evaluation model 124, acquires at least one of the degree of tongue swelling, the degree of dryness, and the color of the tongue coating output by the tongue evaluation model 124 as an evaluation result, and stores it in the storage unit 12. The evaluation unit 135 also acquires a detection result by the tongue coating detection unit 134, and stores it in the storage unit 12 as an evaluation result 126. The evaluation unit 135 also evaluates the state of the tongue as an overall evaluation based on at least one of the degree of tongue swelling, the degree of dryness, the color of the tongue coating, and the position and amount of the tongue coating, and stores it in the storage unit 12 as the evaluation result 126.

[0052] The functional configuration of the tongue evaluation device 1 is not limited to the above example. For example, the tongue evaluation device 1 may have a part of the functional configuration described above, and the user terminal 2 may have the rest. The tongue evaluation device 1 may also have functional configurations other than those described above. As described above, each functional configuration of the tongue evaluation device 1 may be realized by software, or may be realized by hardware such as an IC chip, a SoC (System on Chip), an LSI (Large Scale Integration), or a microcomputer.

[0053] <Functional configuration of user terminal 2> Next, a description will be given of the functional configuration of the user terminal 2. Fig. 9 is a diagram showing an example of the functional configuration of the user terminal 2. As shown in Fig. 9, the user terminal 2 includes a communication unit 21, a storage unit 22, and a control unit 23.

[0054] The communication unit 21 is realized by the communication I / F 204. The communication unit 21 transmits and receives information to and from the user terminal 2 via the network N. The communication unit 21 transmits a face image 221 to the tongue evaluation device 1, and receives an evaluation result 222 from the tongue evaluation device 1.

[0055] The storage unit 22 is realized by a memory 202 and a storage 203. The storage unit 22 stores a face image 221 and an evaluation result 222.

[0056] The facial image 221 is an image of a face of a subject with tongue out. The facial image 221 includes images of the subject's face and tongue. The facial image 221 may also include the subject's mouth and teeth. In the following, an example will be described in which the facial image 221 is captured by the user terminal 2. However, the user terminal 2 may obtain the facial image 221 captured in advance from an external device via the network N.

[0057] The evaluation result 222 is an evaluation result of the condition of the tongue of the subject. The evaluation result 222 includes at least one of the degree of swelling of the tongue of the subject, the degree of dryness, the color of the tongue coating, the position and amount of the tongue coating, and an overall evaluation.

[0058] The control unit 23 is realized by the processor 201 reading and executing a program from the memory 202 and working in cooperation with other hardware configurations. The control unit 23 controls the overall operation of the user terminal 2. The control unit 23 includes an imaging unit 231, an acquisition unit 232, and a display unit 233.

[0059] The photographing section 231 controls the photographing device 205C in response to a user's operation, photographs a face image 221 of the subject, and stores the face image 221 in the storage section 22.

[0060] The acquisition unit 232 acquires the evaluation result 126 from the tongue evaluation device 1 , and stores it in the storage unit 22 as the evaluation result 222 .

[0061] The display unit 233 displays the evaluation result 222 on the display device 206D.

[0062] The functional configuration of the user terminal 2 is not limited to the above example. For example, the user terminal 2 may have a functional configuration other than the above. Furthermore, each functional configuration of the user terminal 2 may be realized by software as described above, or may be realized by hardware such as an IC chip, an SoC, an LSI, or a microcomputer.

[0063] <Processing Executed by Tongue Evaluation System 1000> Next, a description will be given of the processing (tongue evaluation method) executed by the tongue evaluation system 1000. Fig. 10 is a flowchart showing an example of the processing executed by the tongue evaluation system 1000.

[0064] (Step S101) The photographing section 231 of the user terminal 2 controls the photographing device 205C in response to an operation by the user, photographs a face image 221 of the subject, and stores the face image 221 in the storage section 22 (step S101).

[0065] (Step S102) The communication unit 21 of the user terminal 2 transmits the face image 221 stored in the storage unit 22 to the tongue evaluation device 1 (Step S102).

[0066] (Step S103) The acquisition unit 131 of the tongue evaluation device 1 acquires the face image 221 (see FIG. 5) received by the communication unit 11, and stores it as the face image 121 in the storage unit 12 (step S103).

[0067] (Step S104) The tongue detection unit 132 detects the tongue of the subject from the face image 121 by using the tongue detection model 122 (see FIG. 7) (step S104). As a result, the tongue of the subject is segmented from the face image 121.

[0068] (Step S105) The trimming unit 133 trims the partial image 123 (see FIG. 6) from the face image 121 based on the result of the tongue detection in step S104, and stores the partial image 123 in the storage unit 12 (step S105).

[0069] (Step S106) The evaluation unit 135 uses the tongue evaluation model 124 to evaluate the degree of swelling of the subject's tongue from the partial image 123, and stores the evaluation result 126 in the storage unit 12 (step S106). Specifically, the evaluation unit 135 inputs the partial image 123 to the tongue evaluation model 124, and obtains the evaluation result of the degree of swelling output by the tongue evaluation model 124. The degree of swelling is evaluated, for example, by a score or a rank, but is not limited thereto.

[0070] The partial image 123 has less parts (noise) other than the evaluation target (tongue) than the facial image 121. Therefore, by evaluating the degree of swelling based on the partial image 123, the degree of swelling of the tongue can be evaluated more accurately than when the degree of swelling is evaluated based on the facial image 121.

[0071] Furthermore, when the degree of swelling is evaluated based on an image of only the tongue, the evaluation of the degree of swelling varies greatly depending on the shooting environment (brightness, etc.). In contrast, unlike an image of only the tongue, partial image 123 includes an image of the tongue and its surroundings (teeth, oral cavity, lips, etc.), so when the degree of swelling is evaluated based on partial image 123, the degree of swelling is evaluated taking into account the relationship between the tongue and its surroundings. Therefore, by evaluating the degree of swelling based on partial image 123, the influence of the shooting environment can be suppressed and the degree of swelling of the tongue can be evaluated with high accuracy compared to when the degree of swelling is evaluated based on an image of only the tongue.

[0072] (Step S107) The evaluation unit 135 uses the tongue evaluation model 124 to evaluate the degree of dryness of the subject's tongue from the partial image 123, and stores the evaluation result 126 in the storage unit 12 (step S107). Specifically, the evaluation unit 135 inputs the partial image 123 to the tongue evaluation model 124, and obtains the evaluation result of the degree of dryness output by the tongue evaluation model 124. The degree of dryness is evaluated, for example, by a score or a rank, but is not limited thereto.

[0073] Partial image 123 has less parts (noise) other than the evaluation target (tongue) than facial image 121. Therefore, by evaluating the degree of dryness based on partial image 123, the degree of dryness of the tongue can be evaluated more accurately than when the degree of dryness is evaluated based on facial image 121.

[0074] Furthermore, when the degree of dryness is evaluated based on an image of only the tongue, the evaluation of the degree of dryness varies greatly depending on the shooting environment (brightness, etc.). In contrast, unlike an image of only the tongue, partial image 123 includes an image of the tongue and its surroundings (teeth, oral cavity, lips, etc.), so when the degree of dryness is evaluated based on partial image 123, the relationship between the tongue and its surroundings is taken into account in the evaluation. Therefore, by evaluating the degree of dryness based on partial image 123, the influence of the shooting environment can be suppressed and the degree of dryness of the tongue can be evaluated with high accuracy compared to when the degree of dryness is evaluated based on an image of only the tongue.

[0075] (Step S108) The evaluation unit 135 uses the tongue evaluation model 124 to evaluate the color of the tongue coating of the subject's tongue from the partial image 123, and stores the evaluation result 126 in the storage unit 12 (step S108). Specifically, the evaluation unit 135 inputs the partial image 123 to the tongue evaluation model 124, and obtains the evaluation result of the color of the tongue coating output by the tongue evaluation model 124. The color of the tongue coating is evaluated, for example, using four colors (red, white, yellow, and black), but is not limited to this.

[0076] The partial image 123 has less parts (noise) other than the evaluation target (tongue) than the facial image 121. Therefore, by evaluating the color of the tongue coating based on the partial image 123, the color of the tongue coating can be evaluated more accurately than when the color of the tongue coating is evaluated based on the facial image 121.

[0077] Furthermore, when the color of the tongue coating is evaluated based on an image of only the tongue, the evaluation of the color of the tongue coating varies greatly depending on the shooting environment (brightness, etc.). In contrast, partial image 123, unlike an image of only the tongue, includes an image of the tongue and its surroundings (teeth, oral cavity, lips, etc.), so when the degree of dryness is evaluated based on partial image 123, the color of the tongue coating is evaluated taking into account the relationship between the tongue and its surroundings. Therefore, by evaluating the color of the tongue coating based on partial image 123, the influence of the shooting environment can be suppressed and the color of the tongue coating can be evaluated with higher accuracy than when the color of the tongue coating is evaluated based on an image of only the tongue.

[0078] (Step S109) The tongue coating detection unit 134 detects tongue coating from the partial image 123 using the tongue coating detection model 125 (see FIG. 8), and stores the result as the evaluation result 126 in the storage unit 12 (step S109).

[0079] (Step S110) The evaluation unit 135 evaluates the state of the tongue (overall evaluation) based on the degree of tongue swelling, the degree of dryness, the color of the tongue coating, and the detection results of the tongue coating (position and amount), and stores the result as evaluation result 126 in the storage unit 12 (step S110). The overall evaluation may be, for example, a score or rank that comprehensively evaluates the state of the tongue, or a comment that explains the state of the tongue.

[0080] The evaluation unit 135 can evaluate the state of the tongue by referring to a table that indicates the relationship between the degree of tongue swelling, the degree of dryness, the color of tongue coating, and the detection results of tongue coating (position and amount), which are stored in advance in the storage unit 12, and an overall evaluation. The evaluation unit 135 may also generate comments by inputting the degree of tongue swelling, the degree of dryness, the color of tongue coating, and the detection results of tongue coating (position and amount) into a machine learning model such as LLM (Large Language Models). The evaluation unit 135 may also generate an overall evaluation by using the above table and the machine learning model in combination.

[0081] (Step S111) The communication unit 11 of the evaluation device 1 transmits the evaluation result 126 stored in the storage unit 12 to the user terminal 2 (step S111). The communication unit 11 may transmit the partial image 123 together with the evaluation result 126 to the user terminal 2.

[0082] (Step S112) The acquisition unit 232 of the user terminal 2 acquires the evaluation result 126 received by the communication unit 21, and stores it in the storage unit 22 as the evaluation result 222 (step S112).

[0083] (Step S113) The display unit 233 displays the evaluation result 222 on the display device 206D (step S113). The evaluation result 222 is displayed, for example, on an evaluation result screen SC.

[0084] Fig. 11 is a diagram showing an example of an evaluation result screen SC. A face image 221 and an evaluation result 222 are displayed on the evaluation result screen SC of Fig. 11. As the evaluation result 222, a score indicating the degree of swelling (swelling), the degree of dryness (dryness), the amount of tongue coating, and an overall evaluation (overall), the color of the tongue coating, the detection result of the tongue coating displayed superimposed on the partial image 123, and a comment (overall evaluation) are displayed. In this way, by displaying the evaluation result 222 on the user terminal 2, the user can easily understand the evaluation result 222.

[0085] <Summary> As described above, according to this embodiment, a tongue evaluation method is implemented by the tongue evaluation system 1000, including an acquisition step of acquiring a face image 121 obtained by photographing a face with the tongue out, a tongue detection step of detecting the tongue from the face image 121 using the tongue detection model 122, a trimming step of trimming a partial image 123 including the tongue from the face image 121 based on the tongue detection result, and an evaluation step of evaluating the degree of tongue swelling from the partial image 123 using the tongue evaluation model 124. This makes it possible to evaluate the degree of tongue swelling, which could not be evaluated conventionally, and therefore allows the state of the tongue to be evaluated with higher accuracy.

[0086] Moreover, according to the present embodiment, the tongue dryness level is evaluated from the partial image 123 using the tongue evaluation model 124. This makes it possible to evaluate the tongue dryness level, which was not previously evaluated, and therefore allows the tongue condition to be evaluated with higher accuracy.

[0087] Moreover, according to the present embodiment, the tongue coating color is evaluated from the partial image 123 using the tongue evaluation model 124. This makes it possible to evaluate the tongue coating color, which has not been evaluated in the past, and therefore allows the tongue condition to be evaluated with higher accuracy.

[0088] Furthermore, according to the present embodiment, the state of the tongue is evaluated based on at least one of the degree of swelling, the degree of dryness, the color of the tongue coating, and the detection result of the tongue coating. This makes it possible to evaluate the state of the tongue based on a plurality of tongue states that have not been evaluated in the past, and therefore allows the state of the tongue to be evaluated more accurately.

[0089] <Additional Notes> The present embodiment includes the following disclosure.

[0090] (Appendix 1) A tongue evaluation method executed by a tongue evaluation system, comprising: An acquisition step of acquiring a face image of a face with tongue out; a tongue detection step of detecting the tongue from the face image using a tongue detection model; a trimming step of trimming a partial image including the tongue from the face image based on a result of the tongue detection; an evaluation step of evaluating a degree of swelling of the tongue from the partial image by using a tongue evaluation model; A tongue evaluation method comprising:

[0091] (Appendix 2) The evaluation step evaluates a degree of dryness of the tongue from the partial image by utilizing the tongue evaluation model. Tongue evaluation method described in Appendix 1.

[0092] (Appendix 3) The evaluation step includes evaluating a color of the tongue coating from the partial image by using the tongue evaluation model. Tongue evaluation method described in Appendix 1.

[0093] (Appendix 4) The tongue evaluation model is a machine learning model trained to output an evaluation result of the tongue when the partial image is input. Tongue evaluation method described in Appendix 1.

[0094] (Appendix 5) The method further includes a tongue coating detection step of detecting tongue coating of the tongue from the partial image using a tongue coating detection model. Tongue evaluation method described in Appendix 1.

[0095] (Appendix 6) The evaluation step evaluates the state of the tongue based on at least one of the degree of swelling of the tongue, the degree of dryness, the color of the tongue coating, and the detection result of the tongue coating. Tongue evaluation method described in Appendix 1.

[0096] (Appendix 7) An acquisition unit that acquires a facial image of a face with a tongue out; a tongue detection unit that detects the tongue from the face image using a tongue detection model; a trimming unit that trims a partial image including the tongue from the face image based on a result of the tongue detection; an evaluation unit that evaluates a degree of swelling of the tongue from the partial image by using a tongue evaluation model; A tongue evaluation system comprising:

[0097] (Appendix 8) A tongue evaluation method executed by a tongue evaluation system, comprising: An acquisition step of acquiring a face image of a face with tongue out; a tongue detection step of detecting the tongue from the face image using a tongue detection model; a trimming step of trimming a partial image including the tongue from the face image based on a result of the tongue detection; an evaluation step of evaluating a degree of swelling of the tongue from the partial image by using a tongue evaluation model; A tongue evaluation program for causing a computer to execute a tongue evaluation method comprising the steps of:

[0098] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is indicated by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. The present invention is not limited to the above-mentioned embodiments, and various modifications are possible within the scope of the claims, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0099] 1: Tongue evaluation device 2: User terminal 121:Facial image 122: Tongue detection model 123: Partial image 124: Tongue evaluation model 125: Tongue coating detection model 126: Evaluation results 131: Acquisition Department 132: Tongue detection unit 133: Trimming section 134: Tongue coating detection unit 135: Evaluation section

Claims

1. A tongue evaluation method executed by a tongue evaluation system, comprising: An acquisition step of acquiring a face image of a face with tongue out; a tongue detection step of detecting the tongue from the face image using a tongue detection model; a trimming step of trimming a partial image including the tongue from the face image based on a result of the tongue detection; an evaluation step of evaluating a degree of swelling of the tongue from the partial image by using a tongue evaluation model; A tongue evaluation method comprising:

2. The evaluation step evaluates a degree of dryness of the tongue from the partial image by utilizing the tongue evaluation model. The tongue evaluation method according to claim 1 .

3. The evaluation step includes evaluating a color of the tongue coating from the partial image by using the tongue evaluation model. The tongue evaluation method according to claim 1 .

4. The tongue evaluation model is a machine learning model trained to output an evaluation result of the tongue when the partial image is input. The tongue evaluation method according to claim 1 .

5. The method further includes a tongue coating detection step of detecting tongue coating of the tongue from the partial image using a tongue coating detection model. The tongue evaluation method according to claim 1 .

6. The evaluation step evaluates the state of the tongue based on at least one of the degree of swelling of the tongue, the degree of dryness, the color of the tongue coating, and the detection result of the tongue coating. The tongue evaluation method according to claim 1 .

7. An acquisition unit that acquires a facial image of a face with a tongue out; a tongue detection unit that detects the tongue from the face image using a tongue detection model; a trimming unit that trims a partial image including the tongue from the face image based on a result of the tongue detection; an evaluation unit that evaluates a degree of swelling of the tongue from the partial image by using a tongue evaluation model; A tongue evaluation system comprising:

8. A tongue evaluation method executed by a tongue evaluation system, comprising: An acquisition step of acquiring a face image of a face with tongue out; a tongue detection step of detecting the tongue from the face image using a tongue detection model; a trimming step of trimming a partial image including the tongue from the face image based on a result of the tongue detection; an evaluation step of evaluating a degree of swelling of the tongue from the partial image by using a tongue evaluation model; A tongue evaluation program for causing a computer to execute a tongue evaluation method comprising the steps of:

Images