Information processing device, information processing system, information processing method, and control program

Abstract

Provided are an information processing device, an information processing system, an information processing method, and a control program capable of outputting the health condition of a person with higher accuracy. This information processing device includes: an acquisition unit that acquires a tongue image including any of the papillae of the tongue, front and back conditions of the tongue, and the entire tongue of a person; and an output unit that outputs information indicating the health condition of the person based on the tongue image. The information processing system includes a terminal device and an information processing device. The terminal device includes a transmission unit that transmits, to the information processing device, a tongue image including any of the papillae of the tongue, front and back conditions of the tongue, and the entire tongue of a person. The information processing device includes: a reception unit that receives the tongue image from the terminal device; and an output unit that outputs information indicating the health condition of the person based on the tongue image.

Description

Information processing device, information processing system, information processing method, and control program

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and a control program.

[0002] Conventionally, systems have been developed that determine a person's health condition from an image of the person's tongue.

[0003] Patent Document 1 discloses a health monitoring device that divides a tongue region from an image of an individual's tongue, extracts a region of interest from the divided tongue region, and determines the individual's health condition.

[0004] Japanese Patent Application Laid-Open No. 2004-209245

[0005] There is a demand for information processing devices to output a person's health condition with higher accuracy.

[0006] An object of the information processing device, information processing system, information processing method, and control program is to enable a person's health condition to be output with higher accuracy.

[0007] The information processing device according to the embodiment has an acquisition unit that acquires a tongue image including the papillae of a person's tongue, the characteristics of the front and back of the tongue, or the entire tongue, and an output unit that outputs information indicating the person's health condition based on the tongue image.

[0008] In the information processing device according to the embodiment, it is preferable that the output unit outputs information indicating the condition of the person's skin as the information indicating the person's health condition.

[0009] In the information processing device according to the embodiment, the skin condition is preferably one or more of the degree of wrinkles, the degree of pores, the degree of texture, and the degree of spots.

[0010] In the information processing device of the embodiment, it is preferable that the output unit outputs, as information indicating the person's health condition, one or more of information indicating the state of the person's blood vessels and / or blood flow, information indicating antioxidant balance, information indicating the body's nutritional state, information indicating psychological stress, and metabolic abnormalities.

[0011] In the information processing device according to the embodiment, the state of the blood vessels is preferably the distribution of blood flow or the softness of the blood vessels, and the state of blood flow is preferably the quality of blood flow.

[0012] In the information processing device according to the embodiment, the nutritional state of the body is preferably the amount or deficiency of vitamins, minerals, and proteins, or the quality of lipid metabolism.

[0013] In the information processing device according to the embodiment, it is preferable to further include an identification unit that identifies any of the characteristics of the papillae of a person's tongue, the characteristics of the front and back of the tongue, or the overall shape of the tongue based on a tongue image, and a determination unit that determines the person's health condition based on the characteristics.

[0014] In the information processing device according to the embodiment, it is preferable that the determination unit determines the condition of the person's skin as the health condition of the person based on the characteristics of the nipple.

[0015] In the information processing device according to the embodiment, it is preferable that the determination unit determines the state of the person's blood vessels based on the characteristics of the nipple, and determines the state of the person's skin as the person's health state based on the state of the blood vessels.

[0016] In the information processing device according to the embodiment, it is preferable that the specifying unit specifies the shape, size or regularity of the nipple as the nipple feature.

[0017] In the information processing device according to the embodiment, it is preferable that the acquisition unit acquires a plurality of tongue images captured at mutually different times, and the identification unit identifies a change in the papilla as a feature of the papilla.

[0018] In the information processing device according to the embodiment, it is preferable that the identification unit identifies the characteristics of the papillae at each of the end and back of the tongue, and the determination unit determines the health condition of the person based on the characteristics of the papillae identified at each of the end and back of the tongue.

[0019] In the information processing device according to the embodiment, it is preferable that the acquisition unit further acquires the person's subjective physical condition or environmental information, and the determination unit determines the person's health state further based on the person's subjective physical condition or environmental information.

[0020] In the information processing device according to the embodiment, it is preferable that the determination unit determines one or more of the antioxidant balance and the nutritional state of the body of the person based on the characteristics of the fungiform papillae among the papillae.

[0021] In the information processing device according to the embodiment, it is preferable that the identification unit identifies one or more of the shape characteristics and color characteristics of the fungiform papillae as the characteristics of the fungiform papillae among the papillae.

[0022] In the information processing device according to the embodiment, it is preferable that the judgment unit judges one or more of the nutritional state, blood flow state, or blood state of a person's body based on one or more of the amount of tongue coating on the surface of the tongue, the state of blood oozing on the surface, or the state of blood oozing on the back surface.

[0023] In the information processing device according to the embodiment, it is preferable that the identifying unit identifies one or more of the amount of tongue coating on the surface of the tongue, the state of blood oozing on the surface, and the state of blood oozing on the back surface.

[0024] In the information processing device according to the embodiment, it is preferable that the determination unit determines one or more of a nutritional state of the body, psychological stress, and metabolic abnormality based on the entire tongue.

[0025] In the information processing device according to the embodiment, it is preferable that the identification unit identifies one or more of the following as the overall features of the tongue: a color balance feature, a crack feature, a thickness feature, and a teeth mark feature.

[0026] In the information processing device according to the embodiment, it is preferable that the identification unit identifies any of the characteristics of the papillae of a person's tongue, the characteristics of the surface and back properties of the tongue, or the characteristics of the overall shape of the tongue by any of template matching, histogram analysis, analysis using near-infrared spectroscopic images, shape analysis of an object using an image, texture analysis, and image analysis using machine learning.

[0027] In the information processing device according to the embodiment, it is preferable that the output unit outputs the factors of the person's health condition determined by the determination unit.

[0028] The information processing system according to the embodiment is an information processing system having a terminal device and an information processing device, wherein the terminal device has a transmitting unit that transmits a tongue image including either the papillae of a person's tongue, the characteristics of the front and back of the tongue, or the entire tongue to the information processing device, and the information processing device has a receiving unit that receives the tongue image from the terminal device and an output unit that outputs information indicating the person's health condition based on the tongue image.

[0029] In the information processing device according to the embodiment, it is preferable that the terminal device further includes an imaging unit that generates a tongue image.

[0030] In the information processing method according to the embodiment, an information processing device acquires a tongue image including the papillae of a person's tongue, the characteristics of the front and back of the tongue, or the entire tongue, and outputs information indicating the person's health condition based on the tongue image.

[0031] The control program of the embodiment is a control program of an information processing device that acquires a tongue image including the papillae of a person's tongue, the characteristics of the front and back of the tongue, or the entire tongue, and outputs information indicating the person's health condition based on the tongue image.

[0032] The information processing device, information processing system, information processing method, and control program can output a person's health condition with higher accuracy.

[0033] 1 is a diagram illustrating a schematic configuration of an information processing system 1 according to an embodiment. FIG. 1 is a diagram illustrating a schematic configuration of a terminal device 100. FIG. 1 is a diagram illustrating a schematic configuration of a server device 200. FIG. 2 is a sequence illustrating an example of an operation of a determination process. FIG. 3 is a schematic diagram illustrating an example of a tongue image. FIG. 4 is a schematic diagram illustrating an example of a nipple image. FIG. 5 is a schematic diagram illustrating an example of a blood flow distribution image. FIG. 6 is a schematic diagram for explaining the relationship between nipple features and blood vessel conditions. FIG. 7 is a schematic diagram for explaining the relationship between nipple features and skin conditions. FIG. 8 is a schematic diagram for explaining the relationship between nipple features and nutritional status. FIG. 9 is a schematic diagram for explaining the relationship between nipple features and nutritional status. FIG. 10 is a schematic diagram for explaining the relationship between nipple features and nutritional status. FIG. 11 is a schematic diagram for explaining the relationship between nipple features and nutritional status. FIG. 12 is a schematic diagram for explaining the relationship between nipple features and nutritional status. Fig. 1 is a schematic diagram showing an example of a gradation histogram. Fig. 2 is a schematic diagram showing an example of a display screen displayed on a first display device. Fig. 3 is a schematic diagram showing an example of a display screen displayed on a first display device. Fig. 4 is a schematic diagram showing a state in which a target person is imaging his or her tongue.

[0034] Hereinafter, an information processing device, an information processing system, an information processing method, and a control program according to one aspect of an embodiment will be described with reference to the drawings. However, please note that the technical scope of the present invention is not limited to the embodiment, but extends to the inventions set forth in the claims and their equivalents.

[0035] First Embodiment FIG. 1 is a diagram showing a schematic configuration of an information processing system 1 according to an embodiment.

[0036] 1, the information processing system 1 includes a terminal device 100 and a server device 200. The terminal device 100 and the server device 200 are connected to each other so as to be able to communicate with each other via a network N. The network N is a wired network such as the Internet or an intranet. The network N may also be a wireless network such as a wireless LAN (Local Area Network).

[0037] FIG. 2 is a diagram showing a schematic configuration of the terminal device 100. As shown in FIG.

[0038] The terminal device 100 is an example of an information processing device. The terminal device 100 may be a personal computer, a notebook PC (Personal Computer), a tablet PC, a multi-function mobile phone (a so-called smartphone), or the like. For example, the terminal device 100 is placed in a store or the like and used by a store clerk or the like to capture an image of the tongue of a target person whose health condition is to be determined. The terminal device 100 may also be used directly by the target person. The terminal device 100 includes a first input device 101, a first display device 102, a first communication device 103, an imaging device 104, a first storage device 110, a first processing device 120, and the like. The first input device 101, the first display device 102, the first communication device 103, the imaging device 104, the first storage device 110, and the first processing device 120 are connected to each other via a CPU (Central Processing Unit) bus or the like.

[0039] The first input device 101 has input devices such as a keyboard, a mouse, and a touch panel, and an interface circuit that acquires signals from the input devices, and outputs operation signals in response to input operations by the user.

[0040] The first display device 102 has a display such as a liquid crystal display or an organic electroluminescence (EL) display, and an interface circuit that outputs image data to the display, and displays the image data on the display.

[0041] The first communication device 103 has a wired communication interface circuit that complies with a communication protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol). The first communication device 103 is communicatively connected to the network N in accordance with a communication standard such as Ethernet (registered trademark). The first communication device 103 sends data received from the server device 200 via the network N to the first processing device 120. The first communication device 103 transmits data received from the first processing device 120 to the server device 200 via the network N. The first communication device 103 has an antenna that transmits and receives wireless signals and a wireless communication interface circuit that complies with a communication protocol such as a wireless LAN, and may be communicatively connected to the network N in accordance with a communication standard such as a wireless LAN.

[0042] The imaging device 104 has a photoelectric conversion element, such as a charge-coupled device (CCD) element or a complementary metal oxide semiconductor (C-MOS) element, that is sensitive to visible light or infrared light. Infrared light includes near-infrared light having a wavelength of 700 nm or more and 1700 nm or less. The imaging device 104 also has an imaging optical system that forms an image on the photoelectric conversion element and an A / D converter that amplifies and performs analog-to-digital (A / D) conversion on the electrical signal output from the photoelectric conversion element. The imaging device 104 converts the captured analog image of each color of RGB into a digital image in which each pixel has a brightness value ranging from 0 to 255, generates it as an input image, and outputs it to the first processing device 120. The imaging device 104 may also be an external camera connected via a communication interface circuit that complies with a communication protocol such as USB (Universal Serial Bus) or Bluetooth (registered trademark). The external camera includes a camera that does not allow magnification, a camera that allows magnification, an external lens, etc. By including the imaging device 104, the terminal device 100 can easily acquire an input image of a person's tongue by the user of the terminal device 100.

[0043] The imaging device 104 may include multiple imaging devices. For example, if the terminal device 100 is a tablet PC, a multi-function mobile phone, or the like, the imaging device 104 includes a front imaging device and a rear imaging device. The front imaging device is configured to capture an image of the side on which the first input device 101 and the first display device 102 are arranged, and the rear imaging device is configured to capture an image of the rear side of the terminal device 100, i.e., the side on which the first input device 101 and the first display device 102 are not arranged. Each imaging device has a photoelectric conversion element, an imaging optical system, and an A / D converter, and outputs an input image to the first processing device 120. The performance of the photoelectric conversion element, imaging optical system, and A / D converter of the rear imaging device is higher than that of the photoelectric conversion element, imaging optical system, and A / D converter of the front imaging device. For example, the resolution of the input image generated by the rear imaging device is higher than the resolution of the input image generated by the front imaging device, and / or the gradation range of each pixel of the input image generated by the rear imaging device is larger than the gradation range of each pixel of the input image generated by the front imaging device.

[0044] When the photoelectric conversion element is sensitive to near-infrared light having a wavelength of 700 nm or more and 1700 nm or less, the imaging device 104 may further include an illuminator that irradiates near-infrared light having a wavelength of 700 nm or more and 1700 nm or less.

[0045] The first storage device 110 includes a memory device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a fixed disk device such as a hard disk, or a portable storage device such as a flexible disk or optical disk. The first storage device 110 also stores computer programs, databases, tables, and the like used for various processes of the terminal device 100. The computer programs may be installed into the first storage device 110 from a computer-readable portable recording medium using a known setup program or the like. The portable recording medium is, for example, a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or the like. The computer programs may be stored on a recording medium owned by a predetermined server and installed via a network N.

[0046] The first processing device 120 operates based on a program stored in advance in the first storage device 110. The first processing device 120 is, for example, a CPU. A digital signal processor (DSP), a large scale integration (LSI), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or the like may be used as the first processing device 120. The first processing device 120 is connected to the first input device 101, the first display device 102, the first communication device 103, the imaging device 104, the first storage device 110, and the like, and controls each device. The first processing device 120 acquires a tongue image including a person's tongue from the imaging device 104 and transmits the image to the server device 200 via the first communication device 103.

[0047] The first processing device 120 reads the computer program stored in the first storage device 110 and operates in accordance with the read computer program. As a result, the first processing device 120 functions as a transmission unit 121 and a display control unit 122.

[0048] FIG. 3 is a diagram showing a schematic configuration of the server device 200.

[0049] The server device 200 is an example of an information processing device. The server device 200 includes a second communication device 201, a second storage device 210, and a second processing device 220. The second communication device 201, the second storage device 210, and the second processing device 220 are connected to each other via a CPU bus or the like.

[0050] The second communication device 201 is an example of an output unit. The second communication device 201 has a wired communication interface circuit that complies with a communication protocol such as TCP / IP. The second communication device 201 is communicatively connected to the network N in accordance with a communication standard such as Ethernet (registered trademark). The second communication device 201 sends data received from the terminal device 100 via the network N to the second processing device 220. The second communication device 201 transmits data received from the second processing device 220 to the terminal device 100 via the network N. Note that the second communication device 201 may have an antenna that transmits and receives wireless signals and a wireless communication interface circuit that complies with a communication protocol such as a wireless LAN, and may be communicatively connected to the network N in accordance with a communication standard such as a wireless LAN.

[0051] The second storage device 210 includes a memory device such as RAM or ROM, a fixed disk device such as a hard disk, or a portable storage device such as a flexible disk or optical disk. The second storage device 210 also stores computer programs, databases, tables, and the like used for various processes of the server device 200. The computer programs may be installed into the second storage device 210 from a computer-readable portable recording medium such as a CD-ROM or DVD-ROM using a known setup program or the like. The computer programs may be stored in a recording medium owned by a predetermined server and installed via the network N.

[0052] The second processing device 220 operates based on a program stored in advance in the second storage device 210. The second processing device 220 is, for example, a CPU. A DSP, an LSI, an ASIC, an FPGA, etc. may be used as the second processing device 220. The second processing device 220 is connected to the second communication device 201, the second storage device 210, etc., and controls each device. The second processing device 220 receives a tongue image from the terminal device 100 via the second communication device 201. The second processing device 220 determines the person's health condition based on the received tongue image, and transmits information indicating the person's health condition to the terminal device 100 via the second communication device 201.

[0053] The second processing device 220 reads the computer program stored in the second storage device 210 and operates in accordance with the read computer program. As a result, the second processing device 220 functions as a receiving unit 221, an identifying unit 222, a determining unit 223, and an output control unit 224. The receiving unit 221 is an example of an acquiring unit.

[0054] FIG. 4 is a sequence showing an example of the operation of the determination process in the information processing system 1.

[0055] An example of the operation of the determination process will be described below with reference to the flowchart shown in Fig. 4. The operation flow described below is executed mainly by each processing device of each device in cooperation with each element of each device, based on a program stored in advance in each storage device of each device included in the information processing system 1.

[0056] First, the transmitting unit 121 of the terminal device 100 acquires a tongue image including the target person's tongue, subjective physical condition, and environmental information (step S101). The transmitting unit 121 causes the imaging device 104 to capture an image of the target person's tongue in accordance with instructions input by the user using the first input device 101, and acquires the input image generated by the imaging device 104 as a tongue image. The tongue image includes at least a papilla image including papillae of the target person's tongue. The papillae include filiform papillae and fungiform papillae. The transmitting unit 121 may acquire multiple tongue images of the target person's tongue captured at a predetermined time interval, i.e., multiple tongue images captured at different times.

[0057] FIG. 5 is a schematic diagram showing an example of a tongue image.

[0058] 5 shows tongue images 500, 510, and 520 of multiple target persons, respectively. Tongue image 500 includes the tongue of person A who is in good health, tongue image 510 includes the tongue of person B who is in somewhat poor health, and tongue image 520 includes the tongue of person C who is in poor health.

[0059] FIG. 6 is a schematic diagram showing an example of a nipple image.

[0060] 6 shows nipple images 600, 610, and 620 of multiple target persons, respectively. Nipple image 600 is an image in which a portion of tongue image 500 is enlarged to show the nipple in detail, nipple image 610 is an image in which a portion of tongue image 510 is enlarged to show the nipple in detail, and nipple image 620 is an image in which a portion of tongue image 520 is enlarged to show the nipple in detail.

[0061] The transmission unit 121 also acquires the subjective physical condition and environmental information of the target person specified by the user using the first input device 101. The subjective physical condition includes one or more elements related to physical condition, such as fever, pain, stress, chills, fatigue, stiff shoulders, and sleep, and the user specifies whether the subjective physical condition is good or bad and / or the degree of each element as the subjective physical condition. The subjective physical condition may be specified for each body part, such as the head, abdomen, arms, and legs. The environmental information is information indicating the target person's environment, situation, etc., and specifies whether or not the target person is consuming each element and / or the degree of each element, for one or more elements, such as tobacco and sweets.

[0062] Next, the transmitting unit 121 transmits the acquired tongue image, subjective physical condition, and environmental information to the server device 200 via the first communication device 103 (step S102). The transmitting unit 121 transmits the tongue image, subjective physical condition, and environmental information together to the server device 200. The transmitting unit 121 may also asynchronously acquire the tongue image, subjective physical condition, and environmental information and transmit them separately to the server device 200.

[0063] Next, the receiving unit 221 of the server device 200 receives and acquires the tongue image, subjective physical condition, and environmental information of the target person from the terminal device 100 via the second communication device 201 (step S103).

[0064] Next, the identification unit 222 identifies the characteristics of the papillae of the target person's tongue based on the tongue image acquired by the receiving unit 221 (step S104).

[0065] The identification unit 222 first extracts a predetermined region (nipple image) that includes the nipple from the tongue image. The identification unit 222 extracts, as edge pixels, pixels whose difference in gradation value (brightness value, color value, etc.) from horizontally, vertically, or diagonally adjacent pixels in the tongue image is equal to or greater than a predetermined threshold. The identification unit 222 extracts the rightmost edge pixel on each horizontal line extending in the nipple image as the leftmost pixel indicating the left end of the tongue, and the leftmost edge pixel on each horizontal line extending in the nipple image as the rightmost pixel indicating the right end of the tongue. The identification unit 222 also extracts the topmost edge pixel on each vertical line extending in the nipple image as the topmost pixel indicating the top end of the tongue, and the bottommost edge pixel as the bottommost pixel indicating the bottom end of the tongue. Note that, since the tongue image may not include the top end of the tongue, as shown in tongue images 500, 510, and 520 in FIG. 5 , the identification unit 222 may omit extracting the topmost pixel.

[0066] The identification unit 222 extracts a predetermined region within the region surrounded by the extracted leftmost pixel, rightmost pixel, topmost pixel (or the top edge of the tongue image), and bottommost pixel as a papilla image. The predetermined region may be, for example, the center of the tongue. The predetermined region may be the edge of the tongue, which is a region within a predetermined range from the leftmost pixel, rightmost pixel, or bottommost pixel, and / or the back of the tongue, which is a region within a predetermined range from the topmost pixel (or the top edge of the tongue image). At the edge of the tongue, papillae are rapidly lost due to friction within the oral cavity, and it takes a long time for them to regenerate. If the width of the missing papilla is large at the edge of the tongue, it is likely that the papilla has been missing for a long period of time. By having the identification unit 222 identify the papilla characteristics for each of the edge and back of the tongue, the information processing system 1 can determine the person's health condition for each part with different papilla characteristics, thereby enabling more accurate determination of the person's health condition. Furthermore, the specifying unit 222 may perform interpolation processing such as bilinear interpolation or bicubic interpolation within the predetermined region to interpolate pixels and enlarge the predetermined region, that is, the nipple image.

[0067] The identification unit 222 extracts edge pixels in the nipple image and, by labeling or the like, extracts, as a nipple, a group of edge pixels that are adjacent to each other in the horizontal, vertical, or diagonal direction from the extracted edge pixels. The identification unit 222 may extract the nipple based on a binary image. In this case, the identification unit 222 generates a binary image in which pixels in the nipple image whose gradation value is less than a binarization threshold are designated as invalid pixels and pixels whose gradation value is equal to or greater than the binarization threshold are designated as valid pixels. The identification unit 222 extracts, as a nipple, groups of pixels in the nipple image that correspond to groups of valid pixels that are adjacent to each other in the horizontal, vertical, or diagonal direction in the binary image. The identification unit 222 identifies the shape, size, regularity, or color of the nipple as nipple characteristics.

[0068] The identification unit 222 identifies the circularity of the nipple as the shape of the nipple. Circularity indicates the degree of similarity to a circle. For each nipple extracted from the nipple image, the identification unit 222 calculates the maximum value of the similarity between each nipple and circles of various sizes. The similarity may be a normalized cross-correlation, SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), or the like. The identification unit 222 calculates the statistical value (average, median, maximum, minimum, etc.) of the maximum similarity calculated for each nipple as the circularity of the nipple in the nipple image. The identification unit 222 may identify the circularity of the nipple using a learning model that has been pre-trained to output the circularity of the nipple in an input image. The learning model is pre-trained using a neural network, support vector machine, or the like, using combinations of multiple images containing nipples and the circularity of the nipple in each image as training data. The specifying unit 222 inputs the nipple image into the learning model and acquires the information output from the learning model as the circularity.

[0069] 6, the shape of each nipple of person A, who is in good health, is close to a circle, the shape of each nipple of person B, who is in slightly poor health, is slightly different from a circle, and the shape of each nipple of person C, who is in poor health, is significantly different from a circle. In other words, the better the health of the target person, the closer the shape of the nipple of the target person is to a circle, and the information processing system 1 can determine the health condition of the target person with high accuracy based on the shape of the nipple.

[0070] The identification unit 222 calculates, as the nipple size, statistical values ​​(average, median, maximum, minimum, standard deviation, etc.) of the area of ​​each nipple (number of pixels within each nipple) extracted from the nipple image. The identification unit 222 may also calculate, as the nipple size, statistical values ​​of the width of each nipple extracted from the nipple image in a predetermined direction (horizontal, vertical, etc.). The identification unit 222 may also identify the nipple size using a learning model that has been pre-trained to output the size of the nipple in an input image. The learning model is pre-trained using a neural network, support vector machine, etc., using combinations of multiple images containing nipples and the nipple sizes in each image as training data. The identification unit 222 inputs nipple images to the learning model and obtains the information output from the learning model as the size.

[0071] 6, the nipples of person A, who is in good health, are smaller than the nipples of person B, who is in slightly poor health, and the nipples of person C, who is in poor health, are larger than the nipples of person B, who is in slightly poor health. In other words, the better the health of a target person, the smaller the size of the target person's nipples, and the information processing system 1 can determine the target person's health with high accuracy based on the shape of the nipples.

[0072] For example, in a 700 x 700 pixel image, nipple image 600 has an average nipple diameter of 187 pixels and a particle size distribution (SD) of 5468, nipple image 610 has an average nipple diameter of 382 pixels and a particle size distribution (SD) of 5137, and nipple image 620 has an average nipple diameter of 1020 pixels and a particle size distribution (SD) of 14000.

[0073] The identification unit 222 calculates, as the nipple regularity, a statistical value (average, median, maximum, minimum, etc.) of the variance or standard deviation of the distance between each pair of adjacent nipples extracted from the nipple image. The identification unit 222 may perform frequency transformation on the nipple image using a two-dimensional Fourier transform, wavelet transform, etc., and calculate, as the nipple regularity, the difference between the maximum and minimum values ​​of frequency components whose amplitude is equal to or greater than a predetermined threshold. The identification unit 222 may identify the nipple regularity using a learning model that has been pre-trained to output the nipple regularity in an input image. The learning model is pre-trained using a neural network, support vector machine, etc., using combinations of multiple images containing nipples and the nipple regularity in each image as training data. The identification unit 222 inputs nipple images to the learning model and obtains the information output from the learning model as the regularity.

[0074] The specifying unit 222 may calculate the regularity of the nipple after downsizing the nipple image by thinning out pixels in the nipple image.

[0075] The identification unit 222 may also calculate the regularity of the nipple by histogram analysis. The identification unit 222 calculates the number of pixels having each gradation value for each gradation value within the gradation range of each pixel in the nipple image. The gradation value may be a Red value (red component value), a Green value (green component value), or a Blue value (blue component value), etc. The identification unit 222 generates a gradation histogram in which each gradation value is used as a class and the number of pixels having each gradation value is used as a frequency, and the gradation values ​​are arranged in order of magnitude.

[0076] FIG. 13 is a schematic diagram showing an example of a gradation histogram.

[0077] The horizontal axis of the tone histogram 1300 indicates tone values ​​(classes), and the vertical axis indicates the number of pixels having each tone value (frequency). The tone values ​​are arranged in descending order.

[0078] The specifying unit 222 specifies the maximum value M of the number of pixels, which is the frequency, in the gradation histogram, and the maximum frequency gradation value V1, which is the class where the frequency is maximum. Furthermore, the specifying unit 222 scans the classes of the gradation histogram in a direction of increasing gradation value from the maximum frequency gradation value V1, and specifies an upper limit gradation value V2 where the number of pixels is equal to or less than a predetermined percentage α (e.g., 10%) of the maximum value M. Furthermore, the specifying unit 222 scans the classes of the gradation histogram in a direction of decreasing gradation value from the maximum frequency gradation value V1, and specifies a lower limit gradation value V3 where the number of pixels is equal to or less than a predetermined percentage β (e.g., 10%) of the maximum value M. The smaller the difference between the upper limit gradation value V2 and the lower limit gradation value V3, the more regular the grain shape, and the greater the difference between the upper limit gradation value V2 and the lower limit gradation value V3, the more irregular the grain shape and the poorer the condition of the papilla. The specifying unit 222 calculates the difference D between the upper limit gradation value V2 and the lower limit gradation value V3 as the regularity of the nipple.

[0079] 6, the nipples of person A, who is in good health, are arranged more regularly than the nipples of person B, who is in slightly poor health, and the nipples of person C, who is in poor health, are arranged more irregularly than the nipples of person B, who is in slightly poor health. In other words, the better the health of the target person, the more regularly the nipples of the target person, and the information processing system 1 can determine the target person's health condition with high accuracy based on the shape of the nipples.

[0080] The identification unit 222 may identify a change in the nipple as the nipple feature. The identification unit 222 identifies a change in the shape, size, or regularity of the nipple identified from each of the nipple images in the plurality of tongue images captured at mutually different times as the nipple feature.

[0081] If the nipples gradually become more circular, the subject's physical condition is improving (recovering), and if the nipples gradually become less circular, the subject's physical condition is deteriorating. If the nipples are always close to circular, the subject's physical condition is always good, and if the nipples are always less circular, the subject's physical condition is always poor. Similarly, if the nipples gradually become smaller, the subject's physical condition is improving (recovering), and if the nipples lose their shape and fold over adjacent nipples, causing the particle size to increase, the subject's physical condition is deteriorating. If the nipples are always small, the subject's physical condition is always good, and if the nipples are always large, the subject's physical condition is always poor. Furthermore, if the nipples gradually become more regularly spaced, the subject's physical condition is improving (recovering), and if the nipples gradually become more irregularly spaced, the subject's physical condition is deteriorating. If the nipples are always regularly spaced, the subject's physical condition is always good, and if the nipples are always irregularly spaced, the subject's physical condition is always poor. The information processing system 1 can determine the health condition of the target person in more detail based on changes in the nipples.

[0082] The identification unit 222 identifies the gradation values ​​of multiple color components (e.g., red, green, and blue components) of each pixel included in the nipple in the nipple image as the nipple color. For each nipple extracted in the nipple image, the identification unit 222 may identify a statistical value (average, median, maximum, minimum, mode, etc.) of the gradation values ​​of each color component of each pixel included in each nipple as the nipple color in the nipple image.

[0083] As described above, the identification unit 222 may identify the nipple features by image analysis using machine learning. When a learning model is used to identify the nipple features, low-resolution images, particularly images with a lower resolution than the nipple image, may be used as training data for the learning model. This allows the determination unit 223 to appropriately identify the nipple features using the learning model, even if the tongue image acquired in step S101 has a low resolution.

[0084] Furthermore, when a learning model is used to identify nipple features, images captured by a camera mounted on a typical multi-function mobile phone and images captured by a microscope may be used as training data for the learning model. Alternatively, images captured by a camera mounted on a typical multi-function mobile phone and images captured by a single-lens reflex camera may be used as training data for the learning model. On the other hand, when using the learning model, the identification unit 222 identifies nipple features by inputting only images captured by the imaging device 104 (a camera mounted on a typical multi-function mobile phone) into the learning model. This allows the learning model to be trained efficiently, and the identification unit 222 to identify nipple features with high accuracy.

[0085] The identification unit 222 may also identify the characteristics of tongue papillae through template matching. For example, the server device 200 stores multiple sample images containing papillae with various characteristics in the second storage device 210 in advance. The identification unit 222 cuts out regions of the same size as each sample image from various positions on the papilla image and calculates the similarity between each cut-out region and each sample image. If the calculated similarity is equal to or greater than a predetermined threshold, the identification unit 222 determines that the tongue included in the tongue image contains papillae with the characteristics included in the sample image. Alternatively, the identification unit 222 may calculate feature amounts from each region cut out from the papilla image and each sample image, and if the similarity of the feature amounts is equal to or greater than a predetermined threshold, determine that the tongue included in the papilla image contains papillae with the characteristics included in the sample image. The feature amount may be the gradient of the image (e.g., HOG feature amount), etc. The similarity of the feature amount may be cosine similarity, inner product, etc.

[0086] Next, the determination unit 223 determines the state of the blood vessels and / or the state of blood flow of the target person based on the characteristics of the nipple identified by the identification unit 222 (step S105). For example, the determination unit 223 determines the state of the blood vessels of the target person based on the speed of the blood flow, the distribution of the blood flow, the softness of the blood vessels, or the quality of the blood flow. The determination unit 223 determines the quality of the blood flow as the state of the blood flow of the target person. The quality of the blood flow distribution is indicated, for example, by an evaluation value that has a higher value when the blood flow distribution is less uneven (the more uniform the blood vessel distribution) and a lower value when the blood flow distribution is more uneven (the more uneven the blood vessel distribution). The softness of the blood vessels is indicated, for example, by an evaluation value that has a higher value when the blood vessels are softer (the more elastic the blood vessels) and a lower value when the blood vessels are harder (the more elastic the blood vessels). The quality of blood flow refers to the quality of blood circulation (flow). The quality of blood flow is indicated by an evaluation value that, for example, the better the blood circulation (the thinner the blood, i.e., the lower the viscosity of the blood), the higher the value, and the poorer the blood circulation (the thicker the blood, i.e., the higher the viscosity of the blood), the lower the value. On the other hand, with regard to the quality of blood flow, there are cases where the viscosity of blood is high even when the blood circulation is good, and cases where the viscosity of blood is low even when the blood circulation is poor.

[0087] For example, the server device 200 may store in advance in the second storage device 210 a relational expression or table indicating the relationship between nipple features and blood vessel conditions or blood flow conditions. In this table, nipple features and / or blood vessel conditions or blood flow conditions may be classified using one or more thresholds. The determination unit 223 refers to the relational expression or table stored in the second storage device 210 and identifies the blood vessel conditions or blood flow conditions corresponding to the nipple features identified by the identification unit 222 as the blood vessel conditions or blood flow conditions of the target person. The determination unit 223 may determine the blood vessel conditions or blood flow conditions of the target person using a learning model that has been pre-trained to output the blood vessel conditions or blood flow conditions of a specific person when the nipple features of that person are input. The learning model is pre-trained using a neural network, a support vector machine, or the like, using combinations of nipple features and blood vessel conditions or blood flow conditions of multiple people as training data. The determination unit 223 inputs the nipple features identified by the identification unit 222 into the learning model, and identifies the information output from the learning model as the state of blood vessels or blood flow of the target person.

[0088] FIG. 7 is a schematic diagram showing an example of a blood flow distribution image showing the distribution of blood flow volume, captured using LSFG (Laser Speckle Flowgraphy).

[0089] 7 shows blood flow distribution images 700, 710, and 720 of multiple target persons. Blood flow distribution image 700 is that of person A, who is in good health, blood flow distribution image 710 is that of person B, who is in somewhat poor health, and blood flow distribution image 720 is that of person C, who is in poor health. LSFG generates blood flow distribution images by irradiating near-infrared laser light toward the person's skin (blood vessels) and measuring the pattern of light scattered by red blood cells with an imaging sensor such as a CCD.

[0090] Person A's blood flow has an ideal speed, and light is scattered moderately in the blood flow distribution image 700. Person B's blood flow is slightly turbulent relative to the ideal speed, and light is slightly biased in the blood flow distribution image 710. Person C's blood flow is significantly turbulent relative to the ideal speed, and light is significantly biased in the blood flow distribution image 720. In other words, there is a correlation between nipple characteristics, particularly nipple shape, size, or regularity, and blood flow speed. Therefore, the information processing system 1 can accurately determine the blood flow speed of a target person based on the nipple shape, size, or regularity of the target person.

[0091] Furthermore, there is almost no unevenness in the blood flow distribution of person A, and the spatial bias of the blood flow volume is slight in the blood flow distribution image 700. There is some unevenness in the blood flow distribution of person B, and areas with high blood flow volume are spatially localized in the blood flow distribution image 710. There is a lot of unevenness in the blood flow distribution of person C, and the blood flow volume is significantly biased locally in the blood flow distribution image 720. In other words, there is a correlation between nipple features, particularly the shape, size, or regularity of the nipple, and the state of the blood vessels, particularly the distribution of blood flow. Therefore, the information processing system 1 can accurately determine the blood flow distribution of a target person based on the shape, size, or regularity of the target person's nipple.

[0092] FIG. 8 is a schematic diagram for explaining the relationship between the characteristics of the nipple and the flexibility of blood vessels and the quality of blood flow.

[0093] Fig. 8 schematically shows a cross section of a nipple 800 and a blood vessel 801 located immediately below it, and a cross section of a skin surface 802 and a blood vessel 803 located immediately below it, of person A, who is in good health. Fig. 8 also schematically shows a cross section of a nipple 810 and a blood vessel 811 located immediately below it, and a cross section of a skin surface 812 and a blood vessel 813 located immediately below it, of person B, who is in somewhat poor health. Fig. 8 also schematically shows a cross section of a nipple 820 and a blood vessel 821 located immediately below it, and a cross section of a skin surface 822 and a blood vessel 823 located immediately below it, of person C, who is in poor health. Fig. 8 also schematically shows a cross section of a nipple 830 and a blood vessel 831 located immediately below it, and a cross section of a skin surface 832 and a blood vessel 833 located immediately below it, of person D, who has excessive blood circulation.

[0094] As shown in Figure 8, the distance between the nipple and the blood vessels located immediately below it is much smaller than the distance between the skin surface and the blood vessels located immediately below it, and the shape, size, regularity, or color of the nipple is strongly influenced by the blood vessels located immediately below it.

[0095] Person A's blood vessels 801 and 803 are flexible, and blood circulation in the blood vessels 801 and 803 is appropriate. In this case, person A's blood vessels 801 and 803 have an ideal shape. Therefore, the shape of the nipple 800 is close to circular, the size of the nipple 800 is appropriate, and the nipples 800 are regularly arranged. On the other hand, person B's blood vessels 811 and 813 have slightly low elasticity, and blood circulation in the blood vessels 811 and 813 is somewhat poor. In this case, person B's blood vessels 811 and 813 have a distorted shape. Therefore, the shape of the nipple 810 deviates slightly from a circle, and the size of the nipple 810 is large. The shapes and sizes of the multiple nipples 810 are non-uniform, and the multiple nipples 810 are irregularly arranged. Furthermore, blood flow is restricted in some parts of person C's blood vessels 821 and 823, resulting in poor blood circulation in the blood vessels 821 and 823. In this case, person C's blood vessels 821 and 823 have a collapsed shape. As a result, the shape of the nipple 820 is significantly different from a circle, and the size of the nipple 820 is extremely large. Because the shapes and sizes of the multiple nipples 810 are extremely non-uniform, the nipples 820 are arranged in an extremely irregular manner. Furthermore, person D's blood vessels 831 and 833 are dilated, resulting in excessive blood flow in the blood vessels 831 and 833 and excessive blood circulation. Because the blood vessels 831 and 833 are dilated, the size of the nipple 830 is extremely large.

[0096] Thus, there is a correlation between nipple characteristics, particularly nipple shape, size, or regularity, and vascular softness. Therefore, the information processing system 1 can accurately determine the vascular softness of the target person based on the nipple shape, size, or regularity of the target person.

[0097] Furthermore, nipple characteristics, particularly nipple shape, size, or regularity, correlate with blood circulation, so the information processing system 1 can accurately determine the quality of blood flow in a target person based on the nipple shape, size, or regularity.

[0098] Furthermore, when the identification unit 222 identifies the characteristics of the papilla for each of the end and back of the tongue, the determination unit 223 identifies the state of the blood vessels of the target person for each of the end and back of the tongue.

[0099] Next, the determination unit 223 determines the health condition of the target person based on the state of the target person's blood vessels (step S106). For example, the determination unit 223 determines the state of the target person's skin as the health condition of the target person. The determination unit 223 determines the degree of roughness, wrinkles, pores, texture, or blemishes as the skin condition of the target person.

[0100] For example, the server device 200 pre-stores in the second storage device 210 a relational equation or table showing the relationship between vascular conditions and skin conditions. In this table, vascular conditions and / or skin conditions may be classified using one or more thresholds. The determination unit 223 references the relational equation or table stored in the second storage device 210 and identifies the skin condition corresponding to the vascular condition determined in step S105 as the skin condition of the target person. The determination unit 223 may determine the skin condition of the target person using a learning model that has been pre-trained to output the skin condition of a specific person when the vascular condition of that person is input. The learning model is pre-trained using a neural network, a support vector machine, or the like, using combinations of vascular conditions and skin conditions of multiple people as training data. The determination unit 223 inputs the vascular condition determined in step S105 into the learning model and identifies the information output from the learning model as the skin condition of the target person.

[0101] FIG. 9 is a schematic diagram showing an example of a skin image including a person's skin.

[0102] 9 shows skin images 900, 910, and 920 of multiple target persons. Skin image 900 includes the skin of person A, who is in good health; skin image 910 includes the skin of person B, who is in slightly poor health; and skin image 920 includes the skin of person C, who is in poor health. As shown in skin image 900, person A, who is in good health, has fine, healthy skin. As shown in skin image 910, person B, who is in slightly poor health, has slightly rough skin. As shown in skin image 920, person C, who is in poor health, has very rough skin. In other words, there is a correlation between nipple characteristics, particularly the shape, size, or regularity of the nipple, the state of blood vessels, and the skin condition, particularly the degree of roughness. Therefore, information processing system 1 can accurately determine the degree of roughness of the target person's skin based on the shape, size, or regularity of the target person's nipple. In particular, the information processing system 1 determines the condition of the target person's blood vessels based on the shape, size or regularity of the target person's nipples, and then determines the degree of skin roughness of the target person based on the condition of the target person's blood vessels, thereby being able to determine the degree of skin roughness of the target person with higher accuracy.

[0103] FIG. 10 is a schematic diagram for explaining the relationship between nipple characteristics and the degree of wrinkles, pores, texture, or spots.

[0104] 10 shows nipple image 1000, skin image 1001, skin image 1002, and skin image 1003 of person E, and nipple image 1010, skin image 1011, skin image 1012, and skin image 1013 of person F. As shown in nipple image 1000 and nipple image 1010, the circularity of person F's nipple is lower than that of person E, and the regularity of person F's nipple is lower than that of person E. Skin image 1001 and skin image 1011 contain the parts of person E's and person F's cheeks with the most wrinkles W, respectively. As shown in skin image 1001 and skin image 1011, the length, depth, and number of wrinkles W of person F are greater than the length, depth, and number of wrinkles W of person E. Skin image 1002 and skin image 1012 contain the parts of person E's and person F's cheeks with the most pores P, respectively. As shown in skin images 1002 and 1012, the size, depth, and number of pores P of person F are greater than the size, depth, and number of pores P of person E. Skin images 1003 and 1013 contain the portions of person E's and person F's cheeks with the most blemishes B, respectively. As shown in skin images 1003 and 1013, the size and number of blemishes B of person F are greater than the size and number of blemishes B of person E. In other words, there is a correlation between nipple characteristics, particularly the circularity or regularity of the nipple, and the degree of wrinkles, pores, texture, or blemishes. Therefore, information processing system 1 can accurately determine the degree of wrinkles, pores, texture, or blemishes of a target person based on the circularity or regularity of the target person's nipples.

[0105] The determination unit 223 may determine the state of the target person's blood vessels as the target person's health condition. For example, the determination unit 223 may determine the target person's blood vessel condition as the speed of the target person's blood flow, the distribution of the blood flow, the softness of the blood vessels, or the quality of the blood flow. As described above, there is a correlation between nipple characteristics, particularly the circularity, size, or regularity of the nipple, and the state of the blood vessels, and the information processing system 1 can determine the state of the target person's blood vessels with high accuracy based on the circularity, size, or regularity of the target person's nipple.

[0106] The determination unit 223 may also determine the target person's health condition based on the subjective physical condition and / or environmental information of the target person acquired by the receiving unit 221. For example, the server device 200 pre-stores in the second storage device 210 a relational expression or a table showing the relationship between the vascular condition, the subjective physical condition and / or environmental information, and each of the above-mentioned health conditions. In this table, the vascular condition, the subjective physical condition, and / or the environmental information may be classified using one or more thresholds. The determination unit 223 refers to the relational expression or table stored in the second storage device 210 and identifies, as the target person's health condition, each health condition corresponding to the vascular condition determined in step S105 and the subjective physical condition and / or environmental information acquired by the receiving unit 221. The determination unit 223 may determine the target person's health condition using a learning model pre-trained to output the health condition of a specific person when the vascular condition, the subjective physical condition, and / or environmental information of the specific person are input. The learning model is pre-trained using a neural network, a support vector machine, or the like, using combinations of vascular conditions, subjective physical condition and / or environmental information, and health conditions for multiple people as training data. The determination unit 223 inputs the vascular conditions determined in step S105 and the subjective physical condition and / or environmental information acquired by the receiving unit 221 into the learning model, and identifies the information output from the learning model as the health condition of the target person. This allows the information processing system 1 to determine the health condition of the target person with higher accuracy.

[0107] Furthermore, when the target person's health is impaired, the determination unit 223 may determine the cause of the target person's impaired health as the target person's health condition based on the target person's subjective physical condition and / or environmental information. For example, when the subjective physical condition indicates that the target person has a fever or pain, the determination unit 223 determines that the cause of the target person's impaired health is illness or injury. When the environmental information indicates that the target person consumes tobacco, sweets, or the like, the determination unit 223 determines that the cause of the target person's impaired health is advanced oxidation. This allows the information processing system 1 to present the cause of the impaired health to the target person, thereby improving convenience for the target person.

[0108] Furthermore, when the identification unit 222 has identified papilla characteristics for each of the tip and back portions of the tongue, the determination unit 223 determines the health condition of the target person for each of the tip and back portions of the tongue. Note that the determination unit 223 may determine that the target person's health condition is not good if it determines that the target person's health condition is not good at either the tip or back portion of the tongue. This allows the information processing system 1 to prevent erroneous determinations that the target person's health condition is good when it is actually poor. Note that the determination unit 223 may determine that the target person's health condition is not good if it determines that the target person's health condition is not good at both the tip and back portion of the tongue. This allows the information processing system 1 to prevent erroneous determinations that the target person's health condition is poor when it is actually good.

[0109] Next, the output control unit 224 outputs information indicating the health condition of the target person determined by the determination unit 223 by transmitting it to the terminal device 100 via the second communication device 201 (step S107). The output control unit 224 transmits the target person's health condition itself as information indicating the target person's health condition. Note that the output control unit 224 may generate a normalized value of the target person's health condition, an image indicating the target person's health condition, a graph indicating the target person's health condition, or the like as information indicating the target person's health condition, and transmit the generated value, image, graph, or the like.

[0110] Furthermore, the output control unit 224 may transmit, as information indicating the health condition of the target person, information indicating factors of the target person's health condition in addition to or instead of the above-mentioned information. Furthermore, the output control unit 224 may transmit, as information indicating the health condition of the target person, information indicating changes in the target person's health condition in addition to or instead of the above-mentioned information.

[0111] Next, the display control unit 122 of the terminal device 100 acquires information indicating the health condition of the target person by receiving it from the server device 200 via the first communication device 103. The display control unit 122 displays the received information indicating the health condition of the target person on the first display device 102 (step S108). This completes the determination process.

[0112] Note that the acquisition of subjective physical condition and / or environmental information in step S101 and the transmission and reception of subjective physical condition and / or environmental information in steps S102 and S103 may be omitted. In this case, in step S106, the determination unit 223 determines the health condition of the target person without using the subjective physical condition and / or environmental information of the target person.

[0113] Alternatively, step S105 may be omitted. In this case, in step S106, the determination unit 223 directly determines the health condition of the target person based on the nipple features of the target person. For example, the server device 200 pre-stores in the second storage device 210 a relational expression or table showing the relationship between each of the above-described nipple features and each health condition. In this table, the nipple features and / or health information may be classified using one or more thresholds. The determination unit 223 references the relational expression or table stored in the second storage device 210 and identifies, as the health condition of the target person, each health condition corresponding to the nipple feature identified by the identification unit 222 in step S104. The determination unit 223 may determine the health condition of the target person using a learning model pre-trained to output the health condition of a specific person when the nipple features of the specific person are input. The learning model is pre-trained using a neural network, a support vector machine, or the like, using combinations of nipple features and health conditions of multiple people as training data. The determination unit 223 inputs the nipple characteristics identified by the identification unit 222 in step S104 into the learning model, and identifies the information output from the learning model as the health condition of the target person. In this case, the information processing system 1 can also determine the health condition of the target person with high accuracy.

[0114] In this case, the server device 200 may store in advance in the second storage device 210 a relational expression or a table showing the relationship between nipple characteristics, subjective physical condition and / or environmental information, and health condition, and the determination unit 223 may identify the target person's health condition using this relational expression or table. The determination unit 223 may also determine the target person's health condition using a learning model that has been pre-trained to output the health condition of a specific person when the nipple characteristics, subjective physical condition, and / or environmental information of the person are input.

[0115] Furthermore, the determination unit 223 may determine the nutritional state of the target person as the health state of the target person. For example, the determination unit 223 determines, as the nutritional state of the target person, whether or not a predetermined component (such as protein, vitamin B, or zinc) is deficient, or whether or not a predetermined component (such as carbohydrate) is excessive, and / or the degree of either.

[0116] For example, the server device 200 may store in advance in the second storage device 210 a relational expression or table showing the relationship between nipple features and nutritional status. In this table, nipple features and / or nutritional status may be classified using one or more thresholds. The determination unit 223 refers to the relational expression or table stored in the second storage device 210 and determines the nutritional status corresponding to the nipple features identified by the identification unit 222 as the nutritional status of the target person. The determination unit 223 may determine the nutritional status of the target person using a learning model that has been pre-trained to output the nutritional status of a specific person when the nipple features of the specific person are input. The learning model is pre-trained using a neural network, a support vector machine, or the like, using combinations of nipple features and nutritional statuses of multiple people as training data. The determination unit 223 inputs the nipple features identified by the identification unit 222 into the learning model and determines the information output from the learning model as the nutritional status of the target person.

[0117] FIG. 11 is a schematic diagram for explaining the relationship between the characteristics of the nipple and the amounts of protein, vitamin B, and zinc.

[0118] 11 shows nipple images 1100 of person G, 1110 of person H, 1120 of person I, and 1130 of person J. Person G has an appropriate amount of protein and vitamin B, while person H is deficient in protein and vitamin B. Person I has an appropriate amount of zinc, while person J is deficient in zinc. Images 1101, 1111, 1121, and 1131 are enlarged views of portions of nipple image 1100, nipple image 1110, nipple image 1120, and nipple image 1130, respectively. As shown in images 1101 and 1111, the nipples of person H, who is deficient in protein and vitamin B, are closer to yellowish white in color than the nipples of person G, who has an appropriate amount of protein and vitamin B. Furthermore, as shown in images 1121 and 1131, the nipples of person J, who is deficient in zinc, are closer to white in color than the nipples of person I, who has an appropriate amount of zinc. In other words, there is a correlation between nipple characteristics, particularly nipple color, and the amounts of protein, vitamin B, and zinc. Therefore, the information processing system 1 can accurately determine the amounts of protein, vitamin B, and zinc of the target person based on the nipple characteristics of the target person.

[0119] FIG. 12 is a schematic diagram for explaining the relationship between the characteristics of the papilla and the amount of carbohydrates.

[0120] 12 shows a nipple image 1200 of person K, who has excess sugar. As shown in nipple image 1200, the redness of the nipple of person K, who has excess sugar, is lighter than the redness of the nipple of a person with an appropriate amount of sugar. In other words, there is a correlation between nipple characteristics, particularly nipple color, and the amount of sugar. Therefore, the information processing system 1 can accurately determine the amount of sugar of a target person based on the color of the target person's nipple.

[0121] As described above in detail, the information processing system 1 outputs the health condition of the target person based on the image of the tongue papilla of the target person. This enables the information processing system 1 to output the health condition of the target person with higher accuracy. Therefore, the user of the terminal device 100 can correctly recognize the health condition of the target person and provide appropriate advice to the target person. Furthermore, the information processing system 1 can output the health condition of the target person inexpensively by using a terminal device with a general-purpose camera, without using an expensive system such as an LSFG.

[0122] Second Embodiment In this embodiment, in step S104 of FIG. 4 , the identification unit 222 identifies characteristics of the overall shape of the target person's tongue based on the tongue image. In step S106, the determination unit 223 determines the target person's health condition based on the characteristics of the overall shape of the target person's tongue. The processing of step S105 is omitted. The tongue image includes the entire tongue. In particular, the identification unit 222 identifies characteristics of the target person's tongue fissures as characteristics of the overall shape of the target person's tongue. The determination unit 223 determines the target person's physical nutritional state and / or psychological stress as the target person's health condition. For example, the physical nutritional state and / or psychological stress may be whether or not the body is under stress to the extent that it is unable to absorb nutrients. The physical nutritional state may be whether or not the content and / or timing of meals are appropriate. The physical nutritional state may be the amount or deficiency of vitamins, minerals, and proteins. The physical nutritional state may be whether or not there is a vitamin deficiency, particularly whether or not there is a vitamin deficiency in the gastrointestinal tract.

[0123] FIG. 14 is a schematic diagram showing an example of a tongue image according to this embodiment.

[0124] 14 shows tongue images 1400, 1410, and 1420 of multiple target persons, respectively. Tongue image 1400 includes the tongue of a person without a fissure, tongue image 1410 includes the tongue of a person with a small fissure, and tongue image 1420 includes the tongue of a person with a large fissure.

[0125] For example, the identification unit 222 identifies the characteristics of tongue cracks through histogram analysis. The identification unit 222 generates a green tongue image in which the green component value (Green value) is used as the gradation value from the tongue image, which is an RGB image. Alternatively, the identification unit 222 generates a saturation tongue image in which the saturation component value (Saturation value) is used as the gradation value from the tongue image converted from an RGB image to an HSV image. The identification unit 222 calculates the sum of the gradation values ​​of the pixels included in each vertical line for each of multiple vertical lines extending vertically in the green tongue image or the saturation tongue image. The identification unit 222 generates a position histogram in which the horizontal position of each vertical line is used as a class and the calculated sum for each vertical line is used as a frequency, arranging the vertical lines in order of their horizontal position.

[0126] FIG. 15 is a schematic diagram showing an example of a position histogram.

[0127] Position histogram 1500 is a position histogram generated from tongue image 1400, position histogram 1510 is a position histogram generated from tongue image 1410, and position histogram 1520 is a position histogram generated from tongue image 1420. The horizontal axis of each position histogram indicates the horizontal position (rank) of each vertical line, and the vertical axis indicates the sum (frequency) calculated for each vertical line. The sums calculated for each vertical line are arranged in order of the horizontal position of each vertical line.

[0128] The identification unit 222 scans the classes of the position histogram from one end to the other in the horizontal direction and identifies groups G that are sandwiched between vertical lines whose total value, which is a frequency, is less than a reference value R and that have consecutive vertical lines whose total value is equal to or greater than the reference value R. The reference value R is set, for example, to a value obtained by multiplying the average value of the total values ​​of each vertical line by a predetermined coefficient. The identification unit 222 determines that the tongue contains small cracks if the number of groups G is equal to or greater than a predetermined threshold, and determines that the tongue does not contain small cracks if the number of groups G is less than the threshold. Furthermore, the identification unit 222 determines that the tongue contains large cracks if the number W of vertical lines included in any group G is equal to or greater than a predetermined threshold, and determines that the tongue does not contain small cracks if the number W of vertical lines included in both groups G is less than the threshold. The identification unit 222 identifies, as characteristics of tongue cracks, whether the tongue contains small cracks and whether the tongue contains large cracks.

[0129] The identification unit 222 may calculate the sum of the gradation values ​​of pixels included in each horizontal line extending horizontally in the tongue image, and identify groups of consecutive horizontal lines whose sum is equal to or greater than the reference value and sandwiched between horizontal lines whose sum is less than a reference value. In this case, the identification unit 222 calculates the characteristics of tongue fissures from the number of identified groups or the number of horizontal lines included in each group. The identification unit 222 may identify, as the characteristics of tongue fissures, the amount of change in the number of groups or the number of vertical or horizontal lines included in each group, calculated from multiple tongue images in which the tongue of the target person is captured at a predetermined time interval.

[0130] The identification unit 222 may identify the characteristics of the tongue cracks by template matching. For example, the server device 200 stores a plurality of sample images including cracks of various sizes in the second storage device 210 in advance. The identification unit 222 cuts out areas of the same size as each sample image from various positions on the tongue image and calculates the similarity between each cut-out area and each sample image. If the calculated similarity is equal to or greater than a predetermined threshold, the identification unit 222 determines that the tongue included in the tongue image contains a crack of the size included in the sample image. Furthermore, the identification unit 222 may perform histogram matching between each area and each sample image and then calculate the similarity between each area and each sample image.

[0131] For example, the server device 200 may store in advance in the second storage device 210 a relational expression or table showing the relationship between crack features and the body's nutritional state and / or psychological stress. In this table, the crack features and / or the body's nutritional state and / or psychological stress may be classified using one or more thresholds. The determination unit 223 may refer to the relational expression or table stored in the second storage device 210 and identify the nutritional state and / or psychological stress corresponding to the crack features identified by the identification unit 222 as the target person's nutritional state and / or psychological stress. The determination unit 223 may determine the target person's nutritional state and / or psychological stress using a learning model pre-trained to output the person's nutritional state and / or psychological stress when the crack features of a specific person are input. The learning model is pre-trained using a neural network, a support vector machine, or the like, using combinations of crack features and nutritional state and / or psychological stress for multiple people as training data. The determination unit 223 inputs the characteristics of the cracks identified by the identification unit 222 into the learning model, and identifies the information output from the learning model as the nutritional state and / or psychological stress of the target person.

[0132] If a person has small or large cracks on their tongue, their gastrointestinal condition may be poor. The information processing system 1 can determine the person's health condition with high accuracy by using the characteristics of the cracks on the tongue.

[0133] Third Embodiment In this embodiment, in step S104 of FIG. 4 , the identification unit 222 identifies characteristics of the overall shape of the target person's tongue based on the tongue image, and in step S106, the determination unit 223 determines the target person's health condition based on the characteristics of the overall shape of the target person's tongue. The processing of step S105 is omitted. The tongue image includes the entire tongue. The tongue image includes a tongue imaged from the side. In particular, the identification unit 222 identifies characteristics of the thickness of the target person's tongue as a characteristic of the overall shape of the target person's tongue. The determination unit 223 determines metabolic abnormality as the target person's health condition. For example, the metabolic abnormality is an abnormality in body water, an abnormality in carbohydrate metabolism, or an abnormality in lipids.

[0134] FIG. 16 is a schematic diagram showing an example of a tongue image according to this embodiment.

[0135] 16 shows a tongue image 1600. The tongue image 1600 includes a tongue imaged from the side.

[0136] The identification unit 222 identifies the thickness of the target person's tongue by analyzing the shape of the object using the image. The identification unit 222 identifies the orientation of the tongue included in the tongue image using known image processing technology. For example, the identification unit 222 extracts feature points such as the tip, side edges, upper edge, and lower edge of the tongue from the tongue image and identifies the orientation of the tongue based on the positional relationship of the extracted feature points. The identification unit 222 may identify the orientation of the tongue by performing template matching with multiple sample images containing tongues captured in various orientations. If the orientation of the tongue is not within a predetermined range, the identification unit 222 transmits a request signal to the terminal device 100 requesting reacquisition of the tongue image. Upon receiving the request signal, the terminal device 100 reacquires the tongue image. In this case, the terminal device 100 may reacquire the tongue image while displaying a guide on the first display device 102 for aligning the tongue position so that the tongue is properly imaged.

[0137] In steps S101 and S102 of FIG. 4 , the transmitting unit 121 may acquire a facial image of the face along with the tongue image and transmit the acquired image to the server device 200. In this case, in step S103, the receiving unit 221 receives the facial image along with the tongue image. The identifying unit 222 extracts feature points such as the corners of the eyes, the tip of the nose, the base of the nose, the edges of the lips, and the center of the lips from the facial image using known image processing techniques, and identifies the facial orientation based on the positional relationship of the extracted feature points. If the facial orientation is not sideways, the identifying unit 222 transmits a request signal to the terminal device 100 requesting that the user be instructed to turn sideways. Upon receiving the request signal, the terminal device 100 reacquires the tongue image and facial image. In this case, the terminal device 100 notifies the user by displaying an instruction to turn sideways on the first display device 102.

[0138] 4, the transmitting unit 121 may acquire a front tongue image captured from the front, along with a tongue image captured from the side, and transmit the acquired images to the server device 200. In this case, in step S103, the receiving unit 221 receives the front tongue image along with the tongue image. The identifying unit 222 identifies the orientation of the tongue included in the front tongue image using known image processing techniques. The identifying unit 222 identifies the orientation of the tongue based on the positional relationship of feature points, template matching, or the like. If the orientation of the tongue is not horizontal but points upward or downward, the identifying unit 222 transmits a request signal to the terminal device 100 requesting that the user stick out their tongue flat. Upon receiving the request signal, the terminal device 100 reacquires the tongue image. In this case, the terminal device 100 notifies the user by displaying an instruction to stick out their tongue flat on the first display device 102.

[0139] The identification unit 222 extracts edge pixels indicating the edge of the tongue in the same manner as when cutting out a papilla image, and extracts the area surrounded by the extracted edge pixels as the tongue area. The identification unit 222 rotates the tongue image so that the tongue extends horizontally. For example, the identification unit 222 extracts feature points such as the tip, side edges, upper edge, and lower edge of the tongue within the tongue area, performs affine transformation so that each feature point is positioned within a predetermined range, and rotates the tongue image.

[0140] A tongue image 1610 shown in FIG. 16 is an image obtained by rotating the tongue image 1600 so that the tongue extends horizontally.

[0141] The identification unit 222 calculates the distance H between the upper and lower ends of the tongue at predetermined intervals along the horizontal direction from the tip side toward the throat side in the rotated tongue image. The identification unit 222 identifies the amount of change in the distance H calculated at the predetermined intervals as a characteristic of the tongue thickness of the target person. The thicker the tongue, the greater the amount of change in the distance H.

[0142] The identification unit 222 may calculate the circularity of the tongue as a feature of the tongue thickness. The identification unit 222 identifies the center position (center of gravity position) of the tongue, and also identifies the position (tip position of the tongue) where the distance H is smallest, and the upper and lower end positions of the tongue at the horizontal position where the distance H is largest. The identification unit 222 calculates each distance between the center position of the tongue and the tip position, upper end position, and lower end position, and calculates the circularity so that the larger the difference between the distances is, the lower the circularity becomes, and the smaller the difference between the distances is, the higher the circularity becomes. The thicker the tongue, the higher the circularity of the tongue.

[0143] For example, the server device 200 pre-stores in the second storage device 210 a relational expression or table showing the relationship between tongue thickness characteristics and the presence or absence of metabolic disorders. In this table, tongue thickness characteristics and / or the presence or absence of metabolic disorders may be classified using one or more thresholds. The determination unit 223 references the relational expression or table stored in the second storage device 210 and determines the presence or absence of metabolic disorders corresponding to the tongue thickness characteristics identified by the identification unit 222 as the presence or absence of metabolic disorders in the target person. The determination unit 223 may determine the presence or absence of metabolic disorders in the target person using a learning model pre-trained to output the presence or absence of metabolic disorders in a target person when the tongue thickness characteristics of a specific person are input. The learning model is pre-trained using a neural network, a support vector machine, or the like, using combinations of tongue thickness characteristics and the presence or absence of metabolic disorders for multiple people as training data. The determination unit 223 inputs the tongue thickness characteristics identified by the identification unit 222 into the learning model and determines the presence or absence of metabolic disorders in the target person based on the information output from the learning model.

[0144] When the determination unit 223 determines that the target person's health condition is abnormal in intracorporeal water, it determines that the cause of the target person's health condition is kidney abnormality or excessive salt intake.When the determination unit 223 determines that the target person's health condition is abnormal in lipids, it determines that the cause is liver abnormality, excessive intake of sweets, or excessive intake of carbohydrates.

[0145] The thicker the tongue, the more water there is in the body and the more likely it is that the kidneys or gastrointestinal system is in poor condition. Also, the thicker the tongue, the more lipid there is in the body and the possibility that lipid and water metabolism is not good. The information processing system 1 can determine a person's health condition with high accuracy by using the characteristics of tongue thickness.

[0146] Fourth Embodiment In this embodiment, in step S104 of FIG. 4, the identification unit 222 identifies the characteristics of the surface property of the tongue of the target person based on the tongue image, and in step S106, the determination unit 223 determines the health condition of the target person based on the characteristics of the surface property of the tongue of the target person. The processing of step S105 is omitted. The tongue image includes the surface property of the tongue. In particular, the identification unit 222 identifies the amount of tongue coating on the surface of the tongue of the target person as the characteristics of the surface property of the tongue of the target person. The determination unit 223 determines the nutritional state of the body as the health condition of the target person. For example, the nutritional state of the body is good lipid metabolism.

[0147] For example, the identification unit 222 identifies the amount of tongue coating on the surface of the tongue through texture analysis. The identification unit 222 generates a green tongue image or a saturation tongue image from the tongue image. Alternatively, the identification unit 222 generates a chromatic tongue image in which the yellow and blue chromaticity component values ​​(b* values) are used as gradation values ​​from a tongue image converted from an RGB image to a Lab image. The identification unit 222 calculates the number of pixels having each gradation value for each gradation value within the gradation range of each pixel in the green tongue image, saturation tongue image, or chromatic tongue image. The identification unit 222 generates a gradation histogram in which each gradation value is used as a class and the number of pixels having each gradation value is used as a frequency, and the gradation values ​​are arranged in order of magnitude.

[0148] FIG. 17 is a schematic diagram showing an example of a gradation histogram.

[0149] 17, the horizontal axis represents the gray level (class), and the vertical axis represents the number of pixels having each gray level (frequency). The gray levels are arranged in descending order.

[0150] The identification unit 222 scans the classes of the gradation histogram from low to high, and identifies a maximum frequency gradation value U1, which is the gradation value at which the number of pixels (frequency) is greatest, and a maximum gradation value U2, which is the maximum gradation value at which the frequency is greater than 0. The identification unit 222 identifies an internal division gradation value U3, which divides the maximum frequency gradation value U1 and the maximum gradation value U2 internally at a predetermined ratio. The identification unit 222 generates a binary image such that pixels in the tongue image whose gradation value is equal to or greater than the maximum frequency gradation value U1 and equal to or less than the internal division gradation value U3 are valid pixels, and pixels whose gradation value is less than the maximum frequency gradation value U1 or less than the internal division gradation value U3 are invalid pixels.

[0151] 17 shows an example of a binary image. In image 1710, pixels whose gradation values ​​are equal to or greater than the maximum frequency gradation value U1 and equal to or less than the internal division gradation value U3 are white, and pixels whose gradation values ​​are less than the maximum frequency gradation value U1 or less than the internal division gradation value U3 are black.

[0152] The determination unit 222 determines the area (number of pixels) of effective pixels in the generated binary image as the amount of tongue coating on the surface of the tongue. If the resolution of the tongue image is higher than a predetermined threshold, the determination unit 222 may perform noise removal such as smoothing on the tongue image, generate a gradation histogram, and generate a binary image.

[0153] The determination unit 222 may determine the amount of tongue coating on the surface of the tongue based on the number of edge pixels of a specific color component in the tongue image. For example, the determination unit 222 generates a first tongue image from an RGB tongue image, in which a specific color component value (red value, green value, or blue value) is used as a gradation value. Alternatively, the determination unit 222 generates a second tongue image from a tongue image converted from an RGB image to an HSV image, in which a specific component value (hue value, saturation value, or value) is used as a gradation value. Alternatively, the determination unit 222 generates a third tongue image from a tongue image converted from an RGB image to an Lab image, in which a specific component value (L* value, a* value, or b* value) is used as a gradation value. The determination unit 222 calculates the number of pixels having each gradation value for each gradation value within the gradation range of each pixel in the first, second, or third tongue image. The identification unit 222 generates a gradation histogram in which each gradation value represents a class and the number of pixels having each gradation value represents a frequency, arranging the gradation values ​​in order of magnitude. The identification unit 222 sets a reference difference from the generated gradation histogram. For example, in the first embodiment, the identification unit 222 calculates the difference D between the upper limit gradation value V2 and the lower limit gradation value V3, as in the example shown in FIG. 13, and sets the calculated difference D as the reference difference. The identification unit 222 extracts, as edge pixels, pixels whose gradation value difference with adjacent pixels in the horizontal, vertical, or diagonal directions in the first, second, or third tongue image is equal to or greater than the reference difference. The reference difference may be a predetermined fixed value. The identification unit 222 identifies the number of extracted edge pixels as the amount of tongue coating on the tongue surface.

[0154] For example, the server device 200 stores in advance in the second storage device 210 a relational expression or table showing the relationship between the amount of tongue coating on the tongue surface and the nutritional status of the body. In this table, the amount of tongue coating on the tongue surface and / or the nutritional status of the body may be classified using one or more thresholds. The determination unit 223 refers to the relational expression or table stored in the second storage device 210 and identifies the nutritional status of the body corresponding to the amount of tongue coating on the tongue surface identified by the identification unit 222 as the nutritional status of the target person. The determination unit 223 may determine the nutritional status of the target person using a learning model that has been pre-trained to output the nutritional status of the person when the amount of tongue coating on the tongue surface of a specific person is input. The learning model is pre-trained using a neural network, a support vector machine, or the like, using combinations of the amount of tongue coating on the tongue surface and the nutritional status of the body of multiple people as training data. The determination unit 223 inputs the amount of tongue coating on the surface of the tongue identified by the identification unit 222 into the learning model, and identifies the information output from the learning model as the nutritional state of the subject's body.

[0155] If there is a large amount of tongue coating, there may be an excess of carbohydrates in the body. If there is no tongue coating, there may be a lack of minerals. The information processing system 1 can determine a person's health condition with high accuracy by using the amount of tongue coating on the surface of the tongue.

[0156] Fourth Embodiment In this embodiment, in step S104 of FIG. 4 , the identification unit 222 identifies the characteristics of the target person's tongue papillae based on the tongue image, and in step S106, the determination unit 223 determines the target person's health condition based on the characteristics of the target person's tongue papillae. The processing of step S105 is omitted. The tongue image includes tongue papillae. In particular, the identification unit 222 identifies the characteristics of the target person's fungiform papillae as the characteristics of the target person's tongue papillae. The identification unit 222 identifies the characteristics of the shape or color of the fungiform papillae as the characteristics of the target person's tongue papillae. The determination unit 223 determines the target person's antioxidant balance or physical nutritional state as the target person's health condition. For example, the antioxidant balance is oxidative stress. For example, the physical nutritional state is the amount or deficiency of vitamins, minerals, and proteins.

[0157] For example, the identification unit 222 identifies the shape characteristics of the fungiform papillae of the target person's tongue as the characteristics of the fungiform papillae of the target person's tongue. For example, the identification unit 222 generates a blue tongue image in which the blue component value (Blue value) is used as a gradation value from the tongue image, which is an RGB image. Alternatively, the identification unit 222 generates a saturation tongue image from the tongue image. The identification unit 222 calculates the number of pixels having each gradation value for each gradation value within the gradation range of each pixel in the blue tongue image or the saturation tongue image. The identification unit 222 generates a gradation histogram in which each gradation value is used as a class and the number of pixels having each gradation value is used as a frequency, and the gradation histogram is arranged in order of magnitude. The identification unit 222 scans the classes of the gradation histogram from high to low gradation values ​​and calculates a cumulative value by accumulating the frequency (number of pixels) of each class (gradation value). The identification unit 222 identifies the gradation value at which the ratio of the cumulative value to the total number of pixels in the blue tongue image or the saturation tongue image first exceeds a predetermined ratio (e.g., 10%) as the reference value. The identification unit 222 generates a binary image in which pixels in the blue tongue image or the saturation tongue image whose gradation value is equal to or greater than the reference value are defined as valid pixels and pixels whose gradation value is less than the reference value are defined as invalid pixels.

[0158] FIG. 18 is a schematic diagram showing an example of a saturation tongue image and a binary image according to this embodiment.

[0159] 18 shows a saturation tongue image 1800 and a binary image 1810. The binary image 1810 is a binary image generated from the saturation tongue image 1800.

[0160] The identification unit 222 identifies valid pixels included in a predetermined region D corresponding to the tip of the tongue in the binary image as fungiform papillae, and identifies the area (number of pixels) of the identified valid pixels as a characteristic of the shape of the fungiform papillae.

[0161] The identification unit 222 may identify the color of the fungiform papillae of the target person's tongue as a feature of the fungiform papillae of the target person's tongue. For example, the identification unit 222 generates a blue tongue image or a saturation tongue image from the tongue image. The identification unit 222 calculates the sum of the gradation values ​​of the pixels included in each of multiple horizontal lines extending horizontally in the blue tongue image or the saturation tongue image. The identification unit 222 generates a position histogram in which the vertical position of each horizontal line is used as a class and the calculated sum for each horizontal line is used as a frequency, and the horizontal lines are arranged in order of their vertical position.

[0162] FIG. 19 is a schematic diagram showing an example of a saturation tongue image and a position histogram.

[0163] 19 shows a saturation tongue image 1900 and a position histogram 1910. The position histogram 1910 is generated from the saturation tongue image 1900. The horizontal axis of the position histogram 1910 indicates the vertical position (rank) of each horizontal line, and the vertical axis indicates the sum (frequency) calculated for each horizontal line. The sums calculated for each horizontal line are arranged in order of the vertical position of each horizontal line.

[0164] The identification unit 222 identifies, among horizontal lines in the position histogram whose vertical positions (classes) are included in a predetermined range S corresponding to the tongue tip, horizontal lines whose total value (frequency) is equal to or greater than a reference value R as horizontal lines containing fungiform papillae. The identification unit 222 identifies whether or not there are horizontal lines containing fungiform papillae as a characteristic of the color of the fungiform papillae. The reference value R is set to, for example, a value obtained by multiplying the average value of the total values ​​of each horizontal line by a predetermined coefficient.

[0165] The identification unit 222 may calculate the regularity of the fungiform papillae in the same manner as the method used to calculate the regularity of the papillae in the first embodiment, and identify the calculated regularity as a characteristic of the fungiform papillae.

[0166] The identification unit 222 may also identify the features of fungiform papillae through image analysis using machine learning. In this case, the identification unit 222 identifies the features of fungiform papillae using a learning model that has been pre-trained to output the features of fungiform papillae in an input image. The learning model is pre-trained using a neural network, a support vector machine, or the like, using combinations of multiple images containing fungiform papillae and the features of the fungiform papillae in each image as training data. The identification unit 222 inputs a tongue image to the learning model and acquires the information output from the learning model as the features of the fungiform papillae.

[0167] In this case, images captured by a camera mounted on a typical multi-function mobile phone and images captured by a microscope may be used as training data for the learning model. Alternatively, images captured by a camera mounted on a typical multi-function mobile phone and images captured by a single-lens reflex camera may be used as training data for the learning model. On the other hand, when using the learning model, the identification unit 222 identifies the features of fungiform papillae by inputting only images captured by the imaging device 104 (the camera mounted on a typical multi-function mobile phone) into the learning model. This allows the learning model to be trained efficiently, and the identification unit 222 to identify the features of the papillae with high accuracy.

[0168] For example, the server device 200 may store in advance in the second storage device 210 a relational expression or table showing the relationship between fungiform papilla characteristics and antioxidant balance or physical nutritional status. In this table, the fungiform papilla characteristics and / or antioxidant balance and / or physical nutritional status may be classified using one or more thresholds. The determination unit 223 references the relational expression or table stored in the second storage device 210 and identifies the antioxidant balance or physical nutritional status corresponding to the fungiform papilla characteristics identified by the identification unit 222 as the antioxidant balance or physical nutritional status of the target person. The determination unit 223 may determine the antioxidant balance or physical nutritional status of the target person using a learning model that has been pre-trained to output the antioxidant balance or physical nutritional status of a specific person when the fungiform papilla characteristics of that person are input. The learning model is pre-trained using a neural network, a support vector machine, or the like, using combinations of fungiform papilla characteristics and antioxidant balance or physical nutritional status of multiple people as training data. The determination unit 223 inputs the characteristics of the fungiform papillae identified by the identification unit 222 into the learning model, and identifies the information output from the learning model as the antioxidant balance or physical nutritional state of the target person.

[0169] If the fungiform papillae are reddish, there is a possibility that the oxidative stress is high. The information processing system 1 can determine the health condition of a person with high accuracy by using the characteristics of the fungiform papillae.

[0170] Fifth Embodiment In this embodiment, in step S104 of FIG. 4 , the identification unit 222 identifies characteristics of the overall shape of the target person's tongue based on the tongue image, and in step S106, the determination unit 223 determines the target person's health condition based on the characteristics of the overall shape of the target person's tongue. The processing of step S105 is omitted. The tongue image includes the entire tongue. The tongue image includes a front tongue image including the tongue captured from the front, and a side tongue image including the tongue captured from the side (or oblique direction). In particular, the identification unit 222 identifies characteristics of teeth marks on the target person's tongue as characteristics of the overall shape of the target person's tongue. Teeth marks are teeth marks left on the edge of the tongue when the tongue is lowered and the edge of the tongue hits the inside of the lower teeth. The determination unit 223 determines dyslipidemia as the target person's health condition.

[0171] For example, the identification unit 222 identifies the characteristics of teeth marks on the tongue through histogram analysis. The identification unit 222 generates a green tongue image or a saturation tongue image from the tongue image. The identification unit 222 reduces the resolution of the green tongue image or the saturation tongue image by thinning out pixels in the green tongue image or the saturation tongue image. The identification unit 222 calculates the number of pixels having each gradation value for each gradation value within the gradation range of each pixel in the reduced-resolution green tongue image or the saturation tongue image. The identification unit 222 generates a gradation histogram in which each gradation value is defined as a class and the number of pixels having each gradation value is defined as a frequency, and the gradation histogram is arranged in order of magnitude. The identification unit 222 scans the classes of the gradation histogram from low to high gradation values ​​and calculates a cumulative value by accumulating the frequencies (number of pixels) of each class (gradation value). The identification unit 222 identifies as a reference value the gradation value at which the ratio of the cumulative value to the total number of pixels included in the low-resolution green tongue image or saturation tongue image first exceeds a predetermined ratio. The identification unit 222 generates a binary image in the low-resolution green tongue image or saturation tongue image so that pixels whose gradation values ​​are equal to or greater than the reference value are valid pixels and pixels whose gradation values ​​are less than the reference value are invalid pixels. The identification unit 222 calculates the number of invalid pixels included in each of multiple vertical lines extending vertically in the binary image. The identification unit 222 generates a position histogram in which the horizontal position of each vertical line is used as a class and the calculated number of invalid pixels for each vertical line is used as a frequency, arranging the vertical lines in order of their horizontal position.

[0172] FIG. 20 is a schematic diagram showing an example of a binary image and a position histogram according to this embodiment.

[0173] 20 shows a binary image 2000 and a position histogram 2010. The binary image 2000 is generated from a tongue image in which a tongue having teeth marks is captured. In the binary image 2000, valid pixels are shown in white, and invalid pixels are shown in black. The position histogram 2010 is a position histogram generated from the binary image 2000. The horizontal axis of the position histogram 2010 indicates the horizontal position (class) of each vertical line, and the vertical axis indicates the number of invalid pixels (frequency) calculated for each vertical line. The number of invalid pixels calculated for each vertical line is arranged in order of the horizontal position of each vertical line.

[0174] The identification unit 222 scans the classes of the position histogram from one end to the other in the horizontal direction and identifies positions where the number of invalid pixels reaches a maximum value and a minimum value. The identification unit 222 calculates the difference D between adjacent maximum and minimum values ​​as the size of the tooth marks, and identifies the average or maximum value of the calculated tooth mark sizes as the feature of the tooth marks. The identification unit 222 may calculate the number of combinations in which the difference D between adjacent maximum and minimum values ​​is equal to or greater than a threshold as the number of tooth marks, and identify the calculated number of tooth marks as the feature of the tooth marks.

[0175] Furthermore, when the identified tooth mark feature is equal to or greater than a predetermined tooth mark threshold, the identifying unit 222 identifies the tongue thickness feature from the lateral tongue image in the same manner as in the third embodiment.

[0176] The identification unit 222 may identify the characteristics of the teeth marks on the tongue by template matching. For example, the server device 200 stores a plurality of sample images containing teeth marks of various shapes in the second storage device 210 in advance. The identification unit 222 cuts out regions of the same size as each sample image from various positions on the tongue image and calculates the similarity between each cut-out region and each sample image. If the calculated similarity is equal to or greater than a predetermined threshold, the identification unit 222 determines that the tongue included in the tongue image contains teeth marks of the shape (size) contained in the sample image. Alternatively, the identification unit 222 may calculate feature amounts from each region cut out from the tongue image and each sample image, and if the similarity of the feature amounts is equal to or greater than a predetermined threshold, determine that the tongue included in the tongue image contains teeth marks of the size contained in the sample image. The feature amount may be the gradient of the image (e.g., HOG feature amount), etc. The similarity of the feature amount may be cosine similarity, inner product, etc.

[0177] For example, the server device 200 pre-stores in the second storage device 210 a relational expression or table showing the relationship between tongue tooth mark features and the presence or absence of dyslipidemia. In this table, the tongue tooth mark features and / or the presence or absence of dyslipidemia may be classified using one or more thresholds. The determination unit 223 references the relational expression or table stored in the second storage device 210 and identifies the presence or absence of dyslipidemia of the target person based on the presence or absence of dyslipidemia corresponding to the tongue tooth mark features identified by the identification unit 222. The determination unit 223 may determine the presence or absence of dyslipidemia of the target person using a learning model pre-trained to output the presence or absence of dyslipidemia of a specific person when the tongue tooth mark features of the specific person are input. The learning model is pre-trained using a neural network, support vector machine, or the like, using combinations of tongue tooth mark features and the presence or absence of dyslipidemia for multiple people as training data. The determination unit 223 inputs the tongue tooth mark features identified by the identification unit 222 into the learning model and identifies the presence or absence of dyslipidemia of the target person based on the information output from the learning model.

[0178] Alternatively, when the tooth mark feature is equal to or greater than a predetermined tooth mark threshold, the determination unit 223 determines whether the tongue thickness feature is equal to or greater than a predetermined thickness threshold. When the tongue thickness feature is equal to or greater than the thickness threshold, the determination unit 223 determines that swelling exists, and when the tongue thickness feature is less than the thickness threshold, the determination unit 223 determines that stress exists.

[0179] When the determining unit 223 determines that the subject person has dyslipidemia as a health condition, it determines that the subject has consumed too much sweets or fats, or too much carbohydrates.

[0180] The causes of teeth marks include swelling, stress, a low-lying tongue, etc. The information processing system 1 can determine the health condition of a person with high accuracy by using the characteristics of the teeth marks.

[0181] Sixth Embodiment In this embodiment, in step S104 of FIG. 4 , the identification unit 222 identifies characteristics of the front and back properties of the target person's tongue based on the tongue image. In step S106, the determination unit 223 determines the target person's health condition based on the characteristics of the front and back properties of the target person's tongue. The processing of step S105 is omitted. The tongue image includes the back side of the tongue, particularly the back properties, or the front side of the tongue, particularly the front properties. In particular, the identification unit 222 identifies the blood stagnation state of the back side of the target person's tongue as a characteristic of the front and back properties of the target person's tongue from the tongue image capturing the back side of the tongue, and identifies the blood stagnation state of the surface of the target person's tongue as a characteristic of the front and back properties of the target person's tongue from the tongue image capturing the front side of the tongue. Blood stagnation refers to a state in which blood in the body is stagnant and has difficulty flowing. The determination unit 223 determines the state of blood and / or blood flow in the body as the target person's health condition. For example, the state of blood or blood flow may be the presence or absence of abnormalities in neutral fat or cholesterol in the blood. The state of blood flow may be the quality of blood flow.

[0182] For example, the identification unit 222 identifies the state of blood smear on the surface or back of the tongue through histogram analysis. The identification unit 222 generates a saturation tongue image from the tongue image. The identification unit 222 calculates the number of pixels having each gradation value for each gradation value within the gradation range of each pixel in the saturation tongue image. The identification unit 222 generates a gradation histogram in which each gradation value is defined as a class and the number of pixels having each gradation value is defined as a frequency, and the gradation histogram is arranged in order of magnitude. The identification unit 222 scans the classes of the gradation histogram from low to high gradation values ​​and calculates a cumulative value by accumulating the frequency (number of pixels) of each class (gradation value). The identification unit 222 identifies the gradation value at which the ratio of the cumulative value to the total number of pixels included in the saturation tongue image first exceeds a predetermined ratio as a reference value. The identification unit 222 generates a binary image in which pixels in the saturation tongue image whose gradation value is less than a reference value are considered valid pixels and pixels whose gradation value is equal to or greater than the reference value are considered invalid pixels. The identification unit 222 identifies the area (number of pixels) of valid pixels in the binary image as the state of blood oozing on the underside of the tongue.

[0183] FIG. 21 is a schematic diagram showing an example of a binary image according to this embodiment.

[0184] 21 shows binary images 2100 and 2110. Binary image 2100 was generated from a tongue image capturing the underside of a person's tongue with good blood flow, while binary image 2110 was generated from a tongue image capturing the underside of a person's tongue with poor blood flow. In binary images 2100 and 2110, valid pixels are shown in white, and invalid pixels are shown in black. The tongue of a person with poor blood flow contains blood, and the area of ​​the valid image in binary image 2110 is larger than the area of ​​the valid image in binary image 2100.

[0185] The identification unit 222 may identify a bloody state on the surface or back of the tongue based on the number of edge pixels of a specific color component in the tongue image. For example, the identification unit 222 generates a first tongue image from an RGB tongue image, in which the gradation values ​​are the component values ​​of a specific color. Alternatively, the identification unit 222 generates a second tongue image from a tongue image converted from an RGB image to an HSV image, in which the gradation values ​​are the component values ​​of a specific color. Alternatively, the identification unit 222 generates a third tongue image from a tongue image converted from an RGB image to an Lab image, in which the gradation values ​​are the component values ​​of a specific color. The identification unit 222 extracts, as edge pixels, pixels whose gradation value difference with adjacent pixels in the horizontal, vertical, or diagonal direction is equal to or greater than a predetermined threshold in the first, second, or third tongue image. The identification unit 222 identifies the bloody state on the surface or back of the tongue based on the number of extracted edge pixels.

[0186] The identifying unit 222 may also identify the state of blood loss on the surface or back of the tongue by analyzing the near-infrared spectroscopic image. In this case, the photoelectric conversion element of the image capturing device 104 has sensitivity to near-infrared light, and the image capturing device 104 includes an irradiator that irradiates the near-infrared light. In step S101, the transmitting unit 121 causes the irradiator to irradiate the tongue of the target person with near-infrared light while causing the image capturing device 104 to capture an image of the tongue of the target person, and acquires the image generated by the image capturing device 104 as a tongue image.

[0187] The identification unit 222 generates a binary image in the tongue image so that pixels whose luminance values ​​(density values) are less than a predetermined threshold are considered valid pixels and pixels whose luminance values ​​are equal to or greater than the threshold are considered invalid pixels. The identification unit 222 performs a shrinking process on the valid pixels in the binary image, calculates the number of shrinking processes required until the valid pixels disappear as the thickness of the blood vessels, and identifies the blood vessels as a state of blood suffocation. Alternatively, the identification unit 222 calculates the average value of the luminance values ​​of pixels whose luminance values ​​are equal to or greater than a predetermined threshold as the thickness of the blood vessels, and identifies the blood vessels as a state of blood suffocation.

[0188] For example, the server device 200 may store in advance in the second storage device 210 a relational expression or table showing the relationship between the blood stagnation state of the surface or back surface of the tongue and the blood condition or blood flow state of the body. In this table, the blood stagnation state of the surface or back surface of the tongue and / or the blood condition and / or blood flow state of the body may be classified using one or more thresholds. The determination unit 223 may refer to the relational expression or table stored in the second storage device 210 and identify the blood condition or blood flow state of the body corresponding to the blood stagnation state of the surface or back surface of the tongue identified by the identification unit 222 as the blood condition or blood flow state of the target person. The determination unit 223 may determine the blood condition or blood flow state of the target person using a learning model that has been pre-trained to output the blood condition or blood flow state of the target person when the blood stagnation state of the surface or back surface of the tongue of a specific person is input. The learning model is trained in advance by a neural network, a support vector machine, etc., using combinations of the blood-stained state of the surface or back of the tongue and the state of blood or blood flow in the body of a plurality of people as training data. The determination unit 223 inputs the blood-stained state of the surface or back of the tongue identified by the identification unit 222 into the learning model, and identifies the information output from the learning model as the state of blood or blood flow in the body of the target person.

[0189] When the determining unit 223 determines that the target person's health condition is abnormal in blood triglycerides or cholesterol, it determines that the cause of the target person's health condition is malnutrition, stress, or lack of exercise.

[0190] Poor blood flow can occur due to malnutrition, stress, or nutritional deficiency. The information processing system 1 can accurately determine a person's health condition by using the state of blood oozing on the surface or back of the tongue.

[0191] Seventh Embodiment In this embodiment, in step S104 of FIG. 4, the identification unit 222 identifies the overall characteristics of the target person's tongue based on the tongue image, and in step S106, the determination unit 223 determines the target person's health condition based on the color characteristics of the target person's tongue. The processing of step S105 is omitted. The tongue image includes the entire tongue. The identification unit 222 identifies the color characteristics of the tongue, particularly the color balance characteristics of the tongue, as the overall characteristics of the target person's tongue. The determination unit 223 determines the state of blood flow in the body as the target person's health condition. For example, poor blood flow indicates a mineral deficiency.

[0192] For example, the identification unit 222 identifies the color balance characteristics of the tongue through histogram analysis. The identification unit 222 generates a first tongue image from the tongue image, which is an RGB image, in which specific color component values ​​are used as gradation values. Alternatively, the identification unit 222 generates a second tongue image from the tongue image converted from an RGB image to an HSV image in which specific color component values ​​are used as gradation values. The identification unit 222 calculates the number of pixels having each gradation value for each gradation value within the gradation range of each pixel in the first tongue image or the second tongue image. The identification unit 222 generates a gradation histogram in which each gradation value is used as a class and the number of pixels having each gradation value is used as a frequency, and the gradation histogram is arranged in order of magnitude. The identification unit 222 identifies the gradation value in the gradation histogram at which the number of pixels, or the frequency, is maximized, and identifies that gradation value as the color balance characteristic of the tongue.

[0193] FIG. 22 is a schematic diagram showing an example of a gradation histogram.

[0194] Grayscale histogram 2200 is a grayscale histogram generated from a tongue image of a person with good blood flow, and grayscale histogram 2210 is a grayscale histogram generated from a tongue image of a person with poor blood flow and a high hemoglobin concentration. The horizontal axis of each grayscale histogram indicates grayscale value (class), and the vertical axis indicates the number of pixels (frequency) having each grayscale value. The grayscale values ​​are arranged in order of magnitude. Grayscale value M2, which represents the maximum number of pixels in grayscale histogram 2210 generated from a tongue image of a person with poor blood flow, is smaller than grayscale value M1, which represents the maximum number of pixels in grayscale histogram 2200 generated from a tongue image of a person with good blood flow.

[0195] The identification unit 222 may identify the tongue color characteristics by template matching. For example, the server device 200 stores sample gradation histograms, which are gradation histograms calculated from multiple sample images including tongues of various colors, in the second storage device 210 in advance. The identification unit 222 calculates the similarity between the generated gradation histogram (an image showing the shape of the generated gradation histogram) and each sample gradation histogram (an image showing the shape of the generated gradation histogram). If the calculated similarity is equal to or greater than a predetermined threshold, the identification unit 222 determines that the tongue color included in the tongue image is the same as the tongue color included in the sample image from which the sample gradation histogram was generated. The identification unit 222 identifies the determined tongue color as a color balance characteristic.

[0196] For example, the server device 200 may store in advance in the second storage device 210 a relational expression or table indicating the relationship between tongue color balance features and bodily blood flow states. In this table, tongue color balance features and / or bodily blood flow states may be classified using one or more thresholds. The determination unit 223 refers to the relational expression or table stored in the second storage device 210 and identifies the bodily blood flow state corresponding to the tongue color balance features identified by the identification unit 222 as the bodily blood flow state of the target person. The determination unit 223 may determine the bodily blood flow state of the target person using a learning model that has been pre-trained to output the bodily blood flow state of a specific person when the tongue color balance features of the specific person are input. The learning model is pre-trained using a neural network, a support vector machine, or the like, using combinations of tongue color balance features and bodily blood flow states of multiple people as training data. The determination unit 223 inputs the characteristics of the color balance of the tongue identified by the identification unit 222 into the learning model, and identifies the information output from the learning model as the state of blood flow in the body of the target person.

[0197] The information processing system 1 can determine the health condition of a person with high accuracy by using the color balance of the tongue.

[0198] (Eighth embodiment) In this embodiment, in step S107 of Figure 4, the output control unit 224 transmits information indicating a change in the health condition of the target person as information indicating the health condition of the target person, and in step S108, the display control unit 122 displays the information indicating the change in the health condition of the target person on the first display device 102.

[0199] FIG. 23 is a schematic diagram showing an example of a display screen displayed on the first display device.

[0200] 23 shows display screens 2300, 2310, and 2320. Display screen 2300 shows changes in the health condition of a target person whose health condition is improving, display screen 2310 shows changes in the health condition of a target person whose health condition is not changing much, and display screen 2320 shows changes in the health condition of a target person whose health condition is deteriorating. The health conditions include skin condition, blood vessel condition, antioxidant balance, physical nutritional state, and / or psychological stress. Display screens 2300, 2310, and 2320 each show a graph 2301 and an image 2302.

[0201] The horizontal axis of graph 2301 indicates time, and the vertical axis indicates the health condition at each time. By viewing graph 2301, the user can confirm the change in the health condition of the target person over time.

[0202] Image 2302 is an image indicating a change in health status. That is, image 2302 does not indicate whether the current health status is good or bad, but whether the health status is improving (improving), unchanged, or worsening. An image of a tree is shown as image 2302. Any other image, such as grass or flowers, may be shown as image 2302. Display screen 2300 showing the health status of a target person whose health status is improving shows a tree with flowers as image 2302. Image 2302 may be displayed so that the greater the degree of improvement in the health status, the more flowers the tree contains, the larger the size, or the more vivid the color. Display screen 2310 showing the health status of a target person whose health status has changed little shows a tree with lush leaves but no flowers as image 2302. Display screen 2320 showing the health status of a target person whose health status is deteriorating shows a withered tree as image 2302. In image 2302, the tree may be displayed such that the greater the degree of deterioration in the person's health, the shorter the branches, the thinner the thickness, or the darker the color. By viewing image 2302, a user can intuitively recognize changes in the person's health and predict future changes in the person's health.

[0203] Even if a person's current nutritional status is good, if psychological stress is high, there is a possibility that their nutritional status will decline in the future. Poor antioxidant balance is unlikely to manifest as a change in physical condition, but if this state of poor antioxidant balance continues for a long time, it can cause changes in taste and other aspects, and glycation in the body may progress unnoticed. Therefore, the information processing system 1 may display a radar chart showing the balance between the current nutritional status, antioxidant balance, and psychological stress. The information processing system 1 may also display a radar chart showing the balance between changes in nutritional status, changes in antioxidant balance, and changes in psychological stress.

[0204] Furthermore, the information processing system 1 may display an image showing the current health condition instead of or in addition to the image 2302.

[0205] The information processing system 1 may also display suggestions for improving the health condition of the target person. For example, if the antioxidant balance (oxidative stress) is poor, the information processing system 1 may inquire about the target person's snacking or eating out habits and encourage improvement based on the situation. If psychological stress and oxidative stress are poor, the information processing system 1 may encourage improvement by focusing on alleviating mental stress rather than changing the diet. If psychological stress and nutritional status are poor, the information processing system 1 may encourage improvement of psychological stress and sleep and prioritize physical repair.

[0206] In step S107 of Figure 4, the output control unit 224 transmits information indicating the nutritional status of the target person as information indicating the target person's health status, and in step S108, the display control unit 122 may display the information indicating the nutritional status of the target person on the first display device 102.

[0207] FIG. 24 is a schematic diagram showing an example of a display screen displayed on the first display device.

[0208] FIG. 24 shows a display screen 2400. An image 2401 is displayed on the display screen 2400. The image 2401 is an image of a character indicating the nutritional status of a target person. The image 2401 shows an image of a tree. Any other image, such as grass or flowers, may also be displayed as the image 2401. In the image 2401, the leaf portion is divided into regions for each nutrient component, such as minerals, protein, vitamin A, vitamin B, and vitamin C. An image of a representative food representing each nutrient component (e.g., beef for protein, lemon for vitamin C, etc.) may be displayed in each nutrient component region. For example, the more sufficient each nutrient component is, the brighter the color of the region for that nutrient component is displayed, and the more deficient each nutrient component is, the darker the color of the region for that nutrient component is displayed. Furthermore, the more sufficient each nutrient component is, the larger the food is displayed, and the smaller the food is displayed, the smaller the food is displayed.

[0209] If there is a lot of carotene, vitamin A levels will be high, but if there is an imbalance in vitamins or minerals, physical condition will not improve. A lack of minerals will prevent muscle building and result in a lack of strength. Even if a person eats a balanced diet, psychological stress and other factors may prevent the body from obtaining a balanced amount of nutrients. Because a person's nutritional status is reflected in the tongue, the information processing system 1 can accurately determine the state of nutrients being absorbed into the body by determining nutritional status from the tongue.

[0210] Ninth Embodiment In this embodiment, the terminal device 100 is a tablet PC, a multi-function mobile phone, or the like. The imaging device 104 includes a front imaging device and a rear imaging device having higher performance than the front imaging device. The terminal device 100 also has an audio output device, a vibration sensor, or the like. The audio output device is a speaker, or the like. The vibration sensor is an acceleration sensor, a displacement sensor, or the like. In step S101 of FIG. 4 , the transmission unit 121 acquires a tongue image using the rear imaging device. However, since the rear imaging device is configured to capture an image of the rear side of the terminal device 100, i.e., the side where the first input device 101 and the first display device 102 are not located, it is difficult for the target person to capture an image of their own tongue.

[0211] FIG. 25 is a schematic diagram showing a situation in which a target person captures an image of his or her tongue.

[0212] 25 , in this embodiment, the target person P captures an image of his or her tongue using a rear imaging device. At this time, the rear imaging device C of the terminal device 100 is directed toward the target person P, and therefore the first input device 101 and the first display device 102 are directed toward the opposite side of the target person P.

[0213] The transmission unit 121 outputs imaging instruction information from the first display device 102 or the audio output device to instruct the target person P to image the tongue using the rear imaging device, to image in front of the mirror M, and to point the rear imaging device toward the target person P and the first display device 102 toward the mirror M.

[0214] When the first input device 101 or the vibration sensor detects that the user has performed a zoom operation such as a double tap or triple tap on the first input device 101 (screen) or the housing (rear face) of the terminal device 100, the first input device 101 or the vibration sensor outputs a zoom operation signal to the first processing device 120. When the transmission unit 121 receives the zoom operation signal, it adjusts the imaging range of the rear imaging device so that the tongue is captured at an appropriate size by the rear imaging device. The transmission unit 121 uses known image processing technology to detect the tongue from the image captured by the rear imaging device and causes the rear imaging device to zoom in or out so that the detected tongue is captured at an appropriate size. When the first input device 101 or the vibration sensor detects that the user has performed an imaging operation such as a tap on the first input device 101 (screen) or the housing (rear face) of the terminal device 100, the first input device 101 or the vibration sensor outputs an imaging operation signal to the first processing device 120. When the transmission unit 121 receives the imaging operation signal, it causes the rear imaging device to capture an image of the tongue. When capturing an image of the side of the tongue, the transmission unit 121 outputs, from the first display device 102 or the audio output device, orientation instruction information for instructing the target person P to face the side relative to the terminal device 100.

[0215] The terminal device 100 may capture an image of the tongue of the target person using a frontal imaging device. When capturing an image of the tongue of the target person using a frontal imaging device, the transmission unit 121 does not output the imaging instruction information.

[0216] This allows the terminal device 100 to acquire high-quality images at the user's home, allowing the user to easily recognize daily changes in the user's internal condition with high accuracy while at home. Note that when operating the terminal device 100, if the user places the terminal device 100 close to their face, they cannot see a mirror, and if the user moves the terminal device 100 away from their face to look in the mirror, the size of the tongue included in the generated input image becomes smaller. The user can easily control the zoom of the imaging device 104 by tapping the screen or the back, so the terminal device 100 can be operated away from their face. Therefore, the terminal device 100 can acquire high-quality images while improving user operability.

[0217] Although preferred embodiments have been described above, the embodiments are not limited to these. For example, the person's health condition may be determined by the terminal device 100 rather than the server device 200. In this case, the first processing device 120 of the terminal device 100 functions as the transmission unit 121 and the display control unit 122, and also functions as an identification unit and a determination unit having functions similar to those of the identification unit 222 and the determination unit 223 of the server device 200. The identification unit and the determination unit of the terminal device 100 execute the processes of steps S104 to S106 of the sequence shown in FIG. 4. In this case, the terminal device 100 is an example of an information processing device, the first display device 102 is an example of an output unit, and the transmission unit 121 is an example of an acquisition unit.

[0218] 4, steps S102 and S103 are omitted, and in step S104, the identification unit identifies features of the papillae or other features of the target person's tongue based on the tongue image acquired by the transmission unit 121. Furthermore, step S107 is omitted, and in step S108, the display control unit 122 outputs information indicating the target person's health condition determined by the determination unit by displaying it on the first display device 102. In this case, too, the information processing system 1 can output the person's health condition with higher accuracy.

[0219] Furthermore, in the information processing system 1, a plurality of terminal devices 100 and / or a plurality of server devices 200 may cooperate to share the respective steps of the above-described processes.

[0220] The above-described embodiments may be combined and implemented simultaneously. The information processing system 1 may determine the health condition of a target person by performing multiple methods among the methods described in the first to seventh embodiments on the same tongue image.

[0221] 1 Information processing system, 100 Terminal device, 102 First display device, 104 Imaging device, 121 Transmission unit, 200 Server device, 201 Second communication device, 221 Reception unit, 222 Identification unit, 223 Determination unit

Claims

1. An information processing apparatus comprising: an acquisition unit that acquires a tongue image including any one of papillae of a person's tongue, front and back states of the tongue, and the entire tongue; and an output unit that outputs information indicating the health state of the person based on the tongue image.

2. The information processing apparatus according to claim 1, wherein the output unit outputs information indicating the state of the person's skin as information indicating the health state of the person.

3. The information processing apparatus according to claim 2, wherein the skin state is any one or more of the degree of wrinkles, the degree of pores, the degree of texture, or the degree of spots.

4. The information processing apparatus according to claim 1, wherein the output unit outputs any one or more of information indicating the state of the person's blood vessels and / or the state of blood flow, information indicating the antioxidant balance, information indicating the nutritional state of the body, information indicating psychological stress, and metabolic abnormalities as information indicating the health state of the person.

5. The information processing apparatus according to claim 4, wherein the state of the blood vessels is the distribution of blood flow or the softness of the blood vessels, and the state of the blood flow is the quality of the blood flow.

6. The information processing apparatus according to claim 4, wherein the nutritional state of the body is the amount or deficiency of vitamins, minerals, proteins, or the quality of lipid metabolism.

7. The information processing apparatus according to claim 1 or 2, further comprising: a specifying unit that specifies any feature of papillae of the person's tongue, front and back states of the tongue, and the overall shape of the tongue based on the tongue image; and a determining unit that determines the health state of the person based on the feature.

8. The information processing apparatus according to claim 7, wherein the determining unit determines the state of the person's skin as the health state of the person based on the feature of the papillae.

9. The information processing apparatus according to claim 7, wherein the determining unit determines the state of the person's blood vessels based on the feature of the papillae, and determines the state of the person's skin as the health state of the person based on the state of the blood vessels.

10. The information processing apparatus according to claim 7, wherein the specifying unit specifies the shape, size, or regularity of the papillae as the feature of the papillae.

11. The information processing apparatus according to claim 7, wherein the acquisition unit acquires a plurality of the tongue images captured at mutually different timings, and the specifying unit specifies a change in the papillae as the feature of the papillae.

12. The specific part specifies the characteristics of the papillae for each of the tip and the back of the tongue, and the determination part determines the health condition of the person based on the characteristics of the papillae specified for each of the tip and the back of the tongue. The information processing apparatus according to claim 7.

13. The acquisition part further acquires the subjective physical condition or environmental information of the person, and the determination part determines the health condition of the person based on the subjective physical condition or environmental information of the person. The information processing apparatus according to claim 7.

14. The determination part determines at least one of the antioxidant balance and the nutritional state of the body of the person based on the characteristics of the mushroom-shaped papillae among the papillae. The information processing apparatus according to claim 7.

15. The specific part specifies at least one of the shape characteristics and color characteristics of the mushroom-shaped papillae as the characteristics of the mushroom-shaped papillae among the papillae. The information processing apparatus according to claim 7.

16. The determination part determines at least one of the nutritional state of the body, the blood flow state, or the blood state of the person based on at least one of the amount of tongue coating on the surface of the tongue, the congestion state on the surface, or the congestion state on the back surface. The information processing apparatus according to claim 7.

17. The specific part specifies at least one of the amount of tongue coating on the surface of the tongue, the congestion state on the surface, or the congestion state on the back surface. The information processing apparatus according to claim 7.

18. The determination part determines at least one of the nutritional state of the body, psychological stress, or metabolic abnormality based on the entire tongue. The information processing apparatus according to claim 7.

19. The specific part specifies at least one of the color balance characteristics, crack characteristics, thickness characteristics, and tooth mark characteristics as the characteristics of the entire tongue. The information processing apparatus according to claim 7.

20. The specific part specifies any one of the characteristics of the papillae of the person's tongue, the characteristics of the front and back properties of the tongue, or the characteristics of the overall shape of the tongue by any one of template matching, histogram analysis, analysis by near-infrared spectroscopic imaging, shape analysis of an object by an image, texture analysis, or image analysis by machine learning. The information processing apparatus according to claim 7.

21. The output part outputs the factors of the health condition of the person determined by the determination part. The information processing apparatus according to claim 7.

22. An information processing system having a terminal device and an information processing device, wherein the terminal device has a transmission unit that transmits a tongue image including any one of papillae of a person's tongue, front and back properties of the tongue, and the entire tongue to the information processing device, and the information processing device has a reception unit that receives the tongue image from the terminal device and an output unit that outputs information indicating the health state of the person based on the tongue image. An information processing system characterized by this.

23. The information processing system according to claim 22, wherein the terminal device further has an imaging unit that generates the tongue image.

24. An information processing method characterized in that an information processing device acquires a tongue image including any one of papillae of a person's tongue, front and back properties of the tongue, and the entire tongue, and outputs information indicating the health state of the person based on the tongue image.

25. A control program for an information processing device, characterized in that the information processing device is caused to acquire a tongue image including any one of papillae of a person's tongue, front and back properties of the tongue, and the entire tongue, and output information indicating the health state of the person based on the tongue image.

Images