Information processing apparatus, information processing system, information processing method, and control program

Abstract

Information processing apparatus, information processing system, information processing method, and control program are provided, capable of outputting the health status of a person with higher accuracy. The information processing apparatus includes: an acquisition unit that acquires a tongue image including any one of the papillae, surface and dorsal features of the tongue, and the entire tongue; and an output unit that outputs information representing the person's health status based on the tongue image. The information processing system is an information processing system having a terminal device and an information processing apparatus. The terminal device has a transmitting unit that transmits a tongue image including any one of the papillae, surface and dorsal features of the tongue, and the entire tongue to the information processing apparatus. The information processing apparatus includes: a receiving unit that receives the tongue image from the terminal device; and an output unit that outputs information representing the person's health status based on the tongue image.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus, information processing system, information processing method, and control program. Background Technology

[0002] In the past, systems have been developed to determine a person's health status based on images of their tongue.

[0003] Patent document 1 discloses a health monitoring device that segments the tongue region from an individual's tongue image, extracts the region of interest from the segmented tongue region, and determines the individual's health status.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2004-209245 Summary of the Invention

[0007] In information processing devices, the goal is to output the health status of individuals with higher precision.

[0008] The purpose of information processing devices, information processing systems, information processing methods, and control programs is to output a person's health status with higher accuracy.

[0009] The information processing apparatus according to the embodiment includes: an acquisition unit that acquires a tongue image including any one of the papillae of a person's tongue, the surface and dorsal features of the tongue, and the tongue as a whole; and an output unit that outputs information representing the health status of a person based on the tongue image.

[0010] In the information processing apparatus described in the embodiment, it is preferable that the output unit outputs information indicating the skin condition of a person as information indicating the person's health status.

[0011] In the information processing apparatus involved in the implementation, the skin condition is preferably any one or more of the following: degree of wrinkles, degree of pores, degree of texture, and degree of spots.

[0012] In the information processing device involved in the implementation, it is preferable that the output unit outputs information indicating the person's vascular status and / or blood flow status, information indicating antioxidant balance, information indicating the body's nutritional status, information indicating psychological stress, and metabolic abnormalities as information indicating the person's health status.

[0013] In the information processing device involved in the implementation, the preferred vascular state is blood flow distribution or vascular flexibility, and the preferred blood flow state is good blood flow.

[0014] In the information processing device involved in the implementation, the body's nutritional status is preferably that the amount of vitamins, minerals, and proteins is insufficient or that lipid metabolism is good.

[0015] In the information processing apparatus according to the embodiment, it is preferable to further include: a determination unit that determines, based on a tongue image, a feature of any one of the papillae of a person's tongue, the surface and dorsal features of the tongue, and the overall shape of the tongue; and a determination unit that determines, based on the features, the person's health status.

[0016] In the information processing apparatus described in the implementation, it is preferable that the determination unit determines the skin condition of a person as the person's health status based on the characteristics of the nipples.

[0017] In the information processing apparatus involved in the implementation, it is preferable that the determination unit determines the blood vessel status of a person based on the characteristics of the nipples, and determines the skin status of the person based on the blood vessel status to determine the person's health status.

[0018] In the information processing apparatus involved in the implementation, it is preferable that the determining unit determines the shape, size, or regularity of the nipple as a characteristic of the nipple.

[0019] In the information processing apparatus described in the embodiments, it is preferable that the acquisition unit acquires multiple tongue images taken at different time intervals, and the determination unit determines the changes in the nipple as a characteristic of the nipple.

[0020] In the information processing apparatus according to the embodiment, it is preferable that the determination unit determines the characteristics of the nipples for the tip and the back of the tongue respectively, and the determination unit determines the health status of the person based on the characteristics of the nipples determined for the tip and the back of the tongue respectively.

[0021] In the information processing apparatus involved in the implementation, it is preferable that the acquisition unit also acquires the person's self-aware physical condition or environmental information, and the determination unit further determines the person's health status based on the person's self-aware physical condition or environmental information.

[0022] In the information processing apparatus involved in the implementation, it is preferred that the determination unit determines one or more of the antioxidant balance and nutritional status of the individual based on the characteristics of the fungal nipples in the nipples.

[0023] In the information processing apparatus of the embodiment, it is preferable that the determining unit determines one or more of the shape features and color features of the fungal papillae in the papillae as features of the fungal papillae.

[0024] In the information processing device involved in the implementation, the determination unit preferably determines one or more of the following: the nutritional status, blood flow status, and blood status of a person's body based on one or more of the amount of tongue coating on the surface of the tongue, the state of bruising on the surface, and the state of bruising on the back of the tongue.

[0025] In the information processing apparatus according to the embodiments, it is preferable that the determining unit determines one or more of the following: the amount of tongue coating on the surface of the tongue, the state of bruising on the surface, and the state of bruising on the back of the tongue.

[0026] In the information processing device involved in the implementation, the determination unit preferably determines one or more of the body's nutritional status, psychological stress, and metabolic abnormalities based on the tongue as a whole.

[0027] In the information processing apparatus involved in the implementation, it is preferable that the determination unit determines one or more of the following as the overall characteristics of the tongue: color balance characteristics, crack characteristics, thickness characteristics, and teeth mark characteristics.

[0028] In the information processing apparatus involved in the implementation, the determination unit preferably determines the characteristics of the papillae of a person's tongue, the characteristics of the surface and dorsal features of the tongue, and the characteristics of the overall shape of the tongue by any one of template matching, histogram analysis, near-infrared spectral image analysis, image-based object shape analysis, texture analysis, and machine learning-based image analysis.

[0029] In the information processing apparatus according to the embodiment, it is preferable that the output unit outputs the factors of the person's health status determined by the determination unit.

[0030] The information processing system involved in the implementation is an information processing system having a terminal device and an information processing device. The terminal device has a transmitting unit that transmits a tongue image to the information processing device, including any one of the papillae of a person's tongue, the surface and dorsal features of the tongue, and the tongue as a whole. The information processing device has a receiving unit that receives the tongue image from the terminal device and an output unit that outputs information representing the health status of a person based on the tongue image.

[0031] In the information processing apparatus described in the embodiments, the terminal device preferably also has a camera unit for generating tongue images.

[0032] In the information processing method involved in the implementation, the information processing device acquires a tongue image including any one of the papillae of a person's tongue, the surface and dorsal features of the tongue, and the tongue as a whole, and outputs information representing the person's health status based on the tongue image.

[0033] The control program involved in the implementation is a control program for an information processing device, which causes the information processing device to perform the following processing: acquiring a tongue image containing any one of the papillae of a person's tongue, the surface and dorsal features of the tongue, and the tongue as a whole; and outputting information representing the health status of a person based on the tongue image.

[0034] Information processing devices, information processing systems, information processing methods, and control programs can output a person's health status with higher accuracy. Attached Figure Description

[0035] Figure 1 This is a diagram showing the general configuration of the information processing system 1 involved in the implementation method.

[0036] Figure 2 This is a diagram showing the general configuration of the terminal device 100.

[0037] Figure 3 This is a diagram showing the general configuration of server device 200.

[0038] Figure 4 This is a sequence diagram representing an example of a decision-making process.

[0039] Figure 5 This is a schematic diagram illustrating an example of an image of a tongue.

[0040] Figure 6 This is a schematic diagram representing an example of a nipple image.

[0041] Figure 7 This is a schematic diagram illustrating an example of a blood flow distribution image.

[0042] Figure 8 This is a schematic diagram used to illustrate the relationship between nipple characteristics and vascular status.

[0043] Figure 9 This is a schematic diagram representing an example of a skin image.

[0044] Figure 10 This is a diagram used to illustrate the relationship between nipple characteristics and skin condition.

[0045] Figure 11 It is a diagram used to illustrate the relationship between nipple characteristics and nutritional status.

[0046] Figure 12 It is a diagram used to illustrate the relationship between nipple characteristics and nutritional status.

[0047] Figure 13 This is a schematic diagram representing an example of a grayscale histogram.

[0048] Figure 14 This is a schematic diagram illustrating an example of an image of a tongue.

[0049] Figure 15 This is a schematic diagram representing an example of a position histogram.

[0050] Figure 16 This is a schematic diagram illustrating an example of an image of a tongue.

[0051] Figure 17 This is a schematic diagram representing an example of a grayscale histogram.

[0052] Figure 18 This is a schematic diagram representing an example of a tongue image with saturation.

[0053] Figure 19 This is a schematic diagram representing an example of a saturation tongue image and a position histogram.

[0054] Figure 20 This is a schematic diagram representing an example of a binary image and a position histogram.

[0055] Figure 21 This is a schematic diagram representing an example of a binary image.

[0056] Figure 22 This is a schematic diagram representing an example of a grayscale histogram.

[0057] Figure 23 This is a schematic diagram illustrating an example of a display screen shown on a first display device.

[0058] Figure 24 This is a schematic diagram illustrating an example of a display screen shown on a first display device.

[0059] Figure 25 This is an illustration showing a person taking a picture of their own tongue.

[0060] Explanation of reference numerals in the attached figures

[0061] 1. Information processing system; 100. Terminal device; 102. First display device; 104. Imaging device; 121. Transmitting unit; 200. Server device; 201. Second communication device; 221. Receiving unit; 222. Determining unit; 223. Judgment unit. Detailed Implementation

[0062] Hereinafter, an information processing apparatus, information processing system, information processing method, and control program relating to one aspect of the embodiments will be described with reference to the accompanying drawings. However, it should be noted that the scope of the present invention is not limited to those embodiments, but rather covers the invention as described in the claims and its equivalents.

[0063] (First Implementation)

[0064] Figure 1 This is a diagram showing the general configuration of the information processing system 1 involved in the implementation method.

[0065] like Figure 1As shown, the information processing system 1 includes a terminal device 100 and a server device 200. The terminal device 100 and the server device 200 are connected to each other via a network N in a manner that enables them to communicate with each other. The network N is a wired network such as the Internet or an intranet. The network N can also be a wireless network such as a wireless LAN (Local Area Network).

[0066] Figure 2 This is a diagram showing the general configuration of the terminal device 100.

[0067] Terminal device 100 is an example of an information processing device. Terminal device 100 can be a personal computer, a laptop PC, a tablet PC, a multi-functional mobile phone (a so-called smartphone), etc. For example, terminal device 100 is installed in a shop or similar establishment and used by shop employees or others to photograph the tongue of a person being assessed for their health status. Terminal device 100 can also be used directly by the person being assessed. Terminal device 100 includes a first input device 101, a first display device 102, a first communication device 103, a photographing device 104, a first storage device 110, and a first processing device 120, etc. The first input device 101, first display device 102, first communication device 103, photographing device 104, first storage device 110, and first processing device 120 are interconnected via a CPU (Central Processing Unit) bus, etc.

[0068] The first input device 101 includes input devices such as a keyboard, mouse, and touch panel, and an interface circuit that obtains signals from the input devices, and outputs operation signals corresponding to the user's input operations.

[0069] The first display device 102 includes a display such as a liquid crystal or an organic EL (electro-Luminescence) display, and an interface circuit for outputting image data to the display, and displays image data on the display.

[0070] The first communication device 103 has a wired communication interface circuit that conforms to communication protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol). The first communication device 103 is connected to network N according to communication standards such as Ethernet (registered trademark). The first communication device 103 transmits data received from server device 200 via network N to the first processing device 120. The first communication device 103 also transmits data received from the first processing device 120 to server device 200 via network N. The first communication device 103 may also have an antenna for transmitting and receiving wireless signals and a wireless communication interface circuit that conforms to communication protocols such as wireless LAN, and is connected to network N according to communication standards such as wireless LAN.

[0071] The imaging device 104 includes a photoelectric conversion element sensitive to visible or infrared light, such as a CCD (Charge Coupled Device) element or a C-MOS (Complementary Metal Oxide Semiconductor) element. Infrared light includes near-infrared light with wavelengths of 700 nm to 1700 nm. Furthermore, the imaging device 104 includes: an imaging optical system that images the photoelectric conversion element; and an A / D converter that amplifies the electrical signal output from the photoelectric conversion element and performs analog-to-digital (A / D) conversion. The imaging device 104 converts the captured RGB analog images into digital images with brightness values ​​ranging from 0 to 255 for each pixel to generate an input image, and outputs it to the first processing unit 120. Alternatively, the imaging device 104 can also be an external camera connected via a communication interface circuit conforming to communication protocols such as USB (Universal Serial Bus) or Bluetooth. External cameras include cameras that cannot be zoomed in, cameras that can be zoomed in, or external lenses. The terminal device 100, by having a shooting device 104, enables the user of the terminal device 100 to easily obtain an input image of a person's tongue.

[0072] The imaging device 104 may also include multiple imaging devices. For example, if the terminal device 100 is a tablet PC or a multi-functional mobile phone, the imaging device 104 includes a front imaging device and a rear imaging device. The front imaging device is configured to capture images of the side where the first input device 101 and the first display device 102 are located, and the rear imaging device is configured to capture images of the rear side of the terminal device 100, i.e., the side where the first input device 101 and the first display device 102 are not located. Each imaging device has a photoelectric conversion element, an imaging optical system, and an A / D converter, and outputs an input image to the first processing device 120. The performance of the photoelectric conversion element, imaging optical system, and A / D converter of the rear imaging device is higher than that of the photoelectric conversion element, imaging optical system, and A / D converter of the front imaging device. For example, the resolution of the input image generated by the rear imaging device is higher than that of the input image generated by the front imaging device, and / or, the grayscale range of each pixel of the input image generated by the rear imaging device is larger than that of each pixel of the input image generated by the front imaging device.

[0073] When the photoelectric conversion element is sensitive to near-infrared light with a wavelength of 700 nm or more and 1700 nm or less, the imaging device 104 may also include an irradiator that irradiates near-infrared light with a wavelength of 700 nm or more and 1700 nm or less.

[0074] The first storage device 110 may include storage devices such as RAM (Random Access Memory) and ROM (Read-Only Memory), fixed disk devices such as hard disks, or removable storage devices such as floppy disks and optical disks. Furthermore, the first storage device 110 stores computer programs, databases, tables, etc., used in various processes of the terminal device 100. The computer programs can also be installed onto the first storage device 110 from a computer-readable removable recording medium using a well-known installation program. Examples of removable recording media include CD-ROM (compact disc read-only memory) and DVD-ROM (digital versatile disc read-only memory). The computer programs may also be stored on a recording medium provided by a predetermined server and installed via a network N.

[0075] The first processing unit 120 operates based on a program pre-stored in the first storage device 110. The first processing unit 120 is, for example, a CPU. Other possible first processing units include DSPs (digital signal processors), LSIs (large-scale integration), ASICs (application-specific integrated circuits), and FPGAs (field-programmable gate arrays). The first processing unit 120 is connected to the first input device 101, the first display device 102, the first communication device 103, the imaging device 104, and the first storage device 110, and controls each of these devices. The first processing unit 120 acquires an image of a person's tongue from the imaging device 104 and transmits it to the server device 200 via the first communication device 103.

[0076] The first processing unit 120 reads the computer program stored in the first storage device 110 and performs operations according to the read computer program. Thus, the first processing unit 120 can function as a transmission unit 121 and a display control unit 122.

[0077] Figure 3 This is a diagram showing the general configuration of server device 200.

[0078] Server device 200 is an example of an information processing device. Server device 200 includes a second communication device 201, a second storage device 210, and a second processing device 220, etc. The second communication device 201, the second storage device 210, and the second processing device 220 are interconnected via a CPU bus, etc.

[0079] The second communication device 201 is an example of an output unit. The second communication device 201 has a wired communication interface circuit according to communication protocols such as TCP / IP. The second communication device 201 is connected to the network N according to communication standards such as Ethernet (registered trademark). The second communication device 201 transmits data received from the terminal device 100 via the network N to the second processing device 220. The second communication device 201 transmits data received from the second processing device 220 to the terminal device 100 via the network N. Furthermore, the second communication device 201 may also have an antenna for transmitting and receiving wireless signals, and a wireless communication interface circuit according to communication protocols such as Wireless LAN, and be connected to the network N according to communication standards such as Wireless LAN.

[0080] The second storage device 210 includes storage devices such as RAM and ROM, fixed disk devices such as hard disks, or removable storage devices such as floppy disks and optical disks. Furthermore, the second storage device 210 stores computer programs, databases, tables, etc., used in various processes of the server device 200. Computer programs can be installed to the second storage device 210 from computer-readable removable recording media such as CD-ROMs and DVD-ROMs using well-known installation programs. Alternatively, computer programs can be stored on recording media provided by a predetermined server and installed via network N.

[0081] The second processing unit 220 operates based on a program pre-stored in the second storage device 210. The second processing unit 220 is, for example, a CPU. Alternatively, a DSP, LSI, ASIC, FPGA, etc., can also be used as the second processing unit 220. The second processing unit 220 is connected to the second communication device 201 and the second storage device 210, etc., and controls each device. The second processing unit 220 receives a tongue image from the terminal device 100 via the second communication device 201. Based on the received tongue image, the second processing unit 220 determines the person's health status and sends information indicating the person's health status to the terminal device 100 via the second communication device 201.

[0082] The second processing unit 220 reads the computer program stored in the second storage device 210 and performs operations according to the read computer program. Thus, the second processing unit 220 functions as a receiving unit 221, a determining unit 222, a decision-making unit 223, and an output control unit 224. The receiving unit 221 is an example of an acquisition unit.

[0083] Figure 4 This is a timing diagram representing an example of a decision-making action in information processing system 1.

[0084] The following is for reference Figure 4 The flowchart shown illustrates an example of the decision-making process. Furthermore, based on programs pre-stored in the storage devices of each device within the information processing system 1, the following described actions are performed primarily by the processing units of each device in cooperation with the elements of each device.

[0085] First, the transmitting unit 121 of the terminal device 100 acquires a tongue image containing the tongue of the subject, self-awareness of physical condition, and environmental information (step S101). Following instructions input by the user using the first input device 101, the transmitting unit 121 causes the imaging device 104 to capture the subject's tongue, acquiring the input image generated by the imaging device 104 as a tongue image. The tongue image includes at least a papilla image, which contains the papillae of the subject's tongue. The papillae include filiform papillae and fungiform papillae. The transmitting unit 121 may also acquire multiple tongue images obtained by capturing the subject's tongue at predetermined time intervals, i.e., multiple tongue images captured at different time points.

[0086] Figure 5 This is a schematic diagram of an example of a tongue image.

[0087] exist Figure 5 The image shows tongue images 500, 510, and 520 for each of the following individuals. Tongue image 500 contains the tongue of person A, who is in good health; tongue image 510 contains the tongue of person B, who is in slightly poor health; and tongue image 520 contains the tongue of person C, who is in poor health.

[0088] Figure 6 This is a schematic diagram representing an example of a nipple image.

[0089] exist Figure 6 The image shows nipple images 600, 610, and 620 for each of several subjects. Nipple image 600 is a magnified view of a portion of tongue image 500 to show the nipple in detail; nipple image 610 is a magnified view of a portion of tongue image 510 to show the nipple in detail; and nipple image 620 is a magnified view of a portion of tongue image 520 to show the nipple in detail.

[0090] Additionally, the sending unit 121 acquires the self-perceived physical condition and environmental information of the target person, specified by the user using the first input device 101. The self-perceived physical condition includes one or more elements related to physical condition, such as fever, pain, stress, cold, fatigue, stiff shoulders, and sleep. The user specifies whether the self-perceived physical condition is good, bad, and / or its degree based on one or more of these elements. The self-perceived physical condition can also be specified by body parts such as the head, abdomen, arms, and legs. The environmental information represents the target person's environment and condition, specifying whether each element has been ingested and / or its degree based on one or more elements such as cigarettes and sweets.

[0091] Next, the transmitting unit 121 transmits the acquired tongue image, self-aware physical condition, and environmental information to the server device 200 via the first communication device 103 (step S102). The transmitting unit 121 transmits the tongue image, self-aware physical condition, and environmental information to the server device 200 together. The transmitting unit 121 may also acquire the tongue image, self-aware physical condition, and environmental information asynchronously and transmit them to the server device 200 separately.

[0092] Next, the receiving unit 221 of the server device 200 receives the image of the target person's tongue, self-awareness of physical condition, and environmental information from the terminal device 100 via the second communication device 201 (step S103).

[0093] Next, the determining unit 222 determines the nipple features of the tongue of the target person based on the tongue image obtained by the receiving unit 221 (step S104).

[0094] The determination unit 222 first cuts out a predetermined region (nipple image) containing the nipples within the tongue image. The determination unit 222 extracts pixels within the tongue image whose grayscale values ​​(brightness values, color values, etc.) differ from those of adjacent pixels in the horizontal, vertical, or diagonal directions by a predetermined threshold as edge pixels. The determination unit 222 extracts the rightmost edge pixel on each horizontal line extending along the horizontal direction within the nipple image as the left-end pixel representing the left end of the tongue, and the leftmost edge pixel as the right-end pixel representing the right end of the tongue. Furthermore, the determination unit 222 extracts the topmost edge pixel on each vertical line extending along the vertical direction within the nipple image as the top-end pixel representing the top of the tongue, and the bottommost edge pixel as the bottom-end pixel representing the bottom of the tongue. Additionally, there are... Figure 5 Since the upper part of the tongue is not included in the tongue images as shown in images 500, 510, and 520, the extraction of the upper part pixels can be omitted in the determination unit 222.

[0095] The determining unit 222 cuts out a predetermined region within the area surrounded by the extracted left-end pixel, right-end pixel, upper pixel (or the upper part of the tongue image), and lower pixel to obtain a nipple image. The predetermined region is, for example, the center of the tongue. The predetermined region can also be the tip and / or the back of the tongue, where the tip is the area within a predetermined range from the left, right, or lower pixel, and the back is the area within a predetermined range from the upper pixel (or the upper part of the tongue image). At the tip of the tongue, due to friction within the oral cavity, the nipple disappears drastically, and its reconstruction takes a long time. At the tip of the tongue, where the width of the nipple disappearance is large, the likelihood of prolonged nipple disappearance is high. By determining the nipple features separately for the tip and back of the tongue by the determining unit 222, the information processing system 1 can determine the person's health status based on different parts of the nipple features, thus achieving a higher accuracy in determining the person's health status. In addition, the determining unit 222 can also perform interpolation processing such as bilinear interpolation and bidirectional interpolation within the predetermined area to interpolate pixels and enlarge the predetermined area, i.e., the nipple image.

[0096] The determination unit 222 extracts edge pixels within the nipple image and, through labeling or the like, extracts groups of edge pixels that are adjacent to each other in the horizontal, vertical, or diagonal directions as nipples. The determination unit 222 can also extract nipples based on a binary image. In this case, the determination unit 222 generates a binary image in the nipple image where pixels with grayscale values ​​less than a binarization threshold are invalid pixels, and pixels with grayscale values ​​greater than or equal to the binarization threshold are valid pixels. The determination unit 222 extracts groups of pixels in the nipple image that correspond to groups of valid pixels that are adjacent to each other in the horizontal, vertical, or diagonal directions as nipples within the binary image. The determination unit 222 determines the shape, size, regularity, or color of the nipple as nipple features.

[0097] The determination unit 222 determines the roundness of the nipple as the shape of the nipple. Roundness represents the degree of similarity to a circle. The determination unit 222 calculates the maximum value of the similarity between each nipple extracted from the nipple image and circles of various sizes. The similarity is normalized cross-correlation, SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), etc. The determination unit 222 calculates the statistical value (mean, median, maximum, minimum, etc.) of the maximum value of the similarity calculated for each nipple as the roundness of the nipple in the nipple image. The determination unit 222 may also use a learning model to determine the roundness of the nipple. The learning model is pre-learned so that it outputs the roundness of the nipple in the image when an image is input. The learning model is pre-learned using multiple images containing nipples and combinations of the roundness of the nipples in each image as teacher data, and is trained in advance using neural networks, support vector machines, etc. The determination unit 222 inputs a nipple image into the learning model and obtains the information output from the learning model as the roundness.

[0098] like Figure 6 Images 600, 610, and 620 show nipple shapes. The nipples of person A (in good health) are nearly circular, while those of person B (in slightly poorer health) deviate slightly from a circular shape, and those of person C (in poor health) deviate significantly from a circular shape. In other words, the better the health of the subject, the closer the nipple shape is to a circle. Information processing system 1 can accurately determine the health status of a subject based on the nipple shape.

[0099] The determination unit 222 calculates statistical values ​​(mean, median, maximum, minimum, standard deviation, etc.) of the area (number of pixels in each nipple) of each nipple extracted within the nipple image as the nipple size. The determination unit 222 may also calculate statistical values ​​of the width of each nipple extracted within the nipple image in a predetermined direction (horizontal, vertical, etc.) as the nipple size. The determination unit 222 may also use a learning model to determine the nipple size, which has been pre-learned to output the nipple size in an image when an image is input. This learning model uses multiple images containing nipples and combinations of nipple sizes in each image as teacher data, and is pre-learned using neural networks, support vector machines, etc. The determination unit 222 inputs a nipple image to the learning model and obtains the information output from the learning model as the size.

[0100] like Figure 6Images 600, 610, and 620 show nipple images. The nipple of person A (in good health) is smaller than that of person B (in slightly poorer health), while the nipple of person C (in poor health) is larger than that of person B. That is, the better the health of the subject, the smaller the size of their nipples. Information processing system 1 can accurately determine the health status of a subject based on the shape of their nipples.

[0101] For example, in a 700×700 pixel image, the average grain size of the nipple in nipple image 600 is 187 pixels and the grain size distribution (SD) is 5468; the average grain size of the nipple in nipple image 610 is 382 pixels and the grain size distribution (SD) is 5137; and the average grain size of the nipple in nipple image 620 is 1020 pixels and the grain size distribution (SD) is 14000.

[0102] The determination unit 222 calculates the statistical values ​​(mean, median, maximum, minimum, etc.) of the variance or standard deviation of the distances between pairs of adjacent nipples extracted within the nipple image as the regularity of the nipples. The determination unit 222 can also perform frequency transformation on the nipple image using a two-dimensional Fourier transform or wavelet transform, calculating the difference between the maximum and minimum values ​​of frequency components with amplitudes above a predetermined threshold as the regularity of the nipples. The determination unit 222 can also use a learning model to determine the regularity of the nipples. This learning model is pre-learned to output the regularity of the nipples in an image when an image is input. This learning model uses multiple images containing nipples and combinations of the regularities of the nipples in each image as teacher data, and is pre-learned using neural networks, support vector machines, etc. The determination unit 222 inputs a nipple image into the learning model and obtains the information output from the learning model as the regularity.

[0103] Furthermore, the determination unit 222 calculates the regularity of the nipple by performing interval removal of pixels within the nipple image and then downsizing it.

[0104] Alternatively, the determination unit 222 can also calculate the regularity of the nipple through histogram analysis. The determination unit 222 calculates the number of pixels with each grayscale value within the grayscale range of each pixel in the nipple image. Grayscale values ​​are, for example, Red values ​​(red component values), Green values ​​(green component values), or Blue values ​​(blue component values). The determination unit 222 uses each grayscale value as a level and the number of pixels with each grayscale value as a degree, generating a grayscale histogram arranged in order of grayscale value magnitude.

[0105] Figure 13 This is a schematic diagram representing an example of a grayscale histogram.

[0106] The horizontal axis of the grayscale histogram 1300 represents grayscale values ​​(levels), and the vertical axis represents the number of pixels with each grayscale value (degrees). The grayscale values ​​are arranged in order of magnitude.

[0107] The determining unit 222 determines the maximum value M of the number of pixels as a degree and the maximum degree gray value V1 of the highest degree level in the gray-scale histogram. Furthermore, starting from the maximum degree gray value V1, the determining unit 222 scans the levels of the gray-scale histogram in the direction of increasing gray values ​​to determine an upper limit gray value V2 that is less than or equal to a predetermined proportion α (e.g., 10%) of the number of pixels reaching the maximum value M. Additionally, starting from the maximum degree gray value V1, the determining unit 222 scans the levels of the gray-scale histogram in the direction of decreasing gray values ​​to determine a lower limit gray value V3 that is less than or equal to a predetermined proportion β (e.g., 10%) of the number of pixels reaching the maximum value M. The smaller the difference between the upper limit gray value V2 and the lower limit gray value V3, the more uniform the grain shape; conversely, the larger the difference, the more irregular the grain shape and the worse the nipple condition. The determination unit 222 calculates the difference D between the upper limit gray value V2 and the lower limit gray value V3 as the regularity of the nipple.

[0108] like Figure 6 As shown in nipple images 600, 610, and 620, the nipples of person A (whose health is good) are arranged more regularly than those of person B (whose health is slightly worse), while the nipples of person C (whose health is poor) are arranged less regularly than those of person B. That is, the better the health of the subject, the higher the regularity of the subject's nipples. Information processing system 1 can accurately determine the health status of the subject based on the shape of the nipples.

[0109] The determining unit 222 can also determine changes in the nipple as nipple features. The determining unit 222 determines changes in the shape, size, or regularity of the nipple as nipple features, and these changes are determined based on individual nipple images within multiple tongue images captured at different time intervals.

[0110] When nipples gradually change towards a more rounded shape, the subject's physical condition is improving (recovering); when nipples gradually deviate from a round shape, the subject's physical condition is deteriorating. When nipples are always close to a round shape, the subject's physical condition is always good; when nipples are always far from a round shape, the subject's physical condition is always poor. Similarly, when nipples gradually become smaller, the subject's physical condition is improving (recovering); when the shape of the nipples deforms, becoming overlapping with adjacent nipples and increasing in size, the subject's physical condition is deteriorating. When nipples are always small, the subject's physical condition is always good; when nipples are always large, the subject's physical condition is always poor. Furthermore, when nipples gradually change to a regular arrangement, the subject's physical condition is improving (recovering); when nipples gradually change to an irregular arrangement, the subject's physical condition is deteriorating. When nipples are always regularly arranged, the subject's physical condition is always good; when nipples are always irregularly arranged, the subject's physical condition is always poor. Information processing system 1 can determine the health status of a person in more detail based on changes in the nipples.

[0111] The determination unit 222 determines the grayscale values ​​of each color component (e.g., red, green, and blue components) of each pixel contained in the nipple image as the color of the nipple. Alternatively, the determination unit 222 may determine the statistical values ​​(mean, median, maximum, minimum, and most frequent values, etc.) of the grayscale values ​​of each color component of each pixel contained in each nipple extracted from the nipple image as the color of the nipple in the nipple image.

[0112] As described above, the determination unit 222 can also determine nipple features through image analysis based on machine learning. Furthermore, when a learning model is used to determine nipple features, low-resolution images, particularly images with a resolution lower than the nipple image, can be used as teacher data for the learning model. Therefore, even when the tongue image obtained in step S101 has a low resolution, the determination unit 223 can appropriately determine nipple features using the learning model.

[0113] Furthermore, when using a learning model to determine nipple features, images captured by a camera mounted on a typical multifunction mobile phone and images captured by a microscope can be used as teacher data for the learning model. Alternatively, images captured by a camera mounted on a typical multifunction mobile phone and images captured by an SLR camera can also be used as teacher data for the learning model. On the other hand, when using this learning model, the determination unit 222 determines nipple features by inputting only images captured by the imaging device 104 (the camera mounted on a typical multifunction mobile phone) into the learning model. As a result, the learning model is learned efficiently, and the determination unit 222 can determine nipple features with high accuracy.

[0114] Alternatively, the determination unit 222 can also determine the nipple features of the tongue through template matching. For example, the server device 200 pre-stores multiple sampled images containing nipples with various features in the second storage device 210. The determination unit 222 cuts out regions of the same size as each sampled image from various locations in the nipple image and calculates the similarity between each cut-out region and each sampled image. If the calculated similarity is above a predetermined threshold, the determination unit 222 determines that a nipple with the features contained in the sampled image exists on the tongue contained in the tongue image. Alternatively, the determination unit 222 can calculate feature quantities based on each region cut from the nipple image and each sampled image, and determine that a nipple with the features contained in the sampled image exists on the tongue contained in the nipple image if the similarity of the feature quantities is above a predetermined threshold. The feature quantities are image gradients (e.g., HOG feature quantities). The similarity of the feature quantities is cosine similarity, inner product, etc.

[0115] Next, the determination unit 223 determines the vascular state and / or blood flow state of the subject based on the nipple features determined by the determination unit 222 (step S105). For example, the determination unit 223 determines the blood flow velocity, blood flow distribution, vascular flexibility, or blood flow quality of the subject as the vascular state of the subject. The determination unit 223 determines the blood flow quality as the blood flow state of the subject. The quality of blood flow distribution is represented by an evaluation value, for example, the smaller the unevenness of blood flow distribution (the more uniform the vascular distribution), the higher the evaluation value; the greater the unevenness of blood flow distribution (the more uneven the vascular distribution), the lower the evaluation value. Vascular flexibility is represented by an evaluation value, for example, the more flexible the vascular (the higher the elasticity of the vascular), the higher the evaluation value; the more rigid the vascular (the lower the elasticity of the vascular), the lower the evaluation value. Blood flow quality refers to the quality of blood circulation (flow). Blood flow quality can be represented by an evaluation value, for example. The better the blood circulation (the smoother the blood, i.e., the lower the blood viscosity), the higher the evaluation value; conversely, the worse the blood circulation (the thicker the blood, i.e., the higher the blood viscosity), the lower the evaluation value. On the other hand, for blood flow quality, there are cases where blood circulation is good and blood viscosity is high, and there are also cases where blood circulation is poor and blood viscosity is low.

[0116] For example, server device 200 pre-stores a relational expression or table representing the relationship between nipple features and vascular state or blood flow state in second storage device 210. In this table, nipple features and / or vascular state or blood flow state can also be classified using one or more thresholds. Determination unit 223 refers to the relational expression or table stored in second storage device 210 to determine the vascular state or blood flow state corresponding to the nipple features determined by determination unit 222 as the vascular state or blood flow state of the target person. Determination unit 223 may also use a learning model to determine the vascular state or blood flow state of the target person. This learning model is pre-learned to output the vascular state or blood flow state of a specific person when nipple features of that person are input. This learning model uses combinations of nipple features and vascular states or blood flow states involving multiple people as teacher data and is pre-learned using neural networks, support vector machines, etc. Determination unit 223 inputs the nipple features determined by determination unit 222 into the learning model and determines the vascular state or blood flow state of the target person from the information output by the learning model.

[0117] Figure 7 This is a schematic diagram illustrating an example of a blood flow distribution image captured using LSFG (Laser Speckle Flowgraphy) that represents the distribution of blood flow.

[0118] exist Figure 7 The image shows blood flow distribution images 700, 710, and 720 for multiple subjects. Blood flow distribution image 700 is for subject A, who is in good health; blood flow distribution image 710 is for subject B, who is in slightly poorer health; and blood flow distribution image 720 is for subject C, who is in poor health. LSFG generates blood flow distribution images by irradiating near-infrared laser light onto the subject's skin (blood vessels) and using a CCD or other imaging sensor to measure the pattern of light scattered by red blood cells.

[0119] In person A, blood flow has an ideal velocity, and in blood flow distribution image 700, light is moderately scattered. Compared to the ideal velocity, blood flow in person B is slightly erratic, and in blood flow distribution image 710, light is slightly deflected. Compared to the ideal velocity, blood flow in person C is quite erratic, and in blood flow distribution image 720, light is significantly deflected. That is, nipple features, particularly the shape, size, or regularity of the nipples, are correlated with blood flow velocity. Therefore, information processing system 1 can determine the blood flow velocity of a person with high accuracy based on the shape, size, or regularity of the nipples.

[0120] Furthermore, the blood flow distribution of person A is almost uniform; in blood flow distribution image 700, the spatial imbalance of blood flow is minimal. The blood flow distribution of person B is slightly uneven; in blood flow distribution image 710, there are localized areas with high blood flow. The blood flow distribution of person C is highly uneven; in blood flow distribution image 720, the blood flow is locally significantly higher. That is, nipple features, especially the shape, size, or regularity of the nipples, are correlated with vascular status, particularly blood flow distribution. Therefore, information processing system 1 can accurately determine the blood flow distribution of a person based on the shape, size, or regularity of their nipples.

[0121] Figure 8 This is a schematic diagram used to illustrate the relationship between nipple characteristics, vascular flexibility, and blood flow.

[0122] exist Figure 8 The diagram schematically shows a cross-section of a healthy person A's nipple 800 and the blood vessel 801 located directly beneath it, as well as a cross-section of the skin surface 802 and the blood vessel 803 located directly beneath it. Additionally, in... Figure 8 The diagram schematically shows a cross-section of the nipple 810 and the blood vessel 811 directly beneath it, and the skin surface 812 and the blood vessel 813 directly beneath it, belonging to a person B in slightly poor health. Additionally, in Figure 8The diagram schematically shows a cross-section of the nipple 820 and the blood vessel 821 directly beneath it, and a cross-section of the skin surface 822 and the blood vessel 823 directly beneath it, belonging to a person C in poor health. Additionally, in Figure 8 The diagram schematically shows the nipple 830 of a person D with excessive blood circulation, a cross-section of the blood vessel 831 located directly beneath it, and the skin surface 832, a cross-section of the blood vessel 833 located directly beneath it.

[0123] like Figure 8 As shown, the distance between the nipple and the blood vessels located directly beneath it is significantly smaller than the distance between the skin surface and the blood vessels located directly beneath it. The shape, size, regularity, or color of the nipple is strongly influenced by the blood vessels located directly beneath it.

[0124] Character A's blood vessels 801 and 803 are soft, and their blood circulation is adequate. In this case, character A's blood vessels 801 and 803 have an ideal shape. Therefore, nipple 800 is nearly circular in shape, of moderate size, and regularly arranged. On the other hand, character B's blood vessels 811 and 813 have slightly lower elasticity, and their blood circulation is slightly poorer. In this case, character B's blood vessels 811 and 813 have a twisted shape. Therefore, nipple 810's shape deviates slightly from a circle, and its size is larger. Because the shapes and sizes of the multiple nipples 810 are uneven, they are irregularly arranged. Furthermore, regarding character C's blood vessels 821 and 823, some have no blood flow, blood flow is restricted, and blood circulation in blood vessels 821 and 823 is poor. In this case, character C's blood vessels 821 and 823 have a collapsed shape. Therefore, the shape of the nipple 820 deviates significantly from a circle, and its size becomes extremely large. Because the shape and size of the multiple nipples 810 become extremely uneven, the nipples 820 are arranged in an extremely irregular manner. Furthermore, the blood vessels 831 and 833 of character D are dilated, resulting in excessive blood flow and over-circulation. Due to the dilation of blood vessels 831 and 833, the size of the nipple 830 becomes extremely large.

[0125] Thus, nipple characteristics, particularly the shape, size, or regularity of the nipples, are correlated with vascular flexibility. Therefore, information processing system 1 can accurately determine the vascular flexibility of a person based on the shape, size, or regularity of their nipples.

[0126] Furthermore, nipple characteristics, particularly the shape, size, or regularity of the nipples, are correlated with blood circulation. Therefore, the information processing system 1 can accurately determine the blood flow status of a person based on the shape, size, or regularity of their nipples.

[0127] Furthermore, if the determination unit 222 determines the nipple features at the tip and the inside of the tongue respectively, the determination unit 223 determines the blood vessel status of the subject at the tip and the inside of the tongue respectively.

[0128] Next, the determination unit 223 determines the health status of the subject based on the state of the subject's blood vessels (step S106). For example, the determination unit 223 determines the subject's skin condition as the subject's health status. The determination unit 223 determines the degree of skin roughness, wrinkles, pores, texture, or blemishes as the subject's skin condition.

[0129] For example, server device 200 pre-stores a relational expression or table representing the relationship between blood vessel state and skin state in second storage device 210. In this table, blood vessel state and / or skin state can also be classified according to one or more thresholds. Determination unit 223 refers to the relational expression or table stored in second storage device 210 to determine the skin state corresponding to the blood vessel state determined in step S105 as the skin state of the target person. Determination unit 223 can also use a learning model to determine the skin state of the target person. This learning model is pre-learned to output the skin state of a specific person when the blood vessel state of that person is input. This learning model uses combinations of blood vessel states and skin states involving multiple people as teacher data and is pre-learned using neural networks, support vector machines, etc. Determination unit 223 inputs the blood vessel state determined in step S105 to the learning model and determines the skin state of the target person from the information output by the learning model.

[0130] Figure 9 This is a schematic diagram representing an example of a skin image containing a person's skin.

[0131] exist Figure 9The image shows skin images 900, 910, and 920 for multiple individuals. Skin image 900 contains the skin of person A, who is in good health; skin image 910 contains the skin of person B, who is in slightly poor health; and skin image 920 contains the skin of person C, who is in poor health. As shown in skin image 900, the skin of person A, who is in good health, is smooth and is considered good. As shown in skin image 910, the skin of person B, who is in slightly poor health, is slightly rough. As shown in skin image 920, the skin of person C, who is in poor health, is quite rough. That is, nipple features, especially the shape, size, or regularity of the nipples, are correlated with vascular condition and skin condition, especially the degree of skin roughness. The information processing system 1 can determine the degree of skin roughness of an individual with high precision based on the shape, size, or regularity of the nipples. In particular, the information processing system 1 can determine the degree of skin roughness of an individual with even higher precision by determining the vascular condition of the individual based on the shape, size, or regularity of the nipples.

[0132] Figure 10 It is a diagram used to illustrate the relationship between nipple features and the degree of wrinkles, pores, textures, or spots.

[0133] exist Figure 10Images 1000, 1001, 1002, and 1003 of the nipple of person E, and images 1010, 1011, 1012, and 1013 of the nipple of person F are shown. As shown in nipple images 1000 and 1010, the roundness of the nipple of person F is lower than that of person E, and the regularity of the nipple of person F is also lower than that of person E. Skin images 1001 and 1011 respectively contain the areas with the most wrinkles W on the cheeks of person E and person F. As shown in skin images 1001 and 1011, the length, depth, and number of wrinkles W in person F are greater than those in person E. Skin images 1002 and 1012 respectively contain the areas with the most pores P on the cheeks of person E and person F. As shown in skin images 1002 and 1012, the size, depth, and number of pores P in person F are larger than those in person E. Skin images 1003 and 1013 respectively contain the areas with the most spots B on the cheeks of people E and F. As shown in skin images 1003 and 1013, the size and number of spots B in person F are larger than those in person E. That is, nipple features, especially the roundness or regularity of the nipples, are correlated with the degree of wrinkles, pores, texture, or spots. Therefore, the information processing system 1 can accurately determine the degree of wrinkles, pores, texture, or spots in a person based on the roundness or regularity of their nipples.

[0134] The determination unit 223 can also determine the state of the blood vessels of the subject as a measure of the subject's health status. For example, the determination unit 223 determines the subject's blood flow velocity, blood flow distribution, blood vessel flexibility, or blood flow quality as a measure of the subject's blood vessel status. As mentioned above, nipple characteristics, especially the roundness, size, or regularity of the nipples, are correlated with the state of the blood vessels, and the information processing system 1 can determine the state of the subject's blood vessels with high accuracy based on the roundness, size, or regularity of the subject's nipples.

[0135] Furthermore, the determination unit 223 can further determine the health status of the target person based on the self-perceived physical condition and / or environmental information of the target person obtained by the receiving unit 221. For example, the server device 200 pre-stores a relational expression or table representing the relationship between vascular status, self-perceived physical condition and / or environmental information, and the aforementioned health statuses in the second storage device 210. In this table, vascular status, self-perceived physical condition and / or environmental information can also be classified by one or more thresholds. The determination unit 223 refers to the relational expression or table stored in the second storage device 210 to determine each health status corresponding to the vascular status determined in step S105 and the self-perceived physical condition and / or environmental information obtained by the receiving unit 221 as the health status of the target person. The determination unit 223 can also use a learning model to determine the health status of the target person, the learning model being pre-learned so that it outputs the health status of the person when the vascular status and self-perceived physical condition and / or environmental information of a specific person are input. The learning model uses a combination of vascular status, self-perceived physical condition, and / or environmental information, along with health status, from multiple individuals as teacher data, and is pre-learned using neural networks, support vector machines, etc. The determination unit 223 inputs the vascular status determined in step S105 and the self-perceived physical condition and / or environmental information obtained by the receiving unit 221 into the learning model, and determines the health status of the target individual from the information output by the learning model. Thus, the information processing system 1 can determine the health status of the target individual with higher accuracy.

[0136] Furthermore, the determination unit 223 can also determine the cause of the health impairment based on the subject's self-perceived physical condition and / or environmental information, when the subject's health is impaired. For example, if the subject's self-perceived physical condition indicates fever or pain, the determination unit 223 determines that the cause of the health impairment is illness or injury. Additionally, if environmental information indicates the intake of cigarettes, sweets, etc., the determination unit 223 determines that the cause of the health impairment is increased oxidation. Therefore, the information processing system 1 can indicate the cause of the health impairment to the subject, improving convenience for the subject.

[0137] Furthermore, if the determination unit 222 determines the nipple features at both the tip and the back of the tongue, the determination unit 223 determines the health status of the subject at both the tip and the back of the tongue. Additionally, the determination unit 223 can also determine that the subject's health status is poor if it determines at either the tip or the back of the tongue. Therefore, the information processing system 1 can prevent the incorrect determination that the subject's health status is good even if it is actually poor. Furthermore, the determination unit 223 can also determine that the subject's health status is poor if it determines at both the tip and the back of the tongue. Therefore, the information processing system 1 can prevent the incorrect determination that the subject's health status is poor even if it is actually good.

[0138] Next, the output control unit 224 outputs information indicating the health status of the target person determined by the determination unit 223 by sending it to the terminal device 100 via the second communication device 201 (step S107). The output control unit 224 sends the target person's health status itself as information indicating the target person's health status. In addition, the output control unit 224 may also generate values ​​obtained by standardizing the target person's health status, images representing the target person's health status, charts representing the target person's health status, etc., and send the generated values, images, charts, etc. as information indicating the target person's health status.

[0139] Furthermore, the output control unit 224 may also send information indicating factors of the target person's health status, either based on or in place of the aforementioned information, as information indicating the target person's health status. Additionally, the output control unit 224 may also send information indicating changes in the target person's health status, either based on or in place of the aforementioned information, as information indicating the target person's health status.

[0140] Next, the display control unit 122 of the terminal device 100 obtains information indicating the health status of the target person by receiving it from the server device 200 via the first communication device 103. The display control unit 122 displays the received information indicating the health status of the target person on the first display device 102 (step S108). The determination process ends thereafter.

[0141] Furthermore, obtaining self-aware physical condition and / or environmental information in step S101 and sending and receiving self-aware physical condition and / or environmental information in steps S102 and S103 can also be omitted. In this case, in step S106, the determination unit 223 determines the health status of the target person without using the target person's self-aware physical condition and / or environmental information.

[0142] Alternatively, step S105 can be omitted. In this case, in step S106, the determination unit 223 directly determines the health status of the target person based on the nipple features. For example, the server device 200 pre-stores the relational expressions or tables representing the relationships between the nipple features and health statuses in the second storage device 210. In this table, nipple features and / or health information can also be classified by one or more thresholds. The determination unit 223 refers to the relational expressions or tables stored in the second storage device 210 and determines the health statuses corresponding to the nipple features determined by the determination unit 222 in step S104 as the health status of the target person. The determination unit 223 can also use a learning model to determine the health status of the target person. This learning model is pre-learned so that it outputs the health status of a specific person when nipple features of that person are input. This learning model uses combinations of nipple features and health statuses of multiple people as teacher data and is pre-learned through neural networks, support vector machines, etc. The determination unit 223, based on the nipple features determined by the input determination unit 222 in step S104, determines the health status of the subject from the information output by the learning model. In this case, the information processing system 1 can also determine the health status of the subject with high accuracy.

[0143] In this case, the server device 200 may pre-store a relational expression or table representing the relationship between nipple characteristics, self-perceived physical condition and / or environmental information, and health status in the second storage device 210, and the determination unit 223 may use the relational expression or table to determine the health status of the target person. Alternatively, the determination unit 223 may use a learning model to determine the health status of the target person, the learning model being pre-learned to output the health status of the person when nipple characteristics, self-perceived physical condition, and / or environmental information of a specific person are input.

[0144] In addition, the determination unit 223 can also determine the nutritional status of the subject as the subject's health status. For example, the determination unit 223 determines whether predetermined components (protein, vitamin B, or zinc, etc.) are insufficient or whether predetermined components (sugar, etc.) are excessive and / or their degree as the subject's nutritional status.

[0145] For example, server device 200 pre-stores a relational expression or table representing the relationship between nipple features and nutritional status in second storage device 210. In this table, nipple features and / or nutritional status can also be classified using one or more thresholds. Determination unit 223 refers to the relational expression or table stored in second storage device 210 to determine the nutritional status corresponding to the nipple features determined by determination unit 222 as the nutritional status of the target person. Determination unit 223 can also use a learning model to determine the nutritional status of the target person. This learning model is pre-learned to output the nutritional status of a specific person when nipple features of that person are input. This learning model uses combinations of nipple features and nutritional statuses of multiple people as teacher data and is pre-learned using neural networks, support vector machines, etc. Determination unit 223 inputs the nipple features determined by determination unit 222 into the learning model and determines the nutritional status of the target person from the information output by the learning model.

[0146] Figure 11 This is a diagram used to illustrate the relationship between nipple characteristics and the amounts of protein, vitamin B, and zinc.

[0147] exist Figure 11 Images 1100 (nipple image of person G), 1110 (nipple image of person H), 1120 (nipple image of person I), and 1130 (nipple image of person J) are shown. Person G has adequate protein and vitamin B levels, while person H is deficient in both. Person I has adequate zinc levels, while person J is deficient in zinc. Images 1101, 1111, 1121, and 1131 are magnified views of portions of nipple images 1100, 1110, 1120, and 1130, respectively. As shown in images 1101 and 1111, compared to the nipple of person G (who has adequate protein and vitamin B levels), the nipple of person H (who is deficient in both protein and vitamin B levels) is nearly yellowish-white. Furthermore, as shown in images 1121 and 1131, compared to the nipples of person I, who has an adequate amount of zinc, the nipples of person J, who has a zinc deficiency, are nearly white. That is, nipple characteristics, particularly nipple color, are correlated with the amounts of protein, vitamin B, and zinc. Therefore, information processing system 1 can accurately determine the amounts of protein, vitamin B, and zinc in a person based on the nipple characteristics.

[0148] Figure 12 This is a diagram used to illustrate the relationship between nipple characteristics and sugar content.

[0149] exist Figure 12Image 1200 shows the nipples of a person K with excessive sugar intake. As shown in nipple image 1200, the nipples of person K with excessive sugar intake are lighter in color than the nipples of a person with an appropriate amount of sugar. That is, nipple features, especially nipple color, are correlated with the amount of sugar. Therefore, information processing system 1 can accurately determine the amount of sugar in a person based on the nipple color.

[0150] As described in detail above, the information processing system 1 outputs the health status of the subject based on the nipple image of the subject's tongue. Therefore, the information processing system 1 can output the subject's health status with higher accuracy. Consequently, the user of the terminal device 100 can correctly identify the subject's health status and provide appropriate advice, etc. Furthermore, the information processing system 1 can output the subject's health status inexpensively using a general-purpose terminal device with a camera, without relying on expensive systems such as LSFG.

[0151] (Second Implementation)

[0152] In this embodiment, Figure 4 In step S104, the determining unit 222 determines the overall shape features of the subject's tongue based on the tongue image. In step S106, the determining unit 223 determines the subject's health status based on the overall shape features of the subject's tongue. The processing in step S105 is omitted. The tongue image includes the entire tongue. In particular, the determining unit 222 determines the crack features of the subject's tongue as a feature of the overall shape of the subject's tongue. The determining unit 223 determines the subject's nutritional status and / or psychological stress as a measure of the subject's health status. For example, the nutritional status and / or psychological stress may refer to the degree of stress that prevents the body from absorbing nutrients. The nutritional status may also refer to whether the content and / or timing of the diet are appropriate. The nutritional status may also refer to the amount or deficiency of vitamins, minerals, and protein. The nutritional status may also refer to whether there is a vitamin deficiency, especially whether there is a gastrointestinal vitamin deficiency.

[0153] Figure 14 This is a schematic diagram illustrating an example of a tongue image related to this embodiment.

[0154] exist Figure 14 The image shows tongue images 1400, 1410, and 1420 for several subjects. Tongue image 1400 contains the tongue of a subject whose tongue is not cracked, tongue image 1410 contains the tongue of a subject whose tongue has small cracks, and tongue image 1420 contains the tongue of a subject whose tongue has large cracks.

[0155] For example, the determination unit 222 determines the cracked features of the tongue through histogram analysis. The determination unit 222 generates a green tongue image with green component values ​​(Green values) as grayscale values ​​based on the tongue image, which is an RGB image. Alternatively, the determination unit 222 generates a saturated tongue image with saturation component values ​​(Saturation values) as grayscale values ​​based on the tongue image converted from an RGB image to an HSV image. The determination unit 222 calculates the sum of the grayscale values ​​of the pixels contained in each vertical line extending vertically in either the green or saturated tongue image. The determination unit 222 uses the horizontal position of each vertical line as a level and the sum calculated for each vertical line as a degree, generating a position histogram arranged in order of the horizontal position of each vertical line.

[0156] Figure 15 This is a schematic diagram representing an example of a position histogram.

[0157] Position histogram 1500 is generated based on tongue image 1400, position histogram 1510 is generated based on tongue image 1410, and position histogram 1520 is generated based on tongue image 1420. The horizontal axis of each position histogram represents the horizontal position (grade) of each vertical line, and the vertical axis represents the total value (degree) calculated for each vertical line. The total values ​​calculated for each vertical line are arranged in order of their horizontal position.

[0158] The determination unit 222 scans the levels of the position histogram from one end to the other in the horizontal direction, and determines groups G of vertical lines that are consecutive and whose total degree value is less than a reference value R, and whose total degree value is greater than or equal to the reference value R. The reference value R is, for example, set as the average of the total degree values ​​of each vertical line multiplied by a predetermined coefficient. The determination unit 222 determines that small cracks are present on the tongue if the number of groups G is greater than or equal to a predetermined threshold, and determines that small cracks are not present on the tongue if the number of groups G is less than the threshold. Furthermore, the determination unit 222 determines that large cracks are present on the tongue if the number of vertical lines W contained in any group G is greater than or equal to a predetermined threshold, and determines that small cracks are not present on the tongue if the number of vertical lines W contained in any group G is less than the threshold. The determination unit 222 determines whether small cracks and large cracks are present on the tongue as crack characteristics.

[0159] The determination unit 222 can also calculate the total grayscale value of the pixels contained in each horizontal line extending horizontally in the tongue image, and determine a group of continuous horizontal lines whose total value is less than a reference value and whose total value is greater than the reference value. In this case, the determination unit 222 calculates the crack features of the tongue based on the number of determined groups or the number of horizontal lines contained in the groups. The determination unit 222 can also determine the crack features of the tongue as the number of groups calculated from multiple tongue images of a subject obtained by taking pictures of the subject's tongue at predetermined time intervals, or the change in the number of vertical or horizontal lines contained in the groups.

[0160] The determination unit 222 can also determine the crack features of the tongue through template matching. For example, the server device 200 pre-stores multiple sampled images containing cracks of various sizes in the second storage device 210. The determination unit 222 cuts out regions of the same size as each sampled image from various locations in the tongue image and calculates the similarity between each cut-out region and each sampled image. If the calculated similarity is above a predetermined threshold, the determination unit 222 determines that a crack of the size contained in the sampled image exists on the tongue contained in the tongue image. Alternatively, the determination unit 222 can also calculate the similarity between each region and each sampled image after performing histogram matching between each region and each sampled image.

[0161] For example, server device 200 pre-stores a relational expression or table representing the relationship between cracked features and the body's nutritional status and / or psychological stress in second storage device 210. In this table, cracked features and / or the body's nutritional status and / or psychological stress can also be classified using one or more thresholds. Determination unit 223 refers to the relational expression or table stored in second storage device 210 to determine the nutritional status and / or psychological stress corresponding to the cracked features determined by determination unit 222 as the nutritional status and / or psychological stress of the target person. Determination unit 223 can also use a learning model to determine the nutritional status and / or psychological stress of the target person, the learning model being pre-learned to output the nutritional status and / or psychological stress of that person when cracked features of a specific person are input. This learning model uses combinations of cracked features and nutritional status and / or psychological stress involving multiple people as teacher data, and is pre-learned using neural networks, support vector machines, etc. The determination unit 223 inputs the crack features determined by the determination unit 222 into the learning model, and determines the nutritional status and / or psychological stress of the subject from the information output by the learning model.

[0162] The presence of small or large cracks on the tongue may indicate poor gastrointestinal health. Information processing system 1 can accurately determine a person's health status by utilizing the crack characteristics of the tongue.

[0163] (Third Implementation)

[0164] In this embodiment, Figure 4 In step S104, the determining unit 222 determines the overall shape features of the subject's tongue based on the tongue image. In step S106, the determining unit 223 determines the subject's health status based on the overall shape features of the subject's tongue. The processing in step S105 is omitted. The tongue image includes the entire tongue. The tongue image includes a view of the tongue taken from the side. In particular, the determining unit 222 determines the thickness features of the subject's tongue as a feature of the overall shape of the subject's tongue. The determining unit 223 determines metabolic abnormalities as a feature of the subject's health status. For example, metabolic abnormalities include abnormal body water content, abnormal glucose metabolism, or abnormal lipid metabolism.

[0165] Figure 16 This is a schematic diagram illustrating an example of a tongue image related to this embodiment.

[0166] exist Figure 16 The image shows a tongue image 1600. The tongue image 1600 contains a view of the tongue taken from the side.

[0167] The determination unit 222 determines the thickness features of the tongue of the object being viewed by analyzing the shape of the object in the image. The determination unit 222 uses well-known image processing techniques to determine the orientation of the tongue contained in the tongue image. For example, the determination unit 222 extracts feature points such as the front, side, upper, and lower ends of the tongue from the tongue image, and determines the orientation of the tongue based on the positional relationship of the extracted feature points. The determination unit 222 can also determine the orientation of the tongue by performing template matching with multiple sampled images containing the tongue taken under various orientations. If the orientation of the tongue is not within a predetermined range, the determination unit 222 sends a request signal to the terminal device 100 to re-acquire the tongue image. Upon receiving the request signal, the terminal device 100 re-acquires the tongue image. In this case, the terminal device 100 can also re-acquire the tongue image while displaying guidance for aligning the tongue position on the first display device 102 to enable proper tongue imaging.

[0168] In addition, Figure 4In steps S101 to S102, the transmitting unit 121 may also acquire a facial image obtained by capturing a face along with the tongue image and send it to the server device 200. In this case, in step S103, the receiving unit 221 receives the facial image along with the tongue image. The determining unit 222 uses well-known image processing techniques to extract feature points such as the outer corner of the eye, the apex of the nose, the root of the nose, the tip of the lips, and the center of the lips from the facial image, and determines the orientation of the face based on the positional relationship of the extracted feature points. If the orientation of the face is not horizontal, the determining unit 222 sends a request signal to the terminal device 100 to notify the user of the horizontal orientation. Upon receiving the request signal, the terminal device 100 acquires the tongue image and the facial image again. In this case, the terminal device 100 displays the horizontal orientation indication on the first display device 102 to notify the user.

[0169] In addition, Figure 4 In steps S101 to S102, the transmitting unit 121 may also acquire a frontal tongue image obtained from a frontal view along with a tongue image obtained from a side view, and send it to the server device 200. In this case, in step S103, the receiving unit 221 receives the frontal tongue image along with the tongue image. The determining unit 222 uses well-known image processing techniques to determine the orientation of the tongue included in the frontal tongue image. The determining unit 222 determines the tongue orientation by the positional relationship of feature points or template matching, etc. If the tongue orientation is not horizontal but upward or downward, the determining unit 222 sends a request signal to the terminal device 100 requesting the user to extend their tongue. Upon receiving the request signal, the terminal device 100 acquires the tongue image again. In this case, the terminal device 100 displays an instruction to extend the tongue on the first display device 102 to notify the user.

[0170] Similar to the case of cutting out the nipple image, the determination unit 222 extracts the end pixels representing the tip of the tongue, and extracts the region surrounded by the extracted end pixels as the tongue region. The determination unit 222 rotates the tongue image so that the tongue extends in the horizontal direction. For example, the determination unit 222 extracts feature points such as the front tip, side tip, upper tip, and lower tip of the tongue within the tongue region, performs an affine transformation, and rotates the tongue image so that each feature point is positioned within a predetermined range.

[0171] Figure 16 The tongue image 1610 shown is an image in which the tongue image 1600 has been rotated so that the tongue extends in the horizontal direction.

[0172] In the rotated image of the tongue, the determination unit 222 calculates the distance H between the upper and lower ends of the tongue at predetermined intervals along the horizontal direction from the front end towards the throat. The determination unit 222 determines the amount of change in the distance H calculated at each predetermined interval as a thickness feature of the tongue of the subject. The thicker the tongue, the greater the amount of change in distance H.

[0173] The determining unit 222 can also calculate the roundness of the tongue as a characteristic of its thickness. The determining unit 222 determines the center position (center of gravity) of the tongue, and further determines the upper and lower positions of the tongue at the positions where distance H is minimized (the front end of the tongue) and maximized (the horizontal position). The determining unit 222 calculates the distances between the center position and the front, upper, and lower positions of the tongue, calculating the roundness in a manner where a larger difference between these distances indicates lower roundness, and a smaller difference between these distances indicates higher roundness. A thicker tongue generally has a higher roundness.

[0174] For example, server device 200 pre-stores a relational expression or table representing the relationship between tongue thickness features and the presence or absence of metabolic abnormalities in second storage device 210. In this table, tongue thickness features and / or the presence or absence of metabolic abnormalities can also be classified using one or more thresholds. Determination unit 223 refers to the relational expression or table stored in second storage device 210 to determine the presence or absence of metabolic abnormalities corresponding to the tongue thickness features determined by determination unit 222 as the presence or absence of metabolic abnormalities in the target person. Determination unit 223 may also use a learning model to determine the presence or absence of metabolic abnormalities in the target person. This learning model is pre-learned to output the presence or absence of metabolic abnormalities in a specific person when the tongue thickness features of that person are input. This learning model uses combinations of tongue thickness features and the presence or absence of metabolic abnormalities involving multiple people as teacher data and is pre-learned using neural networks, support vector machines, etc. Determination unit 223 inputs the tongue thickness features determined by determination unit 222 into the learning model and determines the presence or absence of metabolic abnormalities in the target person from the information output by the learning model.

[0175] If the determination unit 223 determines that the subject's health condition is due to an abnormality in body water content, the cause of the subject's health condition is determined to be either a kidney abnormality or excessive salt intake. If the determination unit 223 determines that the subject's health condition is due to an abnormality in lipid content, the cause is determined to be either a liver abnormality, excessive sweets intake, or excessive carbohydrate intake.

[0176] A thicker tongue is often associated with higher water retention and potentially poorer kidney or digestive health. Additionally, a thicker tongue may indicate higher lipid levels and less efficient lipid and water metabolism. Information processing system 1 uses tongue thickness as a characteristic to accurately determine a person's health status.

[0177] (Fourth Implementation)

[0178] In this embodiment, Figure 4 In step S104, the determining unit 222 determines the surface characteristics of the subject's tongue based on the tongue image. In step S106, the determining unit 223 determines the subject's health status based on the surface characteristics of the subject's tongue. The processing in step S105 is omitted. The tongue image includes the surface characteristics of the tongue. In particular, the determining unit 222 determines the amount of tongue coating on the subject's tongue surface as a characteristic of the subject's tongue surface characteristics. The determining unit 223 determines the body's nutritional status as the subject's health status. For example, the body's nutritional status is the degree of lipid metabolism.

[0179] For example, the determination unit 222 determines the amount of tongue coating on the tongue surface through texture analysis. The determination unit 222 generates a green tongue image or a saturated tongue image based on the tongue image. Alternatively, the determination unit 222 generates a chromatic tongue image based on a tongue image converted from an RGB image to a Lab image, using the component values ​​(b* values) of yellow and blue chromaticity as grayscale values. In the green tongue image, saturated tongue image, or chromatic tongue image, the determination unit 222 calculates the number of pixels with each grayscale value within the grayscale range of each pixel. The determination unit 222 uses each grayscale value as a level and the number of pixels with each grayscale value as a degree, generating a grayscale histogram arranged in order of grayscale value magnitude.

[0180] Figure 17 This is a schematic diagram representing an example of a grayscale histogram.

[0181] Figure 17 The grayscale histogram 1700 shown has grayscale values ​​(levels) on the horizontal axis and the number of pixels (degrees) with each grayscale value on the vertical axis. The grayscale values ​​are arranged in ascending order.

[0182] The determination unit 222 scans the grayscale histogram levels from the low side to the high side, determining the maximum degree grayscale value U1 and the maximum grayscale value U2. The maximum degree grayscale value U1 is the grayscale value with the largest number of pixels as its degree, and the maximum grayscale value U2 is the maximum grayscale value with a degree greater than 0. The determination unit 222 determines the subdivided grayscale value U3, which is obtained by subdividing the maximum degree grayscale value U1 and the maximum grayscale value U2 at a predetermined ratio. The determination unit 222 generates a binary image in the tongue image by selecting pixels with grayscale values ​​greater than the maximum degree grayscale value U1 and less than the subdivided grayscale value U3 as valid pixels, and pixels with grayscale values ​​less than the maximum degree grayscale value U1 or less than the subdivided grayscale value U3 as invalid pixels.

[0183] Figure 17 Image 1710 shown represents an example of a binary image. In image 1710, pixels with gray values ​​greater than the maximum gray value U1 and less than the inner gray value U3 are displayed as white, while pixels with gray values ​​less than the maximum gray value U1 or less than the inner gray value U3 are displayed as black.

[0184] The determination unit 222 determines the area (number of pixels) of the effective pixels within the generated binary image as the amount of tongue coating on the tongue surface. Alternatively, if the resolution of the tongue image is higher than a predetermined threshold, the determination unit 222 may perform noise reduction such as smoothing on the tongue image, generate a grayscale histogram, and then generate a binary image.

[0185] The determination unit 222 can also determine the amount of tongue coating on the tongue surface based on the number of edge pixels of a specific color component within the tongue image. For example, the determination unit 222 generates a first tongue image with specific color component values ​​(Red, Green, or Blue values) as grayscale values ​​based on a tongue image that is an RGB image. Alternatively, the determination unit 222 generates a second tongue image with specific component values ​​(Hue, Saturation, or Value) as grayscale values ​​based on a tongue image converted from an RGB image to an HSV image. Alternatively, the determination unit 222 generates a third tongue image with specific component values ​​(L*, a*, or b* values) as grayscale values ​​based on a tongue image converted from an RGB image to a Lab image. The determination unit 222 calculates the number of pixels with each grayscale value in the first, second, or third tongue image according to the grayscale value within the grayscale range of each pixel. The determination unit 222 uses each grayscale value as a level and the number of pixels with each grayscale value as a degree to generate a grayscale histogram arranged in order of grayscale value magnitude. The determination unit 222 sets a reference difference based on the generated grayscale histogram. For example, the determination unit 222 is similar to that in the first embodiment. Figure 13Similarly, in the example shown, the difference D between the upper limit grayscale value V2 and the lower limit grayscale value V3 is calculated, and the calculated difference D is set as the reference difference. The determination unit 222 extracts pixels from the first tongue image, second tongue image, or third tongue image whose grayscale value difference with adjacent pixels in the horizontal, vertical, or diagonal direction is greater than or equal to the reference difference as edge pixels. The reference difference can also be a predetermined fixed value. The determination unit 222 determines the number of extracted edge pixels as the amount of tongue coating on the tongue surface.

[0186] For example, server device 200 pre-stores a relational expression or table representing the relationship between the amount of tongue coating on the tongue surface and the nutritional status of the body in second storage device 210. In this table, the amount of tongue coating on the tongue surface and / or the nutritional status of the body can also be classified using one or more thresholds. Determination unit 223 refers to the relational expression or table stored in second storage device 210 to determine the nutritional status of the body corresponding to the amount of tongue coating on the tongue surface determined by determination unit 222, and uses this as the nutritional status of the target person's body. Determination unit 223 can also use a learning model to determine the nutritional status of the target person's body. This learning model is pre-learned to output the nutritional status of a specific person's body when the amount of tongue coating on the tongue surface of that person is input. This learning model uses combinations of tongue coating amounts and nutritional statuses on the tongue surfaces of multiple people as teacher data and is pre-learned using neural networks, support vector machines, etc. Determination unit 223 inputs the amount of tongue coating on the tongue surface determined by determination unit 222 into the learning model and determines the nutritional status of the target person's body from the information output by the learning model.

[0187] A thick coating on the tongue may indicate excessive sugar levels in the body. Conversely, a lack of tongue coating may suggest a deficiency in minerals. Information processing system 1 can accurately determine a person's health status by analyzing the amount of tongue coating on its surface.

[0188] (Fourth Implementation)

[0189] In this embodiment, Figure 4In step S104, the determining unit 222 determines the papillary features of the subject's tongue based on the tongue image. In step S106, the determining unit 223 determines the subject's health status based on the papillary features of the subject's tongue. The processing in step S105 is omitted. The tongue image includes the papillae of the tongue. In particular, the determining unit 222 determines the features of the fungiform papillae of the subject's tongue as the papillary features of the subject's tongue. The determining unit 222 determines the shape or color features of the fungiform papillae of the subject's tongue as the fungiform papilla features of the subject's tongue. The determining unit 223 determines the subject's antioxidant balance or nutritional status as the subject's health status. For example, antioxidant balance refers to oxidative stress. For example, nutritional status refers to the amount or deficiency of vitamins, minerals, and proteins.

[0190] For example, the determination unit 222 determines the shape features of the fungiform papillae of the subject's tongue as features of the fungiform papillae of the subject's tongue. For example, the determination unit 222 generates a blue tongue image with the blue component value (Blue value) as the grayscale value based on the tongue image, which is an RGB image. Alternatively, the determination unit 222 generates a saturated tongue image from the tongue image. In the blue tongue image or the saturated tongue image, the determination unit 222 calculates the number of pixels with each grayscale value according to the grayscale value within the grayscale range of each pixel. The determination unit 222 generates a grayscale histogram arranged in order of grayscale value magnitude, with each grayscale value as a level and the number of pixels with each grayscale value as a degree. The determination unit 222 scans the levels of the grayscale histogram from the side with high grayscale value to the side with low grayscale value, and calculates the accumulated value obtained by accumulating the degree (number of pixels) of each level (grayscale value). The determination unit 222 determines a grayscale value that, for the first time, exceeds a predetermined proportion (e.g., 10%) in proportion to the total number of pixels in the blue tongue image or the saturated tongue image, as a reference value. The determination unit 222 generates a binary image in the blue tongue image or the saturated tongue image, in which pixels with grayscale values ​​above the reference value are considered valid pixels, and pixels with grayscale values ​​below the reference value are considered invalid pixels.

[0191] Figure 18 This is a schematic diagram illustrating an example of a saturated tongue image and a binary image involved in this embodiment.

[0192] exist Figure 18 The image shows a saturated tongue image 1800 and a binary image 1810. The binary image 1810 is a binary image generated from the saturated tongue image 1800.

[0193] Within the binary image, the determination unit 222 determines the effective pixels contained in a predetermined region D corresponding to the tip of the tongue as fungal papillae, and determines the area (number of pixels) of the determined effective pixels as the shape feature of the fungal papillae.

[0194] The determination unit 222 can also determine the color of the fungiform papillae of the subject's tongue as a feature of the fungiform papillae of the subject's tongue. For example, the determination unit 222 generates a blue tongue image or a saturated tongue image based on the tongue image. The determination unit 222 calculates the sum of the gray values ​​of the pixels contained in each horizontal line extending horizontally in the blue tongue image or the saturated tongue image. The determination unit 222 uses the vertical position of each horizontal line as a level and the sum calculated for each horizontal line as a degree, and generates a position histogram arranged in order of the vertical position of each horizontal line.

[0195] Figure 19 This is a schematic diagram representing an example of a saturation tongue image and a position histogram.

[0196] exist Figure 19 The image shows a saturation tongue image 1900 and a position histogram 1910. Position histogram 1910 is a position histogram generated from the saturation tongue image 1900. The horizontal axis of position histogram 1910 represents the vertical position (grade) of each horizontal line, and the vertical axis represents the total value (degrees) calculated for each horizontal line. The total values ​​calculated for each horizontal line are arranged in order of their vertical position.

[0197] Within the position histogram, the determining unit 222 identifies horizontal lines whose vertical position (as a grade) is contained within a predetermined range S corresponding to the tip of the tongue, and whose total degree value is above a reference value R, as horizontal lines containing fungal papillae. The determining unit 222 determines whether a horizontal line containing fungal papillae exists as a color feature of the fungal papillae. The reference value R is, for example, set as the average of the total values ​​of all horizontal lines multiplied by a predetermined coefficient.

[0198] The determining unit 222 may also calculate the regularity of fungal papillae in the same way as the method for calculating the regularity of papillae in the first embodiment, and determine the calculated regularity as a characteristic of fungal papillae.

[0199] Alternatively, the determination unit 222 can also determine the features of the fungiform papillae through image analysis based on machine learning. In this case, the determination unit 222 uses a learning model to determine the features of the fungiform papillae, which has been pre-learned to output the features of the fungiform papillae in an image when an image is input. This learning model uses multiple images containing fungiform papillae and combinations of the features of the fungiform papillae in each image as teacher data, and is pre-learned using neural networks, support vector machines, etc. The determination unit 222 inputs a tongue image to the learning model and obtains the information output from the learning model as the features of the fungiform papillae.

[0200] In this case, the teacher data for the learning model can also be images captured by a camera mounted on a general-purpose mobile phone or images captured by a microscope. Alternatively, the teacher data for the learning model can also be images captured by a camera mounted on a general-purpose mobile phone or images captured by an SLR camera. On the other hand, when using this learning model, the determination unit 222 determines the characteristics of the fungal papillae by inputting only the images captured by the imaging device 104 (the camera mounted on a general-purpose mobile phone) into the learning model. As a result, the learning model is learned efficiently, and the determination unit 222 can determine the papillae characteristics with high accuracy.

[0201] For example, server device 200 pre-stores a relational expression or table representing the relationship between the characteristics of fungal papillae and antioxidant balance or nutritional status in second storage device 210. In this table, the characteristics of fungal papillae and / or antioxidant balance and / or nutritional status can also be classified using one or more thresholds. Determination unit 223 refers to the relational expression or table stored in second storage device 210 to determine the antioxidant balance or nutritional status corresponding to the characteristics of fungal papillae determined by determination unit 222 as the antioxidant balance or nutritional status of the target person. Determination unit 223 can also use a learning model to determine the antioxidant balance or nutritional status of the target person, the learning model being pre-learned to output the antioxidant balance or nutritional status of that person when the characteristics of fungal papillae of a specific person are input. This learning model uses combinations of fungal papillae characteristics and antioxidant balance or nutritional status of multiple people as teacher data, and is pre-learned using neural networks, support vector machines, etc. The determination unit 223 inputs the characteristics of the fungal papillae determined by the determination unit 222 into the learning model, and determines the antioxidant balance or nutritional status of the subject from the information output by the learning model.

[0202] When the fungal papillae are reddish, oxidative stress may be high. Information processing system 1, by utilizing the characteristics of the fungal papillae, can accurately determine a person's health status.

[0203] (Fifth Implementation)

[0204] In this embodiment, Figure 4 In step S104, the determining unit 222 determines the overall shape features of the subject's tongue based on the tongue image. In step S106, the determining unit 223 determines the subject's health status based on the overall shape features of the subject's tongue. The processing in step S105 is omitted. The tongue image includes the entire tongue. The tongue image includes a frontal tongue image taken from the front and a lateral tongue image taken from the side (or oblique direction). In particular, the determining unit 222 determines the teeth marks on the subject's tongue as a feature of the overall shape of the subject's tongue. Teeth marks are tooth-shaped marks imprinted on the tip of the tongue due to the tongue's downward position and the tip of the tongue abutting against the inside of the lower teeth. The determining unit 223 determines lipid abnormalities as a feature of the subject's health status.

[0205] For example, the determination unit 222 determines the teeth marks features of the tongue through histogram analysis. The determination unit 222 generates a green tongue image or a saturated tongue image from the tongue image. The determination unit 222 performs low-resolution reduction of the green tongue image or saturated tongue image by intermittently removing pixels within the green tongue image or saturated tongue image. In the low-resolution green tongue image or saturated tongue image, the determination unit 222 calculates the number of pixels with each grayscale value within the grayscale range of each pixel. The determination unit 222 generates a grayscale histogram arranged in order of grayscale value magnitude, using each grayscale value as a level and the number of pixels with each grayscale value as a degree. The determination unit 222 scans the levels of the grayscale histogram from the low grayscale value side to the high grayscale value side, and calculates a cumulative value obtained by accumulating the degree (number of pixels) for each level (grayscale value). The determination unit 222 determines a reference value as the grayscale value at which the proportion of the accumulated value relative to the total number of pixels in the low-resolution green tongue image or saturated tongue image first exceeds a predetermined proportion. The determination unit 222 generates a binary image by designating pixels with grayscale values ​​above the reference value as valid pixels and pixels with grayscale values ​​below the reference value as invalid pixels in the low-resolution green tongue image or saturated tongue image. In the binary image, the determination unit 222 calculates the number of invalid pixels contained in each vertical line extending in the vertical direction. The determination unit 222 uses the horizontal position of each vertical line as a level and the number of invalid pixels calculated with respect to each vertical line as a degree, generating a position histogram arranged in order of the horizontal position of each vertical line.

[0206] Figure 20 This is a schematic diagram illustrating an example of a binary image and a position histogram involved in this embodiment.

[0207] exist Figure 20 The diagram shows a binary image 2000 and a position histogram 2010. Binary image 2000 was generated from an image of a tongue with teeth marks. In binary image 2000, valid pixels are represented by white, and invalid pixels by black. Position histogram 2010 is a position histogram generated from binary image 2000. The horizontal axis of position histogram 2010 represents the position (level) of each vertical line in the horizontal direction, and the vertical axis represents the number (degrees) of invalid pixels calculated with respect to each vertical line. The number of invalid pixels calculated with respect to each vertical line is arranged in order of position in the horizontal direction of each vertical line.

[0208] The determination unit 222 scans the levels of the position histogram from one end to the other in a horizontal direction to determine the locations of the maximum and minimum values ​​of the number of invalid pixels. The determination unit 222 calculates the difference D between adjacent maximum and minimum values ​​as the size of the serration, and determines the average or maximum value of the calculated serration size as the serration feature. The determination unit 222 may also calculate the number of combinations where the difference D between adjacent maximum and minimum values ​​is above a threshold as the number of serrations, and determine the number of serrations as the serration feature.

[0209] Furthermore, if the determined tooth mark feature is above a predetermined tooth mark threshold, the determination unit 222, similar to the third embodiment, determines the thickness feature of the tongue based on the side tongue image.

[0210] The determination unit 222 can also determine the teeth marks features of the tongue through template matching. For example, the server device 200 pre-stores multiple sampled images containing teeth marks of various shapes in the second storage device 210. The determination unit 222 cuts out regions of the same size as each sampled image from various locations in the tongue image and calculates the similarity between each cut-out region and each sampled image. If the calculated similarity is above a predetermined threshold, the determination unit 222 determines that teeth marks of the shape (size) contained in the sampled image exist on the tongue contained in the tongue image. Alternatively, the determination unit 222 calculates feature quantities based on each region cut out from the tongue image and each sampled image, and if the similarity of the feature quantities is above a predetermined threshold, determines that teeth marks of the size contained in the sampled image exist on the tongue contained in the tongue image. The feature quantities are image gradients (e.g., HOG feature quantities). The similarity of the feature quantities is cosine similarity, inner product, etc.

[0211] For example, server device 200 pre-stores a relational expression or table representing the relationship between tongue teeth marks and the presence or absence of lipid abnormalities in second storage device 210. In this table, tongue teeth marks and / or the presence or absence of lipid abnormalities can also be classified using one or more thresholds. Determination unit 223 refers to the relational expression or table stored in second storage device 210 to determine the presence or absence of lipid abnormalities corresponding to the tongue teeth marks determined by determination unit 222, and uses this as the determination of whether or not a person has lipid abnormalities. Determination unit 223 can also use a learning model to determine the presence or absence of lipid abnormalities in a person. This learning model is pre-learned to output the presence or absence of lipid abnormalities in a person when the tongue teeth marks of a specific person are input. This learning model uses combinations of tongue teeth marks and the presence or absence of lipid abnormalities involving multiple people as teacher data and is pre-learned using neural networks, support vector machines, etc. Determination unit 223 inputs the tongue teeth marks determined by determination unit 222 into the learning model and determines the presence or absence of lipid abnormalities in the person from the information output by the learning model.

[0212] Alternatively, if the teeth mark feature is above a predetermined teeth mark threshold, the determination unit 223 determines whether the tongue thickness feature is above a predetermined thickness threshold. If the tongue thickness feature is above the thickness threshold, the determination unit 223 determines that edema exists; if the tongue thickness feature is below the thickness threshold, the determination unit 223 determines that pressure exists.

[0213] If the judgment unit 223 determines that the person being judged has lipid abnormalities based on their health status, it determines that the person has excessive intake of sweets, lipids, or carbohydrates.

[0214] The causes of teeth marks include edema, stress, and low tongue position. Information processing system 1 can accurately determine a person's health status by using teeth mark features.

[0215] (Sixth Implementation Method)

[0216] In this embodiment, Figure 4In step S104, the determination unit 222 determines the characteristics of the surface and back of the tongue of the subject based on the tongue image. In step S106, the determination unit 223 determines the health status of the subject based on the characteristics of the surface and back of the tongue. The processing in step S105 is omitted. The tongue image includes the dorsal side of the tongue, particularly the dorsal side, or the surface side of the tongue, particularly the surface side. In particular, the determination unit 222 determines the state of congestion on the back of the subject's tongue as a characteristic of the surface and back of the subject's tongue based on the tongue image obtained by photographing the dorsal side of the tongue, and determines the state of congestion on the surface of the subject's tongue as a characteristic of the surface and back of the subject's tongue based on the tongue image obtained by photographing the surface side of the tongue. Congestion refers to a state in which blood in the body stagnates and becomes difficult to flow. The determination unit 223 determines the state of the body's blood and / or blood flow as the health status of the subject. For example, the state of blood or blood flow is the presence or absence of abnormalities in neutral fats or cholesterol in the blood. The state of blood flow can also be the degree of good blood flow.

[0217] For example, the determination unit 222 determines the state of bruising on the surface or the back of the tongue through histogram analysis. The determination unit 222 generates a saturated tongue image from the tongue image. In the saturated tongue image, the determination unit 222 calculates the number of pixels with each grayscale value within its grayscale range. The determination unit 222 generates a grayscale histogram arranged in order of grayscale value magnitude, using each grayscale value as a level and the number of pixels with each grayscale value as a degree. The determination unit 222 scans the levels of the grayscale histogram from the lower grayscale value side to the higher grayscale value side, calculating a cumulative value obtained by accumulating the degree (number of pixels) for each level (grayscale value). The determination unit 222 determines a baseline value as the grayscale value where the proportion of the cumulative value to the total number of pixels in the saturated tongue image exceeds a predetermined proportion for the first time. The determination unit 222 generates a binary image by identifying pixels with gray values ​​less than a reference value as valid pixels and pixels with gray values ​​greater than the reference value as invalid pixels in the saturated tongue image. In the binary image, the determination unit 222 determines the area (number of pixels) of the valid pixels to represent the bruising state of the tongue's dorsal surface.

[0218] Figure 21 This is a schematic diagram illustrating an example of a binary image involved in this embodiment.

[0219] exist Figure 21Binary images 2100 and 2110 are shown. Binary image 2100 is generated from a tongue image obtained by photographing the dorsal side of the tongue of a person with good blood flow, while binary image 2110 is generated from a tongue image obtained by photographing the dorsal side of the tongue of a person with poor blood flow. In binary images 2100 and 2110, valid pixels are represented by white, and invalid pixels are represented by black. The tongue of the person with poor blood flow contains bruising, and the area of ​​the valid image in binary image 2110 is larger than the area of ​​the valid image in binary image 2100.

[0220] The determination unit 222 can also determine the bruising state of the tongue surface or the bruising state of the back of the tongue based on the number of edge pixels of a specific color component within the tongue image. For example, the determination unit 222 generates a first tongue image with the specific color component value as grayscale value based on a tongue image that is an RGB image. Alternatively, the determination unit 222 generates a second tongue image with the specific component value as grayscale value based on a tongue image converted from an RGB image to an HSV image. Alternatively, the determination unit 222 generates a third tongue image with the specific component value as grayscale value based on a tongue image converted from an RGB image to a Lab image. The determination unit 222 extracts pixels in the first, second, or third tongue image whose grayscale value difference with adjacent pixels in the horizontal, vertical, or diagonal direction is greater than or equal to a predetermined threshold as edge pixels. The determination unit 222 determines the number of extracted edge pixels as the bruising state of the tongue surface or the bruising state of the back of the tongue.

[0221] Alternatively, the determining unit 222 can also determine the state of bruising on the surface or the back of the tongue by analyzing the near-infrared spectral image. In this case, the photoelectric conversion element of the imaging device 104 is sensitive to near-infrared light, and the imaging device 104 includes an irradiator that illuminates near-infrared light. In step S101, the transmitting unit 121 illuminates the subject's tongue with near-infrared light while the irradiator is directed towards the subject's tongue, and the imaging device 104 captures an image of the subject's tongue, obtaining an image generated by the imaging device 104 as a tongue image.

[0222] The determination unit 222 generates a binary image in the tongue image by identifying pixels with brightness values ​​(density values) less than a predetermined threshold as valid pixels and pixels with brightness values ​​above the threshold as invalid pixels. The determination unit 222 performs shrinkage processing on the valid pixels in the binary image, calculates the number of shrinkage processes required until the valid pixels disappear, and uses this number as the thickness of the blood vessels, thus determining a state of congestion. Alternatively, the determination unit 222 calculates the average brightness value of pixels with brightness values ​​above the predetermined threshold as the concentration of the blood vessels, and also determines this as a state of congestion.

[0223] For example, server device 200 may pre-store in second storage device 210 a relational expression or table representing the relationship between the bruising state of the tongue surface or the bruising state of the back of the tongue and the blood state or blood flow state of the body. In this table, the bruising state of the tongue surface and / or the bruising state of the back of the tongue, and / or the blood state and / or blood flow state of the body, may also be classified using one or more thresholds. Determination unit 223, referring to the relational expression or table stored in second storage device 210, determines the blood state or blood flow state of the body corresponding to the bruising state of the tongue surface or the bruising state of the back of the tongue determined by determination unit 222, as the blood state or blood flow state of the target person's body. Determination unit 223 may also use a learning model to determine the blood state or blood flow state of the target person's body, the learning model being pre-learned to output the blood state or blood flow state of the person's body when the bruising state of the tongue surface or the bruising state of the back of the tongue of a specific person is input. The learning model uses combinations of bruising on the surface or back of the tongue of multiple individuals, along with the blood state or blood flow state of their bodies, as teacher data, and learns in advance through neural networks, support vector machines, etc. The determination unit 223 inputs the bruising on the surface or back of the tongue determined by the determination unit 222 into the learning model, and determines the blood state or blood flow state of the target individual's body from the information output by the learning model.

[0224] If the determination unit 223 determines that the health status of the subject is abnormal due to the presence of triglycerides or cholesterol in the blood, the cause of the determination is insufficient nutrition, stress, or lack of exercise.

[0225] Sometimes, due to malnutrition, stress, or other factors, blood flow can become impaired. Information processing system 1 can accurately determine a person's health status by observing the state of congestion on the surface or back of the tongue.

[0226] (Seventh Implementation)

[0227] In this embodiment, Figure 4 In step S104, the determination unit 222 determines the overall characteristics of the subject's tongue based on the tongue image. In step S106, the determination unit 223 determines the subject's health status based on the color characteristics of the subject's tongue. The processing in step S105 is omitted. The tongue image includes the entire tongue. The determination unit 222 determines the color characteristics of the tongue, especially the color balance characteristics, as the overall characteristics of the subject's tongue. The determination unit 223 determines the body's blood flow status as the subject's health status. For example, poor blood flow indicates mineral deficiency.

[0228] For example, the determination unit 222 determines the color balance features of the tongue through histogram analysis. Based on a tongue image that is an RGB image, the determination unit 222 generates a first tongue image with specific color component values ​​as grayscale values. Alternatively, based on a tongue image converted from an RGB image to an HSV image, the determination unit 222 generates a second tongue image with specific component values ​​as grayscale values. In the first or second tongue image, the determination unit 222 calculates the number of pixels with each grayscale value within its grayscale range. The determination unit 222 uses each grayscale value as a level and the number of pixels with each grayscale value as a degree, generating a grayscale histogram arranged in order of grayscale value magnitude. The determination unit 222 determines the grayscale value with the largest number of pixels as the degree in the grayscale histogram and identifies this grayscale value as the color balance feature of the tongue.

[0229] Figure 22 This is a schematic diagram representing an example of a grayscale histogram.

[0230] Gray-level histogram 2200 is a gray-level histogram generated from a tongue image obtained from a photograph of a person with good blood flow, and gray-level histogram 2210 is a gray-level histogram generated from a tongue image obtained from a photograph of a person with poor blood flow and high hemoglobin concentration. The horizontal axis of each gray-level histogram represents the gray-level value (level), and the vertical axis represents the number of pixels (degrees) with each gray-level value. The gray-level values ​​are arranged in ascending order. In gray-level histogram 2210, the gray-level value M2 with the largest number of pixels is smaller than the gray-level value M1 with the largest number of pixels in gray-level histogram 2200.

[0231] The determination unit 222 can also determine the color features of the tongue through template matching. For example, the server device 200 pre-stores sampled grayscale histograms in the second storage device 210. These sampled grayscale histograms are calculated based on multiple sampled images containing tongues with various colors. The determination unit 222 calculates the similarity between the generated grayscale histogram (an image representing the shape of the grayscale histogram) and each sampled grayscale histogram (an image representing the shape of each sampled grayscale histogram). If the calculated similarity is above a predetermined threshold, the determination unit 222 determines that the color of the tongue contained in the tongue image is the same as the color of the tongue contained in the sampled image that generated the sampled grayscale histogram. The determination unit 222 then defines the determined tongue color as a color balance feature.

[0232] For example, server device 200 pre-stores a relational expression or table representing the relationship between tongue color balance features and body blood flow status in second storage device 210. In this table, tongue color balance features and / or body blood flow status can also be classified using one or more thresholds. Determination unit 223 refers to the relational expression or table stored in second storage device 210 to determine the body blood flow status corresponding to the tongue color balance features determined by determination unit 222 as the blood flow status of the target person's body. Determination unit 223 can also use a learning model to determine the body blood flow status of the target person's body. This learning model is pre-learned to output the body blood flow status of a specific person when the tongue color balance features of that person are input. This learning model uses combinations of tongue color balance features and body blood flow status involving multiple people as teacher data and is pre-learned using neural networks, support vector machines, etc. Determination unit 223 inputs the tongue color balance features determined by determination unit 222 into the learning model and determines the body blood flow status of the target person from the information output by the learning model.

[0233] Information processing system 1 can accurately determine a person's health status by using tongue color balance.

[0234] (Eighth Implementation)

[0235] In this embodiment, Figure 4 In step S107, the output control unit 224 sends information indicating changes in the health status of the target person as information indicating the health status of the target person. In step S108, the display control unit 122 displays the information indicating changes in the health status of the target person on the first display device 102.

[0236] Figure 23 This is a schematic diagram illustrating an example of a display screen shown on a first display device.

[0237] exist Figure 23 Display screens 2300, 2310, and 2320 are shown. Display screen 2300 shows the change in the health status of a person whose health is improving; display screen 2310 shows the change in the health status of a person whose health is changing little; and display screen 2320 shows the change in the health status of a person whose health is deteriorating. Health status includes skin condition, vascular condition, antioxidant balance, nutritional status, and / or psychological stress, etc. A graph 2301 and an image 2302 are shown in each of the display screens 2300, 2310, and 2320.

[0238] The horizontal axis of graph 2301 represents time, and the vertical axis represents the health status at each time. By viewing graph 2301, users can confirm the changes in the health status of the subject over time.

[0239] Image 2302 is an image representing a change in health status. That is, image 2302 does not indicate whether the current health status is good or bad, but rather whether the health status is improving, unchanged, or deteriorating. Image 2302 shows an image of a tree. Image 2302 could also show any other image such as grass or flowers. In screen 2300, which displays the health status of an object whose health is improving, image 2302 shows a tree in bloom. Image 2302 could also be represented in such a way that the greater the improvement in health, the more numerous, larger, or more vibrant the flowers on the tree. In screen 2310, which displays the health status of an object whose health is changing little, image 2302 shows a tree with abundant leaves but no flowers. In screen 2320, which displays the health status of an object whose health is deteriorating, image 2302 shows a tree withering. Image 2302 can also be represented by the degree of deterioration in health, which corresponds to shorter, thinner, or darker branches. By viewing image 2302, users can intuitively identify changes in the health status of the subject and predict future changes in their health.

[0240] Even with good current nutritional status but high psychological stress, nutritional status may decline in the future. Furthermore, while changes in physical condition may not be readily apparent when antioxidant balance is poor, prolonged periods of poor antioxidant balance can alter sensations such as taste, and glycation may intensify imperceptibly. Therefore, Information Processing System 1 can also display a radar chart representing the balance between current nutritional status, antioxidant balance, and psychological stress. Additionally, Information Processing System 1 can also display a radar chart representing changes in nutritional status, antioxidant balance, and psychological stress.

[0241] Alternatively, the information processing system 1 may display an image representing the current health status based on or in place of the image 2302.

[0242] Furthermore, Information Processing System 1 can also display suggestions for improving the health status of the target individual. For example, when antioxidant balance (oxidative stress) is poor, Information Processing System 1 can inquire about the target individual's snacking or dining out habits and promote improvement based on their situation. Additionally, when psychological stress and oxidative stress are poor, Information Processing System 1 can focus on alleviating psychological stress rather than making dietary changes to promote improvement. Furthermore, when psychological stress and nutritional status are poor, Information Processing System 1 can promote improvements in psychological stress and sleep, thereby prioritizing bodily repair.

[0243] It can also be for Figure 4 In step S107, the output control unit 224 sends information indicating the nutritional status of the target person as information indicating the health status of the target person. In step S108, the display control unit 122 displays the information indicating the nutritional status of the target person on the first display device 102.

[0244] Figure 24 This is a schematic diagram illustrating an example of a display screen shown on a first display device.

[0245] exist Figure 24 Display screen 2400 is shown. Image 2401 is shown on display screen 2400. Image 2401 is an image of a character representing the nutritional status of an object. Image 2401 shows an image of a tree. Image 2401 could also show other arbitrary images such as grass or flowers. In image 2401, the leaves are divided into regions according to nutritional components such as minerals, proteins, vitamin A, vitamin B, and vitamin C. Each nutrient region can also display images of representative foods representing that nutrient (e.g., beef for protein, lemon for vitamin C). For example, the more abundant the nutrient, the more vibrant the color of that nutrient region; the less abundant the nutrient, the darker the color of that nutrient region. Alternatively, the more abundant the nutrient, the larger the food appears; the less abundant the nutrient, the smaller the food appears.

[0246] While a high level of carotene leads to higher vitamin A levels, an imbalance in vitamins or minerals will not improve overall health. Mineral deficiencies can result in muscle loss and general weakness. Even with a balanced diet, psychological stress and other factors can prevent the body from obtaining nutrients in an unbalanced way. Since the tongue reflects a person's nutritional status, Information Processing System 1 uses this information to determine the state of nutrients absorbed by the body with high precision.

[0247] (Ninth Implementation)

[0248] In this embodiment, the terminal device 100 is a tablet PC or a multi-functional mobile phone, etc. The camera device 104 includes a front-facing camera and a rear-facing camera, with the rear-facing camera having higher performance than the front-facing camera. Additionally, the terminal device 100 includes a voice output device and a vibration sensor, etc. The voice output device is a speaker, etc. The vibration sensor is an accelerometer, a displacement sensor, etc. Figure 4 In step S101, the sending unit 121 uses a rear-facing camera to acquire an image of the tongue. However, since the rear-facing camera is configured to capture images of the rear side of the terminal device 100, i.e., the side where the first input device 101 and the first display device 102 are not located, it is difficult for the subject to capture an image of their own tongue.

[0249] Figure 25 This is an illustration showing a person taking a picture of their own tongue.

[0250] like Figure 25 As shown, in this embodiment, the subject P uses a rear-facing camera to photograph his / her tongue. At this time, since the rear-facing camera C of the terminal device 100 is facing the subject P, the first input device 101 and the first display device 102 are facing the opposite side of the subject P.

[0251] The transmitting unit 121 outputs shooting instruction information from the first display device 102 or the voice output device. The shooting instruction information is used to instruct the subject P to use the rear shooting device to shoot the tongue, to take a picture in front of the mirror M, and to turn the rear shooting device toward the subject P and turn the first display device 102 toward the mirror M.

[0252] When the first input device 101 or the vibration sensor detects a zoom operation such as double-clicking or triple-clicking on the first input device 101 (screen) or the casing (back) of the terminal device 100 by a user, it outputs a zoom operation signal to the first processing device 120. Upon receiving the zoom operation signal, the transmitting unit 121 adjusts the shooting range of the rear camera device to capture the tongue at an appropriate size. The transmitting unit 121 uses well-known image processing technology to detect the tongue from the image captured by the rear camera device, and then zooms in or out of the rear camera device to capture the detected tongue at an appropriate size. When the first input device 101 or the vibration sensor detects a shooting operation such as clicking on the first input device 101 (screen) or the casing (back) of the terminal device 100 by a user, it outputs a shooting operation signal to the first processing device 120. Upon receiving the shooting operation signal, the transmitting unit 121 causes the rear camera device to capture the tongue. When the transmitting unit 121 takes a picture of the side of the tongue, it outputs orientation instruction information from the first display device 102 or the voice output device. The orientation instruction information is used to instruct the person P to face the terminal device 100.

[0253] Alternatively, the terminal device 100 can also use a front-facing camera to photograph the tongue of a subject. When using a front-facing camera to photograph the tongue of a subject, the transmitting unit 121 does not output the aforementioned photographing instruction information.

[0254] Therefore, the terminal device 100 can acquire high-quality images in the user's own home, allowing the user to accurately and easily identify daily changes in their internal state. Furthermore, when operating the terminal device 100, if it is positioned next to the user's face, a mirror cannot be observed. When the user moves the terminal device 100 away from their face to observe the mirror, the size of the tongue in the generated input image decreases. The user can easily control the zoom of the imaging device 104 by tapping the screen or the back, thus enabling operation of the terminal device 100 away from the face. Therefore, the terminal device 100 can acquire high-quality images while improving user operability.

[0255] The preferred embodiments have been described above, but the embodiments are not limited to these. For example, instead of the server device 200, the terminal device 100 may determine the health status of the person. In this case, the first processing unit 120 of the terminal device 100 functions as a sending unit 121 and a display control unit 122, and also functions as a determination unit and a determination unit having the same functions as the determination unit 222 and determination unit 223 of the server device 200. The determination unit and determination unit of the terminal device 100 perform... Figure 4 The timing diagram shown illustrates the processing steps S104 to S106. In this case, the terminal device 100 is an example of an information processing device, the first display device 102 is an example of an output unit, and the transmission unit 121 is an example of an acquisition unit.

[0256] exist Figure 4 In the timing diagram of the determination process, steps S102 to S103 are omitted. In step S104, the determination unit determines the nipple features or other features of the subject's tongue based on the tongue image obtained by the transmission unit 121. Furthermore, step S107 is omitted. In step S108, the display control unit 122 outputs information indicating the health status of the subject determined by the determination unit on the first display device 102. In this case, the information processing system 1 can output the subject's health status with higher accuracy.

[0257] In addition, in the information processing system 1, multiple terminal devices 100 and / or multiple server devices 200 may cooperate to share the various steps of the above-mentioned processing.

[0258] Furthermore, the above-described embodiments can be arbitrarily combined and implemented simultaneously. The information processing system 1 can also determine the health status of the subject by executing multiple methods from the first to seventh embodiments on the same tongue image.

Claims

1. An information processing device, characterized in that, have: The acquisition section acquires an image of a tongue, including any one of the following: the papillae of the tongue, the surface or dorsal features of the tongue, or the tongue as a whole; and The output unit outputs information representing the health status of the person based on the tongue image.

2. The information processing device according to claim 1, characterized in that, The output unit outputs information representing the character's skin condition as information representing the character's health status.

3. The information processing device according to claim 2, characterized in that, The skin condition is any one or more of the following: degree of wrinkles, degree of pores, degree of texture, and degree of spots.

4. The information processing apparatus according to claim 1, characterized in that, The output unit outputs information representing the person's vascular status and / or blood flow status, information representing antioxidant balance, information representing the body's nutritional status, information representing psychological stress, and information representing metabolic abnormalities, as information representing the person's health status.

5. The information processing apparatus according to claim 4, characterized in that, The vascular state refers to blood flow distribution or vascular flexibility, and the blood flow state refers to good blood flow.

6. The information processing apparatus according to claim 4, characterized in that, The body's nutritional status refers to the quantity or deficiency of vitamins, minerals, and protein, or the degree of lipid metabolism.

7. The information processing apparatus according to claim 1 or 2, characterized in that, It also has: The determining unit, based on the tongue image, determines the characteristics of any one of the following: the papillae of the tongue, the surface and dorsal features of the tongue, and the overall shape of the tongue; and The determination unit determines the health status of the person based on the aforementioned characteristics.

8. The information processing apparatus according to claim 7, characterized in that, The determination unit uses the characteristics of the nipples to determine the skin condition of the person as a measure of the person's health status.

9. The information processing apparatus according to claim 7, characterized in that, The determination unit determines the blood vessel status of the person based on the characteristics of the nipple, and determines the skin status of the person based on the blood vessel status to determine the person's health status.

10. The information processing apparatus according to claim 7, characterized in that, The determining unit determines the shape, size, or regularity of the nipple as a characteristic of the nipple.

11. The information processing apparatus according to claim 7, characterized in that, The acquisition unit acquires multiple images of the tongue taken at different time intervals. The determining unit determines the changes in the nipple as a feature of the nipple.

12. The information processing apparatus according to claim 7, characterized in that, The defining part defines the characteristics of the nipple for the tip and the inner part of the tongue, respectively. The determination unit determines the person's health status based on the characteristics of the nipples, which are separately identified at the tip and the inner part of the tongue.

13. The information processing apparatus according to claim 7, characterized in that, The acquisition unit also acquires the individual's self-awareness of their physical condition or environmental information. The determination unit also determines the person's health status based on the person's self-perceived physical condition or environmental information.

14. The information processing apparatus according to claim 7, characterized in that, The determination unit determines one or more of the individual's antioxidant balance and nutritional status based on the characteristics of the fungal nipples in the nipples.

15. The information processing apparatus according to claim 7, characterized in that, The determining unit determines one or more of the shape and color features of the fungal papillae in the papillae as features of the fungal papillae.

16. The information processing apparatus according to claim 7, characterized in that, The determination unit determines one or more of the following based on the amount of tongue coating on the surface of the tongue, the state of bruising on the surface, and the state of bruising on the back of the tongue: the nutritional status, blood flow status, and blood status of the person's body.

17. The information processing apparatus according to claim 7, characterized in that, The determining unit determines one or more of the following: the amount of tongue coating on the surface of the tongue, the state of bruising on the surface, and the state of bruising on the back of the tongue.

18. The information processing apparatus according to claim 7, characterized in that, The determination unit, based on the tongue as a whole, determines one or more of the following: the body's nutritional status, psychological stress, and metabolic abnormalities.

19. The information processing apparatus according to claim 7, characterized in that, The determining unit determines one or more of the following features as the overall characteristics of the tongue: color balance, cracking, thickness, and teeth marks.

20. The information processing apparatus according to claim 7, characterized in that, The determining unit determines any one of the following features of the person's tongue: the papillae, the surface and dorsal features of the tongue, and the overall shape of the tongue, through template matching, histogram analysis, near-infrared spectral image analysis, object shape analysis based on the image, texture analysis, and machine learning-based image analysis.

21. The information processing apparatus according to claim 7, characterized in that, The output unit outputs the factors of the character's health status as determined by the determination unit.

22. An information processing system comprising a terminal device and an information processing device, characterized in that, The terminal device has a transmitting unit that transmits to the information processing device an image of a tongue including any one of the following: the papillae of a person's tongue, the surface and dorsal features of the tongue, or the tongue as a whole. The information processing device has: A receiving unit that receives the tongue image from the terminal device; The output unit outputs information representing the health status of the person based on the tongue image.

23. The information processing system according to claim 22, characterized in that, The terminal device also has a camera unit for generating the image of the tongue.

24. An information processing method, characterized in that, Information processing device Obtain a tongue image that includes any one of the following: the papillae of the tongue, the surface and dorsal features of the tongue, or the tongue as a whole. Output information representing the health status of the person based on the tongue image.

25. A control program for an information processing device, characterized in that, The control program causes the information processing device to perform the following processing: Obtain an image of a tongue that includes any one of the following: the papillae of the tongue, the surface or back of the tongue, or the entire tongue. Output information representing the health status of the person based on the tongue image.

Images