Nutrient determination system, nutrient determination device, nutrient determination method, and nutrient determination program

Abstract

A nutrient determination system (1) comprises an image capture device (51), a nutrient determination device (10) that outputs a determination result (D4) of nutrients in a meal, and an output device (52) that presents the determination result to a user. The nutrient determination device (10) comprises: a food area determination unit (11) that determines whether each of pixels constituting a meal image (D0) is a food pixel, and generates a food area image (D1); a channel conversion unit (12) that converts an image signal of each pixel of an image including the food area image (D1) into an HSV signal to thereby generate an HSV image (D2); a nutrient conversion unit (13) that, with reference to data in a nutrient / color DB (15), makes a nutrient determination for each pixel of the HSV image (D2) in a food area, and generates a nutrient image (D3); and a nutrient determination unit (14) that performs a nutrient determination on the meal on the basis of pixel-by-pixel nutrients, and outputs a determination result (D4).

Description

Technical Field

[0001] The present disclosure relates to a nutrient determination system, a nutrient determination device, a nutrient determination method, and a nutrient determination program.

Background Art

[0002] For example, Patent Document 1 proposes an apparatus that extracts tableware from an image of a meal, detects the hue of food for each tableware (in this document, R, G, B), measures the number of pixels for each hue, calculates the amount of each food according to food data stored in advance the correspondence between the ingredients and the hue, and measures the food intake amount from the image of the meal before eating and the image of the meal after eating. Further, this apparatus calculates the intake amount of nutrients by storing nutrient data for each food.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above conventional apparatus, the amount of each food is calculated using a food database that stores the correspondence between the food and the hue, and the nutrients are calculated using data indicating the nutrients for each food. Since the calculation is performed multiple times, there is a problem that the accuracy of the analysis result is low.

[0005] An object of the present disclosure is to solve the above conventional problems and to enable high-precision determination of the nutrients in a meal by simply photographing the meal.

Means for Solving the Problems

[0006] The nutrient determination system according to the present disclosure includes an imaging device, a nutrient determination device that determines the nutrients of a meal based on a meal image that is an image captured by the imaging device or a part of the captured image, and outputs a determination result obtained by the determination, and an output device that outputs the determination result. The nutrient determination device includes a food area determination unit that determines whether each pixel constituting the meal image is a pixel of food and generates a food area image that is an image of a food area composed of the pixels of the food, a channel conversion unit that generates an HSV image by converting the image signal of each pixel of an image including at least the food area image into an HSV signal, a nutrient conversion unit that refers to data in a nutrient-color database indicating the relationship between nutrients and values of hue, saturation, and lightness in the HSV color space, determines the nutrients for each pixel of the HSV image of the food area, and generates a nutrient image, and a nutrient determination unit that determines the nutrients of the meal based on the nutrients for each pixel and outputs the determination result. Then, the nutrient conversion unit refers to the color information of protein, mineral, carotene, vitamin, carbohydrate, and lipid as nutrients stored in the nutrient and color database, and the color information defines the values within the range that each can take for at least two or more items among the items of hue, saturation, and lightness respectively. characterized in that.

[0007] The nutrient determination method according to the present disclosure is implemented by a nutrient determination device that determines the nutrients of a meal based on a meal image that is an image captured by an imaging device or a part of the captured image, and outputs a determination result obtained by the determination. Nutrient determination The method includes a step of determining whether each pixel constituting the meal image is a pixel of food and generating a food area image that is an image of a food area composed of the pixels of the food, a step of generating an HSV image by converting the image signal of each pixel of an image including at least the food area image into an HSV signal, and nutrients In each of protein, mineral, carotene, vitamin, carbohydrate, and lipid which are such nutrients, hue, saturation, and lightness in the HSV color space the values within the range that each can take are defined for at least two or more items among each of the items. a step of referring to data in a nutrient-color database, determining the nutrients for each pixel of the HSV image of the food area, and generating a nutrient image, and a step of determining the nutrients of the meal based on the nutrients for each pixel and outputting the determination result, characterized in that.

Advantages of the Invention

[0008] According to the present disclosure, it is possible to determine the nutrients of a meal with high accuracy just by photographing the meal.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Modes for Carrying Out the Invention

[0010] Hereinafter, a nutrient determination system, a nutrient determination device, a nutrient determination method, and a nutrient determination program according to an embodiment will be described with reference to the drawings. The nutrient determination system is a system that can output information on nutrients contained in a meal just by photographing the meal. Further, by presenting the information on nutrients contained in the meal output by the nutrient determination system to the user, the user can determine the menu of the dish or adjust the serving amount. Note that the “meal” handled by the nutrient determination system may be read as “ingredient”, that is, not limited to the meal after cooking, and the nutrients contained in the ingredients during or before cooking may be determined and output. The nutrient determination system may be a mobile terminal such as a smartphone, a tablet terminal, a personal computer (PC) that can acquire an image photographed by an imaging device, a server computer on a network that can acquire an image photographed by an imaging device, and the like. Further, the nutrient determination system may be a combination of a mobile terminal having a camera and a PC or a server computer that can communicate with the mobile terminal. Note that the following embodiments are merely examples, and it is possible to appropriately combine the embodiments and appropriately change each embodiment.

[0011] 《1》Embodiment 1. 《1-1》Configuration FIG. 1 is a block diagram schematically showing the configuration of the nutrient determination system 1 and the nutrient determination device 10 according to Embodiment 1. The nutrient determination system 1 includes a nutrient determination device 10, an imaging device 51 that captures an image of a meal, and an output device 52 that presents the determination result of the nutrient to the user. The imaging device 51 is a camera that captures a still image or a moving image. The output device 52 is, for example, a video display device, an audio output device, a printing device, one of these, or a combination of two or more of them.

[0012] The nutrient determination device 10 is a device (for example, a computer) that can implement the nutrient determination method according to Embodiment 1. The nutrient determination device 10 determines the nutrients of the meal based on the meal image D0 that is the image captured by the imaging device 51 or a part of the captured image, and outputs the determination result D4 obtained by this determination. The nutrient determination device 10 includes a food region determination unit 11, a channel conversion unit 12, a nutrient conversion unit 13, and a nutrient determination unit 14. The nutrient determination device 10 may have a nutrient-color database (that is, a nutrient-color DB) 15 composed of nutrient-color data showing the relationship between nutrients and colors (the values of hue H, saturation S, and lightness V in the HSV color space). The nutrient-color DB 15 may be a storage device of an external device (for example, a cloud server) different from the nutrient determination device 10. The imaging device 51 captures an image of the meal that is the object of the nutrient determination and outputs a meal image D0 that is a color image. The meal image D0 provided to the food region determination unit 11 may be an image recorded in a recording device (not shown).

[0013] The food area determination unit 11 acquires the meal image D0, and for each of the plurality of pixels constituting the meal image D0, determines whether it is a pixel of food as the subject, thereby generating a food area image D1 that is an image of the food area composed of the pixels of the food. As a method for determining whether a pixel constituting the meal image D0 is a pixel of food, for example, there are a method of determining based on input information by user operation, or a method of determining based on the sensor output of a separately installed thermal image sensor or distance sensor. It is also possible to use a method of recognizing the food area from the meal image D0 by processing using AI (artificial intelligence).

[0014] The channel conversion unit 12 generates the HSV image D2 by converting the image signal of each pixel of the image including at least the food area image D1 into an HSV signal. Specifically, the channel conversion unit 12 converts the image signal (for example, RGB signal or YCbCr signal) of the food area image D1 generated by the food area determination unit 11 into an HSV signal, and outputs the HSV image D2 of the food area. HSV refers to the hue (H) representing the color tone, the saturation (S) representing the vividness, and the value (V) representing the brightness. A certain color can be expressed by three values of H, S, and V. The image signal of the meal image D0 is often expressed by an RGB signal or a YCbCr signal or the like. The conversion between RGB and HSV and the conversion between RGB and YCbCr can be performed using known conversion formulas.

[0015] The nutrient conversion unit 13 refers to the data of the nutrient-color DB15 showing the relationship between nutrients and the values of hue, saturation, and brightness in the HSV color space, and determines the nutrients for each pixel of the HSV image D2 of the food area output from the channel conversion unit 12 to generate a nutrient image D3. When determining the nutrients for each pixel, the nutrient conversion unit 13 refers to, for example, the nutrient-color DB15. As classifications of nutrients contained in foods (that is, items that become food), "the five major nutrients", "the three-color food group", "the six basic food groups", etc. are known.

[0016] "The five major nutrients" represent five types: carbohydrates (sugars), lipids, proteins, inorganic substances (minerals), and vitamins.

[0017] The "Three-color Food Groups" divides foods into three food groups, namely, the food groups of (red), (yellow), and (green) colors, according to the functions or characteristics of nutrients in the human body. The "Three-color Food Groups" consists of nutrients that form the (red) body (e.g., meat, fish, eggs, milk and dairy products, beans, etc.), nutrients that are the source of (yellow) energy (e.g., rice, bread, noodles, tubers, oil, sugar, etc.), and nutrients that regulate the condition of the (green) body (e.g., vegetables, fruits, mushrooms, etc.).

[0018] The "Six Basic Food Groups" classifies nutrients with similar functions into six groups (Group 1 - Group 6) by food. "Group 1 (red)" is a group of nutrients (protein) that build bones and muscles, and includes, for example, meat, fish, eggs, etc. "Group 2 (red)" is a group of nutrients (calcium) that build bones, teeth, etc., and includes, for example, milk, dairy products, seaweeds, small fish, etc. "Group 3 (green)" is a group of nutrients (carotene) that protect the skin and mucous membranes, and includes, for example, green and yellow vegetables. "Group 4 (green)" is a group of nutrients (vitamin C) that regulate the body's condition, and includes, for example, light-colored vegetables, fruits. "Group 5 (yellow)" is a group of nutrients (carbohydrates) that provide energy, and includes, for example, carbohydrates, grains, tubers. "Group 6 (yellow)" is a group of nutrients (lipids) that provide energy, and includes, for example, oils and fats, meat rich in fat, etc. In Embodiment 1, the case of treating the "Six Basic Food Groups" as nutrients will be described.

[0019] The nutrient determination unit 14 determines (i.e., analyzes) the nutrients in the meal from the nutrient image D3 of the food area output from the nutrient conversion unit 13, and outputs a determination result D4. The determination result D4 includes, for example, the amount of each type of nutrient in the target meal. The output device 52 outputs information indicating the determination result D4 in the form of video, audio, printed matter, one of these, or a combination of two or more of them.

[0020] FIG. 2 is a diagram showing an example of the hardware configuration of the nutrient determination device 10. The nutrient determination device 10 includes a processor 101, a memory 102, a storage unit 103 which is a non-volatile storage device, an interface 104, a communication unit 105, and an input unit 106. The processor 101 is, for example, a CPU (Central Processing Unit). The memory 102 is a volatile semiconductor memory such as, for example, a RAM (Random Access Memory). The storage unit 103 is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage unit 103 stores information and programs. The communication unit 105 communicates with other devices via a network. The input unit 106 is an operation unit such as a keyboard which is an input interface. The input unit 106 receives input from a person. Devices such as an imaging device 51 and an output device 52 are connected to the interface 104.

[0021] Each function of the nutrient determination device 10 is realized by a processing circuit. The processing circuit may be dedicated hardware or may be the processor 101 that executes a program stored in the memory 102. The processor 101 may be any of a processing device, an arithmetic device, a microprocessor, a microcomputer, and a DSP (Digital Signal Processor).

[0022] When the processing circuit is dedicated hardware, the processing circuit is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of any of these.

[0023] When the processing circuit is the processor 101, the nutrient determination program according to Embodiment 1 is realized by software, firmware, or a combination of software and firmware. The software and firmware are described as programs and stored in the memory 102. The processor 101 can realize the functions of each part shown in FIG. 1 by reading and executing the nutrient determination program stored in the memory 102. The nutrient determination program is installed in the nutrient determination device 10 by downloading via a network or from a recording medium (i.e., a storage medium) that records information such as an optical disk. Note that part of the nutrient determination device 10 may be realized by dedicated hardware and part may be realized by software or firmware. In this way, the processing circuit can realize the functions of each functional block shown in FIG. 1 by hardware, software, firmware, or any combination thereof.

[0024] 《1-2》Operation FIG. 3 is a flowchart showing the operation of the nutrient determination device 10 according to Embodiment 1. First, a meal image D0 obtained by photographing a meal is input to the food area determination unit 11. In step ST11 of FIG. 3, the food area determination unit 11 acquires the meal image D0, determines whether each pixel constituting the meal image D0 is a pixel of food, and generates a food area image D1 that is an image of a food area composed of pixels of food.

[0025] Figs. 4(A) and (B) are diagrams showing examples of input / output images of the food area determination unit 11. Fig. 4(A) shows a meal image D0 input to the food area determination unit 11, and Fig. 4(B) shows a food area image D1 output from the food area determination unit 11. The food area image D1 is an image of a food area that is an area excluding tableware or the background in the photographed meal. As for the determination as to whether each pixel constituting the meal image D0 is a food pixel, there are a determination based on an area input by the user (for example, an operation of designating an area on a touch panel on which a photographed image is displayed), a determination based on sensor information by the meal image D0 or a separately set sensor (for example, a thermal image sensor or a distance sensor), and the like.

[0026] When using the user input, when the user performs an operation of selecting one food pixel (for example, an operation of touching a point on the food area image D1 displayed on the touch panel), by performing a process of assuming that areas with the same or similar color gamut or texture are the same food, it is possible to reduce the workload and information processing amount of the user related to the determination as to whether it is a food area.

[0027] Also, when using a thermal image sensor as a separately installed sensor, from the temperature information acquired by the thermal image sensor, the temperature occupying most of the image is regarded as the indoor temperature, and pixels with an absolute value of the difference from the indoor temperature being a predetermined value or more are determined as food. For example, when the room temperature is 25°C and the predetermined value is set to +5°C, for example, pixels of 30°C or more are determined as pixels of food (for example, warm drinks, warm food cooked by fire, etc.). Also, when the room temperature is 25°C and the predetermined value is set to -5°C, for example, pixels of 20°C or less are determined as pixels of food (for example, cold drinks, salads, fruits, etc.). Also, when using a distance sensor as a separately installed sensor, taking advantage of the fact that the surface of the food ingredients has more or larger irregularities than the table and tableware, etc., if the difference between the distance indicated by a certain pixel and the distances indicated by its surrounding pixels is equal to or greater than a predetermined threshold value (for example, 5 mm), that is, pixels with large irregularities on the surface of the object being photographed are determined as food.

[0028] In addition, when recognizing a food area from a meal image D0 by means of processing using AI (artificial intelligence), the recognition accuracy of the food area can be improved.

[0029] In step ST12 of FIG. 3, the channel conversion unit 12 generates an HSV image D2, which is an image of the food area after conversion, by converting the image signal of the food area image D1 generated in step ST11 into an HSV signal.

[0030] FIGS. 5(A) to (C) are diagrams showing the formats of the images handled by the channel conversion unit 12 and the nutrient conversion unit 13. FIG. 5(A) shows the RGB signal of the food area image D1, and this RGB signal is input to the channel conversion unit 12. FIG. 5(B) shows the HSV image D2 of the food area, and the HSV image D2 is output from the channel conversion unit 12. FIG. 5(C) shows the nutrient image D3 of the food area, which is a column of numbers (any of the numbers 1, 2, 3, 4, 5, 6) indicating nutrient numbers. Regarding the nutrient numbers in FIG. 5(C), 1 indicates "Group 1: Protein", 2 indicates "Group 2: Mineral", 3 indicates "Group 3: Carotene", 4 indicates "Group 4: Vitamin", 5 indicates "Group 5: Carbohydrate", and 6 indicates "Group 6: Lipid".

[0031] In step ST13 of FIG. 3, the nutrient conversion unit 13 determines nutrients for each pixel of the HSV image D2 of the food area based on the data of the nutrient - color DB15. The nutrient conversion unit 13 generates a nutrient image composed of a column of nutrient numbers.

[0032] FIG. 6 is a diagram showing an example of a nutrient-color DB 15 used by the nutrient determination device 10. The nutrient-color DB 15 defines the hue (H), saturation (S), and lightness (V) characteristics or the range of possible values for each nutrient. Note that the range of possible values for the hue, saturation, and lightness shown in FIG. 6 is an example, and it is also possible to set other value ranges. In the nutrient-color DB 15, for a group of proteins, the lightness is red or yellow, and the saturation is medium saturation. Also, for a group of minerals, the lightness is low lightness. Also, for a group of carotenoids, the hue is red or green, and the saturation is medium-high saturation. Also, for a group of vitamins, the hue is green, and the lightness is high lightness. Also, for a group of carbohydrates, the hue is yellow, and the lightness is high lightness. Also, for a group of lipids, the hue is yellow, and the saturation is low saturation.

[0033] The nutrient-color DB 15 shown in FIG. 6 shows the hue, saturation, and lightness for each nutrient in a range. By referring to the nutrient-color DB 15, it is possible to determine the nutrient indicated by each pixel of the HSV image D2 of the food area from the HSV values of the pixels. Note that the format of the nutrient-color DB 15 is not limited to that in FIG. 6. For example, the nutrient-color DB 15 may define the median values of the hue, saturation, and lightness for each nutrient. In this case, the nutrient conversion unit 13 determines the nutrient with the smallest difference from the median values of the hue, saturation, and lightness of each nutrient defined in the nutrient-color DB 15 as the corresponding nutrient.

[0034] As described above, food is characterized by hue, saturation, and lightness for each nutrient, and it is possible to directly determine the nutrient from the hue, saturation, and lightness. For example, it is possible to incorporate dark green vegetables to ingest carotene or reduce the amount of light-colored carbohydrates. However, when the image signal of a red tomato is represented by an RGB signal, (R, G, B) = (158, 20, 17), and when the image signal of white rice is represented by an RGB signal, (R, G, B) = (246, 242, 233). Thus, if the food or nutrient is judged by the RGB signal, the R value of white rice is higher than the R value of the tomato. Thus, it is not appropriate to directly judge the food or nutrient from the RGB signal. Therefore, in the present disclosure, an algorithm for judging nutrients based on the hue and again the lightness of the HSV image composed of the HSV signal is adopted.

[0035] Also, it is desirable to judge the nutrient in the nutrient conversion unit 13 for all the pixels in the food region, but it is also possible to judge for some of the pixels in the food region. For example, the nutrient conversion unit 13 may judge the nutrient for each region where each signal of H, S, and V is within a predetermined range, and treat this judgment result as the judgment result of the pixels in the region. The predetermined range is, for example, when the hue H is within a predetermined range of 5°, the saturation S is within a predetermined range of 5%, and the lightness V is within a predetermined range of 5%, the judgment results of the pixels in the region within this range may be set to the same value. Note that since the hue H takes a value from 0° to 360° and H = 0° and H = 360° are the same, when judging the hue range near H = 0°, the process may be performed with the value obtained by adding 360° to the value of H.

[0036] In step ST14 of FIG. 3, the nutrient judgment unit 14 acquires the nutrient image D3 from the nutrient conversion unit 13, performs statistical processing on the nutrient for each pixel of the nutrient image D3 to judge (i.e., calculate) the nutrient of the meal, and outputs the judgment result D4. The judgment result D4 is provided to the output device 52 and presented to the user.

[0037] Figs. 7(A) and (B) are diagrams showing examples of determination result D4 output by nutrient determination unit 14 of nutrient determination apparatus 10. Fig. 7(A) is a pie chart showing the percentage [%] of nutrients contained in a meal. Nutrient determination unit 14 generates a histogram of nutrient image D3 and converts the percentage of each nutrient with respect to all pixels of nutrient image D3 into a pie chart. By presenting such a pie chart to the user, the user can intuitively grasp the percentage of nutrients.

[0038] Further, Fig. 7(B) is a radar chart showing the nutritional balance of the target meal with respect to the ideal nutritional balance (100%). Fig. 7(B) shows the deviation of the nutritional balance of the target meal with respect to the ideal balance of nutrients required for health as a hexagonal graph (i.e., a spider graph). By presenting such a radar chart to the user, the user can grasp the deviation of nutrients at a glance.

[0039] 《1-3》Effect As described above, according to Embodiment 1, since the nutrients for each pixel are directly determined from meal image D0, compared with the case where the nutrients are determined using a database of data showing the nutrients for each ingredient after determining the ingredients from the meal image (i.e., when two types of determination processes are performed), the nutrients can be determined with high accuracy with a small amount of computation.

[0040] According to Embodiment 1, a photograph of a prepared dish before a meal can be taken with a camera mounted on a mobile terminal or the like, and the nutrients contained in the dish can be confirmed on the spot and stored in a storage device as a nutrition management record. Since such a nutrient determination system 1 can be carried around, if nutrient determination is performed regardless of the location of the meal, records of all meals can be left. In this case, there is no need for the user to perform operations such as recording the meal content by hand or inputting it from a keyboard, and it is possible to perform daily nutrition management simply by taking a photo. Also, according to Embodiment 1, when a physical discomfort occurs, it is also possible to obtain the effect that the record of the ingested nutrients can contribute to identifying the cause of the physical discomfort.

[0041] Also, according to Embodiment 1, in a school, nursery, hospital, or elderly housing, etc., the user can easily take a photo of the dish before the meal and record the determination result of the nutrients. Such a nutrient determination system 1 can easily record the meal content and the ingested nutrients even when the amount of nutrients ingested or the meal menu differs according to the growth or physical condition of each individual.

[0042] Also, according to Embodiment 1, the state after the meal (that is, the remaining food) can be photographed to confirm the types and amounts of nutrients that were not ingested. Also, by taking the difference from the nutrition determination result using the photo of the dish before the meal and the photo after the meal, there is an effect that the ingested nutrients can be calculated.

[0043] Also, according to Embodiment 1, the user can also use the nutrient determination system 1 in such a way that when considering a menu, the user inputs the completed image of the recipe to confirm the nutrients and decides on a change or arrangement of the menu. In such a nutrient determination system 1, since the user can confirm the nutrients of the meal or ingredients based on the image before cooking, there is an effect that a well-balanced menu can be created without specialized knowledge about the nutrition of each ingredient.

[0044] 《2》Embodiment 2. 《2-1》Configuration FIG. 8 is a block diagram schematically showing the configuration of the nutrient determination system 2 and the nutrient determination device 20 according to Embodiment 2. In FIG. 8, the same or corresponding components as those shown in FIG. 1 are denoted by the same reference numerals as those shown in FIG. 1. The nutrient determination system 2 includes a nutrient determination device 20, an imaging device 51, and an output device 52. The nutrient determination device 20 is a device (for example, a computer) capable of implementing the nutrient determination method according to Embodiment 2. The nutrient determination device 20 includes a food region determination unit 11, a channel conversion unit 22, a nutrient conversion unit 23, and a nutrient determination unit 14.

[0045] The nutrient determination device 20 may have a nutrient-color DB 15. The nutrient-color DB 15 may be a storage device of an external device different from the nutrient determination device 20. The nutrient determination device 20 according to Embodiment 2 differs from the nutrient determination device 10 according to Embodiment 1 in that instead of channel-converting the food region image D1 obtained by determining the food region from the meal image D0, the meal image D0 is channel-converted (that is, the generation of the food region image D1 and the channel conversion are performed in parallel), and in that a nutrient image of the target nutrient input to the nutrient conversion unit 23 from the outside is generated.

[0046] The food region determination unit 11 acquires the meal image D0 and generates a food region image D1. The channel conversion unit 22 acquires the meal image D0 and generates an HSV image D22 of the meal image D0 by converting the image signal of the meal image D0 into an HSV signal.

[0047] The nutrient conversion unit 23 acquires the food area image D1 from the food area determination unit 11, acquires the HSV image D22 of the meal image D0 from the channel conversion unit 22, acquires the target nutrient from the outside (for example, from the user operation unit), and refers to the data in the nutrient - color DB15 to generate a nutrient image D23 of the target nutrient for each pixel of the HSV image of the area within the food area image D1 in the HSV image D22. The target nutrient is, for example, one or more group numbers of the "six basic food groups" are input. If there is no input of the target nutrient, all nutrients may be regarded as the target nutrient to generate the nutrient image D23.

[0048] The nutrient determination unit 14 determines the nutrients of the meal from the nutrient image of the target nutrient in the food area input from the nutrient conversion unit 23 and outputs the determination result D4.

[0049] 《2 - 2》Operation FIG. 9 is a flowchart showing the operation of the nutrient determination device 20 according to the second embodiment. First, the meal image D0 obtained by photographing a meal is input to the food area determination unit 11 and the channel conversion unit 22. In step ST21 of FIG. 3, the channel conversion unit 22 generates an HSV image D22, which is the converted meal image, by converting the image signal of the meal image D0 into an HSV signal.

[0050] In step ST22 of FIG. 9, the food area determination unit 11 acquires the meal image D0, determines whether each pixel constituting the meal image D0 is a pixel of food, and generates a food area image D1, which is an image of the food area composed of food pixels. Note that the order of the processes in steps ST21 and ST22 may be reversed. Also, the processes in steps ST21 and ST22 may be executed in parallel.

[0051] In the next step ST23, the nutrient conversion unit 23 acquires the food area image D1 and the HSV image D22 of the meal image D0, acquires the target nutrient from the outside, and refers to the data of the nutrient - color DB15 to generate a nutrient image D23 of the target nutrient for each pixel of the HSV image in the area within the food area image D1 of the HSV image D22. The target nutrient is the nutrient that the user wants to focus on. When there is an input of the target nutrient, the nutrient conversion unit 23 does not perform the determination of all nutrients, but generates a nutrient image D23 for the input target nutrient.

[0052] In the next step ST24, the nutrient determination unit 14 performs statistical processing on the nutrients for each pixel of the nutrient image D23 to determine (i.e., calculate) the nutrients of the meal and outputs a determination result D4. The determination result D4 is provided to the output device 52 and presented to the user.

[0053] Figs. 10(A) to (C) are diagrams showing examples of images generated by the nutrient determination device 20. Fig. 10(A) shows the meal image D0 input to the food area determination unit 11 and the channel conversion unit 22. Fig. 10(B) shows the food area image D1 output by the food area determination unit 11 as binary information (for example, information with the food area as 1 and the non - food area as 0). Fig. 10(C) is an image diagram showing an example of the nutrient image D23 output from the nutrient conversion unit 23. Fig. 10(C) is an example showing the nutrient numbers of each pixel of the nutrient image D23 using the nutrient numbers 1, 2, 3, 4, 5, 6 of the "six basic food groups".

[0054] 《2 - 3》Effect As described above, according to the second embodiment, since the nutrients for each pixel are directly determined from the meal image D0, compared with the case of determining the nutrients after determining the food ingredients from the meal image and then using a database of data showing the nutrients for each food ingredient (i.e., performing two types of determination processes), the nutrients can be determined with high accuracy with a small amount of calculation.

[0055] Further, according to the second embodiment, since the food region determination and the channel conversion can be processed in parallel, there is an effect that the nutrient determination can be performed faster than in the case of the first embodiment.

[0056] Further, according to the second embodiment, there is an effect that a nutrient can be specified and the nutrient determination specialized for the nutrient can be performed. Note that the specification of the target nutrient can also be applied to the first embodiment or the third or fourth embodiments described later.

[0057] Regarding other aspects, the second embodiment is the same as the first embodiment.

[0058] 《3》Third Embodiment 《3-1》Configuration FIG. 11 is a block diagram schematically showing the configuration of a nutrient determination system 3 and a nutrient determination device 30 according to the third embodiment. In FIG. 11, the same reference numerals as those shown in FIG. 1 are assigned to the same or corresponding configurations as those shown in FIG. 1. The nutrient determination system 3 includes a nutrient determination device 30, an imaging device 51, and an output device 52. The nutrient determination device 30 is a device (for example, a computer) capable of implementing the nutrient determination method according to the third embodiment. The nutrient determination device 30 includes a food region determination unit 11, a channel conversion unit 12, a nutrient conversion unit 13, and a nutrient determination unit 34.

[0059] The nutrient determination device 30 may have a nutrient-color DB 15. The nutrient-color DB 15 may be a storage device of an external device different from the nutrient determination device 30. The nutrient determination device 30 according to the third embodiment is different from the nutrient determination device 10 according to the first embodiment in that information for each region of the nutrient image (for example, information on the weight or size of the food) is input from the outside to the nutrient determination unit 34.

[0060] The food region determination unit 11 acquires the meal image D0 and generates a food region image D1. The channel conversion unit 22 acquires the food region image D1 and generates an HSV image D2 of the food region by converting the image signal of the food region image D1 into an HSV signal.

[0061] The nutrient conversion unit 13 acquires the HSV image D2 of the food area from the channel conversion unit 12, refers to the data in the nutrient - color DB 15, determines the nutrients for each pixel of the HSV image D2, and generates the nutrient image D3 of the food area.

[0062] The nutrient determination unit 34 acquires the nutrient image D3 of the food area input from the nutrient conversion unit 13 and the information for each area of the nutrient image. The information for each area of the nutrient image includes the information on the actual size of the food or the information on the weight of the food. The nutrient determination unit 34 performs the determination (i.e., calculation) of the nutrients in the meal from the nutrient image D3 of the food area input from the nutrient conversion unit 13 and the information for each area of the nutrient image input from the outside, and outputs the determination result D34.

[0063] 《3 - 2》Operation FIG. 12 is a flowchart showing the operation of the nutrient determination device 30 according to the third embodiment. First, the meal image D0 obtained by photographing the meal is input to the food area determination unit 11. Steps ST11 to ST13 in FIG. 11 are the same as the operations in the first embodiment shown in FIG. 3.

[0064] In step ST34 of FIG. 12, the nutrient determination unit 34 acquires the nutrient image D3 from the nutrient conversion unit 13, performs statistical processing for each area of the nutrient image based on the nutrient image D3 and the information for each area of the nutrient image input from the outside, performs the determination (i.e., calculation) of the nutrients in the meal, and outputs the determination result D34. The determination result D34 is provided to the output device 52 and presented to the user.

[0065] The information for each region of the nutrient image is information on the actual size of the food that is the subject or information on the weight of the food. In either case of size and weight, the size per pixel (for example, 1 mm) or the weight (for example, 6 g) is the information described for each region of the image. The size or weight per pixel of the meal image D0 varies depending on the positional relationship between the imaging device 51 that captures the meal and the meal that is the subject. The information for each region of the nutrient image is used to absorb the difference in the way it appears due to the positional relationship between the imaging device 51 and the meal and to accurately measure the amount of nutrients contained in the meal.

[0066] When shooting at an angle overlooking the meal, the distance between the imaging device 51 and the meal is farther the closer it is to the upper part of the screen, and it appears smaller. For this reason, the size or weight of the food per pixel becomes larger the closer it is to the upper part of the screen.

[0067] Also, various ways of dividing the regions can be considered. For example, when dividing the image into 4 vertical divisions and 6 horizontal divisions, the information for each region of the nutrient image stores information on the actual size or weight per pixel for each of the 24 regions (= 4 × 6).

[0068] When the nutrient determination unit 34 determines nutrients from the nutrient image D3 input from the nutrient conversion unit 13, it multiplies the information for each region of the nutrient image D3 to calculate the actual amount of each nutrient and generates the determination result D34.

[0069] 《3-3》Effect As described above, according to the third embodiment, since the nutrients per pixel are directly determined from the meal image D0 and the actual amount is calculated, compared to the case where the nutrients are determined using a database of data indicating the nutrients for each ingredient after determining the ingredients from the meal image (that is, when two types of determination processes are performed), the nutrients can be determined with high accuracy with a small amount of calculation.

[0070] Further, according to Embodiment 3, the actual amount of each nutrient can be included in the determination result D34 by using the information for each region of the nutrient image D3. Therefore, there is an effect that not only the nutrient balance but also the amount of food can be managed.

[0071] Regarding other than the above, Embodiment 3 is the same as Embodiment 1 or 2.

[0072] 《4》Embodiment 4. 《4-1》Configuration FIG. 13 is a block diagram schematically showing the configuration of a nutrient determination system 4 and a nutrient determination device 40 according to Embodiment 4. In FIG. 13, the same or corresponding components as those shown in FIG. 1 are denoted by the same reference numerals as those shown in FIG. 1. The nutrient determination system 4 includes a nutrient determination device 40, an imaging device 51, and an output device 52. The nutrient determination device 40 includes a food region determination unit 11, a channel conversion unit 12, a nutrient conversion unit 43, a nutrient determination unit 44, a preprocessing unit 45, a meal image detection unit 46, and a utensil detection unit 47. The nutrient determination device 20 may have a nutrient-color DB 15. The nutrient determination device 40 according to Embodiment 4 is a device (for example, a computer) that can implement the nutrient determination method according to Embodiment 4. The nutrient determination device 40 according to Embodiment 4 is different from the nutrient determination device 10 according to Embodiment 1 in that it includes a preprocessing unit 45, a meal image detection unit 46, and a utensil detection unit 47, and in terms of the operations of the nutrient conversion unit 43 and the nutrient determination unit 44.

[0073] The preprocessing unit 45 receives an image including a meal as a subject photographed by the imaging device 51, and performs preprocessing for detecting a meal image on this image. The photographed image may be an image in which the subject is not specified.

[0074] The meal image detection unit 46 performs preprocessing on the preprocessed meal image D0 from the preprocessing unit 45 so as to easily detect whether it is the meal image D0, and inputs it to the meal image detection unit 46. The meal image detection unit 46 detects whether the input image from the preprocessing unit 45 is an image in which a meal is shown. If it is the meal image D0, the meal image D0 is input to the food area determination unit 11 and the tableware detection unit 47.

[0075] The food area determination unit 11 obtains the meal image D0, and for each of the plurality of pixels constituting the meal image D0, determines whether it is a pixel of food as a subject, thereby generating a food area image D1 that is an image of the food area composed of the pixels of the food.

[0076] The channel conversion unit 12 converts the image signal (for example, RGB signal or YCbCr signal) of the food area image D1 generated by the food area determination unit 11 into an HSV signal, and outputs the HSV image D2 of the food area. The HSV image D2 of the food area is input to the nutrient conversion unit 43 and the nutrient determination unit 44.

[0077] The nutrient conversion unit 43 obtains the HSV image D2 of the food area from the channel conversion unit 12. In addition, meal menu information is input to the nutrient conversion unit 43 from the outside. The nutrient conversion unit 43 performs nutrient determination for each pixel based on the HSV image D2 of the food area and the menu information, and generates a nutrient image D43 of the food area.

[0078] On the other hand, the tableware detection unit 47 detects tableware (that is, tableware such as a plate, a bowl, etc.) from the meal image D0 input from the meal image detection unit 46, and inputs the feature information D47 of the tableware such as the type, shape, color, size of the tableware to the nutrient determination unit 44.

[0079] The nutrient determination unit 44 performs determination (that is, calculation) of the nutrients of the meal from the HSV image D2 of the food area input from the channel conversion unit 12, the nutrient image D43 of the food area input from the nutrient conversion unit 43, the information for each area of the nutrient image input from the outside, and the feature information D47 input from the tableware detection unit 47, and outputs the determination result D44.

[0080] "4-2" Operation Figure 14 is a flowchart showing the operation of the nutrient determination device according to Embodiment 4. First, an image that is not limited to a meal of the subject is input from the imaging device 51 to the preprocessing unit 45. In step ST41 of Figure 14, the preprocessing unit 45 performs preprocessing for detecting the meal image D0 on the input image. When the subject is photographed by the imaging device 51, due to the influence of light such as illumination or sunlight, phenomena such as part or the whole of the image becoming dark or color bleeding may occur. The preprocessing unit 45 performs processing to reduce the influence of light, for example, by image processing. As this processing method, a known method can be used. In addition, when an out-of-focus or blurred image is input, the preprocessing unit 45 performs processing to enhance the edges.

[0081] In step ST42 of Figure 14, the meal image detection unit 46 detects whether there is a meal as the subject in the image preprocessed by the preprocessing unit 45. If a meal is detected, the process proceeds to step ST43, and if no meal is detected, the process ends.

[0082] When the meal image detection unit 46 detects a meal (YES in step ST42), the meal image detection unit 46 outputs the image input from the preprocessing unit 45 as the meal image D0. The output meal image D0 is input to the food area determination unit 11 and the utensil detection unit 47.

[0083] In step ST43 of Figure 14, the food area determination unit 11 acquires the meal image D0, determines whether each pixel constituting the meal image D0 is a pixel of food, and generates a food area image D1 that is an image of the food area composed of food pixels.

[0084] In step ST44 of Figure 14, the channel conversion unit 12 generates the HSV image D2 of the food area after conversion by converting the image signal of the food area image D1 into an HSV signal. The HSV image D2 of the food area is input to the nutrient conversion unit 43 and the nutrient determination unit 44.

[0085] In step ST45 of FIG. 14, the nutrient conversion unit 43 acquires the HSV image D2 of the food area from the channel conversion unit 12 and acquires meal menu information from the outside. Based on the HSV image D2 of the food area and the menu information, the nutrient conversion unit 43 refers to the data of the nutrient-color DB15, determines the nutrient for each pixel, and generates a nutrient image D43 of the food area composed of a column of nutrient numbers.

[0086] In step ST46 of FIG. 14, the utensil detection unit 47 detects the meal utensil from the meal image D0 input from the preprocessing unit 45 and inputs the utensil feature information D47 to the nutrient determination unit 44. The utensil feature information D47 includes, for example, the shape of the utensil, the color of the utensil, the size of the utensil, and the like. FIG. 15 is a diagram showing an example of the utensil and dish database 47a included in the nutrient determination apparatus 40. Note that the utensil and dish database 47a may be stored in a storage device of an external device that can communicate with the nutrient determination apparatus 40. Further, as shown in FIG. 15, the utensil and dish database 47a has a database 47a indicating the type of utensil, the shape of the utensil, and the dishes served on the utensil, and the dish may be estimated from the utensil features when determining the nutrient to perform the nutrient determination.

[0087] In step ST47 of FIG. 14, the nutrient determination unit 44 determines (i.e., calculates) the nutrients of the meal from the nutrients for each pixel of the nutrient image D43, the information on the size or weight of the nutrient image D43, and the utensil feature information D47, and outputs a determination result D44.

[0088] 《4-3》Effect As described above, according to the fourth embodiment, there is an effect that menu or utensil information can be used to more accurately determine nutrients.

[0089] Regarding other than the above, the fourth embodiment is the same as any one of the first to third embodiments.

Description of Reference Numerals

[0090] 1 to 4 nutrient determination systems, 10, 20, 30, 40 nutrient determination devices, 11 food area determination unit, 12, 22 channel conversion units, 13, 43 nutrient conversion units, 14, 34, 44 nutrient determination units, 15 nutrient - color DB, 45 pre - processing unit, 46 meal image detection unit, 47 utensil detection unit, 51 imaging device, 52 output device, D0 meal image, D1 food area image, D2, D22 HSV images, D3, D23, D43 nutrient images, D4, D34, D44 determination results, D47 feature information.

Claims

1. An imaging device, A nutrient determination device that determines nutrients in a meal image that is an image captured by the imaging device or a part of the captured image, and outputs a determination result obtained by the determination, An output device that outputs the determination result, having, The nutrient determination device includes: A food area determination unit that determines whether each pixel constituting the meal image is a pixel of food, and generates a food area image that is an image of a food area composed of the pixels of the food; A channel conversion unit that generates an HSV image by converting the image signal of each pixel of an image including at least the food area image into an HSV signal; A nutrient conversion unit that refers to data in a nutrient-color database showing the relationship between nutrients and values of hue, saturation, and lightness in the HSV color space, determines nutrients for each pixel of the HSV image of the food area, and generates a nutrient image; A nutrient determination unit that determines the nutrients in the meal based on the nutrients for each pixel and outputs the determination result, having, The nutrient conversion unit refers to the color information of protein, mineral, carotene, vitamin, carbohydrate, and lipid as nutrients stored in the nutrient-color database, The color information defines the values of ranges that can be taken for at least two or more of each of the items of hue, saturation, and lightness. A nutrient determination system characterized by the above.

2. The channel conversion unit generates the HSV image of the food area from the image signal of each pixel of the food area image received from the food area determination unit. The nutrient determination system according to claim 1, characterized by the above.

3. The channel conversion unit generates the HSV image from the image signal of each pixel of the image, Based on the HSV image received from the channel conversion unit and the food area image received from the food area determination unit, the nutrient conversion unit refers to the nutrient / color database to determine the nutrients for each pixel of the HSV image of the food area and generate the nutrient image. The nutrient determination system according to claim 1, characterized in that.

4. The nutrient conversion unit receives a target nutrient from the outside, determines the nutrients for each pixel of the HSV image of the food area, and generates the nutrient image of the target nutrient. The nutrient determination system according to claim 3, characterized in that.

5. The nutrient determination unit receives food information regarding the size or weight of the food for each predetermined area in the nutrient image from the outside, and calculates the amount of nutrients in the meal from the nutrient image using the food information. The nutrient determination system according to claim 4, characterized in that.

6. The nutrient determination device further includes a meal image detection unit that detects the meal image from the image captured by the imaging device. The nutrient determination system according to claim 5, characterized in that.

7. The nutrient determination device further includes a utensil detection unit that detects the utensil on which the food is placed from the meal image, detects the characteristics of the utensil that are any one or more of the size, shape, and color of the utensil, and outputs characteristic information. The nutrient determination unit determines the nutrients in the meal based on the characteristic information. The nutrient determination system according to claim 6, characterized in that.

8. The nutrient conversion unit receives the menu information of the meal from the outside, and determines the nutrients for each pixel of the HSV image of the food area based on the data in the nutrient / color database and the menu information. The nutrient determination system according to any one of claims 1 to 7, characterized in that.

9. A nutrient determination device that determines the nutrients of a meal based on a meal image that is an image captured by an imaging device or a part of the captured image, and outputs a determination result obtained by the determination, comprising: A food area determination unit that determines whether each pixel constituting the meal image is a food pixel, and generates a food area image that is an image of a food area composed of the food pixels; A channel conversion unit that generates an HSV image by converting the image signal of each pixel of an image including at least the food area image into an HSV signal; A nutrient conversion unit that refers to data in a nutrient-color database showing the relationship between nutrients and the values of hue, saturation, and lightness in the HSV color space, determines the nutrients for each pixel of the HSV image of the food area, and generates a nutrient image; A nutrient determination unit that determines the nutrients of the meal based on the nutrients for each pixel and outputs the determination result; and having: The nutrient conversion unit refers to the color information of protein, mineral, carotene, vitamin, carbohydrate, and lipid as nutrients stored in the nutrient-color database; The color information defines the values of the ranges that can be taken for at least two or more of the items of hue, saturation, and lightness respectively. A nutrient determination device characterized by this.

10. A nutrient determination method implemented by a nutrient determination device that determines the nutrients of a meal based on a meal image that is an image captured by an imaging device or a part of the captured image, and outputs a determination result obtained by the determination, comprising: A step of determining whether each pixel constituting the meal image is a food pixel, and generating a food area image that is an image of a food area composed of the food pixels; A step of generating an HSV image by converting the image signal of each pixel of an image including at least the food area image into an HSV signal; Referring to the data in the nutrient-color database that defines the range of values that each can take for at least two or more of the items of hue, saturation, and lightness in the HSV color space for each of the nutrients protein, mineral, carotene, vitamin, carbohydrate, and lipid, determining the nutrient for each pixel of the HSV image of the food region to generate a nutrient image; determining the nutrients of the meal based on the nutrients for each pixel and outputting the determination result; A nutrient determination method characterized by comprising the above steps.

11. On a computer that determines the nutrients of a meal based on a meal image that is an image captured by an imaging device or a part of the captured image and outputs the determination result obtained by the determination, determining whether each pixel constituting the meal image is a pixel of food and generating a food region image that is an image of a food region composed of the pixels of food; generating an HSV image by converting the image signal of each pixel of at least the image including the food region image into an HSV signal; Referring to the data in the nutrient-color database that defines the range of values that each can take for at least two or more of the items of hue, saturation, and lightness in the HSV color space for each of the nutrients protein, mineral, carotene, vitamin, carbohydrate, and lipid, determining the nutrient for each pixel of the HSV image of the food region to generate a nutrient image; determining the nutrients of the meal based on the nutrients for each pixel and outputting the determination result; A nutrient determination program characterized by causing the above steps to be executed.

Images