Information processing apparatus, learning device, and program

Abstract

To provide an information processing apparatus that easily evaluates the concentration of a nutrient.SOLUTION: An information processing apparatus comprises a concentration estimation unit 51 that receives one or more meal items constituting a user's meal, receives, from the user who eats the meal item, evaluation of the strength of the taste of a predetermined nutrient in the meal item, and estimates the concentration of the nutrient in the meal item by applying a model trained in advance to the strength evaluation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a learning device, and a program for processing data related to meals. [Background technology]

[0002] Food record apps and the like analyze nutrients by taking images, and in accordance with this approach, for example, Patent Document 1 uses AI (artificial intelligence) to learn from food images and the measurement results of a salt concentration meter, and estimates salt content from the food images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-031486 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional techniques estimate nutrients such as salt from images, and this estimation is possible under the assumption that the salt content and other nutrients are constant for each type of meal, such as a dish. However, even if the same dish appears the same on an image, it is conceivable that the nutrient density may vary. Conventional techniques, however, are unable to address such situations. This problem arises not only when meal contents are acquired using image analysis. However, solving this problem by performing various chemical analyses of meals to perform nutritional analysis each time is not practical from a cost perspective. It is desirable to be able to evaluate nutrient density using a simple method similar to image analysis.

[0005] In view of the above-mentioned problems of the conventional technology, the first object of the present invention is to provide an information processing device that simply evaluates the concentration of nutrients. The second object is to provide a learning device that learns a model used by the information processing device. The third object is to provide a program corresponding to the information processing device and the learning device. [Means for solving the problem]

[0006] To achieve the first objective, the present invention has a first feature in that it includes an estimation unit that receives one or more meal items comprising a user's meal, receives a taste intensity rating of a predetermined nutrient in the meal item from the user who has eaten the meal item, and estimates the concentration of the nutrient in the meal item by applying a pre-trained model to the intensity rating. To achieve the second objective, the present invention has a second feature in that it is a learning device that reads the taste intensity rating of a predetermined nutrient in the meal item from the user who has eaten the meal item and the concentration of the predetermined nutrient in the meal item as training data, and constructs a model that uses the intensity rating as input and outputs the concentration of the predetermined nutrient in the meal item. The present invention also has a feature in that it is a program corresponding to the information processing device and the learning device. [Effects of the Invention]

[0007] According to the first feature, the concentration of nutrients in a meal item is estimated from the taste intensity evaluation of the nutrients using a pre-trained model, which allows for simple estimation and achieves the first object. According to the second feature, the second object is achieved by training the model. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a functional block diagram of an information processing apparatus according to an embodiment. [Figure 2] 10 is a flowchart illustrating an operation of the information processing apparatus according to the embodiment. [Figure 3]10A and 10B are diagrams illustrating a schematic example of data acquired by a model data acquisition unit. [Figure 4] 1 is a schematic example of a user interface presented to a user to obtain a large amount of data. [Figure 5] 10 is a table showing a schematic example of the processing content of step S2. [Figure 6] FIG. 10 is a diagram showing a schematic example of a screen interface when an additional estimation unit interactively obtains information from a user. [Figure 7] FIG. 1 is a diagram illustrating an example of a hardware configuration of a general computer. DETAILED DESCRIPTION OF THE INVENTION

[0009] FIG. 1 is a functional block diagram of an information processing device 100 according to an embodiment, and the information processing device 100 includes a model data acquisition unit 11, a model construction unit 12, a model storage unit 13, a photographing unit 21, a recognition unit 22, an allocation unit 23, a food DB (database) 24, a concentration estimation unit 31, an additional amount estimation unit 41, a user DB (database) 42, and a result output unit 51.

[0010] 1. For example, the model data acquisition unit 11, the model construction unit 12, and the model storage unit 13 constitute a learning device 10 (learning unit 10) that learns a model, but these may be configured to be provided in an external server device (not shown) separate from the information processing device 100, and the information processing device 100 may read a model (described later) from the server device.

[0011] Fig. 2 is a flowchart of the operation of the information processing device 100 according to the embodiment. Below, the details of the processing content of each functional block of the information processing device 100 in Fig. 1 will be described while explaining each step in Fig. 2.

[0012] In step S1, a model is constructed, and then the process proceeds to step S2. The model constructed in step S1 estimates the objective concentration of a specific nutrient, such as salt, that corresponds to the subjective taste intensity of the specific nutrient as perceived by a specific user. Note that the information processing device 100 of this embodiment can estimate the concentration of a specific nutrient in the user's diet, but the following explanation mainly focuses on the case where the nutrient is salt.

[0013] Specifically, in step S1, the model data acquisition unit 11 acquires model construction data as shown schematically in FIG. 3, the model construction unit 12 constructs a model using the data, and the constructed model is stored in the model storage unit 13.

[0014] In Figure 3, the upper part shows model-building data for estimating the "actual objective concentration of salt" for a specific user's meal item, "miso soup," from the user's "subjective taste evaluation" based on the taste intensity of the nutrient "salt." Such data can be collected, for example, by recording the user's subjective taste evaluation of the nutrient in question in many cases where the meal item in question was actually served, using a meal recording app used by the user on a mobile device such as a smartphone.

[0015] That is, for the same food item, "miso soup," for example, when the objective value of its salt content (for example, 2.5g of salt is contained in 250g of salt water that makes up the miso soup, which means that the mass percentage concentration of the salt in the salt water in the miso soup, or a concentration of 1% by weight percentage), can be obtained as known information, the user's subjective evaluation of how the user perceived the saltiness of the "miso soup" when consuming it is recorded as an evaluation on a predetermined scale (7 scales in this example), such as 3 scales on the "weak" side, 1 scale for "taste" as the appropriate strength, and 3 scales on the "strong" side, as shown in the bottom of Figure 3.

[0016] In this example, the subjective evaluation is recorded in seven discrete stages, but it is also possible to record the subjective evaluation as a continuous value using a visual analogue scale (VAS). That is, a line is presented to the user, and the subjective evaluation is recorded as a continuous value in the range [-3, +3] by having the user specify a point on the line, for example, the left end is "lightest" with a value of -3, the middle is "preferred" with a value of 0, and the right end is "strongest" with a value of +3.

[0017] In this way, a large amount of data can be obtained, such as "miso soup from chain store A" having an objective salt concentration of 1% and the user's subjective rating of "-1 (slightly weak)", "miso soup from manufacturer B" having an objective salt concentration of 1.2% and the user's subjective rating of "0 (normal)", "miso soup from chain store C" having an objective salt concentration of 1.5% and the user's subjective rating of "+3 (very strong)", and so on.

[0018] Figure 4 is a schematic example of a user interface presented to the user to obtain this large amount of data. As shown in the upper section, the contents of the user's meals eaten out or at home can be identified by automatically recognizing images taken by the user, or by the user directly specifying the contents from a menu, and then an objective evaluation of the nutrients such as salt in the identified food items can be obtained from the user, as shown in the lower section.

[0019] The model construction unit 12 uses the model construction data as exemplified in FIG. 3 to perform learning on an arbitrary estimation model, such as a regression model, to construct a function y=f for calculating the objective concentration y of a specific nutrient β from the subjective evaluation value x of the taste intensity of the specific nutrient β in various food items α for a specific user U. U,α,β (x), the model can be constructed. The model storage unit 13 stores the constructed model y=f U,α,β Save (x).

[0020] The above describes step S1 in Fig. 2. From step S2 onwards, the model previously constructed in step S1 is used to estimate the nutrient concentrations and other aspects of a meal when the user is actually eating.

[0021] In step S2, dietary data is acquired from the user who is actually eating, the dietary content is recognized, and whether or not it is a target for nutrient estimation is determined. This step S2 can be specifically realized by the photographing unit 21, the recognition unit 22, the allocation unit 23, and the food DB 24 as follows.

[0022] The hardware configuration of the photographing unit 21 is composed of an RGB camera or an RGB-D camera (a camera that acquires depth images linked to normal RGB color images), and acquires images of the user's meal (as long as the image shows the food on the dining table, etc., the user himself does not necessarily have to be in the image).

[0023] The recognition unit 22 recognizes the individual food items that make up the meal by applying object recognition using machine learning, deep learning, or any other existing method that has been previously trained to the food image obtained by the photographing unit 21. For example, a recognition result may be obtained that the meal image contains four types of food items: "rice," "miso soup," "tofu," and "kushikatsu (3 pieces)." Note that the recognition unit 22 may also obtain the volume of each recognized food item as its quantity by utilizing depth information from the RGB-D camera, etc.

[0024] The food DB 24 is prepared in advance and stores, for each food item, information that indicates that if the weight of the food of that food item is W (g), then the food contains a certain nutrient in a weight w (g). By dividing this information by w / W, the concentration of the nutrient in that food item is obtained.

[0025] The information in this food DB24 is essentially the same as the concentration information in the model constructed in step S1, but unlike the example in Figure 3 where the "salt concentration" for "miso soup" can vary depending on the restaurant or manufacturer, the food DB24 records information equivalent to the "salt concentration" in "miso soup" as a single value based on a representative or typical value.

[0026] The sorting unit 23 refers to the food DB 24 for each of the food item recognition results (where N food items α are recognized, and α=α1, α2, ..., αN) obtained by the recognition unit 22, and calculates the concentration C β (α)=C β (α1),C β (α2),…,C β (αN) and obtain the concentration C β It is determined whether (α) is equal to or greater than a certain value using a predetermined threshold.

[0027] Food items determined by sorting unit 23 to be equal to or greater than the threshold are output to concentration estimation unit 31 as being subject to concentration estimation of the nutrient β, and are then processed in step S3, which will be described later. Food items not determined by sorting unit 23 to be equal to or greater than the threshold are output to additional estimation unit 41 as not being subject to concentration estimation of the nutrient β, and are then processed in step S4, which will be described later.

[0028] 5 is a table showing a schematic example of the processing content of step S2 described above, where the first column is each of the food items recognized by the recognition unit 22, the second column is the predetermined concentration of the nutrient "salt" obtained from the food DB 24 for each of the recognized food items, and the third column shows the results of sorting by the sorting unit 24 by performing threshold judgment on the "salt concentration." Here, a sorting result of "1" indicates that the food falls under the concentration estimation target and is output to the concentration estimation unit 31, and a sorting result of "0" indicates that the food does not fall under the concentration estimation target and is output to the additional estimation unit 41.

[0029] In the example of Fig. 5, the food items "tofu miso soup" and "pork cutlet" are classified as targets for concentration estimation of the nutrient "salt," while the other food items "rice," "hiyayakko" (cold tofu), and "assorted cabbage" are classified as not targets for concentration estimation of the nutrient "salt." As can be seen from the example of Fig. 5, the significance of the classification unit 24 for each food item α is to determine whether, even if this food item α contains nutrient β, the amount is so small that it can be ignored in reality when collecting dietary data, or whether it cannot be ignored and the concentration of nutrient β needs to be estimated.

[0030] Step S2 has been described above. In step S3, for each food item α sorted by sorting unit 23 in step S2 as a target for nutrient β concentration estimation, concentration estimation unit 31 presents a question to user U (user U who has eaten the food item α and whose meal image was captured by imaging unit 21) asking about the intensity of the taste of nutrient β in the food item α, obtains a subjective evaluation result of the taste of nutrient β in the food item α as a response from user U, and estimates the objective concentration of nutrient β in the food item α by applying the model stored in model storage unit 13 to this subjective evaluation result.

[0031] As explained in the model construction in step S1, the concentration estimation unit 31 estimates the objective concentration y of a specific nutrient β from the subjective evaluation value x (question answer result) of the user U regarding the strength of the taste of the specific nutrient β in the food item α, using the function y=f U,α,β (x) can be used as a model for calculation. The subjective assessment value x may be specified as a discrete value, as in the case of model construction, or as a continuous value using a VAS.

[0032] In step S4, for each food item α that was classified by the classification unit 23 in step S2 as not being a target for nutrient β concentration estimation, the additional estimation unit 41 asks the user whether or not the user added the nutrient β to the meal in the form of a seasoning or the like, and if so, how much. By receiving a response from the user, an estimation result of the added amount of nutrient β is obtained.

[0033] As described above, the significance of the additional estimation unit 41 is that when a specific food item α is first served in a meal, it does not contain the nutrient β (it is a negligible amount due to its small amount), but the user may add the nutrient β to the food item α later during the meal in the form of seasonings, etc., and such nutrient β can be acquired interactively.

[0034] In the example of Figure 5, for example, after the meal, the user is presented with a question in text or the like regarding each of the "rice," "hiyayakko," and "kabuki platter," such as "Did you add any seasonings containing salt?", and the user can respond that they added "koikuchi soy sauce" as a seasoning containing salt to the "hiyayakko," but did not add any seasonings to the "rice" or "kabuki platter."

[0035] In this example, the user is further asked how much "strong soy sauce" was added to the "hiyayakko" by being presented with a menu of options: "0.5 teaspoon, 0.75 teaspoon, 1 teaspoon, 1.25 teaspoon, 1.5 teaspoon," and the answer can be accepted.

[0036] For example, if the user responds that they added only one teaspoon of the seasoning "strong soy sauce," the additional estimation unit 41 can estimate that the corresponding nutrient "salt" is 0.86 g. Information about the amount of nutrient β contained in a quantity v (which may be weight or volume) of various seasonings γ (γ = γ1, γ2, γ3, ...) is pre-constructed as a database and recorded in the seasoning DB 42. The additional estimation unit 41 can then perform estimation processing by referring to the seasoning DB 42. FIG. 6 shows an example of a schematic screen interface when the additional estimation unit 41 makes the above-described interactive inquiries and obtains answers from the user. The additional estimation unit 41 poses questions Q1 and Q2 in the form of text and / or images, etc., and the user provides answers A1 and A2 in the form of text and / or menu selection, etc., respectively. Answer A1 to question Q1 reveals that the "hiyayakko" was topped with a salty seasoning, "Company A's dark soy sauce" (the salt concentration of which is pre-recorded in seasoning DB42), and question Q2 asks how much of the seasoning was used in terms of "tablespoons" or "teaspoons," to which answer A2 is obtained.

[0037] As shown in the schematic example of question Q2, the additional estimation unit 41 of this embodiment may display measuring devices for seasonings so that their sizes can be visually confirmed as a screen interface when inquiring about the specific amount of additional seasoning containing salt that the user has added. In the example of Fig. 6, the user can visually confirm that a "tablespoon" as a lightweight utensil is about 5 cm in size and a "teaspoon" is about 3.5 cm in size by an image of a scale (ruler) superimposed on each spoon.

[0038] By answering whether the user added a certain amount of seasoning using a measuring device (tablespoon or teaspoon), the additional estimation unit 41 can obtain information on the nutrients that the user has consumed and that are contained in the seasonings that the user added, based on information that is pre-recorded in the additional estimation unit 41 regarding the amount of seasoning in each measuring device that corresponds to the actual amount of nutrients (for example, information that one cup of seasoning "Company A's dark soy sauce" using a measuring device (teaspoon) corresponds to the nutrient "0.86g of salt").

[0039] In step S5, the result output unit 51 outputs all or part of the processing results of the information processing device 100, particularly the results of the concentration estimation unit 31 and the additional amount estimation unit 41, and the flow of FIG. 2 ends.

[0040] As described above, according to the present invention, by utilizing a learning model, the concentration of a specific nutrient in a specific food item eaten by a user can be simply estimated from an evaluation of the taste strength of the nutrient in the food item.

[0041] Various supplementary examples, alternative examples, additional examples, etc. will be described below.

[0042] (1) By incorporating an embodiment of the present invention into a dietary management app that a user runs on a mobile device such as a smartphone, it becomes possible to simply and accurately estimate nutrient concentrations such as salt concentration, and as a result, it becomes possible to provide the user with simple and accurate dietary advice, etc., which will contribute to Goal 3 of the United Nations-led Sustainable Development Goals (SDGs) “Ensure healthy lives and promote well-being for all at all ages.”

[0043] (2) Each meal item constituting the user's meal is obtained by analyzing the captured image using the photographing unit 21 and the recognition unit 22. However, instead of this, the meal items may be obtained by accepting text input directly from the user, by accepting input by selecting from a predetermined menu, or by reading a one-dimensional code or two-dimensional code that stores the meal contents rather than a meal image using image analysis. The processing after the allocating unit 23, where the meal items are obtained, can be common.

[0044] (3) While the above description has primarily focused on salt (table salt) as a nutrient, the present invention can be applied to any nutrient whose concentration correlates with taste intensity, such as salt, sugar (which may be limited to particularly sweet nutrients like sugar), lipids (whose taste intensity corresponds to whether they are fatty or light), nutrients related to spiciness, nutrients related to sourness, and nutrients related to bitterness. Nutrients related to sugar, lipids, spiciness, sourness, and bitterness may be composed of multiple substances with varying proportions (rather than a single substance like NaCl in the case of salt). The concentration of nutrient β contained in various target meal items α, modeled by the learning unit 10 and estimated by the concentration estimation unit 31, may be any concentration that allows for a certain level of nutritional evaluation. Regardless of whether the meal items α are solid and / or liquid, the weight percent concentration of the nutrient β contained in the food items α may be used. Alternatively, a uniform definition of concentration may be set for each meal item α, and a concentration other than weight percent may be used.

[0045] (4) When the learning unit 10 learns in step S1, it may learn using only the data of each user, but it may also accelerate model construction by supplementing the learning data with data from other users whose attribute information matches (e.g., attribute information determined in whole or in part by questionnaire data on gender, age, residential area, dietary habits, health checkup data, etc.).

[0046] (5) In step S1, the learning unit 10 may periodically update the model using objective measurements (measurements of nutrients such as salt in the food the user actually eats, whether eating out or eating at home), rather than continuing to use the model once constructed.

[0047] (6) FIG. 7 is a diagram showing an example of the hardware configuration of a general computer device 70. The information processing device 100 can be realized as one or more computer devices 70 having such a configuration. When the information processing device is realized using two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the CPU 71's execution instructions in place of or in cooperation with the CPU 71, a RAM 73 as a main memory device that provides a work area for the CPU 71 (and GPU 72), a ROM 74 as an auxiliary memory device, a communication interface 75, a display 76 that displays output, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, etc., a speaker 78 that outputs audio, and a bus BS for transmitting and receiving data among these.

[0048] Each functional unit in the information processing device 100 can be realized by a CPU 71 and / or a GPU 72 that reads a predetermined program corresponding to the function of each unit from a ROM 74 and executes it. Both the CPU 71 and the GPU 72 are types of arithmetic devices (processors). Here, when display-related processing is performed, a display 76 also operates in conjunction with the CPU 71 and / or the GPU 72. When communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the CPU 71 and the GPU 72. When audio output-related processing is performed, a speaker 78 also operates in conjunction with the CPU 71 and the GPU 72. [Explanation of symbols]

[0049] 100...information processing device, 11...model data acquisition unit, 12...model construction unit, 13...model storage unit, 21...photography unit, 22...recognition unit, 23...sorting unit, 24...food DB, 31...concentration estimation unit, 41...addition estimation unit, 42...seasoning estimation unit, 51...result output unit

Claims

1. receiving one or more meal items that comprise a meal for the user; receiving a taste intensity rating of a predetermined nutrient in the meal item from a user who has eaten the meal item; An information processing device comprising: a concentration estimation unit that estimates the concentration of the nutrient in the meal item by applying a pre-trained model to the strength evaluation.

2. 2. The information processing device according to claim 1, further comprising a sorting unit that selects, from the one or more received meal items, meal items whose concentrations are to be estimated by the concentration estimation unit, based on the meal items whose concentration values ​​are equal to or greater than a threshold value.

3. The information processing device according to claim 2, further comprising an additional amount estimation unit that estimates the amount of the nutrient in the added seasoning by asking the user whether the user has added a seasoning containing the nutrient to a meal item not selected by the sorting unit and, if so, the amount of the seasoning added, and accepting the answer.

4. The information processing device according to claim 3, characterized in that the addition estimation unit displays a measuring device for the seasoning so that the size of the measuring device can be visually confirmed as a screen interface when asking the user about the amount of the seasoning added.

5. 2. The information processing device according to claim 1, wherein the nutrients relate to salt, sugar, lipid, sourness, bitterness, or spiciness.

6. 2. The information processing device according to claim 1, further comprising a learning unit that reads, as learning data, an evaluation of the taste intensity of a predetermined nutrient in a meal item from a user who has eaten the meal item and the concentration of the nutrient in the meal item, and constructs the model to be used in the concentration estimation unit.

7. A learning device that reads in, as learning data, a taste strength evaluation of a specific nutrient in a meal item from a user who has eaten the meal item and the concentration of the specific nutrient in the meal item, and constructs a model that uses the strength evaluation as input and outputs the concentration of the specific nutrient in the meal item.

8. 8. A program for causing a computer to function as the information processing device according to claim 1 or the learning device according to claim 7.

Images