Information processing device, information processing method, and program

Abstract

The goal is to provide technology that offers users an interactive dining experience. [Solution] One aspect of the present disclosure relates to an information processing device comprising: an acquisition unit that acquires images relating to a meal by a user; a determination unit that determines either or both of the eating behavior and / or ingredients based on the images; and a provision unit that determines content according to the determination result and provides the content to the user.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] "Eating" has a great impact on mental well-being. Through feelings of satisfaction such as deliciousness and enjoyment, or connections with family and society through meals, one can obtain a sense of self-esteem that one wants to cherish oneself and that one is cherished. This is fundamental for healthy growth in infancy and childhood, and also becomes a sense of purpose that supports an active daily life in old age, enabling one to spend an active old age (active aging).

[0003] It is recognized that "food communication" plays a very important role in leading to an improvement in meal satisfaction. Food communication refers to communication between people through food. Specifically, it means enjoying meals or cooking with someone, learning and understanding about food (food products, food cultures, cooking, etc.) together, or exchanging information about food.

[0004] Since the COVID-19 pandemic, communication through food has decreased. Preventing the decrease in food communication may lead to a reduction in stress and mental fatigue, and an improvement in life satisfaction and sense of purpose. Also, for the increasing number of single people, it may become necessary to consider alternative means of new food communication.

[0005] With the development of information and communication technologies, various efforts related to food using information and communication technologies such as food tech have been progressing.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] Against this backdrop, it is desirable to offer meals not just as individual food items, but as a dining experience.

[0008] The objective of this disclosure is to provide technology for offering users an interactive dining experience. [Means for solving the problem]

[0009] One aspect of the present disclosure relates to an information processing device comprising: an acquisition unit that acquires images relating to a user's meal; a determination unit that determines either or both of the eating behavior and / or ingredients based on the images; and a provision unit that determines content according to the determination result and provides the content to the user. [Effects of the Invention]

[0010] According to this disclosure, we can provide technology to offer users an interactive dining experience. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a schematic diagram showing a food experience provision process according to one embodiment of the present disclosure. [Figure 2] Figure 2 shows an example of a configuration that realizes a food experience provision process according to one embodiment of the present disclosure. [Figure 3] Figure 3 shows an example of a configuration that realizes a food experience provision process according to one embodiment of the present disclosure. [Figure 4] Figure 4 is a block diagram showing the hardware configuration of an information processing device according to one embodiment of the present disclosure. [Figure 5] Figure 5 is a block diagram showing the functional configuration of an information processing device according to one embodiment of the present disclosure. [Figure 6] Figure 6 shows a food ingredient identification model according to one embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram showing correspondence information according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a content image diagram according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is an image diagram showing a food experience providing process according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram showing a food satisfaction determination model according to an embodiment of the present disclosure. [Figure 11] FIG. 11 is a flowchart showing a food experience providing process according to an embodiment of the present disclosure.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0013] In the following embodiments, an information processing apparatus that provides content corresponding to a user's eating action and food is disclosed.

[0014] [Food Experience Providing Process] First, a food experience providing process according to an embodiment of the present disclosure will be described. FIG. 1 is a schematic diagram showing a food experience providing process according to an embodiment of the present disclosure.

[0015] The information processing apparatus 100 realizes an interactive food experience providing process according to an embodiment of the present disclosure. As shown in FIG. 1, when the information processing apparatus 100 acquires an image related to a user's meal, it provides the user with content related to the user's eating action and the food used in the meal.

[0016] For example, in the example shown in FIG. 1, when the user makes a "itadakimasu" pose when starting to eat, the information processing apparatus 100 acquires an image of the pose as shown in the figure, and determines that the user has started eating. Also, as shown in the figure, when the user is eating rice using chopsticks, the information processing apparatus 100 acquires an image of the user eating rice, and determines that the user is eating rice. In order to determine the user's eating behavior based on such an image or video, for example, the information processing apparatus 100 can detect the skeleton of the user imaged in the video, and determine the user's behavior according to the movement of the detected skeleton. Such determination of behavior by skeleton detection can be realized by using any known technology.

[0017] Also, when acquiring an image of the food content, the information processing apparatus 100 determines the food ingredients and dishes imaged. For example, in the example shown in FIG. 1, the information processing apparatus 100 acquires an image of sushi, and determines that the user is eating a tuna nigiri. Also, the information processing apparatus 100 acquires an image of tempura, and determines that the user is eating shrimp tempura. For example, the information processing apparatus 100 may use a food ingredient determination model 50 trained using a training dataset composed of an image of the food content and labels indicating the names of the food ingredients and dishes imaged, to predict the food ingredients and dishes in the inference target image.

[0018] In this way, when determining the user's eating behavior and food ingredients, the information processing apparatus 100 determines the content corresponding to the determined eating behavior and food ingredients, and provides the content to the user. In the example shown in FIG. 1, when the food ingredient is tempura, the information processing apparatus 100 may provide the user with content of an image of a tempura cooking scene.

[0019] Furthermore, the information processing device 100 may determine both the eating behavior and the food ingredients, decide on content corresponding to both the eating behavior and the food ingredients, and provide that content to the user. For example, if the eating behavior of making the "itadakimasu" gesture is determined in addition to determining a certain food ingredient, the user may be provided with guidance information such as the order in which to eat that food ingredient deliciously. Also, if the eating behavior of making the "gochisousama deshita" gesture is determined in addition to determining a certain food ingredient, the user may be provided with information recommending a suitable dessert to eat after that food ingredient. This makes it possible to provide useful information that is more closely related to the user's situation compared to determining only one of the eating behavior or the food ingredients, and can potentially improve user satisfaction.

[0020] Such an information processing device 100 may be implemented as a server located on the cloud. In this case, as shown in Figure 2, the user operates a user terminal to send images related to the meal, such as video of the meal, to the server 100. Based on the images received from the user terminal, the server 100 determines the user's eating actions and ingredients in the meal, and sends content corresponding to the determined eating actions and ingredients to the user terminal. Upon receiving content from the server 100, the user terminal displays content related to the ingredients and cooking methods corresponding to the eating actions and ingredients to the user.

[0021] Alternatively, the information processing device 100 may be implemented on a user terminal, specifically by an application, software, or program installed on the user terminal that implements the above-described food experience provision process. In this case, as shown in Figure 3, the user operates a food experience provision app installed on the user terminal 100 to capture images related to the meal, such as video of the meal, using the camera function of the user terminal 100. Based on the captured images, the food experience provision app determines the user's eating actions and ingredients during the meal, and determines content corresponding to the determined eating actions and ingredients. The user terminal 100 then displays content related to the ingredients and cooking methods corresponding to the eating actions and ingredients to the user.

[0022] Thus, the information processing device 100 can acquire images related to the user's meal, determine content corresponding to the user's eating actions and ingredients based on the acquired images, and provide the user with interactive dining experience content.

[0023] Here, the information processing device 100 may have a hardware configuration such as that shown in Figure 4. That is, the information processing device 100 has a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, all interconnected via bus B.

[0024] The programs or instructions that implement various functions and processes in the information processing device 100 may be stored on a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or flash memory. When the storage medium is set in the drive device 101, the programs or instructions are installed from the storage medium to the storage device 102 or memory device 103 via the drive device 101. However, the programs or instructions do not necessarily have to be installed from the storage medium; they may also be downloaded from any external device via a network or the like.

[0025] The storage device 102 is implemented by a hard disk drive or the like, and stores files, data, etc., used to execute the installed program or instructions, along with the installed program or instructions.

[0026] The memory device 103 is implemented using random access memory, static memory, etc., and when a program or instruction is activated, it reads the program or instruction, data, etc. from the storage device 102 and stores it. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory storage medium.

[0027] The processor 104 may be implemented by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuits, etc., which may consist of one or more processor cores, and executes various functions and processes of the information processing device 100 according to data such as programs, instructions, and parameters necessary to execute said programs or instructions stored in the memory device 103.

[0028] The user interface (UI) device 105 may consist of input devices such as a keyboard, mouse, camera, and microphone, output devices such as a display, speaker, headset, and printer, and input / output devices such as a touch panel, and realizes an interface between the user and the information processing device 100. For example, the user operates the information processing device 100 by operating a GUI (Graphical User Interface) displayed on the display or touch panel using a keyboard, mouse, etc.

[0029] The communication device 106 is implemented by various communication circuits that perform wired and / or wireless communication processing with external devices, the Internet, LAN (Local Area Network), cellular networks, and other communication networks.

[0030] However, the hardware configuration described above is merely an example, and the information processing device 100 according to this disclosure may be implemented by any other suitable hardware configuration.

[0031] [Information Processing Device] Next, an information processing device 100 according to one embodiment of the present disclosure will be described. Figure 5 is a block diagram showing the functional configuration of the information processing device 100 according to one embodiment of the present disclosure. As shown in Figure 5, the information processing device 100 has an acquisition unit 110, a determination unit 120, and a provision unit 130. Each of the functional units of the acquisition unit 110, the determination unit 120, and the provision unit 130 can be realized by the processor 104 of the information processing device 100 executing a program stored in the memory device 103.

[0032] The acquisition unit 110 acquires images related to the user's meal. Specifically, the user uses the camera function of the user terminal to capture images or videos related to the meal, such as images or videos of the food the user is about to eat and / or is eating, and images or videos of the user's actions from the time the user sits down to eat until the time the user finishes eating and leaves the table. The user then transmits the captured images or videos to the information processing device 100.

[0033] In this embodiment, the acquisition unit 110 may acquire images or videos of the user's meal in real time or near real time so that the food experience content can be interactively determined according to the user's progress in eating and provided to the user terminal. The acquisition unit 110 then acquires images or videos related to the user's meal from the user terminal and transmits the acquired images or videos to the determination unit 120.

[0034] The determination unit 120 determines either or both of the eating actions and the food ingredients based on the image. Here, eating actions may include, for example, the user saying "Itadakimasu" when starting a meal, the action of picking up chopsticks or tableware during the meal, the action of holding a glass of drink, the action of expressing opinions about the food such as "delicious", and the action of saying "Gochisousama deshita" when finishing a meal. Ingredients here may include the dishes served with the meal, ingredients, drinks, tableware, etc.

[0035] Specifically, the determination unit 120 may determine eating behavior by performing skeletal detection on the acquired image. Any known skeletal detection method may be used for skeletal detection. For example, the determination unit 120 can perform skeletal detection on an image or video captured during the user's meal and determine the user's eating behavior based on the detected movement of the user's skeleton. For example, if the determination unit 120 detects a user bringing their hands together, it may determine that the user is saying "Itadakimasu" or "Gochisousama deshita". For example, the determination unit 120 may determine that the first detected hand-bragging motion is the user motion for "Itadakimasu", and the next detected hand-bragging motion is the user motion for "Gochisousama deshita".

[0036] Furthermore, the determination unit 120 may use an ingredient determination model 50, which has been trained to determine the ingredients served in a meal from images related to the meal, to determine the ingredients the user is eating in the meal from the acquired images. That is, as shown in Figure 6, the ingredient determination model 50 may accept images related to the meal, such as images or videos taken during the user's meal, as input, and determine the names of the ingredients, the names of the dishes, etc., of the ingredients captured in the input images.

[0037] Such a food ingredient identification model 50 may be implemented as, for example, a neural network model, and may be trained using a training dataset consisting of training images of food ingredients and labels indicating the names of those food ingredients. Specifically, a training device (not shown) inputs training images from the training dataset into the food ingredient identification model 50 to be trained, and adjusts the parameters of the food ingredient identification model 50 according to backpropagation so that the error between the processing result output from the food ingredient identification model 50 and the corresponding label is reduced. Once such parameter adjustment is complete, the finally acquired food ingredient identification model 50 is made available to the information processing device 100.

[0038] The trained food ingredient identification model 50 may be provided in the information processing device 100, or it may be stored and operated in an external device connected to the information processing device 100. In the latter case, the determination unit 120 transmits the image or video acquired from the acquisition unit 110 to the external device and receives food ingredient information as the inference result of the food ingredient identification model 50 executed by the external device.

[0039] Once the eating behavior and / or food information is determined in this manner, the determination unit 120 provides the determined eating behavior and / or food information to the provision unit 130 as a determination result.

[0040] The provisioning unit 130 determines content according to the determination result and provides the content to the user. For example, the provisioning unit 130 may determine food experience content corresponding to the determined eating action and / or food ingredient based on correspondence information indicating the association between eating action, food ingredient, and food experience content, and provide the food experience content to the user terminal.

[0041] Such correspondence information may be provided, for example, in a table format as shown in Figure 7. According to the illustrated correspondence information, if the determined eating action is "Itadakimasu" (a Japanese phrase said before eating), the corresponding food experience content C01 is determined. For example, food experience content C01 may be video content in which a character or avatar responds with "Please enjoy your meal."

[0042] Alternatively, if the identified food item is crab, the corresponding food experience content C02 is determined. For example, food experience content C02 may be a video explaining how to eat crab. In other words, the content corresponding to the identification result may display how to eat the identified food item.

[0043] Alternatively, if the determined eating action is holding a knife and fork, and the food item is roast chicken, then the food experience content C03 may be video content that explains the characteristics of the chicken, such as its origin, cut, and brand, as well as how to cook roast chicken. In other words, the content corresponding to the determination result may display the cooking method for the determined food item.

[0044] Based on images or videos of each scene during the user's meal, the eating behavior and / or ingredients are determined, and the provision unit 130 provides the user terminal with food experience content corresponding to each scene. The food experience content provided in this way is interactive content that corresponds to the user's eating behavior and / or the ingredients the user eats in each scene, and is in response to the determined eating behavior.

[0045] Specifically, as shown in Figure 8, such interactive food experience content may display a series of food experience contents to the user corresponding to each scene of eating and / or food. For example, the character might respond with "Enjoy your meal" in response to the user saying "Itadakimasu" (thank you for the meal), then display a comment in response to the food offered by the user, and then display another comment in response to the user's eating actions.

[0046] For example, as shown in Figure 9, a user can place a user device such as a tablet on a table and, while eating, view food experience content displayed in response to the eating actions and / or ingredients determined for each scene captured by the user device, and enjoy their meal while interacting with a character.

[0047] In one embodiment, the determination unit 120 may use a food satisfaction determination model 60, which has been trained to determine the user's satisfaction with a meal from images related to the meal, to determine the user's satisfaction with the meal from the acquired images. For example, if a user eats a certain food and finds it delicious, the user may show facial expressions or gestures such as joy or surprise, or eat faster. Alternatively, if a user eats a certain food and finds it unappetizing, the user may show facial expressions or actions such as frowning or shaking their head, or eat slower or stop eating altogether. As shown in Figure 10, the food satisfaction determination model 60 is a machine learning model that accepts images or videos capturing the user's facial expressions and actions when eating food as input, and is trained to output the satisfaction level of the meal (food satisfaction level).

[0048] Such a food satisfaction judgment model 60 may be implemented as, for example, a neural network model, and may be trained using a training dataset consisting of training images of users during meals and user satisfaction ratings (e.g., a 5-point scale such as very satisfied, satisfied, average, dissatisfied, very dissatisfied). Specifically, a training device (not shown) inputs training images from the training dataset into the food satisfaction judgment model 60 to be trained, and adjusts the parameters of the food satisfaction judgment model 60 to be trained according to backpropagation so that the error between the processing result output from the food satisfaction judgment model 60 and the corresponding satisfaction level is reduced. Once such parameter adjustment is complete, the finally acquired food satisfaction judgment model 60 is made available to the information processing device 100.

[0049] The trained food satisfaction determination model 60 may be provided in the information processing device 100, or it may be stored and operated in an external device connected to the information processing device 100. In the latter case, the determination unit 120 transmits the image or video acquired from the acquisition unit 110 to the external device and receives the food satisfaction result as an inference result of the food satisfaction determination model 60 executed by the external device.

[0050] The judgment unit 120 may provide the food satisfaction data obtained in this manner to any appropriate feedback recipient. The feedback recipient may be, for example, the food manufacturer, restaurant, or producer of the ingredient in question. This allows the feedback recipient to know the user's satisfaction with the ingredients they manufacture, cook, or produce, and can use this information for future product development, etc.

[0051] According to the information processing device 100 described above, it is possible to acquire images related to the user's meal, determine content corresponding to the user's eating actions and ingredients based on the acquired images, and provide the user with interactive food experience content.

[0052] [Food experience provision processing] Next, a food experience provision process according to one embodiment of the present disclosure will be described. This food experience provision process can be realized by the information processing device 100, more specifically by the processor 104 of the information processing device 100 executing a program stored in the memory device 103. Figure 11 is a flowchart of the food experience provision process according to one embodiment of the present disclosure.

[0053] As shown in Figure 11, in step S101, the information processing device 100 acquires images related to the user's meal. Specifically, the information processing device 100 acquires images or videos of the user eating, captured by the camera function of the user terminal, from the user terminal.

[0054] In step S102, the information processing device 100 determines the eating behavior and / or ingredients based on the image. Specifically, the information processing device 100 can determine the user's eating behavior by performing skeletal detection on the acquired image. In addition, the information processing device 100 can determine the ingredients the user is eating in the meal from the acquired image by using an ingredient determination model 50 that has been trained to determine the ingredients provided in the meal from images related to the meal.

[0055] In step S103, the information processing device 100 determines content according to the determination result. For example, the information processing device 100 may determine food experience content corresponding to the determined eating action and / or ingredient by referring to correspondence information that shows the association between eating action, ingredients, and food experience content. Here, the food experience content may be a video explaining the cooking method or how to eat the determined ingredient.

[0056] In step S104, the information processing device 100 provides the determined content to the user. For example, the information processing device 100 provides the user terminal with food experience content corresponding to the determined eating behavior and / or ingredients, and plays the food experience content on the user terminal. Specifically, the information processing device 100 can provide the user with food experience content corresponding to the eating behavior and / or ingredients in each scene of the user captured during the meal, and can provide the food experience content to the user as interactive content.

[0057] According to the food experience provision process described above, images related to the user's meal are acquired, and based on the acquired images, content corresponding to the user's eating actions and ingredients is determined, and interactive food experience content can be provided to the user.

[0058] Although embodiments of this disclosure have been described in detail above, this disclosure is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of this disclosure as described in the claims. [Explanation of Symbols]

[0059] 50 Food Ingredient Identification Models 60 Food Satisfaction Assessment Model 100 Information Processing Devices 110 Acquisition Department 120 Judgment section 130 Provision Department

Claims

1. An acquisition unit that acquires images related to meals from the user, A determination unit that determines either or both the eating behavior and the food ingredients based on the aforementioned image, A provisioning unit that determines content according to the judgment result and provides the content to the user, An information processing device having

2. The information processing apparatus according to claim 1, wherein the determination unit determines the eating behavior by performing skeletal detection on the acquired image.

3. The information processing apparatus according to claim 1, wherein the determination unit determines the ingredients that the user is eating in the meal from the acquired image, using an ingredient determination model trained to determine the ingredients that are provided in the meal from an image of the meal.

4. The information processing device according to claim 1, wherein the content displays the cooking method for the determined food ingredient.

5. The information processing device according to claim 1, wherein the content displays how to eat the determined food item.

6. The information processing apparatus according to claim 1, wherein the content is interactive content for the determined eating behavior.

7. The information processing apparatus according to claim 1, wherein the determination unit determines the user's satisfaction with the meal from the acquired images using a food satisfaction determination model trained to determine the user's satisfaction with the meal from images related to the meal.

8. Obtaining images related to meals from users, Based on the aforementioned image, determine either the eating behavior or the food ingredients, The content is determined according to the judgment result, and the content is provided to the user. A method of information processing that a computer performs.

9. Obtaining images related to meals from users, Based on the aforementioned image, determine either the eating behavior or the food ingredients, or both. The content is determined according to the judgment result, and the content is provided to the user. A program that causes a computer to execute something.

Images