Information processing device and information processing method

Abstract

To provide an information processing device and information processing method that make it easier for users to visualize cooking menus. [Solution] In an information processing system in which a server, a refrigerator, a television, and a smartphone are each connected to each other via a network such as the Internet, the server 14 includes an information acquisition unit 144 that acquires inventory information indicating the stock of food and ingredient information that is generated based on cooking videos viewed by the user and indicates the ingredients of the dishes made in the cooking videos, and a control unit 141 that performs a notification to the user based on the comparison result of comparing the inventory information and the ingredient information.

Description

Technical Field

[0007]

[0001] The present disclosure relates to an information processing apparatus and an information processing method.

Background Art

[0002] Conventionally, a technology for proposing menus based on ingredients in a refrigerator has been known. For example, Patent Document 1 discloses a system that displays recommended cooking menus together with completed photos on a display panel based on the ingredient information in the refrigerator.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the system of Patent Document 1, although the cooking menu and the completed photo are displayed, by themselves, it may be difficult for the user to imagine the cooking process and the menu after completion.

[0005] The present disclosure has been made in view of the above problems. An object of the present disclosure is to provide an information processing apparatus and an information processing method that make it easier for a user to imagine a cooking menu.

Means for Solving the Problems

[0006] An information processing apparatus according to an aspect of the present disclosure includes an information acquisition unit that acquires inventory information indicating the inventory of food and material information generated based on a cooking video viewed by a user and indicating the ingredients of the dish made in the cooking video, and a control unit that executes a notification to the user based on a comparison result of comparing the inventory information and the material information.

[0007] An information processing method according to one form of the present disclosure is an information processing method performed by an information processing device, which acquires inventory information indicating the inventory of food products and ingredient information generated based on a cooking video viewed by a user and indicating the ingredients of a dish made in the cooking video, and performs a notification to the user based on the comparison result obtained by comparing the inventory information and the ingredient information. [Effects of the Invention]

[0008] According to this disclosure, it is possible to provide an information processing device and an information processing method that make it easier for users to visualize cooking menus. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing an example of the overall configuration of an information processing system according to one embodiment. [Figure 2] Figure 1 is a functional block diagram showing the schematic configuration of the refrigerator. [Figure 3] Figure 1 is a functional block diagram showing the schematic configuration of the television. [Figure 4] Figure 1 is a functional block diagram showing the schematic configuration of a smartphone. [Figure 5] Figure 1 is a functional block diagram showing the general configuration of the server. [Figure 6] This is a sequence diagram showing the processing procedure for a first example of processing performed by the information processing system in Figure 1. [Figure 7] This figure shows an example of comparing material information and inventory information. [Figure 8] This figure shows an example of a display screen in the display unit of a television. [Figure 9] This figure shows an example of a display screen in the display unit of a smartphone. [Figure 10] This figure shows another example of comparing material information with inventory information. [Figure 11] This figure shows an example of a display screen in the display unit of a television. [Figure 12] This figure shows an example of a display screen in the display unit of a smartphone. [Figure 13] This is a flowchart showing the processing executed by the server in FIG. 1 in the first processing example. [Figure 14] This is a sequence diagram showing the processing procedure of the second processing example executed by the information processing system in FIG. 1. [Figure 15] This is a diagram showing an example of a screen for receiving input of a category. [Figure 16] This is a diagram showing an example of a screen for receiving input of a category. [Figure 17] This is a diagram showing an example of comparison between material information and inventory information. [Figure 18] This is a diagram showing an example of a display screen on the display unit of a television. [Figure 19] This is a diagram showing an example of a display screen on the display unit of a smartphone. [Figure 20] This is a flowchart showing the processing executed by the server in FIG. 1 in the second processing example. [Figure 21] This is a diagram showing a modified example of a display screen on the display unit of a television. [Figure 22] This is a diagram showing a modified example of a display screen on the display unit of a television.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, the same or equivalent components are denoted by the same reference numerals, and descriptions of the same or equivalent components will be omitted as appropriate when they overlap.

[0011] [Overall Configuration] FIG. 1 is a schematic diagram showing an example of the overall configuration of an information processing system 1 according to an embodiment. As shown in FIG. 1, the information processing system 1 includes a refrigerator 11, a television receiver (TV) 12, a smartphone 13, and a server 14. The server 14 is connected to the refrigerator 11, the TV 12, and the smartphone 13 so as to be able to communicate information with each other via a network such as the Internet.

[0012] The arrows in Figure 1 indicate information processing in Information Processing System 1, and the symbols attached to each arrow correspond to the symbols shown in the sequence diagram in Figure 6, which will be described later. The specific details of the information processing in Information Processing System 1 will be described in detail in the explanation of Figure 6, which will be described later.

[0013] Refrigerator 11 stores food in a refrigerated state. Refrigerator 11 is an example of a food storage facility. The information processing system 1 is not limited to refrigerator 11; it can be any food storage facility capable of storing food. Therefore, the food storage facility is not limited to one that stores food in a refrigerated state; it may also be one that stores food at room temperature or in a freezer.

[0014] Television 12 is equipped with a display screen for displaying information. Television 12 is an example of a display device for displaying information. The information processing system 1 is not limited to television 12 and may be equipped with any display device capable of displaying information.

[0015] The smartphone 13 is a terminal device used by the user. The smartphone 13 is an example of a terminal device. The information processing system 1 is not limited to the smartphone 13 and may be equipped with any terminal device. For example, the information processing system 1 may be equipped with a computer or tablet terminal, etc., instead of the smartphone 13, or in conjunction with the smartphone 13.

[0016] The server device 14 executes predetermined processing based on the acquired information. Details of the processing performed by the server device 14 will be described later. The server device 14 may be a so-called physical server device or a virtual server device located in the cloud.

[0017] The information processing system 1 does not necessarily have to be equipped with both the television 12 and the smartphone 13. The information processing system 1 only needs to be equipped with at least one of the television 12 or the smartphone 13.

[0018] Figure 2 is a functional block diagram showing the schematic configuration of the refrigerator 11 shown in Figure 1. As shown in Figure 2, the refrigerator 11 comprises a control unit 111, a storage unit 112, a communication unit 113, an imaging unit 114, and a weight measuring unit 115 as functional blocks.

[0019] The control unit 111 controls and manages the entire refrigerator 11, including each of its functional units. The control unit 111 performs various controls, for example, by operating a control program stored in the memory unit 112. For example, the control unit 111 can be composed of a control device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit).

[0020] The storage unit 112 is a storage medium capable of storing programs and data. The storage unit 112 can be configured as, for example, a semiconductor memory or a magnetic memory. Specifically, the storage unit 112 can be configured as, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory). The storage unit 112 may store, for example, a program for operating the control unit 111.

[0021] The communication unit 113 performs information communication with external devices. In this embodiment, the communication unit 113 performs information communication with the server device 14. The communication unit 113 performs information communication, for example, wirelessly. The communication unit 113 sends and receives various types of information through information communication.

[0022] The imaging unit 114 is a device capable of capturing images, specifically, for example, a camera. The imaging unit 114 is positioned to capture images of the inside of the refrigerator 11. The refrigerator 11 may be equipped with an appropriate number and arrangement of imaging units 114 so that the entire inside of the refrigerator 11 can be captured.

[0023] The weight measuring unit 115 is a device that measures the weight of food stored inside the refrigerator 11, and specifically, it is a weight sensor, for example. The weight measuring unit 115 is positioned at an appropriate location so as to be able to measure the weight of food stored inside the refrigerator 11. The refrigerator 11 may be equipped with an appropriate number and arrangement of weight measuring units 115 so as to be able to measure the gravity of all food stored inside the refrigerator 11.

[0024] Figure 3 is a functional block diagram showing the schematic configuration of the television 12 shown in Figure 1. As shown in Figure 3, the television 12 comprises a control unit 121, a storage unit 122, a communication unit 123, a display unit 124, an input unit 125, and an audio output unit 126 as functional blocks. For example, the control unit 121, the storage unit 122, and the communication unit 123 can be provided in a set-top box (STB).

[0025] The control unit 121 controls and manages the entire television 12, including each of its functional units. The control unit 121 performs various controls, for example, by operating a control program stored in the memory unit 122. For example, the control unit 121 can be configured using a control device such as a CPU or MPU.

[0026] The storage unit 122 is a storage medium capable of storing programs and data. The storage unit 122 can be configured as, for example, a semiconductor memory or a magnetic memory. Specifically, the storage unit 122 can be configured as, for example, an EEPROM. The storage unit 122 may store, for example, a program for operating the control unit 121.

[0027] The communication unit 123 performs information communication with external devices. In this embodiment, the communication unit 123 performs information communication with the server device 14. The communication unit 123 performs information communication, for example, wirelessly. The communication unit 123 sends and receives various types of information through information communication. The communication unit 123 may also include an antenna capable of receiving television broadcast signals. The antenna 82 is, for example, a parabolic antenna or a Yagi antenna.

[0028] The display unit 124 is a device that displays images. The display unit 124 is composed of a known display such as a liquid crystal display (LCD), an organic electro-luminescence display (OELD), or an inorganic electro-luminescence display (IELD). The display unit 124 displays various information under the control of the control unit 121. For example, the display unit 124 displays video based on a television broadcast signal based on the control of the control unit 121.

[0029] The input unit 125 is a mechanism capable of receiving user input for the television 12. The input unit 125 is composed of, for example, a group of operation buttons provided on the main body of the television 12. The input unit 125 also accepts signals transmitted from a remote controller (hereinafter also referred to as "remote control") for remotely operating the television 12.

[0030] The sound output unit 126 is a device that outputs sound. The sound output unit 126 is composed of, for example, a speaker that outputs sound. The sound output unit 126 outputs sound based on a television broadcast signal, for example, based on control by the control unit 121.

[0031] Figure 4 is a functional block diagram showing the schematic configuration of the smartphone 13 shown in Figure 1. As shown in Figure 4, the smartphone 13 comprises a control unit 131, a storage unit 132, a communication unit 133, a display unit 134, an input unit 135, and an audio output unit 136 as functional blocks.

[0032] The control unit 131 controls and manages the entire smartphone 13, including each of its functional units. The control unit 131 performs various controls, for example, by operating a control program stored in the memory unit 132. For example, the control unit 131 can be configured using a control device such as a CPU or MPU.

[0033] The storage unit 132 is a storage medium capable of storing programs and data. The storage unit 132 can be configured as, for example, a semiconductor memory or a magnetic memory. Specifically, the storage unit 132 can be configured as, for example, an EEPROM. The storage unit 132 may store, for example, a program for operating the control unit 131.

[0034] The communication unit 133 performs information communication with external devices. In this embodiment, the communication unit 133 performs information communication with the server device 14. The communication unit 133 performs information communication, for example, wirelessly. The communication unit 133 sends and receives various types of information through information communication.

[0035] The display unit 134 is a device that displays images. The display unit 134 is composed of a known display such as an LCD, OLED, or IELD. The display unit 134 displays various information under the control of the control unit 131. For example, the display unit 134 displays video based on content data acquired via the Internet, based on the control of the control unit 131.

[0036] The input unit 135 is a mechanism capable of receiving user input to the smartphone 13. The input unit 135 is composed of, for example, a group of operation buttons provided on the main body of the smartphone 13. If the terminal device is not a smartphone 13 but a computer or the like, the input unit 135 may be composed of, for example, a keyboard. Also, if the display unit 134 is equipped with a touch sensor, the touch sensor provided on the display unit 134 may function as the input unit 135.

[0037] The sound output unit 136 is a device that outputs sound. The sound output unit 136 is composed of, for example, a speaker that outputs sound. The sound output unit 136 outputs sound based on content data acquired via the Internet, for example, based on control by the control unit 131.

[0038] Figure 5 is a functional block diagram showing the schematic configuration of the server 14 shown in Figure 1. As shown in Figure 5, the server 14 comprises a control unit 141, a storage unit 142, a communication unit 143, and an information acquisition unit 144 as functional blocks.

[0039] The control unit 141 controls and manages the entire server 14, including each of its functional units. The control unit 141 performs various controls, for example, by operating a control program stored in the memory unit 142. For example, the control unit 141 can be configured using a control device such as a CPU or MPU.

[0040] The storage unit 142 is a storage medium capable of storing programs and data. The storage unit 142 can be configured as, for example, a semiconductor memory or a magnetic memory. Specifically, the storage unit 142 can be configured as, for example, an EEPROM. The storage unit 142 may store, for example, a program for operating the control unit 141.

[0041] The communication unit 143 performs information communication with external devices. In this embodiment, the communication unit 143 performs information communication with the refrigerator 11, the television 12, and the smartphone 13. The communication unit 143 performs information communication, for example, wirelessly.

[0042] The information acquisition unit 144 acquires information. Specifically, the information acquisition unit 144 acquires inventory information and material information, which will be described later. For example, the information acquisition unit 144 functions as part of the communication unit 143 and acquires information from the refrigerator 11, television 12, and smartphone 13. Alternatively, the information acquisition unit 144 may acquire information stored in the storage unit 142.

[0043] <Example 1 of processing> Next, we will describe some examples of processing performed in the information processing system 1. Figure 6 is a sequence diagram showing the processing procedure of the first example of processing performed by the information processing system 1 in Figure 1.

[0044] In the first processing example, the refrigerator 11 generates inventory information (step S11). The inventory information is information indicating the stock of food, and in this example, it is information about the food stored in the refrigerator 11. The inventory information includes, for example, the type of food and the stock quantity for each type of food. As an example, the inventory information may be expressed as, for example, "onion: 1" and "milk: 75 ml".

[0045] The refrigerator 11 can generate inventory information by, for example, performing image analysis of the image of the inside of the refrigerator 11 captured by the imaging unit 114 to identify the food stored inside the refrigerator 11 and its quantity. Alternatively, the refrigerator 11 may further use, for example, the weight of the food measured by the weight measuring unit 115 to identify the amount of inventory included in the inventory information. The refrigerator 11 may also generate inventory information using known methods other than those described herein.

[0046] The refrigerator 11 generates inventory information, for example, by periodically executing step S11. Alternatively, the refrigerator 11 may generate inventory information when certain conditions are met. These conditions are those that may cause the inventory information to change, such as detecting the opening or closing of the refrigerator 11 door. The refrigerator 11 can update the generated inventory information by repeatedly generating it.

[0047] The refrigerator 11 sends the inventory information generated in step S11 to the server 14 (step S12). As a result, the server 14 acquires the inventory information. The refrigerator 11 may, for example, periodically send the inventory information to the server 14, or it may send the inventory information to the server 14 when the refrigerator 11 generates (updates) the inventory information. The server 14 may store the acquired inventory information in, for example, the storage unit 142.

[0048] The television 12 outputs a cooking video based on user input, for example (step S13). For example, the television 12 may output a cooking program or play a cooking video available on the internet.

[0049] The television 12 generates ingredient information based on the cooking video it is outputting (step S14). The ingredient information is generated based on the cooking video the user is watching and indicates the ingredients for the dish made in the cooking video. The ingredient information includes, for example, the type of ingredient and the required amount of each type of ingredient. As an example, the ingredient information may be expressed in a similar manner to inventory information, such as "onion: 1" and "milk: 75 ml".

[0050] The television 12 generates ingredient information by performing image OCR (Optical Character Recognition / Reader) on an image showing ingredients from among the images displayed on the display unit 124, and reading the information about the ingredients. The ingredient information may further include the name of the dish, the number of people that can be made with the required amount of ingredients included in the ingredient information, and the cooking procedure.

[0051] The television 12 transmits the ingredient information generated in step S14 to the server 14 (step S15). As a result, the server 14 obtains the inventory information. The refrigerator 11 transmits the inventory information to the server 14, for example, when it determines that ingredient information has been generated. The timing of determining that ingredient information has been generated can be appropriately determined, for example, when the reading of characters in the image showing the ingredients is completed, when the display of images related to the ingredients in the cooking video has finished, or when the cooking video has finished.

[0052] Server 14 may store the acquired material information, for example, in a storage unit 142. When the material information includes a dish name, the materials required for the dish indicated by that dish name are stored in the storage unit 142. By repeating this process, information about dishes and their ingredients is accumulated in the storage unit 142 of Server 14.

[0053] Server 14 compares the acquired inventory information with the material information (step S16). Through this comparison, Server 14 determines, for example, whether all the materials indicated by the material information are included in the food items indicated by the inventory information. In this case, Server 14 may compare the types of materials included in the material information with the types of food items included in the inventory information.

[0054] Figure 7 shows an example of comparing material information and inventory information. In the table shown in Figure 7, the left column is an example of material information, and the right column is an example of inventory information. That is, in the table shown in Figure 7, the left column lists the types of ingredients used in the dishes made in the cooking video, and the right column lists the types of food stored inside the refrigerator 11. The control unit 141 of the server 14 determines whether all the types of ingredients shown in the material information in the left column of Figure 7 are included in the food items shown in the inventory information in the right column. In the example shown in Figure 7, all the types of ingredients shown in the material information in the left column of Figure 7 are included in the food items shown in the inventory information in the right column, so the control unit 141 determines that all the ingredients shown in the material information are included in the food items shown in the inventory information.

[0055] Server 14 notifies the user based on the comparison results in step S16. For example, Server 14 sends comparison result information to TV 12 indicating the comparison results (step S17). The comparison result information is, for example, information on whether the dish made in the cooking video output in step S13 can be made with the food stored inside the refrigerator 11 (stock food). In this case, the comparison result information indicates whether the dish made in the cooking video can be made with the stock food.

[0056] The TV 12 notifies the user based on the comparison result information. For example, the TV 12 displays information on the display unit 124 (step S18). The TV 12 displays, for example, cooking information about the dish made in the cooking video. The cooking information includes, for example, the name of the dish, ingredient information (types of ingredients and required quantities), an image of the dish, and information indicating that the dish can be made. Figure 8 is a diagram showing an example of the display screen on the display unit 124 of the TV 12. For example, as shown in Figure 8, the TV 12 displays a display screen showing cooking information on the display unit 124. In the example in Figure 8, the information indicating that the dish can be made is expressed by the sentence, "It can be made with the ingredients in the oven." Note that the display screen is not limited to the example in Figure 8, and can be any screen that the user can recognize. For example, the information indicating that the dish can be made is not limited to the sentence shown in Figure 8. The information indicating that the dish can be made may be displayed, for example, by a pop-up display on the screen.

[0057] In this way, by displaying the results on the display unit 124, the user can be notified based on the comparison results. However, the notification does not necessarily have to be made by displaying the results on the display unit 124; it may be made in other ways. For example, the notification may be made by voice guidance from the sound output unit 126.

[0058] Television 12 may display the information shown in Figure 8 only if the dish made in the cooking video can be made with the food items in stock. Alternatively, server 14 may transmit comparison result information only if it determines in step S16 that all of the ingredients indicated by the ingredient information are included in the food items indicated by the stock information.

[0059] Referring again to Figure 6, the server 14 may send comparison result information indicating the comparison result to the smartphone 13 (step S19). Based on the comparison result information, the smartphone 13 displays information on the display unit 134 (step S20). The smartphone 13 displays, for example, cooking information. The cooking information includes, for example, the name of the dish, ingredient information (types of ingredients and required quantities), and information indicating that the dish can be made. As shown in Figure 9, the smartphone 13 displays a display screen showing the cooking information on the display unit 134. The smartphone 13 may also display an image of the dish on the display unit 134, similar to the example shown in Figure 8.

[0060] The server 14 may perform the display on either the television 12 or the smartphone 13, or both. Therefore, the server 14 may perform either or both of steps S17 and S19, or both of steps S17 and S19. If the server 14 performs both steps S17 and S19, either or both of steps S18 and S20 may be performed.

[0061] In step S16 of Figure 6, the server 14 may further compare the required quantity with the inventory quantity, in addition to comparing the types of ingredients included in the ingredient information with the types of food included in the inventory information. Figure 10 shows another example of comparing ingredient information with inventory information, specifically an example of further comparing the required quantity with the inventory quantity. In the table shown in Figure 10, the two left columns are an example of ingredient information, and the two right columns are an example of inventory information. Specifically, the two left columns list the types of ingredients and required quantities for the dishes made in the cooking video, and the two right columns list the types of food stored inside the refrigerator 11 and their inventory quantities. The control unit 141 of the server 14 determines whether all the types of ingredients shown in the ingredient information on the left column of Figure 7 are included in the ingredients shown in the inventory information on the right column, and whether the inventory quantity of each type of food reaches the required quantity of the corresponding ingredient type.

[0062] In the example shown in Figure 10, all the types of ingredients listed in the left column of the material information are included in the ingredients listed in the right column of the inventory information. On the other hand, while the inventory of all food types except milk reaches the required amount for the corresponding ingredient type, the inventory of milk does not reach the required amount for the corresponding ingredient type and is therefore insufficient. In other words, in the example shown in Figure 10, the inventory of milk is 75 ml, while the required amount is 150 ml, resulting in a shortage.

[0063] In this case, in steps S17 and S19 of Figure 6, the server 14 transmits, as comparison result information, information on the type of ingredient that is lacking, or information on the quantity of food that can be made with the available stock. The information on the type of ingredient that is lacking may include the type of ingredient that is lacking and the amount that is lacking. In the example of Figure 10, the type of ingredient that is lacking is "milk," and the amount that is lacking is "75 ml." The type of ingredient and the amount that is lacking can be determined by the control unit 141 of the server 14 by comparing the ingredient information with the stock information. The information on the quantity of food that can be made with the available stock indicates how many servings of the dish made in the cooking video can be made with the amount of food stored inside the refrigerator 11. The information on the quantity of food that can be made with the available stock can be calculated by the control unit 141 of the server 14 based on the amount of stock relative to the required amount of the type of food that is lacking. The server 14 may transmit both the information on the type of ingredient that is lacking and the information on the quantity of food that can be made with the available stock as comparison result information.

[0064] In steps S18 and S20, the TV 12 and smartphone 13 respectively notify the user based on the information received from the server 14. For example, the TV 12 and smartphone 13 notify the user of information about the types of ingredients that are lacking, or the amount of food that can be made with the available ingredients. The TV 12 and smartphone 13 may also notify the user of both the types of ingredients that are lacking and the amount of food that can be made with the available ingredients.

[0065] Figures 11 and 12 show examples of display screens in the display unit 124 of a television 12 and the display unit 134 of a smartphone 13, respectively. Specifically, these figures show examples of displaying information about missing ingredients. In the display screen examples shown in Figures 11 and 12, compared to the display screens shown in Figures 8 and 9, information about missing ingredients and information about the amount of food that can be made with the available ingredients are further displayed. In the examples shown in Figures 11 and 12, the information about missing ingredients is expressed by the sentence, "Missing ingredient: Milk 75ml." The information about the amount of food that can be made with the available ingredients is expressed by the sentence, "Two servings can be made with the ingredients in the refrigerator."

[0066] Figure 13 is a flowchart showing the processes executed by the server 14 in Figure 1 in the first processing example. As shown in Figure 13, the server 14 acquires inventory information using the information acquisition unit 144 (step S21). The server 14 also acquires material information using the information acquisition unit 144 (step S22). The order in which steps S21 and S22 are executed does not matter. Therefore, steps S21 and S22 may be executed simultaneously, or step S22 may be executed before step S21. Steps S21 and S22 correspond to steps S12 and S15 in Figure 6, respectively, and are processes executed by the server 14.

[0067] The control unit 141 of the server 14 compares the acquired inventory information with the material information (step S23). Then, the control unit 141 transmits the comparison result information to the television 12 or smartphone 13 (step S24). This allows the control unit 141 to notify the user based on the comparison result using the television 12 or smartphone 13. Note that the processing in steps S23 and S24 is the same as steps S16 and S17 in Figure 6, respectively, so a detailed explanation is omitted here.

[0068] <Example 2 of processing> Next, we will describe the second processing example. Figure 14 is a sequence diagram showing the processing procedure of the second processing example executed by the information processing system 1 in Figure 1.

[0069] In the second processing example, the server 14 stores recipe information indicating a recipe for a predetermined dish in the storage unit 142 beforehand. The storage unit 142 can store multiple recipe information entries. The predetermined dish may be a representative or typical dish. The recipe information includes, for example, information on the ingredients of the dish and the procedure for how to make it. Each recipe information entry is associated with a category of dish. That is, the storage unit 142 stores multiple recipe information entries associated with a predetermined category of dish. The classification of categories can be determined as appropriate; for example, there can be three categories: "Japanese food," "Western food," and "Chinese food." As an example, the recipe information for "curry" is associated with "Western food," and the recipe information for "nikujaga" (meat and potato stew) is associated with "Japanese food."

[0070] Furthermore, in the second processing example, a predetermined dish category is associated with the user before the server 14 performs a comparison between the inventory information and the ingredient information. That is, the storage unit 142 of the server 14 stores the dish category and the user in association. The nature of the dish category can be determined as appropriate, for example, it can be a category of the user's favorite dishes.

[0071] For example, when a user starts using the information processing system 1, they perform a process to associate the user with a category of cooking and store it in the storage unit 142 beforehand. In the example shown in Figure 14, steps S31 to S33 perform the process of associating the user with a category of cooking and storing it in the storage unit 142.

[0072] Specifically, the television 12 accepts category input from the user (step S31). For example, the television 12 displays a screen for accepting category input. Figure 15 shows an example of a screen for accepting category input (category selection screen). The television 12 displays the category selection screen shown in Figure 15 on the display unit 124. The category selection screen shown in Figure 15 shows three categories: "Japanese food," "Western food," and "Chinese food." The user selects one or more categories from those displayed on the category selection screen by operating the input unit 125 of the television 12 or by using the remote control. For example, the user selects their preferred category. The television 12 can accept category input by receiving the user's input for selecting a category.

[0073] Furthermore, category input may be performed based on multiple levels. For example, in the category selection screen shown in Figure 15, after the user selects one of "Japanese food," "Western food," and "Chinese food," the display unit 124 may display multiple categories one level below the selected category. For example, after the user selects "Japanese food," the display unit 124 may display categories such as "Meat dishes," "Fish dishes," and "Vegetable dishes." In this case, the user selects one or more categories from the displayed categories one level below. Category input may be performed along multiple levels in this manner.

[0074] Figure 15 and the above-mentioned explanation of categories are merely examples. The classification of categories is not limited to the above example and may be determined as appropriate. For example, categories may be classified by dish name. Furthermore, category input is not necessarily limited to selecting from multiple categories; other methods may be used. For example, categories may be freely entered by the user.

[0075] When TV 12 receives a category input, it sends category information indicating the input category to server 14 (step S32). The category information includes user information and information about the input category. User information is information that can identify the user, such as a user ID or username.

[0076] Based on the acquired category information, the control unit 141 of the server 14 associates the user indicated by the user information with the category and stores it in the storage unit 142.

[0077] Note that the acceptance of category input as described in step S31 may also be performed by the smartphone 13. For example, the category selection screen shown in Figure 16 is displayed on the display unit 134 of the smartphone 13. The smartphone 13 accepts category input from the user in the same procedure as the television 12 described above, and sends the category information to the server 14 in the same manner as in step S32.

[0078] Then, in the information processing system 1, inventory information and material information are transmitted to the server 14 through the processing in steps S11 to S15. The details of the processing in steps S11 to S15 may be the same as in the first processing example.

[0079] Server 14 compares the acquired inventory information with the material information (step S16). Similar to the first processing example, Server 14 determines, for example, whether all the materials indicated by the material information are included in the food indicated by the inventory information. If all the materials indicated by the material information are included in the food indicated by the inventory information, Server 14 notifies the user of cooking information regarding the dish, similar to the case described in the first processing example.

[0080] In the second processing example, if at least some of the types of ingredients included in the ingredient information are not included in the types of food included in the inventory information, the control unit 141 of the server 14 extracts from among the multiple recipe information stored in the storage unit 142 recipe information in which at least some of the ingredients are common to the types of ingredients included in the ingredient information and all of the ingredients are included in the inventory information.

[0081] Figure 17 shows an example of comparing material information and inventory information. In the table shown in Figure 7, the left column is an example of material information, and the right column is an example of inventory information. The control unit 141 of server 14 determines whether all of the material types shown in the material information in the left column of Figure 7 are included in the ingredients shown in the inventory information in the right column. In the example shown in Figure 17, among the material types shown in the material information in the left column of Figure 17, "stew roux" is not included in the inventory information.

[0082] In this case, the control unit 141 extracts from among the multiple recipe information pre-stored in the storage unit 142 the recipe information in which at least some of the ingredients in the recipe information are common to the types of ingredients included in the ingredient information, and all of the ingredients in the recipe information are included in the inventory information. For example, suppose that recipe information for "curry" is pre-stored. Suppose that the ingredients in the "curry" recipe information are the same as those in the ingredient information, except that they do not include "stew roux" but do include "curry roux". Also, as shown in Figure 17, suppose that "curry roux" is among the types of food included in the inventory information. In this case, the "curry" recipe information pre-stored corresponds to recipe information in which at least some of the ingredients in the recipe information are common to the types of ingredients included in the ingredient information, and all of the ingredients in the recipe information are included in the inventory information. Therefore, the control unit 141 extracts the "curry" recipe information.

[0083] In extracting recipe information, the control unit 141 may extract recipe information from among the multiple recipe information stored in the storage unit 142 that has at least some of the ingredients in common with the dish made in the cooking video, and that is associated with a category related to the user. In this case, the server 14 obtains user information of the user watching the cooking video from the television 12. The television 12 transmits user information to the server 14 by, for example, identifying the user based on a predetermined input or identifying the user watching the television 12 using a camera (not shown).

[0084] Here, for example, let's assume that the user has been associated with the category "Western food" through the processing from steps S31 to S33. Let's also assume that the recipe information for "curry" is associated with the category "Western food". In this case, the control unit 141 extracts from the multiple recipe information stored in the memory unit 142 the recipe information that has at least some of the ingredients in common with the dish made in the cooking video and is associated with the category "Western food" that is associated with the user. Here, since the recipe information for "curry" is associated with the category "Western food", the control unit 141 extracts the recipe information for "curry" that is associated with "Western food". If the user is associated with the category "Japanese food" or "Chinese food", the control unit 141 extracts the recipe information associated with "Japanese food" or "Chinese food" that is associated with the user, rather than the recipe information for "curry" that is associated with "Western food". In this way, the control unit 141 can extract recipe information for the category associated with the user. If the categories are associated according to the user's preferences, the control unit 141 can extract recipe information that matches the user's preferences.

[0085] Furthermore, when extracting recipe information, the control unit 141 may extract from among the multiple recipe information stored in the storage unit 142 recipe information that has at least some of the ingredients in common with the dish made in the cooking video, and that also has a common category with the dish made in the cooking video. In this case, for example, in step S14, when ingredient information is generated based on the cooking video, the category of the dish made in the cooking video is determined. The category can be determined using appropriate information. For example, the television 12 determines the category based on the string displayed in the cooking video. As an example, if the string "Western-style dishes you can make at home" is displayed in the cooking video, the television 12 determines the category of the dish made in that cooking video to be "Western-style dishes". Alternatively, the television 12 may estimate the category using a predetermined algorithm from the cooking process and ingredient information. The information on the category of the dish made in the cooking video is transmitted from the television 12 to the server 14. Based on the acquired information, server 14 can extract recipe information that shares at least some of the ingredients with the dish made in the cooking video, and that also shares the same category as the dish made in the cooking video. For example, if the category of the dish made in the cooking video is "Western food," control unit 141 will extract recipe information for "curry," which shares the same category. In this way, control unit 141 can extract recipe information that has a common category with the dish made in the cooking video that the user is watching, i.e., recipes that have a certain degree of similarity.

[0086] Server 14 notifies the user based on the comparison results in step S16. Server 14 transmits comparison result information to the television 12 (step S17). Similar to the first processing example, the comparison result information can be information on whether or not the dish made in the cooking video output in step S13 can be made with the food stored inside the refrigerator 11 (stock food). In the second processing example, if recipe information is extracted in step S16 as described above, the extracted recipe information (the "curry" recipe information in the above example) can be used as the comparison result information.

[0087] The TV 12 performs a notification to the user based on the comparison result information. For example, the TV 12 displays information on the display unit 124 (step S18). The TV 12 displays, for example, cooking information about the dish made in the cooking video. The TV 12 may also perform a display about the recipe information extracted by the server 14. For example, as shown in Figure 18, the TV 12 displays information indicating that the dish shown in the recipe information can be made. In the example in Figure 18, the information indicating that the dish shown in the recipe information can be made is expressed by the sentence "Curry can be made." Note that the display screen is not limited to the example in Figure 8, and can be any screen that the user can recognize the content of. The information indicating that the dish shown in the recipe information can be made may be displayed, for example, by a pop-up display on the screen.

[0088] Similar to the first processing example, the server 14 may transmit comparison result information to the smartphone 13 (step S19). The smartphone 13 displays information on the display unit 134 based on the comparison result information (step S20). The smartphone 13 displays the recipe information extracted by the server 14, for example, as shown in Figure 19. Similar to the first processing example, the server 14 may have the display performed on either the television 12 or the smartphone 13, or both.

[0089] Figure 20 is a flowchart showing the process that server 14 in Figure 1 executes in the second processing example. Similar to the first processing example, server 14 acquires inventory information (step S21), acquires material information (step S22), and compares the acquired inventory information with the material information (step S23).

[0090] In the second processing example, the control unit 141 of the server 14 determines whether all of the materials indicated by the material information are included in the food indicated by the inventory information (step S41).

[0091] If the control unit 141 determines that all the ingredients indicated by the ingredient information are included in the food indicated by the inventory information (Yes in step S41), it transmits the cooking information as comparison result information to the TV 12 or smartphone 13 (step S24). This allows the control unit 141 to notify the user based on the comparison result using the TV 12 or smartphone 13.

[0092] On the other hand, if the control unit 141 determines that at least a portion of the ingredients indicated by the ingredient information are not included in the food indicated by the inventory information (No. in step S41), it extracts recipe information from among the multiple recipe information stored in the storage unit 142 in which at least a portion of the ingredients are common to the types of ingredients included in the ingredient information, and all of the ingredients are included in the inventory information. At this time, the control unit 141 may extract recipe information from among the multiple recipe information stored in the storage unit 142 in which at least a portion of the ingredients are common to the dish made in the cooking video, and which is associated with a category associated with the user (step S42). Then, the control unit 141 transmits the extracted recipe information as comparison result information to the TV 12 or smartphone 13 (step S24). As a result, the control unit 141 can use the TV 12 or smartphone 13 to notify the user based on the comparison result.

[0093] <Other display examples> In the above embodiment, the control unit 141 of the server 14 can notify result information in various forms. For example, the control unit 141 can display result information on the television 12 and the smartphone 13 in various forms.

[0094] Figures 21 and 22 show modified examples of the display screen in the display unit of the television 12. The control unit 141 can display on the display screen in different ways the types of materials included in the material information have reached the required inventory level and the types of materials have not reached the required inventory level. For example, suppose the inventory level of "milk" included in the material information has not reached the required level, while the inventory levels of the other types of materials have reached the required level. In this case, the control unit 141 can display "milk" which has not reached the required inventory level in a different way on the display screen (the display unit 124 of the television 12 or the display unit 134 of the smartphone 13), for example, as shown in Figure 21, in a different way from the other types of materials. The display manner can be varied in any way appropriate, for example, by changing the font or font size, changing the display color, or enclosing it in a border. However, the examples of changes to the display manner are not limited to those shown here. In the example shown in Figure 21, only "milk," whose inventory level has not reached the required amount, is displayed in a different manner from the other ingredients by being enclosed in a border. This display makes it easier for users to recognize which ingredients are running low.

[0095] The control unit 141 may display information on the display screen regarding the difference between the required amount and the stock amount for materials whose stock amount does not reach the required amount among the types of materials included in the material information. For example, suppose the required amount of "milk" is 150 ml, but the stock amount is 75 ml. In this case, the control unit 141 can display information such as "75 ml shortage," which is information regarding the difference between the required amount and the stock amount, on the display screen (the display unit 124 of the TV 12 or the display unit 134 of the smartphone 13), as shown in Figure 22, for example. By displaying it in this way, the user can more easily recognize the amount of the shortage of the material.

[0096] <Variation> In the above embodiment, ingredient information may be generated based on a cooking video that the user is watching in real time. For example, the television 12 may generate ingredient information for the cooking video that the user is currently watching (playing). In this case, the control unit 141 notifies the user that if all the ingredients indicated by the real-time generated ingredient information are included in the food indicated by the inventory information, it is possible to make the dish. This allows the user to know whether the dish made in the cooking video they are currently watching can be made with the food stored in the refrigerator 11.

[0097] In the above embodiment, it was explained that the server 14 performs the comparison between inventory information and material information. However, the comparison between inventory information and material information does not necessarily have to be performed by the server 14. For example, a refrigerator 11, a television 12, or a smartphone 13 may perform the comparison between inventory information and material information. In this case, the inventory information and material information are appropriately transmitted to the device performing the comparison.

[0098] In the above embodiment, it was explained that the refrigerator 11 generates inventory information. However, inventory information does not necessarily have to be generated by the refrigerator 11. For example, a user may check the contents of the refrigerator 11 and input the food stored in the refrigerator 11 using a smartphone 13 or other device.

[0099] In the information processing system 1 described above, the information acquisition unit 144 acquires inventory information indicating the stock of food items and ingredient information generated based on the cooking video viewed by the user, indicating the ingredients used in the cooking video. The control unit 141 then compares the inventory information and the ingredient information and notifies the user based on the comparison result. Therefore, the control unit 141 notifies the user about the dishes that are actually made in the cooking video viewed by the user. As a result, the user can receive notifications about the menus whose cooking process they have watched in the video, making it easier for them to visualize the dishes being prepared.

[0100] In the above embodiment, cooking videos viewed by the user were used as an example, but the genre of video is not limited to cooking videos. For example, the information acquisition unit 144 may acquire ingredient information of dishes displayed in videos of genres other than cooking videos. In this case, well-known technologies such as AI-based image analysis technology may be applied to acquire the ingredient information of dishes in the video. Furthermore, the video referred to here may be a so-called broadcast program or a video distributed via a network, and it does not matter whether it is recorded or real time. Moreover, at least one of the functions of the control unit 141, storage unit 142, communication unit 143, and information acquisition unit 144 of the server 14 may be implemented in the STB of the television 12.

[0101] While this disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions included in each functional part or step can be rearranged in a logically consistent manner, and multiple functional parts or steps can be combined into one or separated. [Explanation of Symbols]

[0102] 1. Information Processing System 11 Refrigerator 12 TV 13 Smartphones 14 Servers 111,121,131,141 Control Unit 112,122,132,142 Storage section 113,123,133,143 Communications Department 114 Imaging Unit 115 Weight measurement section 124,134 Display section 125,135 Input section 126,136 Sound output section 144 Information Acquisition Department

Claims

1. An information acquisition unit acquires inventory information indicating the stock of food products, and ingredient information generated based on cooking videos viewed by the user, which indicates the ingredients of the dishes made in the cooking videos. A control unit that executes a notification to the user based on the comparison result obtained by comparing the inventory information and the material information, An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, wherein the control unit notifies the user of cooking information regarding the cooking if all of the materials indicated by the material information are included in the food indicated by the inventory information.

3. The aforementioned inventory information includes the type of food and the quantity of each type of food in stock. The material information includes the type of material and the required amount for each type of material. The information processing apparatus according to claim 1, wherein the control unit notifies the user of information about the insufficient type of material, or information about the quantity of the dish that can be made with the inventory amount, if all the types of materials included in the material information include all the types of food included in the inventory information, and the inventory quantity of at least some of the types of food included in the inventory information is insufficient to meet the required quantity.

4. It further includes a memory unit that stores multiple recipe information entries for a given dish. The aforementioned inventory information includes the type of food and the quantity of each type of food in stock. The material information includes the type of material and the required amount for each type of material. The information processing apparatus according to claim 1, wherein the control unit, if at least a portion of the types of materials included in the material information are not included in the types of food included in the inventory information, performs a notification regarding a plurality of recipe information in which at least a portion of the materials are common to the types of materials included in the material information and all of the materials are included in the inventory information.

5. The memory unit stores multiple recipe information and the predetermined category of dishes in association with each other. The information processing apparatus according to claim 4, wherein the control unit performs a notification regarding recipe information among the plurality of recipe information, which has at least a portion of the ingredients in common with the dish made in the cooking video, and which is associated with a category associated with the user.

6. The aforementioned inventory information includes the type of food and the quantity of each type of food in stock. The material information includes the type of material and the required amount for each type of material. The information processing apparatus according to claim 1, wherein the control unit displays on the display screen in different ways the types of materials whose inventory has reached the required amount and the types of materials whose inventory has not reached the required amount.

7. The information processing apparatus according to claim 6, wherein the control unit causes the display screen to display information regarding the difference between the required quantity and the inventory quantity for the types of materials whose inventory quantity has not reached the required quantity.

8. The aforementioned material information is generated based on the cooking video that the user watches in real time. The information processing apparatus according to claim 1, wherein the control unit notifies the user of information indicating that it is possible to prepare the dish if all of the materials indicated by the material information are included in the food indicated by the inventory information.

9. An information processing method performed by an information processing device, The system obtains inventory information indicating the availability of food products, and ingredient information generated based on the cooking videos viewed by the user, which indicates the ingredients used in the dishes prepared in those cooking videos. Based on the comparison results obtained by comparing the inventory information and the material information, a notification is sent to the user. Information processing methods.

Images