Program, terminal, and control method of terminal

Abstract

To provide a program, a terminal, and a control method of the terminal, capable of improving convenience for a user in addition of food products to a stock list.SOLUTION: A program causes a processor of a terminal comprising a display part to function as a display control part for displaying by the display part, an addition screen for adding a stored food product which is stored in a refrigerator and is recognized on the basis of a photographic image in the refrigerator, to a stock list which is a list of food products which a user possesses. The display control part displays in list, non-added stored food products which are not added to the stock list among the stored food products recognized on the basis of the photographic image, on the addition screen.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a program, a terminal device, and a control method for a terminal device. [Background technology]

[0002] Patent Document 1 discloses a technique in which a display unit of a mobile terminal displays images stored as inventory information in a storage unit of a control device side by side. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-081762 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a program, a terminal device, and a method for controlling a terminal device that can improve user convenience when adding food items to an inventory list. [Means for solving the problem]

[0005] The program of the present disclosure causes a processor of a terminal device having a display unit to function as a display control unit that displays on the display unit an addition screen for adding stored foods recognized from a photographed image of the refrigerator's interior to an inventory list, which is a list of foods owned by the user, and the display control unit displays a list of unadded stored foods on the addition screen, which are stored foods recognized from the photographed image that have not been added to the inventory list.

[0006] In addition, the terminal device of the present disclosure includes a display unit and a display control unit that displays, on the display unit, an addition screen for adding stored foods recognized from a photographed image showing the interior of the refrigerator to an inventory list, which is a list of foods owned by the user, and the display control unit displays a list of unadded stored foods on the addition screen, which are stored foods recognized from the photographed image that have not been added to the inventory list.

[0007] In addition, the control method of a terminal device in the present disclosure displays an addition screen for adding stored foods recognized from a photographed image showing the interior of the refrigerator to an inventory list, which is a list of foods owned by the user, and displays a list of unadded stored foods, which are stored foods recognized from the photographed image that have not been added to the inventory list, on the addition screen. Effect of the Invention

[0008] The program, the terminal device, and the control method for the terminal device disclosed herein allow a user to easily know which foods recognized from a captured image have not been added to an inventory list, thereby improving the convenience of the user when adding foods to an inventory list. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 shows a configuration of a food management system according to a first embodiment. [Diagram 2] FIG. 1 is a plan view of a refrigerator and an image capturing device according to a first embodiment, as viewed from the front. [Diagram 3] FIG. 1 shows the configuration of a terminal device and a food management server according to a first embodiment. [Figure 4] FIG. 13 is a diagram showing an example of a first inventory screen in the first embodiment. [Diagram 5] FIG. 1 shows an example of a text input field and a search screen according to the first embodiment. [Figure 6] FIG. 1 shows an example of a text input field and a search screen according to the first embodiment. [Figure 7] FIG. 13 shows an example of a search condition change screen in the first embodiment. [Figure 8] FIG. 1 shows an example of a menu screen in the first embodiment. [Figure 9] FIG. 13 is a diagram showing an example of a second inventory screen in the first embodiment. [Figure 10] FIG. 13 shows an example of a query screen according to the first embodiment. [Figure 11] FIG. 13 is a diagram showing an example of a second inventory screen in the first embodiment. [Figure 12] FIG. 1 shows an example of a search screen in the first embodiment. [Figure 13] FIG. 1 shows an example of a menu screen in the first embodiment. [Figure 14] FIG. 13 is a diagram showing an example of a third inventory screen in the first embodiment. [Figure 15] FIG. 1 shows an example of an enlarged display screen in the first embodiment. [Figure 16] FIG. 13 is a diagram showing an example of a first food detail screen in the first embodiment. [Figure 17] FIG. 13 is a diagram showing an example of a second food detail screen in the first embodiment. [Figure 18] FIG. 13 is a diagram showing an example of a use-up confirmation screen in the first embodiment; [Figure 19] FIG. 13 shows an example of an addition screen in the first embodiment. [Figure 20] FIG. 13 shows an example of a correction screen in the first embodiment. [Figure 21] FIG. 13 shows an example of a correction screen in the first embodiment. [Figure 22] FIG. 13 shows an example of an addition screen in the first embodiment. [Figure 23] FIG. 1 shows examples of various databases according to the first embodiment. [Figure 24] FIG. 1 shows examples of various databases according to the first embodiment. [Diagram 25] A flowchart showing the operation of a food management system in the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (The knowledge and other information that formed the basis of this disclosure) At the time when the inventors came up with the present disclosure, there was a technology in which the display unit of a mobile terminal displays images stored as inventory information in the memory unit of a control device side by side. In such a technology for displaying foods in a refrigerator as an inventory list, a mechanism has been considered in which foods are recognized from a photographed image of the interior of the refrigerator and the recognized foods are added to the inventory list by the user. However, when the recognized foods are displayed as a list as in Patent Document 1 when the user adds the recognized foods to the inventory list, the user cannot easily know which foods have not been added to the inventory list. Therefore, the inventors found a problem that there is room for improvement in improving the convenience of the user in the mechanism for adding the recognized foods to the inventory list, and in order to solve the problem, the subject of the present disclosure has been constituted. Therefore, the present disclosure provides a program, a terminal device, and a method for controlling a terminal device that can improve user convenience when adding food items to an inventory list.

[0011] Hereinafter, the embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed explanations than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations of substantially the same configurations may be omitted. It should be noted that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0012] (Embodiment 1) [1-1. Configuration] [1-1-1. Food control system configuration] FIG. 1 is a diagram showing the configuration of a food management system 1000 in the first embodiment. The food management system 1000 is a system that manages food F owned by a user P and food F that was owned by the user P. The food F owned by the user P includes food F whose weight is detected by a weight sensor 1 and food F stored in a refrigerator 2. The food management system 1000 of this embodiment manages food F by, for example, managing the stock amount of food F, the estimated best-before date or expiration date of food F (hereinafter referred to as "estimated expiration date"), the name of food F (hereinafter referred to as "food name"), and the like. The weight sensor corresponds to an example of a "weight detection device."

[0013] Hereinafter, food F whose weight is detected by weight sensor 1 will be referred to as "detection target food" and will be designated with the symbol "KF" and appropriately expressed as "detection target food KF." Also, food F stored in refrigerator 2 will be referred to as "contained food" and will be designated with the symbol "SF" and appropriately expressed as "contained food SF."

[0014] The food management system 1000 of this embodiment comprises two weight sensors 1: a first weight sensor 11 and a second weight sensor 12. Weight sensor 1 is a plate-shaped device, and detects the weight of the detection target food KF placed on its upper surface. The first weight sensor 11 is provided in the refrigerator compartment 21 of the refrigerator 2 installed in the facility H. A container filled with milk F1 is placed on the upper surface of the first weight sensor 11, and the first weight sensor 11 detects the weight of the milk F1 filled in the placed container. The second weight sensor 12 is provided on a shelf 3 set in the facility H. The second weight sensor 12 has a case 4 placed on its upper surface, and detects the total weight of the rice F2 contained in the case 4.

[0015] The number and installation positions of weight sensors 1 provided in food management system 1000 of the present embodiment are merely an example, and are not limited to the number and installation positions shown in FIG.

[0016] The weight sensor 1 communicates with a food management server 5 connected to a network NW. The network NW includes the Internet, a telephone network, and other communication networks. The weight sensor 1 detects the weight of the target food KF at a predetermined period or at a predetermined timing, and transmits first upload data D1 to the food management server 5 every time the weight is detected. The first upload data D1 is data to be uploaded to the food management server 5, and includes the detected weight and a sensor ID (Identification) that uniquely identifies the weight sensor 1. The sensor ID is stored in a predetermined storage area of ​​the weight sensor 1.

[0017] The food management system 1000 includes a refrigerator 2 . The refrigerator 2 includes a main box 20 with an opening at the front. The main box 20 includes a refrigerator compartment 21, an ice making compartment 22, a fresh freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25. These storage compartments are cooled by cold air. A revolving door 21A is provided at an opening 201 at the front of the refrigerator compartment 21. The door 21A includes a revolving right door 21B and a revolving left door 21C. The ice making compartment 22, the fresh freezing compartment 23, the freezer compartment 24, and the vegetable compartment 25 are provided with drawers 22A, 23A, 24A, and 25A, respectively. Each of the drawers 22A to 25A is open at the top when installed. The drawer 25A in this embodiment has two storage cases, a lower case and an upper case disposed above the lower case. Moreover, drawer 24A in this embodiment has three storage cases: a lower case, a middle case arranged on the lower case, and an upper case arranged on the middle case.

[0018] Hereinafter, when there is no need to distinguish between drawers 22A, 23A, 24A, and 25A, the drawers will be designated by the symbol "26A" and appropriately referred to as "drawer 26A."

[0019] The food management system 1000 is equipped with a photographing device 6 that photographs the interior of the refrigerator 2. The photographing device 6 is a device that obtains a photographed image PG by photographing. The photographing device 6 is attached to the top surface 20A of the main box 20 of the refrigerator 2. The photographing device 6 will be described with reference to FIG.

[0020] FIG. 2 is a plan view of the refrigerator 2 and the camera device 6 attached to the refrigerator 2, viewed from the front. FIG. 2 illustrates the X-axis, Y-axis, and Z-axis. The X-axis direction corresponds to the width direction of the camera device 6 and the left-right direction of the camera device 6. The positive direction of the X-axis is to the right. The Y-axis direction corresponds to the front-rear direction of the camera device 6. The positive direction of the Y-axis is forward. The Z-axis direction corresponds to the up-down direction of the camera device 6. The positive direction of the Z-axis is upward.

[0021] Fig. 2 is a plan view of the refrigerator 2 and the image capturing device 6 attached to the refrigerator 2, seen from the front. The refrigerator 2 shown in Fig. 2 has its door 21A in an open state.

[0022] The image capturing device 6 includes a wide-angle camera 60 and a narrow-angle camera 61. When the image capturing device 6 is attached to the refrigerator 2 and the door 21A is in a closed state, the wide-angle camera 60 and the narrow-angle camera 61 are located in front of the door 21A.

[0023] The wide-angle camera 60 has a wider angle than the narrow-angle camera 61. With the imaging device 6 attached to the refrigerator 2, the wide-angle camera 60 captures an image of the refrigerating compartment 21 from above and in front of the refrigerator 2. 2 when viewed from the front of the refrigerator 2. Range RA1 is a range that includes the door pockets of right door 21B and left door 21C and opening 201 of refrigerator compartment 21 when right door 21B and left door 21C are in the maximum open state. In addition, the shooting range of wide-angle camera 60 is a range that includes, in the front-to-rear direction, from the front ends of right door 21B and left door 21C in the open state to the front end of shelf 21D when viewed from the side (right or left) of the refrigerator 2.

[0024] Narrow-angle camera 61 is a camera with a narrower angle than wide-angle camera 60. With imaging device 6 attached to refrigerator 2, narrow-angle camera 61 captures an image of drawer 26A from above and in the front of refrigerator 2. 2 when viewed from the front of refrigerator 2. Range RA2 is a range that includes drawer 26A when viewed from the front of refrigerator 2. In addition, for example, the shooting range of narrow-angle camera 61 is a range that includes drawer 26A when it is pulled out to the maximum in the front-to-rear direction when viewed from the side of refrigerator 2.

[0025] The devices included in the image capturing device 6 are not limited to the wide-angle camera 60 and the narrow-angle camera 61. The image capturing device 6 may further include other devices such as a first distance measuring sensor for detecting the opening and closing of the right door 21B, a second distance measuring sensor for detecting the opening and closing of the left door 21C, a thermopile for detecting the opening and closing of the drawer 26A, a human sensor, an illuminance sensor, and various types of LEDs (Light Emitting Diodes).

[0026] Returning to the description of Fig. 1, the photographing device 6 connects to the network NW and communicates with the food management server 5. The photographing device 6 transmits second upload data D2 to the food management server 5 every time the wide-angle camera 60 captures an image.

[0027] The second upload data D2 is data to be uploaded to the food management server 5. In the second upload data D2, the photography related information of the refrigerator compartment 21, the photography related information of the right door 21B, the photography related information of the left door 21C, and the photography device ID that uniquely identifies the photography device 6 are recorded. The photographing related information for the refrigerator compartment 21 includes a photographed image PG of the interior of the refrigerator compartment 21 photographed by the wide-angle camera 60 and the photographing date and time of the photographed image PG. The photographing related information for the right door 21B includes a photographed image PG of the interior of the right door 21B photographed by the wide-angle camera 60 and the photographing date and time of the photographed image PG. The photographing related information for the left door 21C includes a photographed image PG of the interior of the left door 21C photographed by the wide-angle camera 60 and the photographing date and time of the photographed image PG. The photographing device ID is recorded in a predetermined storage area of ​​the photographing device 6.

[0028] The photographing device 6 transmits the third upload data D3 to the food management server 5 every time the narrow-angle camera 61 captures an image.

[0029] The third upload data D3 is data to be uploaded to the food management server 5. The third upload data D3 includes information related to photography of the storage case of the drawer 26A. The information related to photography of the storage case includes a captured image PG of the top surface of the storage case captured by the narrow-angle camera 61 and the date and time when the captured image PG was captured. In addition, the third upload data D3 records the type of storage case shown in the captured image PG recorded in the photography related information, the type of storage room in which the storage case shown in this captured image PG is located, and the photography device ID.

[0030] The food management system 1000 includes a terminal device 7. The terminal device 7 is a PC (Personal Computer) such as a smartphone, a tablet terminal, etc. An application program related to the management of food F is installed on the terminal device 7, and the terminal device 7 communicates with the food management server 5 by the function of this application program. In the following description, this application program is referred to as the "food management application" and is given the reference number "711." The food management application 711 corresponds to an example of a "program".

[0031] 1, a user P who is inside facility H is indicated by a solid line, and a user P who is outside facility H is indicated by a dotted line. When used by user P who is inside facility H, terminal device 7 communicates with food management server 5 via communication device 8 or without communication device 8. When used by user P who is outside facility H and a communication connection with communication device 8 cannot be established, terminal device 7 communicates with food management server 5 without via communication device 8.

[0032] The communication device 8 is connected to the network NW and communicates with the food management server 5 via the network NW. The communication device 8 functions as an interface device for connecting the weight sensor 1, the photographing device 6, and the terminal device 7 to the network NW. The communication device 8 establishes a local network in the facility H.

[0033] The food management system 1000 includes a food management server 5. The food management server 5 is a device that manages food F. The food management server 5 is connected to a network NW, and processes information using devices connected to the network NW as clients. Note that in each figure, the food management server 5 is represented by a single block, but this does not necessarily mean that the food management server 5 is composed of a single device.

[0034] [1-1-2. Terminal device configuration] FIG. 3 is a diagram showing the configuration of the terminal device 7 and the food management server 5. First, the configuration of the terminal device 7 will be described. The terminal device 7 includes a terminal control device 70, a terminal communication unit 71, and a touch panel 72. The touch panel 72 corresponds to an example of a "display unit."

[0035] The terminal control device 70 includes a terminal processor 700 such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), a terminal memory 710, and an interface circuit to which other devices and sensors are connected. The terminal processor 700 corresponds to an example of a "processor."

[0036] The terminal memory 710 is a memory that stores programs and data. The terminal memory 710 stores a food management application 711 and data to be processed by the terminal processor 700. The terminal memory 710 has a non-volatile storage area. The terminal memory 710 may also have a volatile storage area and configure a work area for the terminal processor 700. The terminal memory 710 is configured, for example, by a ROM (Read Only Memory) or a RAM (Random Access Memory).

[0037] The terminal communication unit 71 includes hardware such as a communication circuit that complies with a predetermined communication standard, and communicates with the food management server 5 under the control of the terminal processor 700 . The touch panel 72 includes a display panel and a touch sensor that is provided over or integral with the display panel.

[0038] The terminal processor 700 reads out and executes a food management application 711 stored in the terminal memory 710, thereby functioning as a terminal communication control unit 701, a display control unit 702, and a reception unit 703.

[0039] The terminal communication control unit 701 communicates with the food management server 5 via the terminal communication unit 71. The display control unit 702 causes a user interface related to management of the food F whose weight is detected by the weight sensor 1 and the food F stored in the refrigerator 2 to be displayed on the touch panel 72. In the following description, this user interface is referred to as an "application UI (User Interface)" and is denoted by the reference numeral "720." The reception unit 703 receives various inputs from the user P via the application UI 720 .

[0040] [1-1-3. App UI configuration] Next, the configuration of the application UI 720 will be described with reference to FIGS. In Fig. 4 to Fig. 23, the upward direction in the figure is indicated by the symbol "UP", the downward direction in the figure is indicated by the symbol "DW", the leftward direction in the figure is indicated by the symbol "L", and the rightward direction in the figure is indicated by the symbol "R". The up-down direction in Fig. 4 to Fig. 23 is a direction approximately parallel to the side of the display surface of the touch panel 72 when the display surface of the touch panel 72 is viewed. Moreover, the left-right direction in Fig. 4 to Fig. 22 is a direction approximately parallel to a side perpendicular to the side of the display surface of the touch panel 72 when the display surface of the touch panel 72 is viewed.

[0041] [1-1-3-1. Configuration of the first inventory screen] The display control unit 702 switchably displays a first inventory screen SG1, a second inventory screen SG2, and a third inventory screen SG3 in the application UI 720. The first inventory screen SG1, the second inventory screen SG2, and the third inventory screen SG3 are inventory screens SG that display food F held by the user P. First, the first stock screen SG1 will be described. FIG. 4 is a diagram showing an example of the first inventory screen SG1.

[0042] The first inventory screen SG1 is a screen that displays an inventory list, which is a list of food items F that the user P owns. The first inventory screen SG1 has a first switching button B1, a second switching button B2, and a third switching button B3. Note that the display positions and designs of the first switching button B1, the second switching button B2, and the third switching button B3 are not limited to those shown in FIG.

[0043] The first switching button B1 is a software button for switching the screen displayed by the application UI 720 to the third inventory screen SG3. When the reception unit 703 receives an operation of selecting the first switching button B1 on the first inventory screen SG1, the display control unit 702 switches the screen displayed by the application UI 720 from the first inventory screen SG1 to the third inventory screen SG3.

[0044] The second switching button B2 is a software button for switching the screen displayed by the application UI 720 to the second inventory screen SG2. When the reception unit 703 receives an operation of selecting the second switching button B2 on the first inventory screen SG1, the display control unit 702 switches the screen displayed by the application UI 720 from the first inventory screen SG1 to the second inventory screen SG2.

[0045] The third switching button B3 is a software button for switching the screen displayed by the application UI 720 to a shopping list screen. When the reception unit 703 receives an operation of selecting the third switching button B3 on the first inventory screen SG1, the display control unit 702 switches the screen displayed by the application UI 720 from the first inventory screen SG1 to the shopping list screen. The shopping list screen is a screen that displays a shopping list. The shopping list is a list that displays a list of food items F that the user P plans to purchase.

[0046] The first inventory screen SG1 has a first area A1. The first area A1 is an area that displays a food icon IC1 for each food F owned by the user P. The food icon IC1 is an icon that indicates the food F. FIG. 4 illustrates an example in which the first area A1 displays 15 food icons IC1. Note that, if there are a large number of food icons IC1 to be displayed, they may be displayed by scrolling downward. Food icon IC1 is an example of a "food image."

[0047] The first area A1 displays the food icon IC1 of the detection target food KF above the food icon IC1 of the stored food SF. Also, the first area A1 displays the food icon IC1 higher for foods F whose approximate expiration date is closer to the present.

[0048] The first area A1 displays the food icon IC1 by changing the state of the food icon IC1 according to the approximate expiration date of the food F. For example, in the first area A1, when the number of days from the present until the estimated expiration date of food F is N or less (N is an integer equal to or greater than 1), the color of the food icon IC1 is made lighter or darker than the original color. Also, for example, in the first area A1, when the number of days from the present until the estimated expiration date of the food F is N or less (N is an integer of 1 or more), the size of the food icon IC1 is made larger or smaller than the original size. Also, for example, in the first area A1, when the number of days from the present until the estimated expiration date of food F is N or less (N is an integer equal to or greater than 1), the food icon IC1 is moved (eg, shaken).

[0049] Also, for example, in the first area A1, the closer the estimated expiration date of the food F is to the present, the lighter or darker the color of the food icon IC1 is made than the original color. Also, for example, in the first area A1, the closer the estimated expiration date of the food F is to the present, the larger or smaller the size of the food icon IC1 is made than the original size.

[0050] For example, the lower the probability of image recognition based on the image recognition result of the food F, the lighter or darker the color of the food icon IC1 in the first area A1 may be than the original color. That is, it is preferable to display the food icon IC1 in a manner different from other food icons IC1. In this case, when the receiving unit 703 receives an operation to select the food icon IC1, the display control unit 702 may display a correction screen FG for the food indicated by the selected food icon IC1. The correction screen FG will be described later. Also, for example, the first area A1 may change the display mode of the food icon IC1 or display it at the top based on the consumption information or nutritional balance of the user P. Also, for example, the first area A1 may change the display mode of the food icon IC1 or display it at the top based on the freshness or hardness information of the food F.

[0051] The first area A1 displays the food icon IC1 by changing the state of the food icon IC1 according to the amount of food F in stock. For example, when the stock amount of food F is less than half of the stock amount of food F, the first area A1 displays a food icon IC1 that is half missing.

[0052] The food icon IC1 displayed in the first area A1 is selectable. When the reception unit 703 receives an operation to select the food icon IC1 on the first inventory screen SG1 that does not display a search screen SEG described later, the display control unit 702 displays a food details screen FDG for the food indicated by the selected food icon IC1. The food details screen FDG will be described later.

[0053] The first inventory screen SG1 has a search button B4 and a menu button B5. In Fig. 4, the search button B4 and the menu button B5 are superimposed on the first area A1, but the display positions of the search button B4 and the menu button B5 are not limited to the positions shown in Fig. 4. Furthermore, the designs of the search button B4 and the menu button B5 are not limited to the designs shown in Fig. 4.

[0054] The search button B4 is a software button for searching for a recipe. When the reception unit 703 receives an operation to select the search button B4, the display control unit 702 displays the text input field C1 and the search screen SEG in a superimposed manner on the first inventory screen SG1. The menu button B5 is a software button for displaying various menus. When the reception unit 703 receives an operation for selecting the menu button B5, the display control unit 702 displays the menu screen MG in a superimposed manner on the first inventory screen SG1.

[0055] FIG. 5 is a diagram showing an example of the text input field C1 and the search screen SEG.

[0056] The text input field C1 is a field for inputting recipe search conditions by text. The accepting unit 703 accepts the input of text in the text input field C1. The search screen SEG is a screen for searching for a recipe.

[0057] The search screen SEG has an operation bar BR1 and a hide button B6. The operation bar BR1 is an image that accepts an operation to change the display area of ​​the search screen SEG. When the accepting unit 703 accepts an operation to select the operation bar BR1 and move the operation bar BR1 upward or downward, the display control unit 702 changes the display area of ​​the search screen SEG in accordance with the moving direction of the operation bar BR1. 5, when the reception unit 703 receives an operation to move the operation bar BR1 upward, the display control unit 702 widens the display area of ​​the search screen SEG so that the search screen SEG can display more recipes. At this time, the display control unit 702 changes the display position of the text input field C1 in accordance with the change in the display area of ​​the search screen SEG so that the position of the text input field C1 is located at the top of the search screen SEG.

[0058] The hide button B6 is a software button for hiding the text input field C1 and the search screen SEG. When the reception unit 703 receives an operation for selecting the hide button HB1, the display control unit 702 hides the text input field C1 and the search screen SEG.

[0059] The search screen SEG has a search condition display area DA1 and a search result display area DA2. The search condition display area DA1 is an area that displays the search conditions for a recipe. When recipe search conditions are set, the search condition display area DA1 displays a search target icon IC2 indicating the search target for each search target that constitutes the set search conditions. When recipe search conditions are not set on the search screen SEG, the search condition display area DA1 displays information that prompts the user to set search conditions for the recipe. For example, as shown in FIG. 5, the search condition display area DA1 displays the character string "Tap to select ingredients."

[0060] The search result display area DA2 is an area that displays the search results for recipes. The search result display area DA2 displays recipes that meet the search criteria indicated in the search criteria display area DA1. More specifically, the search result display area DA2 displays a recipe icon IC3 indicating the recipe for each recipe that meets the search criteria indicated in the search criteria display area DA1. The recipe icon IC3 is a selectable icon. When the receiving unit 703 receives an operation to select the recipe icon IC3, the display control unit 702 displays the recipe corresponding to the selected recipe icon IC3 in the application UI 720. In addition, the search result display area DA2 displays recipes recommended to the user P if a recipe search condition is not set.

[0061] When the search screen SEG is displayed on the first inventory screen SG1, the receiving unit 703 receives an operation to select a food icon IC1. When the receiving unit 703 receives an operation to select a food icon IC1 while the search screen SEG is displayed, the display control unit 702 displays a search target icon IC2 corresponding to the selected food icon IC1 in the search condition display area DA1. When the receiving unit 703 receives an operation to select a food icon IC1 while the search screen SEG is displayed, the display control unit 702 displays a recipe for a dish using the food F indicated by the selected food icon IC1 in the search result display area DA2.

[0062] FIG. 6 is a diagram showing an example of the text input field C1 and the search screen SEG. The search screen SEG shown in FIG. 6 shows a case where recipe search conditions are set. The search screen SEG shown in FIG. 6 shows a case where a food icon IC1 indicating mushrooms and a food icon IC1 indicating beef are selected in the first area A1, and recipes using mushrooms and recipes using beef are included in the search targets. The search screen SEG shown in FIG. 6 also shows a case where time-saving recipes are included in the search targets. Time-saving recipes refer to recipes that aim to reduce the time required for cooking. From the above, the search condition display area DA1 shown in FIG. 6 displays three search target icons IC2, indicating that the search targets that make up the search conditions include recipes using mushrooms, recipes using beef, and time-saving recipes.

[0063] As shown in FIG. 6, when a food icon IC1 is selected while the search screen SEG is displayed, the display control unit 702 superimposes a selected icon IC4 indicating that the food icon IC1 has been selected.

[0064] When displaying a recipe that matches the search conditions shown in the search condition display area DA1, the search result display area DA2 displays a condition change button B7. The condition change button B7 is a software button for changing the search conditions shown in the search condition display area DA1. When the reception unit 703 receives an operation to select the condition change button B7, the display control unit 702 displays a search condition change screen CCG on the first inventory screen SG1.

[0065] When a food icon IC1 is selected while the search screen SEG is displayed, a food icon IC1 of a food F that is compatible with the selected food icon IC1 may be displayed in a state where the display mode is changed from that of the other food icons IC1. This allows the user P to select a compatible food F, improving the convenience of the user P. Also, a food icon IC1 of a food F that is not compatible with the selected food icon IC1 may be displayed in a state where the display mode is changed from that of the other food icons IC1.

[0066] FIG. 7 is a diagram showing an example of the search condition change screen CCG. The search condition change screen CCG has a hide button B8. The hide button B8 is a software button for hiding the search condition change screen CCG. When the reception unit 703 receives an operation of selecting the hide button B8, the display control unit 702 hides the search condition change screen CCG.

[0067] The search condition change screen CCG has a first selection item SK1, a second selection item SK2, a third selection item SK3, and a fourth selection item SK4. The first selection item SK1 is an item for selecting the range of recipes to search. The first selection item SK1 has a first radio button B9 and a second radio button B10. The first radio button B9 is a radio button for setting the range of recipes to search to the entire searchable range. The second radio button B10 is a radio button for setting the range of recipes to search to recipes that can be cooked using only food F owned by the user P.

[0068] The second selection item SK2 is an item for narrowing down the range of recipes to be searched by the type of menu. The second selection item SK2 has icons for each type of menu, such as main dish, side dish, staple food, soup, and sweets. These icons can be selected by the user P.

[0069] The third selection item SK3 is an item for narrowing down the range of recipes to be searched for by type of food F. The third selection item SK3 has an icon for each type of food F, such as meat, fish, vegetables, eggs, etc. These icons can be selected by the user P.

[0070] The fourth selection item SK4 is an item for narrowing down the range of recipes to be searched by cuisine genre, such as Japanese cuisine, Western cuisine, Chinese cuisine, French cuisine, etc. The fourth selection item SK4 has an icon for each cuisine genre. These icons can be selected by the user P.

[0071] The search condition change screen CCG has a setting button B11. The setting button B11 is a software button for adding the content selected in at least one of the first selection item SK1 to the fourth selection item SK4 to the search conditions for the recipe. The setting button B11 may be configured as an inoperable button when no selection is made in any of the selection items. When the reception unit 703 receives an operation to select the setting button B11, the display control unit 702 hides the search condition change screen CCG and updates the display content of the search screen SEG to content that reflects the content selected on the search condition change screen CCG.

[0072] Returning to the explanation of FIG. 4, as described above, when the reception unit 703 receives an operation to select the menu button B5, the display control unit 702 displays the menu screen MG on the first inventory screen SG1.

[0073] FIG. 8 is a diagram showing an example of the menu screen MG. The menu screen MG has a fourth switching button B12, a fifth switching button B13, and a non-display button B14. The menu screen MG has areas other than the fourth switching button B12, the fifth switching button B13, and the non-display button B14 grayed out. Note that the display positions and patterns of the fourth switching button B12, the fifth switching button B13, and the non-display button B14 are not limited to those shown in FIG.

[0074] The fourth switching button B12 is a software button for switching the screen displayed by the application UI 720 to a use-up confirmation screen UCG. When the reception unit 703 receives an operation to select the fourth switching button B12, the display control unit 702 switches the screen displayed by the application UI 720 from the first inventory screen SG1 to the use-up confirmation screen UCG. The use-up confirmation screen UCG will be described later.

[0075] The fifth switching button B13 is a software button for switching the screen displayed by the application UI 720 to an additional screen AG. When the reception unit 703 receives an operation for selecting the fifth switching button B13, the display control unit 702 switches the screen displayed by the application UI 720 from the first inventory screen SG1 to the additional screen AG. The additional screen AG will be described later.

[0076] The hide button B14 is a software button for hiding the menu screen MG. When the reception unit 703 receives an operation for selecting the hide button B14, the display control unit 702 hides the menu screen MG on the first inventory screen SG1.

[0077] [1-1-3-2. Configuration of the second inventory screen] Next, the second inventory screen SG2 will be described. FIG. 9 is a diagram showing an example of the second inventory screen SG2.

[0078] The second inventory screen SG2 is a screen that displays a list of food items F in inventory. The second inventory screen SG2 has a first switching button B1, a third switching button B3, and a sixth switching button B15. Note that the display positions and designs of the first switching button B1, the third switching button B3, and the sixth switching button B15 on the second inventory screen SG2 are not limited to the display positions and designs shown in FIG.

[0079] The sixth switching button B15 is a software button for switching the screen displayed by the application UI 720 to the first inventory screen SG1. When the reception unit 703 receives an operation of selecting the sixth switching button B15 on the second inventory screen SG2, the display control unit 702 switches the screen displayed by the application UI 720 from the second inventory screen SG2 to the first inventory screen SG1.

[0080] The second inventory screen SG2 has a pull-down selection field C2. The selection field C2 is a field for changing the display order of the food information FJ of the stored foods SF among the food information FJ displayed in the second area A2 to the order of the most recent date that the user P put the food F into the refrigerator (hereinafter referred to as the "put-in date"), the order of the shortest estimated expiration date of the food F based on the current date, or the order of the oldest put-in date. The selection field C2 displays a list of display order options and accepts an operation to select one option from the list. When the receiving unit 703 accepts an operation to select one option in the selection field C2, the display control unit 702 changes the display order of the food information FJ displayed in the second area A2 to the display order indicated by the option selected in the selection field C2. Selection column C2 corresponds to an example of a "fourth reception section."

[0081] The second inventory screen SG2 has a second area A2. The second area A2 is an area that displays food information FJ for each food F. The food information FJ has a food icon IC1, food name information FJ1, holding start date information FJ2, estimated expiration date information FJ3, inventory amount information FJ4, a slider SR, a register button B16, and an add button B17. The slider SR corresponds to an example of a "first receiving portion."

[0082] The food name information FJ1 is information that indicates the name of the food. The holding start date information FJ2 is information that indicates the holding start date, which is the date on which the user P started holding the food. The holding start date of the stored food SF is the date of entry into storage. The estimated expiration date information FJ3 is information that indicates the estimated expiration date of the food F. The inventory quantity information FJ4 is information that indicates the inventory quantity of food F. The inventory quantity indicated by the inventory quantity information FJ4 is composed of a numerical value and a unit. The numerical value indicated by the inventory quantity information FJ4 can be changed using the slider SR or the food details screen FDG. In addition, the unit indicated by the inventory quantity information FJ4 can be changed using the food details screen FDG. The slider SR is a UI component that accepts an operation to change the stock amount indicated by the stock amount information FJ4. The registration button B16 is a software button for registering the food F indicated by the corresponding food information FJ as a favorite food F. The add button B17 is a software button for adding the food F indicated by the corresponding food information FJ to the shopping list.

[0083] In the second inventory screen SG2, a food icon IC1 and food name information FJ1 are selectable for each food information FJ. When the reception unit 703 receives an operation to select a food icon IC1 or food name information FJ1 in the second inventory screen SG2, the display control unit 702 displays a food details screen FDG for the food F indicated by the selected food icon IC or food name information FJ1.

[0084] The second area A2 displays the food information FJ of the detection target food KF above the food information FJ of the contained food SF. The second area A2 also displays the food information FJ of the contained food SF in the display order of the options received in the selection box C2.

[0085] The slider SR indicates the stock amount indicated by the stock amount information FJ4 by positioning the control SS at a position corresponding to the stock amount indicated by the stock amount information FJ4. The slider SR accepts an operation to change the numerical value of the stock amount indicated by the stock amount information FJ4 via the control SS. When the slider SR accepts an operation to change the numerical value of the stock amount to zero, the display control unit 702 displays an inquiry screen IG on the second stock screen SG2.

[0086] FIG. 10 is a diagram showing an example of the inquiry screen IG. The inquiry screen IG is a screen for inquiring as to whether or not the food F has been used up. The inquiry screen IG includes a first screen IG1 and a second screen IG2. When the slider SR receives a change to change the stock amount to zero, the display control unit 702 first displays the first screen IG1.

[0087] The first screen IG1 has a non-display button B18, inquiry information IJ, a first selection button B19 for selecting that the food has been discarded, and a second selection button B20 for selecting that the food F has been used up. The first selection button B19 corresponds to an example of a “third reception section.” The second selection button B20 corresponds to an example of a “second reception section.”

[0088] The hide button B19 is a software button for hiding the inquiry screen IG. When the reception unit 703 receives an operation for selecting the hide button B19, the display control unit 702 hides the inquiry screen IG.

[0089] The inquiry information IJ is information inquiring as to whether or not the food F has been used up. In Fig. 11, the character string "Have you used up the pumpkin?" is the inquiry information IJ.

[0090] The first selection button B19 is a software button for selecting "discarded" in response to the inquiry indicated by the inquiry information IJ. The second selection button B20 is a software button for selecting "used up" in response to the inquiry indicated by the inquiry information IJ.

[0091] When the receiving unit 703 receives an operation to select the first selection button B19 or the second selection button B20, the display control unit 702 displays the second screen IG2. The second screen IG2 has information indicating that the food F has been consumed and a non-display button B21. The hide button B21 is a software button that hides the inquiry screen IG on the second inventory screen SG2. When the reception unit 703 receives an operation to select the hide button B21, the display control unit 702 hides the inquiry screen IG on the second inventory screen SG2.

[0092] The information entered on the inquiry screen IG is collected and utilized by at least one of the food management application 711 and the food management server 5. For example, the food management application 711 collects the information entered on the inquiry screen IG and reports to the user P by displaying foods F that are often used up or often wasted. Also, for example, the food management application 711 collects the information entered on the inquiry screen IG and prompts the user P to check the stock of foods F that are often wasted.

[0093] Returning to the explanation of FIG. 9, the second inventory screen SG2 has a selection button B22. The selection button B22 is a software button for selecting the food F to be deleted from the inventory list displayed by the application UI 720. When the reception unit 703 receives an operation to select the selection button B22, the display control unit 702 changes the display content of the second inventory screen SG2.

[0094] FIG. 11 is a diagram showing an example of the second inventory screen SG2 when the SELECT button B22 is selected. As shown in FIG. 11, when the SELECT button B22 is selected, the second inventory screen SG2 displays a CANCEL button B23 instead of the SELECT button B22.

[0095] The cancel button B23 is a software button for ending the selection of the food F to be deleted from the inventory list. When an operation of selecting the cancel button B23 is accepted, the display control unit 702 returns the display content of the second inventory screen SG2 to the content before the selection button B22 was selected.

[0096] As shown in Fig. 11, when the selection button B22 is selected, the food information FJ does not include the registration button B16 or the add button B17 on the second inventory screen SG2. Also, as shown in Fig. 11, when the selection button B22 is selected on the second inventory screen SG2, check boxes CB1 are displayed corresponding to each piece of food information FJ.

[0097] 11, when the SELECT button B22 is selected, the second inventory screen SG2 displays the third area A3. The third area A3 is an area having a DISPOSE button B24, a USE UP button B25, and a DELETE button B26.

[0098] The discard button B24 is a software button for inputting that the food F indicated by the food information FJ corresponding to the checked check box CB1 has been discarded. When the reception unit 703 receives an operation to select the discard button B24, the display control unit 702 deletes the food F indicated by the food information FJ corresponding to the checked check box CB1 from the inventory list.

[0099] The use up button B25 is a software button for inputting that the food F indicated by the food information FJ corresponding to the checked check box CB1 has been used up. When the reception unit 703 receives an operation to select the use up button B25, the display control unit 702 deletes the food F indicated by the food information FJ corresponding to the checked check box CB1 from the inventory list.

[0100] The Delete button B26 is a software button for deleting the food F indicated by the food information FJ corresponding to the checked check box CB1 from the inventory list. When the reception unit 703 receives an operation to select the Delete button B26, the display control unit 702 deletes the food F indicated by the food information FJ corresponding to the checked check box CB from the inventory list.

[0101] When the discard button B24, use up button B25, or delete button B26 is selected on the second inventory screen SG2, the app UI720 may delete the food F from the inventory list after making an inquiry such as on the inquiry screen IG.

[0102] The food F deleted by the discard button B24 and the use up button B25 is collected and utilized by at least one of the food management application 711 and the food management server 5. For example, the food management application 711 collects the food F deleted by the discard button B24 and the use up button B25, and reports to the user P by displaying the food F that is used up a lot or the food F that is discarded a lot. Also, for example, the food management application 711 collects the food F deleted by the discard button B24, and prompts the user P to check the inventory of the food F that is discarded a lot.

[0103] Returning to the explanation of Fig. 9, the second inventory screen SG2 displays a search button B4 and a menu button B5. Note that the display positions and designs of the search button B4 and the menu button B5 on the second inventory screen SG2 are not limited to the display positions and designs shown in Fig. 9.

[0104] When the reception unit 703 receives an operation to select the search button B4, the display control unit 702 displays the text input field C1 and the search screen SEG on the second inventory screen SG2 in the same manner as the text input field C1 and the search screen SEG on the first inventory screen SG1. When the acceptance unit 703 accepts an operation to select the menu button B5, the display control unit 702 displays the menu screen MG in the second inventory screen SG2 in the same manner as the display of the menu screen MG in the first inventory screen SG1.

[0105] FIG. 12 is a diagram showing an example of the text input field C1 and the search screen SEG. FIG. 12 shows a case where a text input field C1 and a search screen SEG are displayed on the second inventory screen SG2.

[0106] As shown in FIG. 12, in the second inventory screen SG2 that displays the text input field C1 and the search screen SEG, a check box CB2 is displayed corresponding to each piece of food information FJ that does not have a registration button B16 or an add button B17.

[0107] The receiving unit 703 receives an operation to select the check box CB2 when the search screen SEG is displayed on the second inventory screen SG2. When the receiving unit 703 receives an operation to select the check box CB2 while the search screen SEG is displayed, the display control unit 702 includes a recipe for food F indicated by food information FJ corresponding to the checked check box CB2 in the search conditions displayed in the search condition display area DA1. Furthermore, when the receiving unit 703 receives an operation to select the check box CB2 while the search screen SEG is displayed, the display control unit 702 displays a recipe for a dish using food F indicated by food information FJ corresponding to the checked check box CB2 in the search result display area DA2.

[0108] FIG. 13 is a diagram showing an example of the menu screen MG. FIG. 13 shows a case where the menu screen MG is displayed on the second inventory screen SG2.

[0109] As shown in Fig. 13, the menu screen MG displayed on the second inventory screen SG2 is the same as the menu screen MG displayed on the first inventory screen SG1. When the reception unit 703 receives an operation to select the hide button B12 in Fig. 13, the display control unit 702 hides the menu screen MG on the second inventory screen SG2.

[0110] [1-1-3-3. Configuration of the 3rd inventory screen] Next, the configuration of the third inventory screen SG3 will be described. FIG. 14 is a diagram showing an example of the third inventory screen SG3.

[0111] The third inventory screen SG3 is a screen for displaying the photographed image PG.

[0112] The third inventory screen SG3 has a hide button B27. The hide button B27 is a software button for hiding the third inventory screen SG3. When the reception unit 703 receives an operation for selecting the hide button B27, the display control unit 702 returns the screen displayed by the application UI 720 to the screen displayed immediately before the transition from the third inventory screen SG3 to the third inventory screen SG3. For example, when the hide button B27 is operated on the third inventory screen SG3 switched from the second inventory screen SG2 by the first switching button B1, the display control unit 702 switches the screen displayed by the application UI 720 from the third inventory screen SG3 to the second inventory screen SG2.

[0113] The third inventory screen SG3 has a fourth area A4. The fourth area A4 is an area for displaying a photographed image PG, the type of which is selected in the fifth area A5.

[0114] Here, the fifth area A5 of the third inventory screen SG3 will be described. The fifth area A5 is an area for changing the type of captured image PG displayed in the fourth area A4. The fifth area A5 has a door selection button B28, a refrigerator compartment selection button B29, a vegetable compartment selection button B30, and a freezer compartment selection button B31. The fifth area A5 also displays a selection bar BR2. The selection bar BR2 is an image indicating which storage compartment the captured image PG displayed in the fourth area A4 is. The captured image PG of the storage compartment may be switched by swiping left and right in the fourth area A4.

[0115] The door selection button B28 is a software button for displaying a captured image PG of the door 21A in the fourth area A4. When the reception unit 703 receives an operation to select the door selection button B28, the display control unit 702 displays the captured image PG of the door 21A in the fourth area A4. When the reception unit 703 receives an operation to select the door selection button B28, the display control unit 702 displays a selection bar BR2 in a position corresponding to the door selection button B28 in the fifth area A5.

[0116] The refrigerator compartment selection button B29 is a software button for displaying a captured image PG of the refrigerator compartment 21 in the fourth area A4. When the reception unit 703 receives an operation to select the refrigerator compartment selection button B29, the display control unit 702 displays the captured image PG of the refrigerator compartment 21 in the fourth area A4. When the reception unit 703 receives an operation to select the refrigerator compartment selection button B29, the display control unit 702 displays a selection bar BR2 in a position corresponding to the refrigerator compartment selection button B29 in the fifth area A5.

[0117] The vegetable compartment selection button B30 is a software button for displaying a captured image PG of the vegetable compartment 25 in the fourth area A4. When the reception unit 703 receives an operation to select the vegetable compartment selection button B30, the display control unit 702 displays the captured image PG of the vegetable compartment 25 in the fourth area A4. When the reception unit 703 receives an operation to select the vegetable compartment selection button B30, the display control unit 702 displays a selection bar BR2 in the fifth area A5 at a position corresponding to the vegetable compartment selection button B30.

[0118] The freezer selection button B31 is a software button for displaying the captured image PG of the freezer compartment 24 in the fourth area A4. When the reception unit 703 receives an operation to select the freezer selection button B31, the display control unit 702 displays the captured image PG of the freezer compartment 24 in the fourth area A4. When the reception unit 703 receives an operation to select the freezer selection button B31, the display control unit 702 displays a selection bar BR2 in the fifth area A5 at a position corresponding to the freezer selection button B31.

[0119] When displaying the captured images PG of the vegetable compartment 25 in the fourth area A4, the display control unit 702 displays the captured images PG arranged for each storage case included in the vegetable compartment 25. When displaying the captured images PG of the freezer compartment 24 in the fourth area A4, the display control unit 702 displays the captured images PG arranged for each storage case included in the freezer compartment 24. FIG. 14 illustrates an example in which the captured images PG of the freezer compartment 24 having three storage cases is displayed. Therefore, in FIG. 14, the third area A3 displays the captured images PG of the three storage cases arranged.

[0120] The fourth area A4 displays, for each photographed image PG, photographing date and time information NJ. The photographing date and time information NJ is information indicating the photographing date and time, and indicates the date and time when the corresponding photographed image PG was obtained.

[0121] The photographed images PG displayed in the fourth area A4 can be selected for each photographed image PG. When the reception unit 703 receives an operation to select one photographed image PG in the fourth area A4, the display control unit 702 displays an enlarged display screen EDG on the third inventory screen SG3.

[0122] FIG. 15 is a diagram showing an example of the enlarged display screen EDG. The enlarged display screen EDG has a hide button B32. The hide button B32 is a software button for hiding the enlarged change screen EDG. When the reception unit 703 receives an operation of selecting the hide button B32, the display control unit 702 hides the enlarged display screen EDG on the third inventory screen SG3.

[0123] The enlarged display screen EDG displays the photographed image PG selected in the third area A3. The enlarged display screen EDG accepts an operation to enlarge the photographed image PG to be displayed. When the reception unit 703 accepts an operation to enlarge the photographed image PG on the enlarged display screen EDG, the display control unit 702 enlarges the photographed image PG on the enlarged display screen EDG. When the reception unit 703 accepts an operation to reduce the photographed image PG on the enlarged display screen EDG, the display control unit 702 reduces the enlarged photographed image PG on the enlarged display screen EDG.

[0124] [1-1-3-4. Food details screen configuration] Next, an example of the food details screen FDG is shown. The food details screen FDG is a screen that displays detailed information about one food F.

[0125] FIG. 16 is a diagram showing an example of the first food details screen FDG1. The first food details screen FDG1 is a food details screen FDG relating to the stored food SF. As shown in FIG. 16, the first food details screen FDG1 has a back button B33, a first switch button B1, a register button B16, an add button B17, and a delete button B34.

[0126] The back button B33 is a software button for returning the screen displayed by the app UI 720 to the screen displayed immediately before transitioning to the first food details screen FDG1. When the reception unit 703 receives an operation of selecting the back button B33 on the first food details screen FDG1, the display control unit 702 returns the screen displayed by the app UI 720 to the screen displayed immediately before transitioning to the first food details screen FDG1.

[0127] The Delete button B34 is a software button for deleting the food F, the details of which are displayed on the first food details screen FDG1, from the inventory list displayed by the application UI720.

[0128] The first food details screen FDG1 has a sixth area A6. The sixth area A6 is an area for displaying detailed information about the stored food SF. The sixth area A6 includes a food icon IC1, food name information FJ1, inventory information FJ4, holding start date information FJ2, estimated expiration date information FJ3, storage location information KJ, and first notification timing information TJ1.

[0129] The storage location information KJ is information that indicates the location where the food F is stored. The first notification timing information TJ1 is information indicating the first notification timing, which is the timing for notifying that the food F has been forgotten to be used.

[0130] The sixth area A6 has a first change button B35, a second change button B36, a third change button B37, a fourth change button B38, and a fifth change button B39.

[0131] The first change button B35 is a software button for changing the numerical value and unit of the inventory quantity indicated by the inventory quantity information FJ4 on the first food details screen FDG1. When the reception unit 703 receives an operation to select the first change button B35, the display control unit 702 displays a screen for changing the numerical value and unit of the inventory quantity indicated by the inventory quantity information FJ4. When at least one of the numerical value and unit of the inventory quantity is changed on the screen, the display control unit 702 reflects the changed content in the inventory quantity indicated by the inventory quantity information FJ4.

[0132] The second change button B36 is a software button for changing the holding start date indicated by the holding start date information FJ2 on the first food details screen FDG1. When the reception unit 703 receives an operation to select the second change button B36, the display control unit 702 displays a screen for changing the holding start date indicated by the holding start date information FJ2. When the holding start date is changed on the screen, the display control unit 702 reflects the changed content in the holding start date indicated by the holding start date information FJ2.

[0133] The third change button B37 is a software button for changing the estimated expiration date indicated by the estimated expiration date information FJ3 on the first food details screen FDG1. When the reception unit 703 receives an operation to select the third change button B37, the display control unit 702 displays a screen for changing the estimated expiration date indicated by the estimated expiration date information FJ3. When the estimated expiration date is changed on the screen, the display control unit 702 reflects the changed content in the estimated expiration date indicated by the estimated expiration date information FJ3.

[0134] The fourth change button B38 is a software button for changing the storage location indicated by the storage location information KJ on the first food details screen FDG1. When the reception unit 703 receives an operation to select the fourth change button B38, the display control unit 702 displays a screen for changing the storage location indicated by the storage location information KJ. When the storage location is changed on this screen, the display control unit 702 reflects the changed content in the storage location indicated by the storage location information KJ. Note that when the storage location information KJ is changed, the expiration date estimate information FJ3 may be automatically changed.

[0135] The fifth change button B39 is a software button for changing the first notification timing indicated by the first notification timing information TJ1 of the first food details screen FDG1. When the reception unit 703 receives an operation for selecting the fifth change button B39, the display control unit 702 displays a screen for changing the first notification timing indicated by the first notification timing information TJ1. When the notification timing is changed on the screen, the display control unit 702 reflects the changed content in the first notification timing indicated by the first notification timing information TJ1.

[0136] The sixth area A6 has an area change button B40. The area change button B40 is a software button for changing the display area of ​​the sixth area A6. For example, when the area change button B40 is operated on the first food detail screen FDG1 shown in FIG. 16, the display control unit 702 narrows the display area of ​​the fifth area A5 to reduce the information displayed in the sixth area A6.

[0137] The first food details screen FDG1 has a seventh area A7. The seventh area A7 is an area that displays recipes that use the food F displayed on the first food details screen FDG1. The seventh area A7 displays one or more recipe icons IC3 for recipes that use the food F displayed on the first food details screen FDG1.

[0138] The first food details screen FDG1 has a confirm button B41. The Confirm button B41 is a software button for confirming the contents displayed on the first food details screen FDG1. The reception unit 703 receives an operation of selecting the Confirm button B41. The Confirm button B41 is displayed on the first food details screen FDG1 when information related to the food F, such as the stock amount or the first notification timing, is changed on the first food details screen FDG1.

[0139] The first food details screen FDG1 has an eighth area A8. The eighth area A8 is an area for displaying information relating to key points regarding storage of the food F displayed on the first food details screen FDG1.

[0140] In addition, the first food details screen FDG1 may display other information about the target contained food SF, such as the characteristics of the contained food SF.

[0141] FIG. 17 is a diagram showing an example of the second food details screen FDG2. The second food details screen FDG2 is a food details screen FDG relating to the detection target food KF. As shown in FIG. 17, the second food details screen FDG2 has a back button B42, a first switching button B1, a register button B16, an add button B17, and a delete button B43.

[0142] The back button B42 is a software button for returning the screen displayed by the app UI 720 to the screen displayed immediately before transitioning to the second food details screen FDG2. When the reception unit 703 receives an operation of selecting the back button B42 on the second food details screen FDG2, the display control unit 702 returns the screen displayed by the app UI 720 to the screen displayed immediately before transitioning to the second food details screen FDG2.

[0143] The Delete button B43 is a software button for deleting the food F, the details of which are displayed on the second food details screen FDG2, from the inventory list displayed by the application UI720.

[0144] The second food details screen FDG2 has a ninth area A9. The ninth area A9 is an area that displays detailed information about the food F. The ninth area A9 has a food icon IC1, food name information FJ1, holding start date information FJ2, inventory amount information FJ4, estimated expiration date information FJ3, storage location information KJ, first notification timing information TJ1, and second notification timing information TJ2.

[0145] The second notification timing information TJ2 is information indicating the second notification timing, which is the timing for notifying the stock amount of the detection target food KF.

[0146] The ninth area A9 has a first change button B35, a second change button B36, a third change button B37, a fourth change button B38, a fifth change button B39, and a sixth change button B44.

[0147] The sixth change button B44 is a software button for changing the second notification timing. When the accepting unit 703 accepts an operation to select the sixth change button B44, the display control unit 702 displays a screen for changing the second notification timing indicated by the second notification timing information TJ2. When the second notification timing is changed on the screen, the display control unit 702 reflects the changed content in the second notification timing indicated by the second notification timing information TJ2.

[0148] The ninth area A9 has a graph GF. Graph GF is a graph showing the change in weight detected by weight sensor 1 over the past week.

[0149] The ninth area A9 has an area change button B45. The area change button B45 is a software button for changing the display area of ​​the ninth area A9. For example, when the area change button B45 is operated on the second food detail screen FDG2 shown in FIG. 17, the display control unit 702 narrows the display area of ​​the ninth area A9 to reduce the amount of information displayed in the ninth area A9.

[0150] The second food details screen FDG2 has a tenth area A10. The tenth area A10 is a software button for confirming the content displayed on the second food details screen FDG2. The tenth area A10 displays one or more recipe icons IC3 of recipes using the food F displayed on the second food details screen FDG2.

[0151] The second food details screen FDG2 has an eleventh area A11. The eleventh area A11 is an area for displaying information relating to key points regarding storage of the food F displayed on the second food details screen FDG2.

[0152] In addition, the first food details screen FDG1 and the second food details screen FDG2 may have a third switching button B3.

[0153] It should be noted that, when information related to the food F, such as the stock amount or the notification timing, is changed on the second food details screen FDG2, a confirm button B41 is displayed on the second food details screen FDG2.

[0154] In addition, the second food details screen FDG2 may display other information about the target detection target food KF, such as the characteristics of the detection target food KF.

[0155] [1-1-3-5. Configuration of the confirmation screen for using up] Next, the use-up confirmation screen UCG will be described. FIG. 18 is a diagram showing an example of the use-up confirmation screen UCG. The use-up confirmation screen UCG is a screen for confirming whether the food F has been used up or discarded.

[0156] As shown in FIG. 18, the use-up confirmation screen UCG has a back button B46. The back button B46 is a software button for returning the screen displayed by the application UI 720 to the screen displayed immediately before transitioning to the use-up confirmation screen UCG. When the reception unit 703 receives an operation of selecting the back button B46 on the use-up confirmation screen UCG, the display control unit 702 returns the screen displayed by the application UI 720 to the first inventory screen SG1 displaying the menu screen MG or the second inventory screen SG2 displaying the menu screen MG.

[0157] The use-up confirmation screen UCG has a twelfth area A12. The twelfth area A12 is an area that displays a list of stored foods SF that are no longer captured in the captured image PG, among the stored foods SF included in the inventory list displayed by the application UI 720. The twelfth area A12 displays food-related information SKJ for each stored food SF that is no longer captured in the captured image PG. The food-related information SKJ includes a food icon IC1, food name information FJ1, holding start date information FJ2, a discard button B47, and a use up button B48.

[0158] The discard button B47 is a software button for selecting that the stored food SF has been discarded. When the reception unit 703 receives an operation to select the discard button B47, the display control unit 702 displays the food-related information SKJ as being checked. An example of the checked state is a state in which a strikethrough is placed on the food name indicated by the food name information FJ1. The strikethrough may be a horizontal line or a diagonal line.

[0159] The use up button B48 is a software button for selecting that the stored food SF has been used up. When the reception unit 703 receives an operation to select the use up button B48, the display control unit 46 displays the food related information SKJ as being checked.

[0160] The twelfth area A12 has an inside check button B49. The inside check button B49 is a software button similar to the first switching button B1. When the reception unit 703 receives an operation to select the inside check button B49, the display control unit 702 switches the screen displayed by the application UI 720 to the third inventory screen SG3.

[0161] The use-up confirmation screen UCG has a thirteenth area A13. The thirteenth area A13 is an area that displays a list of stored foods SF included in the inventory list displayed by the application UI 720. The thirteenth area A13 displays food-related information SKJ for each stored food SF.

[0162] The use-up confirmation screen UCG has a Complete button B50. The completion button B50 is a software button for inputting the completion of confirmation of the disposal of the contained food SF and confirmation of the use up of the contained food SF.

[0163] The contents entered on the use-up confirmation screen UCG are collected and utilized by at least one of the food management application 711 and the food management server 5. For example, the food management application 711 collects the contents entered on the use-up confirmation screen UCG and displays to notify the user P of foods F that are often used up or foods F that are often discarded. Also, for example, the food management application 711 collects the contents entered on the use-up confirmation screen UCG and prompts the user P to check the stock of foods F that are often discarded.

[0164] [1-1-3-6. Additional screen configuration] Next, the additional screen AG will be described. FIG. 19 is a diagram showing an example of the addition screen AG.

[0165] The addition screen AG has a back button B51. The back button B51 is a software button for returning the screen displayed by the application UI 720 to the screen displayed immediately before transitioning to the addition screen AG. When the reception unit 703 receives an operation of selecting the back button B51 on the addition screen AG, the display control unit 702 returns the screen displayed by the application UI 720 to the first inventory screen SG1 displaying the menu screen MG or the second inventory screen SG2 displaying the menu screen MG.

[0166] The additional screen AG has a fourteenth area A14. The fourteenth area A14 has a plurality of switching buttons such as a first food switching button B52 and a second food switching button B53. The switching buttons in the fourteenth area A14 are software buttons for switching the food F displayed in the fifteenth area A15.

[0167] The first food switching button B52 is a software button for switching the food F displayed in the list in the fifteenth area A15 to a not-added stored food. The not-added stored food is a stored food SF recognized from the photographed image PG that has not been added to the inventory list displayed by the application UI720.

[0168] The second food switching button B53 is a software button for switching the food F displayed in the list in the fifteenth area A15 to the food F that the user P has set as already purchased on the shopping list screen.

[0169] The addition screen AG has a fifteenth area A15. The fifteenth area A15 is an area that displays a list of food items F that have not been added to the inventory list displayed by the application UI 720. 19 shows the addition screen AG when the first food switching button B52 is operated. When the first food switching button B52 is operated, the addition screen AG displays a list of unadded contained foods in the fifteenth area A15.

[0170] The fifteenth area A15 that displays the not-added stored foods has, for each not-added stored food, a check box CB2, a food icon IC1, food name information FJ1, and a modify button B55. The modify button B55 is a software button for modifying the food icon IC1 and food name that are set when adding the not-added stored food to the inventory list. Furthermore, the fifteenth area A15 displaying the not-added stored food items has a multiple correction button B56. The multiple correction button B56 is a software button for correcting the food icon IC1 and food name that are set when adding multiple not-added stored foods displayed in the fifteenth area A15 to the inventory list.

[0171] The additional screen AG displayed when the first food switching button B52 is operated has a 16th area A16. The 16th area A16 is an area that displays a list of stored foods SF that have been recognized from the captured image PG and have been added to the inventory list displayed by the application UI 720. The 16th area A16 has a storehouse check button B49.

[0172] When the reception unit 703 receives an operation to select the edit button B55 on the addition screen AG, the display control unit 702 displays a edit screen FG.

[0173] FIG. 20 is a diagram showing an example of the correction screen FG. The modification screen FG is a screen for modifying the food icon IC1 and food name that are set for one stored food item SF when adding it to the inventory list.

[0174] The edit screen FG has a hide button B57. The hide button B57 is a software button for hiding the edit screen FG. When the accepting unit 703 accepts an operation to select the hide button B57, the display control unit 702 hides the edit screen FG.

[0175] The modified screen FG has a seventeenth area A17. The 17th area A17 is a rectangular area that displays the photographed image PG showing the not-added contained food that is the object of correction on the correction screen FG. The 17th area A17 displays the photographed image PG with either the top or bottom side or the left or right side of the photographed image PG aligned with the edge of the 17th area A17. The 17th area A17 highlights the image of the not-added contained food that is the object of correction on the correction screen FG in the photographed image PG. In this embodiment, the image of the not-added contained food is highlighted by graying out the area other than the image of the not-added contained food.

[0176] The modified screen FG has an eighteenth area A18. The 18th area A18 displays multiple combinations KM of the food icon IC1 and food name information FJ1, thereby displaying multiple candidates for the food icon IC1 and food name to be set when adding to the inventory list. The 18th area A18 of this embodiment displays four candidates. Note that the number of candidates displayed in the 18th area A18 is not limited to four and may be less or more. The 18th area A18 displays combinations KM for the top four foods F that match the unadded stored food highlighted in the 17th area A17.

[0177] The accepting unit 703 accepts an operation to select one combination KM in the eighteenth area A18. When the reception unit 703 receives an operation to select a combination KM, the display control unit 702 switches from the edit screen FG to the add screen AG, and changes the food icon IC1 and food name corresponding to the edit button B55 operated during the transition to the edit screen FG to the contents of the combination KM selected on the edit screen FG. Furthermore, when the receiving unit 703 receives an operation to select a combination KM, the display control unit 702 checks the check box CB2 corresponding to the edit button B55 operated when the edit screen FG was transitioned.

[0178] The edit screen FG has a text search button B58. The text search button B58 is a software button for searching for a combination KM by text input. When the accepting unit 703 accepts an operation to select the text search button B58, the display control unit 703 displays a screen for searching for a combination KM by text input. When one combination KM is selected from the combination KMs searched on the screen, the display control unit 702 hides the screen for searching for a combination KM by text input, and changes the food icon IC1 and food name corresponding to the edit button B55 operated when the edit screen FG transitions to the selected food icon IC1 and food name. In addition, the display control unit 702 checks the check box CB2 corresponding to the edit button B55 operated when the edit screen FG transitions.

[0179] Returning to the explanation of Fig. 19, when the reception unit 703 receives an operation to select the multiple edit button B56 on the additional screen AG, the display control unit 702 displays multiple edit screens FG in a switchable manner. More specifically, the display control unit 702 displays edit screens FG related to foods whose check boxes CB2 are checked in a switchable manner, the number of which corresponds to the number of check boxes CB2.

[0180] FIG. 21 is a diagram showing an example of the correction screen FG. The edit screen FG shown in FIG. 21 is the edit screen FG when the multiple edits button B56 is selected.

[0181] When the multiple revision button B56 is selected, the revision screen FG has progress information SCJ, a cancel button B59, and a skip button B60.

[0182] The progress information SCJ is information that indicates which of the multiple correction targets, the unadded stored food, is being corrected.

[0183] The cancel button B59 is a software button for changing the screen displayed by the application UI 720 to the edit screen FG from which the currently displayed edit screen FG is to be transitioned. Note that when the progress information SCJ indicates the first food item that has not yet been added, the cancel button B59 is not displayed.

[0184] The skip button B60 is a software button for changing the screen displayed by the application UI 720 to a correction screen SG related to the next food item that has not been added. Note that when the progress information SCJ indicates the last food item that has not been added, the skip button B60 is not displayed.

[0185] When the reception unit 703 receives an operation to select one combination KM in the eighteenth area A18 of the modification screen FG in which the multiple modification button B56 is selected, the display control unit 702 performs the following process.

[0186] That is, when the progress information SCJ indicates that the last stored food item not yet added has been corrected, the display control unit 702 hides the correction screen FG and changes the food icon IC1 and food name with the check box CB2 checked on the addition screen AG to the corrected food icon IC1 and food name on the corresponding correction screen FG. On the other hand, when the progress information SCJ indicates that the last stored food item not yet added has been corrected, the display control unit 702 displays the correction screen FG for the next stored food item not yet added.

[0187] Returning to the explanation of Figure 19, when the receiving unit 703 receives an operation to select the second food switching button B53, the display control unit 702 switches the food F displayed in the fifteenth area A15 to the food F that is set as purchased by user P on the shopping list screen.

[0188] FIG. 22 is a diagram showing an example of the addition screen AG. 22 shows the additional screen AG when the second food switching button B53 is operated. When the second food switching button B53 is operated, the fifteenth area A15 has a check box CB2, a food icon IC1, and food name information FJ1 for each food F.

[0189] 19 and 22, the fifteenth area A15 of the addition screen AG displays a check box CB2 for each food F. In the addition screen AG, when the check box CB2 for a certain food F is checked, the check box CB2 for the same food F remains checked even if the type of food F displayed in the fifteenth area A15 is switched in the fourteenth area A14.

[0190] 19 and 22 includes an Add button B59. The Add button B59 is a software button for adding a food item whose check box CB2 in the fifteenth area A15 is checked to the inventory list displayed by the application UI 720.

[0191] [1-1-4. Food management server configuration] Returning to the explanation of FIG. 3, the food management server 5 includes a server control device 50 and a server communication unit 51.

[0192] The server control device 50 includes a server processor 500 such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), a server memory 510, and an interface circuit to which other devices and sensors are connected.

[0193] The server memory 510 is a memory that stores programs and data. The server memory 510 stores a control program 511, a first management DB (database) 512, a second management DB 513, a third management DB 514, a fourth management DB 515, a fifth management DB 516, a sixth management DB 517, and data to be processed by the server processor 500. The server memory 510 has a non-volatile storage area. The server memory 510 may also have a volatile storage area and constitute a work area of ​​the server processor 500. The server memory 510 is constituted by, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory).

[0194] 23 and 24 are diagrams showing an example of a database stored in the server memory 510. FIG.

[0195] The first management DB 512 is a database for managing the photographed images PG. Each record R1 stored in the first management DB 512 has an account ID and an imaging device ID that uniquely identify the user P. Furthermore, the record R1 has a first field, a second field, and a third field.

[0196] The first field is a field for storing data related to the refrigerator compartment 21, and has image capturing related information for the refrigerator compartment, image capturing related information for the right door 21B, and image capturing related information for the left door 21C. The second field is a field for storing data relating to the freezer compartment 24, and includes imaging-related information for the upper case of the freezer compartment 24, imaging-related information for the middle case of the freezer compartment 24, and imaging-related information for the lower case of the freezer compartment 24. The third field is a field for storing data related to the vegetable compartment 25, and has photography-related information for the upper case of the vegetable compartment 25 and photography-related information for the lower case of the vegetable compartment 25.

[0197] The second management DB 513 is a database for managing stored food SF owned by the user P. Each record R2 stored in the second management DB 513 has an account ID and an imaging device ID. In addition, the record R2 records, for each stored food SF owned by the user P, a food ID, a food icon IC1, food name information FJ1, inventory amount information FJ4, holding start date information FJ2, estimated expiration date information FJ3, storage location information KJ, and first notification timing information TJ1.

[0198] The third management DB 514 is a database that manages detection target foods KF held by the user P. Each record R3 stored in the third management DB 514 includes an account ID, a sensor ID, a food ID, a food icon IC1, food name information FJ1, inventory amount information FJ4, holding start date information FJ2, estimated expiration date information FJ3, storage location information KJ, first notification timing information TJ1, second notification timing information TJ2, and group data. The group data includes weight values ​​detected by the weight sensor 1 for the most recent week.

[0199] The fourth management DB 515 is a database for managing unadded stored food. Each record R4 stored in the fourth management DB 515 includes an account ID and an image capturing device ID. The record R4 also includes, for each unadded stored food, recognition-related information, candidate information, holding start date information FJ2, estimated expiration date information FJ3, and storage location information KJ.

[0200] The recognition related information includes a photographed image PG in which a not-yet-added contained food is recognized, and the position of the not-yet-added contained food in the photographed image PG.

[0201] The candidate information is information related to candidates for the food icon IC1 and food name information FJ1 that are set when adding unadded stored food to the inventory list. The candidate information records the food ID, food icon IC1, and food name information FJ1 for each candidate.

[0202] The fifth management DB 516 is a database that manages the food F displayed in the twelfth area A12 of the use-up confirmation screen UCG. Each record R5 stored in the fifth management DB 516 has an account ID and an image capturing device ID. In addition, the record R5 stores a food ID, a food icon IC1, food name information FJ1, and possession start date information FJ2 for each food F displayed in the twelfth area A12 of the use-up confirmation screen UCG.

[0203] The sixth management DB 517 is a database that manages the food F displayed in the fifteenth area A15 of the additional screen AG when the second food switching button B53 is operated. Each record R6 stored in the sixth management DB 517 includes an account ID and an image capturing device ID. In addition, the record R6 includes a food ID, a food icon IC1, and food name information FJ1 for each food F.

[0204] Returning to the explanation of FIG. 3, the server communication unit 51 includes hardware such as a communication circuit conforming to a predetermined communication standard, and communicates with devices connected to the network NW under the control of the server control device 50.

[0205] The server processor 500 reads and executes a control program 511 stored in a server memory 510 , thereby functioning as a server communication control unit 501 and a server processing unit 502 .

[0206] The server communication control unit 501 communicates with the weight sensor 1, the image capturing device 6, and the terminal device 7 via the server communication unit 51.

[0207] The server processing unit 502 performs various information processes on the weight sensor 1, the image capturing device 6, and the terminal device 7 as clients.

[0208] When server communication control unit 501 receives first upload data D1, server processing unit 502 executes the following process. The server processing unit 502 identifies a record R3 having the sensor ID recorded in the received first upload data D1 from the third management DB 514. Next, the server processing unit 502 converts the weight recorded in the first upload data D1 into the inventory amount of food F using a predetermined algorithm. Next, the server processing unit 502 updates the inventory amount indicated by the inventory amount information FJ4 recorded in the identified record R2 to the converted inventory amount. In addition, the server processing unit 502 updates the group data in the identified record R3 to include the weight value recorded in the received first upload data D1.

[0209] When server communication control unit 501 receives second upload data D2, server processing unit 502 executes the following process. The server processing unit 502 identifies, from the first management DB 512, a record R1 having the image capturing device ID recorded in the received second upload data D2. Next, the server processing unit 502 updates the image capturing related information of the refrigerator compartment 21 contained in the identified record R2 to the image capturing related information of the refrigerator compartment 21 recorded in the received second upload data D2. The server processing unit 502 also updates the image capturing related information of the right door 21B contained in the identified record R2 to the image capturing related information of the right door 21B recorded in the received second upload data D2. The server processing unit 502 also updates the image capturing related information of the left door 21C contained in the identified record R2 to the image capturing related information of the left door 21C recorded in the received second upload data D2.

[0210] When server communication control unit 501 receives third upload data D3, server processing unit 502 executes the following process.

[0211] The server processing unit 502 identifies the records R2 and R4 of the image capturing apparatus ID recorded in the third upload data D3 from the second management DB 513 and the fourth management DB 515. Next, the server processing unit 502 detects food F from the photographed image PG of the storage case recorded in the received third upload data D3 by a predetermined image recognition process, and recognizes each of the detected foods F. In recognizing the foods F, the server processing unit 502 obtains the food ID, food icon IC1, and food name information FJ1 for the food F that has the highest degree of match with the detected food F from a predetermined database.

[0212] Next, the server processing unit 502 determines whether or not there is any food stored that has not been added among the food F recognized from the photographed image PG of the third upload data D3. The server processing unit 502 acquires, from the identified record R2, a food ID that corresponds to the storage location information KJ indicating the type of storage chamber and the type of storage case recorded in the third upload data D3. Next, if there is a food ID other than the food ID acquired from record R2 among the food IDs acquired when recognizing the food F, the server processing unit 502 determines that there is any food stored that has not been added among the food F recognized from the photographed image PG of the third upload data D3.

[0213] When the server processing unit 502 determines that there is a not-added stored food, it retrieves from a specified database the food ID, food icon IC1, and food name information FJ1 for the top four foods F that match the not-added stored food. Then, the server processing unit 502 generates recognition related information, candidate information, holding start date information FJ2, and storage location information KJ for each not-added stored food, and stores the combination of this generated information in the specified record R4. When storing in record R4, the candidate information includes four items recorded: a food ID obtained from a specified database, a food icon IC1, and food name information FJ1. Furthermore, when storing in record R4, the retention start date indicated in retention start date information FJ2 is set to the shooting date and time recorded in the received third upload data D3. Furthermore, when storing in record R4, the storage location indicated by storage location information KJ is set to the combination of the type of accommodation room and the type of accommodation case recorded in the received third upload data D3.

[0214] Furthermore, when server communication control unit 501 receives third upload data D3, server processing unit 502 executes the following process.

[0215] The server processing unit 502 identifies the records R2 and R5 of the image capturing apparatus ID recorded in the third upload data D3 from the second management DB 513 and the fifth management DB 516.

[0216] Next, the server processing unit 502 detects food items F from the photographed image PG of the storage case recorded in the received third upload data D3 by a predetermined image recognition process, and recognizes each of the detected food items F.

[0217] Next, the server processing unit 502 determines whether or not there is any food F in the inventory list that is no longer captured in the captured image PG recorded in the third upload data D3. The server processing unit 502 acquires a food ID corresponding to the storage location information KJ indicating the type of storage chamber and the type of storage case recorded in the third upload data D3 from the identified record R2. Next, if there is a food ID in the identified record R2 other than the food ID acquired when recognizing the food F, the server processing unit 502 determines that there is any food F that is no longer captured in the captured image PG recorded in the third upload data D3.

[0218] When the server processing unit 502 determines that there is a food F that is no longer captured in the captured image PG recorded in the third upload data D3, it obtains from the identified record R2 the food ID, food icon IC1, and possession start date information FJ2 of the food F. Then, the server processing unit 502 records the food ID, food icon IC1, and possession start date information FJ2 obtained from record R2 in the identified record R5.

[0219] When server communication control unit 501 receives third upload data D3, server processing unit 502 executes the following process.

[0220] The server processing unit 502 identifies a record R1 having the photographing device ID recorded in the received third upload data D3 from the first management DB 512. Next, the server processing unit 502 identifies the photographing related information to be updated from the photographed image PG of the identified record R1 based on the type of the storage room and the type of the storage case recorded in the received third upload data D3. Next, the server processing unit 502 updates the identified photographing related information to be updated to the photographing related information of the storage case recorded in the third upload data D3.

[0221] Furthermore, when server communication control unit 501 receives third upload data D3, server processing unit 502 executes the following process.

[0222] The server processing unit 502 identifies, from the second management DB 513, the record R2 having the image capturing apparatus ID recorded in the received third upload data D3. Next, the server processing unit 502 detects food items F from the photographed image PG of the storage case recorded in the received third upload data D3 by a predetermined image recognition process, and recognizes each of the detected food items F. Next, the server processing unit 502 estimates the stock amount of each recognized food item F from the size of the food item F in the photographed image PG and the like, using a predetermined algorithm.

[0223] Next, the server processing unit 502 determines whether the food F in the inventory list includes the food F that appears in the photographed image PG recorded in the third upload data D3. The server processing unit 502 acquires the food ID corresponding to the storage location information KJ indicating the type of storage chamber and the type of storage case recorded in the third upload data D3 from the identified record R2. Next, if the food ID acquired when recognizing the food F is present in the food IDs in the identified record R2, the server processing unit 502 determines that the food F in the inventory list includes the food F that appears in the photographed image PG recorded in the third upload data D3.

[0224] If the server processing unit 502 determines that the food F included in the inventory list is a food F that appears in the photographed image PG recorded in the third upload data D3, it updates the inventory quantity information FJ4 contained in the identified record R2, which corresponds to the food ID of the food F that appears in the photographed image PG, to indicate the estimated inventory quantity.

[0225] [1-2. Operation] Next, the operation of each part of the food management system 1000 related to the application UI 720 will be described. Fig. 25 is a flowchart showing the operation of the food management system 1000. In Fig. 25, a flowchart FA shows the operation of the terminal device 7, and a flowchart FB shows the operation of the food management server 5.

[0226] The terminal communication control unit 701 determines whether or not to request the food management server 5 to transmit a record (step SA1).

[0227] For example, when the food management application 711 is launched, the terminal communication control unit 701 makes an affirmative determination in step SA1. Also, for example, when a predetermined period (for example, one-minute period) arrives while the application UI 720 is displayed, the terminal communication control unit 701 makes an affirmative determination in step SA1. Also, for example, when a predetermined trigger occurs for updating the display contents of the application UI 720, the terminal communication control unit 701 makes an affirmative determination in step SA1. Note that, as an example of the predetermined trigger, an operation of an update button for updating the display contents of the application UI 720 can be given. Also, for example, if screen switching is to be performed, the terminal communication control unit 701 makes an affirmative decision in step SA1.

[0228] When the terminal communication control unit 701 determines to request transmission of the record (step SA1: YES), it transmits record request information, which is information requesting the record, to the food management server 5 (step SA2). The record request information includes the account ID stored in the terminal memory 710.

[0229] As shown in the flowchart FB, the server communication control unit 501 receives record request information from the terminal device 7 (step SB1).

[0230] Next, server processing unit 502 identifies records R1 to R6 corresponding to the account ID included in the record request information received in step SB1 from each database (step SB2).

[0231] Next, the server processing unit 502 transmits the records R1 to R6 identified in step SB2 to the terminal device 7 (step SB3).

[0232] As shown in the flowchart FA, the terminal communication control unit 701 receives records R1 to R6 from the food management server 5 (step SA3).

[0233] Next, the display control unit 702 generates various screens to be displayed by the application UI 720 based on the records R1 to R6 received in step SA3 (step SA4).

[0234] Step SA4 will now be described in detail. The display control unit 702 generates a first inventory screen SG1 that displays the food icon IC1 recorded in records R2 and R3. In the first inventory screen SG1, the display control unit 702 positions the food icon IC1 recorded in record R3 above the food icon IC1 recorded in record R2. The display control unit 702 also arranges the food icons IC1 of the contained foods SF based on the estimated expiration date indicated by the estimated expiration date information FJ3 recorded in record R2. The display control unit 702 also changes the state of the food icon IC1 displayed on the first inventory screen SG1 based on the estimated expiration date information FJ3 and inventory amount information FJ4 recorded in record R2.

[0235] Step SA4 will now be described in detail. The display control unit 702 generates a second inventory screen SG2 based on records R2 and R3. In generating the second inventory screen SG2, the display control unit 702 generates food information FJ based on various information contained in records R2 and R3. Then, the display control unit 702 displays the food information FJ generated from record R3 above the food information FJ generated from record R2.

[0236] Step SA4 will now be described in detail. The display control unit 702 generates a third inventory screen SG3 based on various types of photography related information recorded in the record R1.

[0237] Step SA4 will now be described in detail. The display control unit 702 generates a first food details screen FDG1 that displays various information recorded in the record R2.

[0238] Step SA4 will now be described in detail. The display control unit 702 generates a graph GF from the group data recorded in record R3. Then, the display control unit 702 displays various information recorded in record R2 and generates a second food details screen FDG2 that displays the generated graph GF.

[0239] Step SA4 will now be described in detail. The display control unit 702 generates a use-up confirmation screen UCG based on records R2 and R5. In generating the use-up confirmation screen UCG, the display control unit 702 displays a list of the food icon IC1, food name information FJ1, and holding start date information FJ recorded in record R5 in the 12th area A12. In generating the use-up confirmation screen UCG, the display control unit 702 displays a list of the food icon IC1, food name information FJ1, and holding start date information FJ2 recorded in record R2 in the 13th area A13.

[0240] Step SA4 will now be described in detail. The display control unit 702 generates the addition screen AG based on the records R2, R4, and R6. In generating the addition screen AG, the display control unit 702 displays the food icon IC1 and food name information FJ1 recorded in record R4 in a list in the 15th area A15 when the first food switching button B52 is operated. In addition, in generating the addition screen AG, the display control unit 702 displays the food icon IC1 and food name information FJ1 recorded in record R6 in a list in the 15th area A15 when the second food switching button B53 is operated. In addition, in generating the addition screen AG, the display control unit 702 displays the food icon IC1 and food name information FJ1 recorded in record R2 in a list in the 16th area A16.

[0241] Step SA4 will now be described in detail. The display control unit 702 generates a correction screen FG for each of the not-added stored foods based on the record R4. The display control unit 702 generates a correction screen FG in which the captured image PG recorded in the recognition related information is displayed in the 17th area A17, and the food icon IC1 and the food name information FJ1 recorded in the candidate information are displayed in the 18th area A18.

[0242] Returning to the explanation of the flowchart FA, the display control unit 702 displays the screen generated in step SA4 in response to the operation on the application UI 720 accepted by the accepting unit 703 (step SA5). The display in step SA5 includes a display for switching between the first inventory screen SG1, the second inventory screen SG2, and the third inventory screen SG3.

[0243] Next, the operation of food management system 1000 when a software button displayed by application UI 720 is operated will be described.

[0244] When the receiving unit 703 receives an operation to select the first selection button B19 or the second selection button B20, the terminal communication control unit 701 transmits first deletion request information to the food management server 5. The first deletion request information is information requesting deletion of various information recorded in record R2. The first deletion request information records an account ID and a food ID that is the subject of an inquiry on the inquiry screen IG. When the server processing unit 502 receives the first deletion request information, it identifies a record R2 having the account ID recorded in the first deletion request information from the second management DB 513. Then, the server processing unit 502 deletes the food ID recorded in the first deletion request information and various information associated with this food ID from the identified record R2.

[0245] When the receiving unit 703 receives an operation to select any one of the discard button B24, the use up button B25, and the delete button B26, the terminal communication control unit 701 transmits second deletion request information to the food management server 5. The second deletion request information is information requesting the deletion of various information recorded in record R2. The second deletion request information records the account ID and the food ID of food F for which the check box CB1 is checked on the second inventory screen SG2. When the second deletion request information is received, the server processing unit 502 deletes various information from record R2 in the same manner as the first deletion request information.

[0246] When the reception unit 703 receives an operation to select the confirm button B41, the terminal communication control unit 701 transmits change request information to the food management server 5. The change request information is information requesting a change to various information recorded in the record R2. The change request information includes the food ID of the food F targeted on the food details screen FDG. The change request information also includes an account ID. The change request information also includes the content changed on the food details screen FDG. When the server communication control unit 501 receives the change request information, the server processing unit 502 identifies the record R2 of the account ID recorded in the change request information from the second management DB 513. Then, the server processing unit 502 appropriately updates various information associated with the food ID recorded in the change request information from the identified record R2 to the change content recorded in the change request information.

[0247] When the receiving unit 703 receives an operation to select the completion button B50, the terminal communication control unit 701 transmits the third deletion request information to the food management server 5. The third deletion request information is information requesting deletion of various information from the records R2 and R5. The account ID is recorded in the third deletion request information. The food ID of the food F for which the discard button B47 or the use-up button B48 is selected in the 12th area A12 is also recorded in the third deletion request information. When the server communication control unit 501 receives the third deletion request information, the server processing unit 502 identifies the records R2 and R5 of the account ID recorded in the third deletion request information from the second management DB 513 and the fifth management DB 516. Then, the server processing unit 502 deletes the food ID recorded in the first deletion request information and various information associated with this food ID from the identified records R2 and R5.

[0248] When the receiving unit 703 receives an operation to select the Add button B59, the terminal communication control unit 701 transmits the addition request information to the food management server 5. The addition request information is information requesting the addition of food F to the inventory list. The addition request information records an account ID. The addition request information also records the food ID of food F, for which the check box CB2 on the addition screen AG is checked. When the server communication control unit 501 receives the addition request information, the server processing unit 502 identifies records R2 and R4 of the account ID recorded in the addition request information from the second management DB 513 and the fourth management DB 515. Then, the server processing unit 502 deletes the food ID recorded in the addition request information and information associated with the food ID (food icon IC1, food name information FJ1, holding start date information FJ2, estimated expiration date information J3, and storage location information KJ) from the identified record R4, and adds the various information deleted from record R4 to the identified record R2.

[0249] The operation of the food management system 1000 described above is configured such that the food management server 5 transmits records R1-R6. However, the records transmitted by the food management server 5 to the terminal device 7 do not have to be all of records R1-R6. The food management server 5 may be configured such that the terminal device 7 transmits records necessary for displaying the screen. For example, when the terminal device 7 displays the first inventory screen SG1, the terminal device 7 requests records R2 and R3 from the food management server 5, and the food management server 5 transmits records R2 and R3 to the terminal device 7.

[0250] [1-3. Effects, etc.] As described above, the food management application 711 causes the terminal processor 700 of the terminal device 7 equipped with the touch panel 72 to function as the display control unit 702 that displays the inventory screen SG displaying the food F held by the user P on the touch panel 72. The display control unit 702 switchably displays the first inventory screen SG1, the second inventory screen SG2, and the third inventory screen SG3. The food F includes the stored food SF stored in the refrigerator 2. The first inventory screen SG1 is an inventory screen SG that displays a food icon IC1 indicating the food F for each food F. The second inventory screen SG2 is an inventory screen SG that displays food information FJ including the food icon IC1 and information related to the food F other than the food icon IC1 for each food F. The third inventory screen SG3 is an inventory screen SG that displays a captured image PG of the interior of the refrigerator 2.

[0251] According to this, the first inventory screen SG1, the second inventory screen SG2, and the third inventory screen SG3 are switchably displayed, so that the inventory screen SG with the content desired by the user P can be displayed depending on the situation in which the user P wants to grasp the food products F he or she owns. This improves the convenience of the user P in grasping the food products F he or she owns.

[0252] The first inventory screen SG1 changes the state of the food icon IC1 according to the estimated expiration date of the food F and displays the food icon IC1.

[0253] This allows the user P to easily understand the approximate expiration date of the food represented by the food icon IC1 on the first inventory screen SG1. This further improves the convenience of the user P in understanding the food F he or she owns.

[0254] The first inventory screen SG1 changes the state of the food icon IC1 according to the amount of food F in stock, and displays the food icon IC1.

[0255] This allows the user P to easily grasp the stock status of the food F indicated by the food icon IC1 on the first inventory screen SG1. This further improves the convenience of the user P in grasping the food F he / she owns.

[0256] The closer the estimated expiration date of the food F is to the present, the closer the food icon IC1 is displayed on the first inventory screen SG1.

[0257] According to this, on the first inventory screen SG1, food icons IC1 are displayed in order according to the length from the present to the estimated expiration date, so that the user P can grasp the length from the present to the estimated expiration date of the food F while comparing it with other foods F. This further improves the convenience of the user P in grasping the food F he owns.

[0258] The foods F include detection target foods KF whose weight is detected by the weight sensor 1. The first inventory screen SG1 displays the food icons IC1 of the detection target foods KF above the food icons IC1 of the contained foods SF.

[0259] According to this, the food icons IC1 of the foods F whose weights are being detected intentionally by the user P can be displayed together at the top of the first inventory screen SG1, so that the user P can easily understand which foods F whose weights are being detected intentionally. This further improves the convenience of the user P in understanding the foods F that he or she owns.

[0260] The food information FJ includes inventory information FJ4 indicating the inventory amount of food F, and a slider SR that accepts an operation to change the inventory amount indicated by the inventory information FJ4.

[0261] This allows the user P to intuitively operate the stock amount of food F displayed on the stock screen SG. This improves the convenience of the user P in understanding the food F he or she owns, and also provides the user P with the stock screen SG that allows the user P to easily change the stock amount of food F.

[0262] When the slider SR accepts a change to change the inventory amount indicated by the inventory amount information FJ4 to zero, the second inventory screen SG2 displays a second selection button B20 to accept input that the food F whose inventory amount has been changed to zero has been used up, and a first selection button B19 to accept input that the food F whose inventory amount has been changed to zero has been discarded.

[0263] This allows the terminal device 7 to determine whether the food F whose inventory has been changed to zero is food F that has been used up or discarded, making it possible to utilize the fact that user P has used up the food F or discarded the food F.

[0264] The second inventory screen SG2 displays a selection field C2 that accepts operations to change the display order of the food information FJ for the stored foods SF to at least the order of the shortest estimated expiration date of the food, the order of the most recent date the stored foods SF were stored in the refrigerator 2, or the order of the oldest date the stored foods SF were stored.

[0265] This allows the user P to change the display order of the food information FJ of the stored food SF to a display order desired by the user P, thereby improving the convenience of the user P in understanding the food F he or she owns.

[0266] The foods F include detection target foods KF whose weight is detected by the weight sensor 1. The second inventory screen SG2 displays food information FJ of the detection target foods KF above the food information FJ of the contained foods SF.

[0267] According to this, the food information FJ of the food F whose weight is being detected intentionally by the user P can be displayed all together at the top of the second inventory screen SG2, so that the user P can easily understand which food F whose weight is being detected intentionally. This further improves the convenience of the user P in understanding the food F that he / she owns.

[0268] The terminal device 7 includes a touch panel 72 and a display control unit 702 that uses the touch panel 72 to display an inventory screen SG that displays food items F held by the user P. The display control unit 702 switchably displays a first inventory screen SG1, a second inventory screen SG2, and a third inventory screen SG3. The food items F include stored foods SF stored in the refrigerator 2. The first inventory screen SG1 is an inventory screen SG that displays a food icon IC1 indicating the food item F for each food item F. The second inventory screen SG2 is an inventory screen SG that displays food information FJ including the food icon IC1 and information related to the food item F other than the food icon IC1 for each food item F. The third inventory screen SG3 is an inventory screen SG that displays a captured image PG of the interior of the refrigerator 2.

[0269] This provides the same effects as the food management application 711 described above.

[0270] The control method of the terminal device 7 switchably displays a first inventory screen SG1 displaying food F owned by the user P, a second inventory screen SG2 displaying food F, and a third inventory screen SG3 displaying food F. Food F includes stored food SF stored in the refrigerator 2. The first inventory screen SG1 is a screen that displays a food icon IC1 indicating the food F for each food F. The second inventory screen SG2 is a screen that displays food information FJ including the food icon IC1 and information related to the food F other than the food icon IC1 for each food F. The third inventory screen SG3 is a screen that displays a captured image PG captured of the interior of the refrigerator 2.

[0271] This provides the same effects as the food management application 711 described above.

[0272] The food management application 711 causes the terminal processor 700 of the terminal device 7 equipped with the touch panel 72 to function as a display control unit 702 that displays, on the touch panel 72, an addition screen AG for adding stored foods SF stored in the refrigerator 2 recognized from the photographed image PG of the interior of the refrigerator 2 to an inventory list, which is a list of foods F held by the user P. On the addition screen AG, the display control unit 702 displays a list of not-added stored foods, which are stored foods SF recognized from the photographed image PG that have not been added to the inventory list.

[0273] This allows the user P to easily know which food items F recognized from the captured image PG have not been added to the inventory list. This improves the convenience of the user P when adding food items F to the inventory list.

[0274] The food management application 711 causes the terminal processor 700 to function as a reception unit 703 that receives the selection of a not-added stored food on the addition screen AG. When the reception unit 703 receives the selection of a not-added stored food, the display control unit 702 displays a correction screen FG on the touch panel 72. The correction screen FG is a screen for correcting the food name and food icon IC1 that are set when adding the selected not-added stored food to the inventory list.

[0275] This allows the user P to set an appropriate food name and food icon IC1 for the food F to be added to the inventory list when adding the food F. This further improves the convenience of the user P when adding the food F to the inventory list.

[0276] The correction screen FG displays the photographed image PG in which the selected not-yet-added contained food appears, and highlights the image of the selected not-yet-added contained food in the displayed photographed image PG.

[0277] This allows the user P to easily understand which food F is the food F to be corrected on the correction screen FG. Therefore, when adding a food F to the inventory list, the user P can more appropriately set a food name and a food icon IC1 for the food F to be added.

[0278] The correction screen FG highlights the image of the selected not-added contained food by graying out the area other than the image of the selected not-added contained food in the photographed image PG.

[0279] This allows the user P to easily and accurately grasp which food F is the food F to be corrected on the correction screen FG.

[0280] The correction screen FG displays multiple selectable candidates for the food name and food icon IC1 to be set when adding to the inventory list. When a candidate is selected on the correction screen FG, the display control unit 702 sets the food name and food icon IC1 corresponding to the selected candidate as the food name and food icon IC1 to be set when adding the not-added contained food to the inventory list.

[0281] According to this, by displaying selectable candidates on the edit screen FG, the user P can easily determine the food name and food icon IC1 to be set on the edit screen FG without performing a search or the like.

[0282] The terminal device 7 includes a touch panel 72 and a display control unit 702 that displays, via the touch panel 72, an addition screen AG for adding stored foods SF stored in the refrigerator 2 that are recognized from a captured image PG showing the interior of the refrigerator 2 to an inventory list that is a list of foods F owned by the user P. The display control unit 702 displays a list of stored foods that have not been added on the addition screen AG.

[0283] This provides the same effects as the food management application 711 described above.

[0284] The control method of the terminal device 7 displays an addition screen AG for adding stored foods SF stored in the refrigerator 2 recognized from the photographed image PG showing the interior of the refrigerator 2 to an inventory list, which is a list of foods F owned by the user P. The control method of the terminal device 7 also displays a list of stored foods that have not been added on the addition screen AG.

[0285] This provides the same effects as the food management application 711 described above.

[0286] (Other embodiments) As described above, the above-mentioned first embodiment has been described as an example disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above-mentioned first embodiment to create a new embodiment. Therefore, other embodiments will be exemplified below.

[0287] In the above-mentioned first embodiment, the "weight detection device" has been described as a plate-shaped device, but the shape of the "weight right device" is not limited to a plate-shaped device.

[0288] In the above-described first embodiment, the vegetable compartment 25 is provided below the freezer compartment 24. However, the freezer compartment 24 may be provided below the vegetable compartment 25.

[0289] In the above-described embodiment 1, the selection column C2 displays three options, but the number of options displayed by the selection column C2 is not limited to three and may further include options indicating different display orders.

[0290] In the above-described first embodiment, a pull-down selection field C2 is used as an example of the "fourth reception section", but the "fourth reception section" is not limited to the pull-down selection field C2 and may be, for example, a radio button.

[0291] In the above-described first embodiment, the slider SR is exemplified as the "first reception section", but the "first reception section" is not limited to the slider SR and may be, for example, a pull-down selection field or radio buttons.

[0292] In the above-described embodiment 1, the freezer compartment 24 has three storage cases and the vegetable compartment 25 has two storage cases, but the number of storage cases provided in the freezer compartment 24 and the vegetable compartment 25 is not limited to that in embodiment 1.

[0293] In the above-described first embodiment, the food icon IC1 is exemplified as the "food image", but the "food image" may be an image cut out from the photographed image PG.

[0294] In the above-described first embodiment, the terminal device 7 is a smartphone or a tablet PC. In other embodiments, the terminal device 7 may be a desktop or laptop PC.

[0295] In the above-described first embodiment, an application program that can be installed in the terminal device 7 is exemplified as the "program", but the "program" may be a software program that already exists in the terminal device 7, such as a browser.

[0296] The server processor 500 and the terminal processor 700 may be configured with a single processor or multiple processors. These processors may be hardware programmed to realize the corresponding functional units. That is, these processors may be configured with, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0297] The configurations of the food management server 5 and the terminal device 7 shown in Fig. 3 are merely examples, and the specific implementation form is not particularly limited. In other words, it is not necessarily necessary to implement hardware corresponding to each unit individually, and it is also possible to implement a configuration in which one processor executes a program to realize the functions of each unit. Also, some of the functions realized by software in the above-mentioned embodiment may be hardware, or some of the functions realized by hardware may be software.

[0298] The step units of the operation shown in FIG. 25 are divided according to the main processing contents in order to facilitate understanding of the operation, and the operation is not limited by the manner in which the processing units are divided or the names thereof. The operation may be divided into more step units according to the processing contents. In addition, one step unit may be divided so as to include more processing. In addition, the order of the steps may be appropriately changed within a range that does not interfere with the purpose of this disclosure.

[0299] It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, substitutions, additions, omissions, and the like can be made within the scope of the claims or their equivalents.

[0300] (Additional Note) The above description of the embodiments discloses the following techniques.

[0301] (Technology 1) A program that causes a processor of a terminal device equipped with a display unit to function as a display control unit that displays, on the display unit, an addition screen for adding stored foods recognized from a photographed image of the interior of the refrigerator to an inventory list, which is a list of foods owned by a user, and the display control unit displays a list of unadded stored foods on the addition screen, which are stored foods recognized from the photographed image that have not been added to the inventory list. This allows the user to easily know which foods among those recognized from the captured image have not been added to the inventory list, thereby improving the convenience for the user when adding foods to the inventory list.

[0302] (Technology 2) The program described in Technology 1 causes the processor to function as a reception unit that receives the selection of the not-added stored food on the addition screen, and when the reception unit receives the selection of the not-added stored food, the display control unit displays a correction screen on the display unit, the correction screen being a screen for correcting the food name and food image that are set when the selected not-added stored food is added to the inventory list. This allows the user to set an appropriate food name and food image for the food to be added to the inventory list when adding the food to the inventory list, thereby improving the convenience for the user when adding the food to the inventory list.

[0303] (Technology 3) The program described in Technology 2, in which the correction screen displays the photographed image in which the selected not-added stored food appears, and highlights the image of the selected not-added stored food in the displayed photographed image. This allows the user to easily identify which food is to be edited on the edit screen, and therefore allows the user to set more appropriate food names and food images for the food to be added to the inventory list.

[0304] (Technology 4) The program described in Technology 3, in which the correction screen highlights the image of the selected not-added contained food by graying out areas other than the image of the selected not-added contained food in the photographed image. This allows the user to easily and accurately grasp which food item is the food item to be corrected on the correction screen.

[0305] (Technology 5) The program described in any one of Technology 2 to Technology 4, wherein the edit screen displays a plurality of selectable candidates for food names and food images to be set when adding to the inventory list, and when a candidate is selected on the edit screen, the display control unit sets the food name and food image corresponding to the selected candidate as the food name and food image to be set when adding the not-added contained food to the inventory list. According to this, by displaying selectable candidates on the edit screen, the user can easily determine the food name and food image to set on the edit screen without performing a search or the like.

[0306] (Technology 6) A terminal device comprising: a display unit; and a display control unit that displays, via the display unit, an addition screen for adding stored foods recognized from a photographed image showing the interior of the refrigerator to an inventory list, which is a list of foods owned by a user, and the display control unit displays a list of unadded stored foods on the addition screen, which are stored foods recognized from the photographed image that have not been added to the inventory list. This provides the same effect as the program of Technology 1.

[0307] (Technology 7) A method for controlling a terminal device, which displays an addition screen for adding stored foods stored in a refrigerator that are recognized from a photographed image showing the interior of the refrigerator to an inventory list, which is a list of foods owned by a user, and displays a list of unadded stored foods on the addition screen, which are stored foods that are recognized from the photographed image but have not been added to the inventory list. This provides the same effect as the program of Technology 1. [Industrial Applicability]

[0308] As described above, the program, terminal device, and control method for the terminal device according to the present invention can be used to add food recognized from a captured image to an inventory list, which is a list of food items owned by the user. [Explanation of symbols]

[0309] 1. Weight sensor (weight detection device) 2. Refrigerator 5. Food Management Server 6. Imaging Device 7 Terminal Equipment 11 First weight sensor 12 Second weight sensor 700 Terminal Processor (Processor) 701 Terminal communication control unit 702 Display control unit 703 Reception 710 Terminal Memory 711 Food management app (program) 1000 Food Control System AG addition screen B19 1st selection button (3rd reception area) B20 Second selection button (second reception area) C2 Selection column (4th reception area) F Food FDG food details screen FDG1 1st food details screen FDG2 2nd food details screen FG correction screen FJ food information FJ4 Stock Information IC1 Food Icon (Food Image) KF Detection target foods PG Shooting Image SF Contained Food SG Inventory Screen SG1 First inventory screen SG2 2nd inventory screen SG3 3rd inventory screen SR Slider (First Reception Area)

Claims

1. A processor of a terminal device having a display unit, a display control unit that displays, on the display unit, an addition screen for adding food items recognized from a photographed image of the interior of the refrigerator to an inventory list that is a list of food items held by the user; The display control unit is on the addition screen, a list of unadded stored foods, which are stored foods that have not been added to the inventory list among the stored foods recognized from the photographed image, is displayed. program.

2. The processor functions as a reception unit that receives a selection of the not-added stored food on the addition screen; The display control unit displays a correction screen on the display unit when the reception unit receives the selection of the not-added-stored food, The correction screen is a screen for correcting a food name and a food image set when adding the selected not-added stored food to the inventory list. The program according to claim 1.

3. The correction screen is Displaying the photographed image showing the selected unadded food; highlighting an image of the selected food item not yet added in the photographed image to be displayed; The program according to claim 2.

4. the correction screen highlights the image of the selected not-to-be-added contained food by graying out an area other than the image of the selected not-to-be-added contained food in the photographed image; The program according to claim 3.

5. the correction screen displays a plurality of selectable candidates for food names and food images to be set when adding to the inventory list; When a candidate is selected on the correction screen, the display control unit sets a food name and a food image corresponding to the selected candidate as a food name and a food image to be set when the not-added stored food is added to the inventory list. The program according to any one of claims 2 to 4.

6. A display unit; a display control unit that displays, on the display unit, an addition screen for adding food items recognized from a photographed image showing the interior of the refrigerator to an inventory list that is a list of food items held by the user, The display control unit is on the addition screen, a list of unadded stored foods, which are stored foods that have not been added to the inventory list among the stored foods recognized from the photographed image, is displayed. Terminal device.

7. displaying an addition screen for adding food items contained in the refrigerator, which are recognized from a photographed image showing the interior of the refrigerator, to an inventory list, which is a list of food items held by the user; on the addition screen, a list of unadded stored foods, which are stored foods that have not been added to the inventory list among the stored foods recognized from the photographed image, is displayed. A method for controlling a terminal device.

Images