Program and information display method

Abstract

To provide a program and information display method, which improve usability.SOLUTION: A program according to an embodiment causes a computer to display first information on foods stored in a refrigerator on a display screen, display a special display showing foods recommended to be purchased by a user of the refrigerator on the display screen, and when a predetermined condition is satisfied due to a decrease in the amount of food kept as a regular item by the user, display the food on the special display.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] FIELD An embodiment of the present invention relates to a program and an information display method. [Background technology]

[0002] There have been proposed food management systems for managing food stored in refrigerators. Programs used in such food management systems are expected to become easier to use. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-146120 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a program and an information display method that can improve usability. [Means for solving the problem]

[0005] The program of the embodiment is a program for causing a computer to display first information regarding ingredients stored in a refrigerator on a display screen, display a special display on the display screen showing ingredients that the user of the refrigerator is encouraged to purchase, and, when the amount of an ingredient that the user keeps as regular items decreases and a specified condition is satisfied, display the ingredient in the special display. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram showing an overall configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a front view showing the refrigerator according to the embodiment. [Diagram 3]FIG. 2 is a block diagram showing the functional configuration of the smart speaker according to the embodiment. [Figure 4] FIG. 2 is a block diagram showing a functional configuration of a terminal device according to the embodiment. [Diagram 5] FIG. 2 is a block diagram showing the functional configuration of the server system according to the embodiment. [Figure 6] FIG. 4 is a diagram showing an example of the contents of an inventory list DB according to the embodiment. [Figure 7] FIG. 4 is a diagram showing an example of the contents of a shopping list DB according to the embodiment. [Figure 8] FIG. 4 is a diagram showing an example of the contents of a trash can DB according to the embodiment. [Figure 9] FIG. 4 is a diagram showing an example of the contents of a candidate list DB according to the embodiment. [Figure 10] FIG. 4 is a diagram showing an example of the contents of an ingredient DB according to the embodiment. [Figure 11] FIG. 4 is a diagram showing an image displayed on a display screen of the embodiment. [Figure 12] FIG. 13 is a diagram showing an image of a top page of foodstuff management according to an embodiment. [Figure 13] FIG. 13 is a diagram showing an image of a modified example of the top page of foodstuff management according to the embodiment. [Figure 14] FIG. 13 is a diagram showing an image displaying a complete ingredient list according to an embodiment. [Figure 15] FIG. 13 is a diagram showing an image displaying a food ingredient list for each storage compartment according to the embodiment. [Figure 16] FIG. 13 is a diagram showing an image of a modified example in which an ingredient list is displayed according to the embodiment. [Figure 17] 11A and 11B are diagrams showing images when a new ingredient is registered in the ingredient list of the embodiment. [Figure 18] FIG. 13 is a diagram showing an image of a modified example of registering a new ingredient in the ingredient list of the embodiment. [Figure 19] FIG. 13 is a diagram showing an image when changing an ingredient list according to the embodiment. [Figure 20] FIG. 13 is a diagram showing an image including a trash list according to an embodiment. [Figure 21] 1A and 1B are diagrams illustrating images including a shopping list and candidate list according to an embodiment. [Figure 22] FIG. 13 illustrates another image including a shopping list according to an embodiment. [Figure 23] 4 is a flowchart showing a process flow relating to a voice dialogue function of the embodiment. [Figure 24] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Diagram 25] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Figure 26] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Figure 27] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Figure 28] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Figure 29] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Diagram 30] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Diagram 31] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Diagram 32] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Diagram 33] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Diagram 34] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Diagram 35] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Diagram 36] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Figure 37] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. [Figure 38] 13A and 13B are diagrams showing modified examples of images displayed on the display screen of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an information display device according to an embodiment will be described with reference to the drawings. In the following description, the same reference numerals are used for components having the same or similar functions. Then, duplicated descriptions of those components may be omitted. "Based on XX" means "based on at least XX" and may include a case where it is based on another element in addition to XX. "Based on XX" is not limited to a case where XX is directly used, and may include a case where it is based on XX that has been calculated or processed. "XX or YY" is not limited to a case where either XX or YY is used, and may include a case where both XX and YY are used. "XX" and "YY" are any element (for example, any information). In the following, "database" is written as "DB". In the following, a case where a voice dialogue function is realized will be described, but speech from the device side is not required. For this reason, "voice dialogue" in the following description may be read as "voice recognition". Each of "best before date" and "expiration date" described below is an example of "intake date" or "first date".

[0008] <1. Overall composition> FIG. 1 is a diagram showing the overall configuration of an information processing system 1 according to an embodiment. The information processing system 1 is an example of an "ingredient management system" and is a system for managing the inventory of ingredients held by a user. The information processing system 1 includes, for example, a refrigerator 100, a smart speaker 200, a home router 300, a terminal device 400, and a server system 500. A network NW, which will be described later, includes, for example, one or more of the Internet, a cellular network, a Wi-Fi network, a Wide Area Network (WAN), a Local Area Network (LAN), a public line, a telephone line, a wireless base station, and the like. The router 300 is, for example, a wireless router, but is not limited thereto.

[0009] The refrigerator 100 is placed in a user's home. The refrigerator 100 is connected to a network NW via, for example, a router 300, and is capable of communicating with a server system 500. In this embodiment, the refrigerator 100 includes a display device 130, which will be described later. In this case, the refrigerator 100 corresponds to an example of an "information display device." The refrigerator 100 includes a microphone and a speaker, and may have a voice dialogue function similar to that of the smart speaker 200. In this case, the refrigerator 100 corresponds to an example of a "voice dialogue device (or voice recognition device)."

[0010] The smart speaker 200 is placed in a user's home. The smart speaker 200 is connected to a network NW via, for example, a router 300, and is capable of communicating with a server system 500. The smart speaker 200 may be capable of communicating with a refrigerator 100 via, for example, the router 300. The smart speaker 200 is an example of a "voice dialogue device (or voice recognition device)."

[0011] The terminal device 400 is, for example, a mobile terminal device owned by a user, such as a smartphone or a tablet terminal device. The terminal device 400 is connected to a network NW and can communicate with the server system 500. An application program (hereinafter, may be referred to as an "app") for managing ingredients and remotely controlling the refrigerator 100 is installed in the terminal device 400. The terminal device 400 is an example of an "information display device". However, the terminal device 400 is not limited to the above example, and may be a personal computer or a television receiver with an input function. The terminal device 400 may include a microphone and a speaker, and may have a voice dialogue function similar to that of the smart speaker 200. In this case, the terminal device 400 corresponds to an example of a "voice dialogue device (or voice recognition device)".

[0012] The server system 500 is a server system that records various data for food management, and is composed of, for example, multiple server devices 501 and 502. The server devices 501 and 502 are cloud servers connected to the network NW. For example, the server system 500 includes a first server device 501 for voice interaction that controls the smart speaker 200, and a second server device 502 for device control that controls the refrigerator 100. However, the server system 500 may be composed of one server device, or may be composed of three or more server devices. The server system 500 may include an information processing unit that performs edge computing or fog computing, such as an information processing unit included in the router 300 and / or an information processing unit included in a router in the network NW. Various functional units included in the server system 500, which will be described later, may be separated and exist in two or more devices.

[0013] The refrigerator 100, the smart speaker 200, the terminal device 400, and the server system 500 will be described in detail below.

[0014] <2. Refrigerator> FIG. 2 is a front view showing the refrigerator 100. The refrigerator 100 includes, for example, a housing 110, a plurality of doors 130, a display device 140, an actuator 150, and a control unit 160. A plurality of storage compartments (storage compartments) 120 are provided inside the housing 110. The plurality of storage compartments 120 include, for example, a refrigerator compartment 121, a chilled compartment 122, a vegetable compartment 123, an ice making compartment 124, a small freezing compartment 125, and a main freezing compartment 126. In this embodiment, the refrigerator compartment 121 is arranged at the top, the vegetable compartment 123 is arranged below the refrigerator compartment 121, the ice making compartment 124 and the small freezing compartment 125 are arranged below the vegetable compartment 123, and the main freezing compartment 126 is arranged below the ice making compartment 124 and the small freezing compartment 125. However, the arrangement of the storage compartments 120 is not limited to the above example. These plurality of storage compartments 120 are closed by a plurality of doors 130 so as to be able to be opened and closed, respectively.

[0015] Display device 140 is a liquid crystal display or an organic EL (Electro Luminescence) display, etc., and includes a display screen 141 capable of displaying images and / or videos. Display device 140 includes a touch sensor 142 provided on top of display screen 141. Actuator 150 includes a compressor that compresses a refrigerant for cooling and sends it to the cooler, a blower that sends the cold air cooled by the cooler to each storage chamber 120, etc.

[0016] The control unit 160 controls the display device 140 and the actuator 150. For example, the control unit 160 can accept a user's operation on the display screen 141 via the touch sensor 142. By controlling the actuator 150, the control unit 160 can execute functions of the refrigerator 100, such as changing the set temperature zone of each storage compartment 120 of the refrigerator 100, changing the control mode of the chilled compartment 122, "quick freezing" for cooling the freezers 126, 126 more quickly than in normal operation, and "quick ice making" for making ice more quickly than in normal operation.

[0017] However, the configuration of refrigerator 100 is not limited to the above example. Refrigerator 100 may have a camera 170 installed inside housing 110 or on door 130. In this case, images of ingredients taken by camera 170 may be sent to server system 500 and used for ingredient management. This will be described later.

[0018] <3. Smart speaker> 3 is a block diagram showing the functional configuration of the smart speaker 200. The smart speaker 200 has, for example, a microphone 210, a speaker 220, a wireless communication module 230, a storage unit 240, and an information processing unit 250.

[0019] The microphone 210 acquires the voice of the user spoken to the smart speaker 200. The speaker 220 outputs the voice to the outside of the smart speaker 200. The wireless communication module 230 includes a high-frequency circuit and an antenna, and is capable of wireless communication with the router 300, for example.

[0020] The storage unit 240 stores a voice DB 241. The voice DB 241 includes, for example, a simple voice recognition DB 241a. The simple voice recognition DB 241a registers a trigger word, which is a voice command that activates the smart speaker 200. "Activation" means changing from a dormant state (a state in which the power consumption is suppressed below a predetermined level and the device does not react to user utterances other than the trigger word) to a standby state (a state in which the device can receive a voice command) in which the device can react (can have a dialogue) to user utterances other than the trigger word. The "standby state" may be referred to as an "activation state." The trigger word is a predetermined specific word or phrase, such as "Hello, XXX."

[0021] The information processing unit 250 has, for example, a voice receiving unit 251, an activation determination unit 252, a data transmission / reception unit 253, and a voice output unit 254. The voice receiving unit 251 receives voice (user's speech) input to the microphone 210 as voice data. The activation determination unit 252 determines whether or not activation of a voice dialogue function is necessary. When the voice data received by the voice receiving unit 251 includes a trigger word registered in the simple voice recognition DB 241a, the activation determination unit 252 determines that activation is necessary and activates the voice dialogue function.

[0022] The data transmitter / receiver 253 controls the wireless communication module 230 to transmit and receive control signals and data to and from the server system 500. The data transmitter / receiver 253 transmits voice data other than the trigger word acquired by the voice receiver 251 to the server system 500. The data transmitter / receiver 253 receives response voice data prepared by the server system 500 from the server system 500. The data transmitter / receiver 253 outputs the received response voice data to the voice output unit 254. The voice output unit 254 outputs the voice data received from the data transmitter / receiver 253 to the outside as voice (utterance of the smart speaker 200) through the speaker 220.

[0023] <4. Terminal equipment> 4 is a block diagram showing the functional configuration of the terminal device 400. The terminal device 400 has, for example, a display device 410, a wireless communication module 420, a storage unit 430, and an information processing unit 440. The display device 410 is a liquid crystal display or an organic EL display, etc., and includes a display screen 411 capable of displaying images and / or videos. The display device 410 includes a touch sensor 412 provided over the display screen 411. The wireless communication module 420 includes a high-frequency circuit, an antenna, etc., and is connectable to a network NW.

[0024] The storage unit 430 stores application programs for managing ingredients and remotely controlling the refrigerator 100. All or a part of the information processing unit 440 is realized by a hardware processor such as a CPU (Central Processing Unit) mounted on the terminal device 400 executing the application programs.

[0025] The information processing unit 440 includes, for example, an operation receiving unit 441, a display device control unit 442, and a data transmission / reception unit 443. The information processing unit 440 is an example of a "processing unit."

[0026] The operation reception unit 441 receives a user's operation on the display screen 411 via the touch sensor 412. The data transmission / reception unit 443 transmits and receives control signals and data to and from the server system 500 by controlling the wireless communication module 420. The data transmission / reception unit 443 transmits information indicating the user's operation content received by the operation reception unit 441 to the server system 500. The data transmission / reception unit 443 outputs a control instruction and display data related to the display received from the server system 500 to the display device control unit 442. The display device control unit 442 displays an image or video included in the display data received from the server system 500 on the display screen 411 based on the control instruction received from the server system 500.

[0027] However, the configuration of the terminal device 400 is not limited to the above example. For example, the terminal device 400 may have a function as a voice dialogue device similar to the above smart speaker 200. The terminal device 400 may also have a camera 450. In this case, an image of the ingredient captured by the camera 450 may be transmitted to the server system 500 and used for ingredient management. This will be described later.

[0028] <5. Server System> FIG. 5 is a block diagram showing a functional configuration of the server system 500. In the following, an example will be described in which the server system 500 controls a terminal device 400, which is an "information display device", and a smart speaker 200, which is a "voice dialogue device". However, the "information display device" controlled by the server system 500 may be the refrigerator 100. In this case, the "terminal device 400" in the following description may be read as the "refrigerator 100". The "voice dialogue device" controlled by the server system 500 may be the terminal device 400 or the refrigerator 100. In this case, the "smart speaker 200" in the following description may be read as the "terminal device 400" or the "refrigerator 100".

[0029] The server system 500 includes a communication module 510, an information processing unit 520, and a storage unit 530. For convenience, functional units other than those for voice dialogue will be described first, followed by functional units for voice dialogue. The communication module 510 includes a communication circuit and the like, and can be connected to a network NW.

[0030] <5.1 Configuration of the information processing section> Information processing unit 520 includes, for example, instruction receiving unit 521, registration information management unit 522, display control unit 523, voice processing unit 524, control mode changing unit 525, control instruction generation unit 526, and response generation unit 527. However, some or all of these functional units may be provided in terminal device 400 or refrigerator 100 instead of server system 500. Information processing unit 520 is an example of a "processing unit".

[0031] The instruction receiving unit 521 receives information indicating a user's operation (user's operation on the display screen 411) received by the operation receiving unit 441 of the terminal device 400 from the terminal device 400 via the communication module 510. The instruction receiving unit 521 outputs the information received from the terminal device 400 to the registration information management unit 522.

[0032] The registration information management unit 522 updates various DBs (such as an inventory list DB 601, a shopping list DB 602, a candidate list DB 604, a trash can DB 603, a regular item DB 607, a user setting DB 606, and a remote operation registration DB 609) described later based on information received from the instruction reception unit 521. In addition, in this embodiment, the registration information management unit 522 receives information indicating contents (user's instruction by speech) analyzed by the voice processing unit 524 described later from the voice processing unit 524. The registration information management unit 522 updates the various DBs described above based on information received from the voice processing unit 524. The registration information management unit 522 will be described in detail later. In the following description, "user's operation" or "user's input" means a user's operation (user's operation on the terminal device 400) received by the instruction reception unit 521, or a user's operation (operation by speech on the smart speaker 200) analyzed by the voice processing unit 524 described later.

[0033] The display control unit 523 generates display data and a control signal including an image or video to be displayed on the display screen 411 of the terminal device 400 based on the contents of the various DBs. The display control unit 523 transmits the generated display data and control signal to the terminal device 400 via the communication module 510. The control signal generated by the display control unit 523 is a control signal for operating the display device control unit 442 of the terminal device 400. Here, "display on the display screen" is not limited to the meaning of displaying an image or video on the display screen 411 by the display device control unit 442 of the terminal device 400, but may also include displaying an image or video on the display screen 411 by providing display data and / or a control signal from outside to the terminal device 400.

[0034] The voice processing unit 524 receives voice data acquired by the smart speaker 200 from the smart speaker 200 via the communication module 510. For example, the voice processing unit 524 receives voice data other than the trigger word from among the voice data acquired by the smart speaker 200. The voice processing unit 524 analyzes the voice data received from the smart speaker 200. For example, the voice processing unit 524 converts the voice data into text by performing voice recognition processing, and performs semantic interpretation on the text voice data by performing natural language processing. The voice processing unit 524 outputs the analysis result of the voice data to the control mode changing unit 525, the control instruction generating unit 526, and the response generating unit 527.

[0035] The control mode change unit 525 changes the operation state of the smart speaker 200 based on the analysis result of the voice data by the voice processing unit 524. The control mode change unit 525 will be described in detail later.

[0036] Based on the analysis result of the voice data by the voice processing unit 524, if the user's utterance includes an instruction for the refrigerator 100 (for example, an instruction to change the set temperature range of each storage compartment 120, change the control mode of the chilled compartment 122, "quick freezing" or "quick ice making"), the control instruction generating unit 526 accepts the instruction. The control instruction generating unit 526 generates a control instruction corresponding to the accepted instruction for the refrigerator 100. The control instruction generating unit 526 transmits the generated control instruction to the refrigerator 100 via the communication module 510. The control unit 160 of the refrigerator 100 realizes the control desired by the user by driving the actuator 150 based on the control instruction generated by the control instruction generating unit 526.

[0037] When there is content to be answered in response to the user's utterance analyzed by the voice processing unit 524 or when there is content to be notified to the user, the response generation unit 527 refers to the voice DB 610 (described later) and generates voice data to be output from the smart speaker 200. The response generation unit 527 transmits the generated voice data to the smart speaker 200 via the communication module 510. The smart speaker 200 drives the speaker 220 based on the voice data generated by the response generation unit 527 to answer or notify the user by voice.

[0038] <5.2 Various DBs> The storage unit 530 stores an inventory list DB 601, a shopping list DB 602, a trash can DB 603, a candidate list DB 604, an ingredient DB 605, a user setting DB 606, a regular item DB 607, a history DB 608, a remote operation registration DB 609, and a voice DB 610. The inventory list DB 601, the shopping list DB 602, the trash can DB 603, the candidate list DB 604, the user setting DB 606, the regular item DB 607, and the history DB 608 are prepared for each user.

[0039] For the sake of convenience, the following will show an example in which the above-mentioned multiple DBs exist as a DB for managing ingredients, but these DBs may be provided as an integrated unit and managed by adding identification information (e.g., a tag) to the registration of each ingredient to indicate that it will be displayed in a specific list (a shopping list or a candidate list) described below. Note that the processes for the various DBs described below (e.g., registering, deleting, and checking ingredients) are executed by the registration information management unit 522.

[0040] <5.2.1 Inventory list DB> FIG. 6 is a diagram showing an example of the contents of stock list DB601. The "stock list" is a list in which ingredients owned by the user are managed. The "stock list" is an example of a "first list." In the following description, the "stock list" may be referred to as an "ingredient list." Inventory list DB601 registers ingredients that are in stock at the user's home. Inventory list DB601 registers not only ingredients stored inside refrigerator 100, but also ingredients stored outside refrigerator 100 (e.g., stored at room temperature). Stock list DB601 is an example of "stock information" that indicates the quantities of ingredients owned by the user.

[0041] In the inventory list DB 601, for each ingredient, the "management ID," "ingredient name," "ingredient category," "storage location," "purchase date," "best before date" (or "best before date"), "quantity," "unit," and "history" are registered in association with each other. "For each ingredient" refers to, for example, the same ingredient purchased at the same time. In other words, even if it is the same type, an ingredient purchased on one date and time and an ingredient purchased on a different date and time are registered as different ingredients. Note that the "purchase date" here refers to the registration date when a specific ingredient is newly registered, i.e., the date when the inventory amount increases, and is used to include cases where the ingredient was not obtained by purchasing it.

[0042] "Food classification" refers to the classification of food ingredients. In this embodiment, 16 types of food classifications are set as "food classifications", including "vegetables", "fruits", "meat", "seafood / seaweed", "eggs / dairy products", "beans / bean products", "processed foods", "rice / breads", "noodles", "beverages", "frozen foods", "sweets", "seasonings", "spices", "side dishes", and "others". However, the above classifications are examples, and the classifications of food ingredients can be set as appropriate.

[0043] "Storage location" refers to the location where food ingredients are stored. In this embodiment, four types of storage locations are set as "storage locations", namely, "refrigerator compartment", "vegetable compartment", "freezer compartment", and "other". "Refrigerator compartment" corresponds to the area combining refrigerator compartment 121 and chilled compartment 122. "Vegetable compartment" corresponds to vegetable compartment 123. "Freezer compartment" corresponds to the area combining small freezer compartment 125 and main freezer compartment 126. "Other" corresponds to storage locations outside refrigerator 100.

[0044] However, the above classification is an example, and the classification of the storage location can be set as appropriate. For example, the refrigerator compartment 121 and the chilled compartment 122 may be classified as different storage locations. The small freezer compartment 125 and the main freezer compartment 126 may be classified as different storage locations. Furthermore, the classification of the storage location may be finely classified, such as "the top shelf of the refrigerator compartment" and "the right door pocket of the refrigerator compartment." The "storage location" may be called a "storage room" or a "storage area." For the sake of convenience in the following description, when "refrigerator compartment 121" is referred to, it is a concept that may include the chilled compartment 122, and when "freezer compartment 126" is referred to, it is a concept that may include the small freezer compartment 125.

[0045] The latest quantity of an ingredient is registered in "quantity." When an ingredient is reduced, information indicating the reduced quantity and the date and time of reduction is registered in "history."

[0046] The registration information management unit 522 performs registration, deletion, and confirmation of ingredients in the stock list DB 601 based on, for example, a user's operation. In this embodiment, the registration information management unit 522 notifies the terminal device 400 that there is an ingredient whose expiration date (or expiration date) is approaching a predetermined number of days (for example, one day) before the expiration date (or use-by date) of the ingredient registered in the stock list DB 601. This notification is displayed as a push notification on the display screen 411 of the terminal device 400. When the expiration date (or use-by date) of an ingredient registered in the stock list DB 601 has passed, the registration information management unit 522 automatically deletes the ingredient whose expiration date (or use-by date) has passed from the stock list DB 601, and transfers the registration of the ingredient to the trash bin DB 603 and candidate list DB 604 described later. The "stock list" is displayed as an "ingredient list" on the display screen 411 of the terminal device 400 under the control of the server system 500. A display example of the "stock list (ingredient list)" will be described in detail later.

[0047] <5.2.2 Shopping list DB> FIG. 7 is a diagram showing an example of the contents of the shopping list DB 602. The "shopping list" is a list that the user can refer to when shopping to prevent forgetting to buy, and is a list of ingredients that the user has confirmed to purchase (for example, an explicit purchase intention). The "shopping list" is an example of a "third list", and is a list that is more likely to be purchased than a candidate list, which will be described later. However, the "shopping list" is not limited to the user's explicit purchase intention, and may include ingredients that are registered as presumptions based on the user's regular stock intention (intention to keep a certain ingredient as a regular item). For example, ingredients registered in the "shopping list" are once registered in a "candidate list" to be described later, and then moved from the "candidate list" to the "shopping list" based on the user's intention (user operation) to be registered in the "shopping list". However, ingredients registered in the "shopping list" may be registered directly in the "shopping list" based on the user's operation without going through the candidate list. The "shopping list" is displayed on the display screen 411 of the terminal device 400 under the control of the server system 500. A display example of the "shopping list" will be described in detail later.

[0048] Ingredients to be displayed as a "shopping list" are registered in the shopping list DB 602. In the shopping list DB 602, each ingredient is registered in association with a "management ID," "ingredient name," "ingredient category," "quantity," and "unit." The registration information management unit 522 registers, deletes, and checks ingredients in the shopping list DB 602 based on user operations.

[0049] <5.2.3 Trash DB> FIG. 8 is a diagram showing an example of the contents of the trash can DB 603. In the trash can DB 603, ingredients deleted from the stock list DB 601 are temporarily registered. For example, the trash can DB 603 registers ingredients that were registered in the stock list DB 601, but were consumed and deleted from the stock list DB based on a user's operation. In this embodiment, the trash can DB 603 also registers ingredients that were registered in the stock list DB and were automatically deleted from the stock list DB by the registration management unit 522 because their best-before date (or expiration date) has passed. The ingredients registered in the trash can DB 603 can be re-registered in the stock list DB by a user's operation. The ingredients registered in the trash can DB 603 are automatically deleted from the trash can DB 603 when a predetermined period (e.g., three days) has passed since the date of registration in the trash can DB 603. Even if an ingredient whose best-before date (or expiration date) has passed is registered in the stock list DB 601, it may be deleted by waiting for a user's operation rather than automatically deleting it from the stock list DB 601. Similarly, ingredients registered in the trash DB 603 may be deleted upon user operation without setting a period until automatic deletion. In the trash DB 603, a "management ID," "ingredient name," "ingredient classification," and "registration date in the trash DB 603" are registered in association with each ingredient.

[0050] <5.2.4 Candidate List DB> FIG. 9 is a diagram showing an example of the contents of the candidate list DB604. The "candidate list" is a list that the user can refer to as an auxiliary to prevent forgetting to buy an ingredient, and is a list of ingredients that are candidates for purchase that the user has not confirmed to purchase (e.g., an explicit intention to purchase). The "candidate list" is an example of a "second list". The "candidate list" is an example of information that prompts the user to purchase an ingredient based on a decrease in the stock of an ingredient. The "candidate list" is purchase suggestion information, and is generated, for example, by using a deletion operation of an ingredient registered in the stock list DB601. The "candidate list" is displayed on the display screen 411 of the terminal device 400 under the control of the server system 500. A display example of the "candidate list" will be described in detail later. In the candidate list DB604, ingredients to be displayed as the "candidate list" are registered. In the candidate list DB604, for each ingredient, a "management ID", "ingredient name", "ingredient classification", and "registration date to the candidate list DB604" are registered in association with each other. As described below, in the case where a specific ingredient registered in the stock list DB 601 is registered in the candidate list DB 604 when the quantity of that ingredient decreases to a specified quantity before it becomes zero, the "quantity" of that ingredient (quantity in stock) may also be registered in the candidate list DB 604 in association with that ingredient.

[0051] In this embodiment, when the quantity of an ingredient registered in the stock list DB 601 is reduced and a predetermined condition is satisfied, the reduced ingredient is automatically registered in the candidate list DB 604. The "predetermined condition" is, for example, that the number of ingredients after the reduction falls below a predetermined threshold. In this embodiment, the "predetermined condition" is that the number of ingredients after the reduction falls to zero (i.e., the ingredient is out of stock). The "predetermined condition" is an example of the "first condition."

[0052] For example, when an ingredient registered in the stock list DB 601 is deleted and registered in the trash bin DB 603, the same ingredient as the ingredient registered in the trash bin DB 603 is automatically registered in the candidate list DB 604. "Deletion of registration" means that the stock of a specific ingredient is exhausted (changed from available to unavailable). That is, the candidate list DB 604 automatically registers an ingredient registered in the stock list DB 601 that has been consumed and deleted from the stock list DB based on a user's operation, and an ingredient registered in the stock list DB that has automatically been deleted from the stock list DB by the registration information management unit 522 because the expiration date (or use-by date) has passed. In both the shopping list DB 602 and the candidate list DB 604, ingredients that are exhausted (or reduced) in stock are registered. The shopping list DB 602 and the candidate list DB 604 are updated at different times. In the server system 500 of this embodiment, when a user deletes a registration of an ingredient in the inventory list DB 601 in response to eating / throwing away an ingredient, the ingredient would normally no longer need to be displayed thereafter, but the server system 500 uses this information as data to recommend shopping.

[0053] The registration information management unit 522 can accept a user operation to move an ingredient registered in the candidate list DB 604 to the shopping list DB 602. For example, the user can select an ingredient to be moved in the candidate list described below by tapping (single pressing) the display of the ingredient, and move the selected ingredient to the shopping list. When the selected ingredient is registered in the shopping list DB 602, the registration of the ingredient is deleted from the candidate list DB 604, which is the source of the move.

[0054] Here, when an ingredient is already registered in the candidate list DB604, the same ingredient may be directly registered in the shopping list DB602 by a user operation (i.e., an ingredient is registered in the shopping list DB602 without going through the candidate list DB604). In this case, the registration information management unit 522 registers the ingredient in the shopping list DB602 and deletes the registration of the ingredient from the candidate list DB604. In other words, an ingredient already registered in the shopping list DB602 will not be registered in the candidate list DB604 even if the inventory quantity in the inventory list DB601 satisfies the above-mentioned "predetermined condition."

[0055] When the number of ingredients registered in the inventory list DB601 is reduced and the above-mentioned specified condition is satisfied, and the ingredient is a special ingredient previously registered by the user (e.g., an ingredient registered in the regular item DB607), the registration information management unit 522 may directly register the ingredient in the shopping list DB602 without registering the ingredient in the candidate list DB604.

[0056] In this embodiment, a maximum number (e.g., 20) of ingredients that can be registered in the candidate list DB 604 is set. When a predetermined number (e.g., the maximum number) of ingredients are already registered in the candidate list DB 604, the registration information management unit 522 sequentially deletes old registrations in the candidate list DB 604 as new ingredients are registered in the candidate list DB 604.

[0057] The registration information management unit 522 deletes old registrations from the candidate list DB 604 in order of priority based on one or more of the registration date in the candidate list DB 604, the purchase date of the ingredient, the expiration date of the ingredient, or the expiration date of the ingredient. For example, the registration information management unit 522 deletes old registrations from among the ingredients registered in the candidate list DB 604 in order of the oldest purchase date of the ingredient, the oldest expiration date of the ingredient, or the oldest expiration date of the ingredient. Note that, when a specific ingredient registered in the stock list DB 601 is registered in the candidate list DB 604 when the quantity of the ingredient is reduced to a predetermined quantity before it becomes zero as described above, among the ingredients still remaining in stock in the stock list DB 601, the ingredients may be deleted in order of the newest expiration date of the ingredient, or the newest expiration date of the ingredient.

[0058] Instead of / in addition to the above priority order, the registration information management unit 522 deletes old registrations from the candidate list DB 604 according to a priority order based on the ingredient classification to which the ingredient belongs. For example, if an ingredient belongs to a specific ingredient classification, the registration information management unit 522 lowers the priority order for deletion of that ingredient (or removes it from the list of ingredients to be deleted), making it harder for that ingredient to be deleted from the candidate list DB 604. A "specific ingredient classification" is an ingredient classification that is similar to a regular ingredient, such as seasonings or spices.

[0059] Instead of / in addition to the above priority order, the registration information management unit 522 deletes old registrations in the candidate list DB 604 according to a priority order based on the past inventory frequency of the ingredient and / or whether the ingredient is included in an ingredient group designated by the user. If the past inventory frequency of the ingredient is equal to or greater than a threshold, the registration information management unit 522 lowers the priority for deleting the ingredient (or removes it from the list of ingredients to be deleted), making it difficult for the ingredient to be deleted from the candidate list DB 604. The past inventory frequency can be obtained based on the history DB 608, which will be described later. If the ingredient is included in an ingredient group designated by the user, the registration information management unit 522 lowers the priority for deleting the ingredient (or removes it from the list of ingredients to be deleted), making it difficult for the ingredient to be deleted from the candidate list DB 604. The "designated ingredient group" is, for example, an ingredient group registered in the stock item DB 607.

[0060] By providing the candidate list, it is possible to reduce forgetting to add items to the shopping list. In addition, the candidate list also assists in adding items to the shopping list, which reduces the workload of adding items to the shopping list.

[0061] <5.2.4 Ingredients DB> FIG. 10 is a diagram showing an example of the contents of the ingredient DB 605. The ingredient DB 605 is a default DB set up in common for multiple users. The ingredient DB 605 is used for automatic input of ingredient information, which will be described later. The ingredients registered in the ingredient DB are a large number of commonly distributed ingredients. In the ingredient DB 605, for each ingredient, an "ingredient name," "accepted ingredient name," "ingredient classification," "ingredient classification icon," "default storage location (recommended storage location)," and "best before date (or expiration date)" are registered in association with each other. Each of the "ingredient classification," "ingredient classification icon," "default storage location," and "best before date (or expiration date)" is an example of "associated information." The ingredient DB 605 is an example of "related information" in which an ingredient name and associated information are associated with each other, and is an example of "first information." However, the "first information" referred to here is a different concept from the "first information I1," which will be described later. In this embodiment, an ingredient classification icon is registered as one of the associated information, but an image indicating the ingredient classification may be used instead. The ingredient DB 605 is information that stores general correspondences between ingredients and storage locations of the ingredients (standard correspondences that are not different for each individual user but are set in advance to be common to all users unless otherwise specified by the user). The correspondences are an example of the "first correspondences."

[0062] As the "reception candidate for ingredient name", one or more reception candidates are registered taking into consideration input variation of ingredient name. For example, if the ingredient is cherry tomato, in addition to "cherry tomato (written in katakana)", "petit tomato (written in hiragana)", "tomato", etc. are registered. As the "default storage location", a storage location inside or outside the refrigerator 100 that is suitable for storing ingredients is registered. As described above, if the refrigerator compartment 121 has shelves, the "storage location" may be a category indicating a part of a specific storage compartment 120 that is distinguished from other locations, such as "the top shelf of the refrigerator compartment 121". Note that the "reception candidate for ingredient name" is used only for association with "accompanying information" (i.e., use of "related information"), and the name of the ingredient to be shown to the user may be the name registered by the user itself without being converted by the "reception candidate for ingredient name". For example, if the user inputs "Tomato from XX", then "Tomato" is registered as a "Candidate for Accepted Ingredient Name", and associated information such as the ingredient classification "Vegetable" and storage location "Vegetable compartment" prepared in association with "Tomato" is registered in the inventory list DB 601, while the name of the ingredient shown to the user on the display screen 411 of the terminal device 400 may be "Tomato from XX" or the like, without modification, as the name input by the user. Alternatively, the name may be displayed with only a portion that is considered to be a spelling variation changed, such as "Tomato from XX".

[0063] Here, the arrangement of storage compartments 120 of refrigerators 100 differs depending on the model. For this reason, the recommended storage location for ingredients may differ depending on the model. Therefore, in this embodiment, ingredient DB 605 is prepared for each model of refrigerator 100, for example. In other words, ingredient DB 605 differs depending on the model of refrigerator 100. Registration information management unit 522 identifies the model of refrigerator 100 of the user based on model information of refrigerator 100 obtained by referring to remote control registration information DB 609 described later, and selects ingredient DB 605 corresponding to the identified model. Note that the information that differs depending on the model of refrigerator 100 may be prepared not for the entire ingredient DB 605, but only for a part of the information in ingredient DB 605, such as information indicating storage locations.

[0064] 2, the description will continue. In addition to the above-mentioned DBs, storage unit 530 stores a user setting DB 606, a stock item DB 607, a history DB 608, a remote operation registration DB 609, and a voice DB 610.

[0065] The user setting DB 606 is information registered by a user. In order to provide flexibility in the automatic input of ingredient information, which will be described later, the user-specific change settings for the ingredient DB are registered in the user setting DB 606. For example, the user setting DB 606 registers a user-specific correspondence between ingredients and storage locations. This correspondence is an example of a "second correspondence". Specifically, in the ingredient DB 605, "refrigerator" is registered as the default storage location for "chicken". If the user purchases chicken and stores it in the freezer more than in the refrigerator, the user can register "chicken" and "freezer" in association with each other in the user setting DB 606. The user setting DB 606 is another example of "association information" in which an ingredient name and associated information are associated with each other, and is an example of "second information". However, the "second information" referred to here is a different concept from the "second information I2" described later. The registration information management unit 522 registers, deletes, and checks the user setting DB 606 based on the user's operation.

[0066] The stock DB 607 stores ingredients that the user frequently uses (e.g., ingredients that the user does not want to run out of). The stock DB 607 stores ingredients (special ingredients) that the user "declares" to keep in stock. The registration information management unit 522 registers, deletes, and checks the stock DB 607 based on the user's operation.

[0067] The history DB 608 can be used instead of the user setting DB 606 or in combination with the user setting DB 606. The history DB 608 registers the user's history (the usage history of the refrigerator 100) regarding the storage of ingredients, for example, without the user's explicit registration. The registration information management unit 522 can obtain information regarding which storage compartment 120 is frequently used for a certain ingredient by referring to the history DB 608. The history DB 608 is generated based on the user's registration results in the inventory list DB 601 and / or the results of photography by the camera 170 of the refrigerator 100, etc.

[0068] In the remote operation registration DB 607, a "management ID," a "model," and a "management ID of a terminal device registered for remote operation" are registered in association with each other for each refrigerator 100. The display control section 523 identifies the model of the user's refrigerator 100 from the remote operation registration DB 607 based on the management ID of the terminal device 400 received from the terminal device 400, and changes the image of the refrigerator 100 displayed on the display screen 411 of the terminal device 400 based on the identified model of the refrigerator 100. This will be described in detail later.

[0069] The voice DB 610 stores a dictionary for interpreting the meaning of user utterances, voice information for generating voice data for responses, etc. Furthermore, the storage unit 530 stores image data of icons of food classifications.

[0070] <6.Screen display> Next, a description will be given of screen displays displayed on the display screen 411 of the terminal device 400 when managing ingredients in the refrigerator 100 using the terminal device 400. The screen displays described below are realized by the display device control unit 442 of the terminal device 400 executing control based on display data and control instructions transmitted from the server system 500.

[0071] <6.1 Top page of refrigerator management screen> FIG. 11 is a diagram showing images displayed on display screen 411. On the left side of FIG. 11 is image S1 showing a top page displayed when an application program is started on terminal device 400. Image S1 includes a pattern R0 which shows refrigerator 100 in a schematic manner, and a plurality of indicators D11-D16. Pattern R0 includes indicators R0a-R0f which show each storage compartment 120 of refrigerator 100 in a schematic manner. The indicators R0a-R0f are indicators which show refrigerator compartment 121, chilled compartment 122, vegetable compartment 123, ice compartment 124, small freezer compartment 125, and main freezer compartment 126, respectively. Indicators D11-D16 show the setting contents (operating state) for each storage compartment 120.

[0072] When the user presses display D17 indicating "Settings" located in the upper right of image S1, display control unit 523 displays image S2 indicating a menu superimposed on image S1. When the user operates (e.g., taps) "Ingredients Management" included in image S2, display control unit 523 displays image S3 (FIG. 13) of the top page of ingredients management on display screen 411.

[0073] <6.2 Food Management Top Page> 12 shows an image S3 of the top page of ingredient management. The image S3 includes a first picture R1, a second picture R2, first to fourth display boxes D1 to D4, and first to fourth icons K1 to K4.

[0074] In this embodiment, in image S3, patterns R1a-R1e showing each storage compartment (storage location) 120 included in a first pattern R1 that typically shows the refrigerator 100 are displayed in association with first information I1 about ingredients stored in the storage compartment 120. Image S3 is an example of a "first display." Information included in each of the first to fourth display boxes D1-D4 is an example of "first information I1." The first information I1 includes information about all the ingredients stored in each storage compartment 120. For example, the first information I1 is the number of ingredients stored in each storage compartment 120.

[0075] However, the first information I1 may be information indicating the degree of filling (degree of use) of each storage chamber 120. The "degree of filling" is information based on the calculation result of dividing the number of ingredients stored in each storage chamber 120 by the number of ingredients when each storage chamber 120 is full, and is expressed as a filling rate (use rate) or a filling amount ("large", "normal", "small")). The filling rate of each storage chamber 120 can be estimated more accurately according to the type and quantity of ingredients currently registered as being stored, if the volume and dimensions occupied by each type of food are determined. Instead of the volume and dimensions of the ingredients, each combination of the type and quantity of ingredients that can be stored in each storage chamber 120 may be determined in advance. The first information I1 may include the names of ingredients stored in each storage chamber 120 instead of / in addition to the number of ingredients and the degree of filling.

[0076] An example of this embodiment will be described in detail below. The first pattern R1 is a pattern that typically shows the refrigerator 100. The first pattern R1 includes patterns R1a-R1e that typically show each storage compartment 120 of the refrigerator 100 in addition to the external shape of the refrigerator 100. The first pattern R1 is composed of, for example, a line that shows the external shape of the refrigerator 100 and a line that shows the boundary between the multiple storage compartments 120. The first pattern R1 includes a pattern R1a that shows the refrigerator compartment 121, a pattern R1b that shows the vegetable compartment 123, a pattern R1c that shows the ice compartment 124, a pattern R1d that shows the small freezer compartment 125, and a pattern R1e that shows the main freezer compartment 126.

[0077] In this embodiment, the first pattern R1 does not include a pattern indicating the chilled compartment 122, but includes one pattern R1a corresponding to the refrigerator compartment 121 and the chilled compartment 122. In this embodiment, the first pattern R1 omits the boundary line between the left and right refrigerator compartment doors even in the double-door refrigerator 100. As a result, the first pattern R1 is a neat pattern. However, the first pattern R1 may include a pattern indicating the chilled compartment 122, or may have a boundary line between the left and right refrigerator compartment doors 130 in the double-door refrigerator 100. Modifications of the first pattern R1 will be described in detail later. The first pattern R1 does not need to accurately represent the dimensions of each door of the actual refrigerator 100, and the dimensional ratio may be appropriately adjusted to make it easier to see or to display information.

[0078] The display control unit 523 identifies the model of the user's refrigerator 100 based on the remote operation registration DB 609, and includes in the image S3 the first pattern R1 of the arrangement of the storage compartments 120 corresponding to the model of the user's refrigerator 100. In this case, the display control unit 523 may use the same first pattern R1 even if there is a slight difference in the dimensional ratio, as long as the arrangement of the doors 130 of the refrigerator 100 is the same.

[0079] The second pattern R2 is arranged outside the first pattern R1. For example, the second pattern R2 is arranged to the side of the first pattern R1. In this embodiment, the second pattern R2 is arranged to the side of the first pattern R1 at the same height as the height of the bottom of the refrigerator included in the first pattern R1. The second pattern R2 diagrammatically indicates a storage location outside the refrigerator 100. The second pattern R2 includes, for example, a pattern indicating a box-shaped storage section.

[0080] The first to fourth display boxes D1 to D4 include, for example, pictures showing speech bubbles. The first display box D1 includes first information I1 related to ingredients stored in the refrigerator compartment 121 and the chilled compartment 122. For example, the first display box D1 displays information showing the number of ingredients stored in the refrigerator compartment 121 and the chilled compartment 122. The first display box D1 is disposed to the side of the refrigerator compartment picture R1a included in the first picture R1. From another perspective, the first display box D1 is disposed at the same height as the refrigerator compartment picture R1a included in the first picture R1.

[0081] The second display box D2 includes first information I1 related to ingredients stored in the vegetable compartment 123. For example, the second display box D2 displays information indicating the number of ingredients stored in the vegetable compartment 123. The second display box D2 is disposed to the side of the pattern R1b of the vegetable compartment 123 included in the first pattern R1. When viewed from another perspective, the second display box D2 is disposed at the same height as the pattern R1b of the vegetable compartment 123 included in the first pattern R1.

[0082] The third display box D3 includes the first information I1 related to the ingredients stored in the freezer compartments 125, 126. For example, the third display box D3 displays information indicating the number of ingredients stored in the freezer compartments 125, 126. The third display box D3 is disposed to the side of the patterns R1d, R1e of the freezer compartments 125, 126 included in the first pattern R1. From another perspective, the third display box D3 is disposed at the same height as the patterns R1d, R1e of the freezer compartments 125, 126 included in the first pattern R1.

[0083] The fourth display box D4 includes first information I1 related to ingredients stored outside the refrigerator 100 (ingredients not contained in the refrigerator 100). For example, the fourth display box D4 displays information indicating the number of ingredients stored outside the refrigerator 100. The fourth display box D4 is positioned directly above the second picture R2. The first information I1 included in the fourth display box D4 is an example of "outside information."

[0084] The information displayed in the first to fourth display boxes D1 to D4 is generated, for example, by the display control unit 523 based on the inventory list DB 601. With this configuration, each storage compartment 120 of the refrigerator 100 is visually linked to the food ingredient information, thereby improving the usability of the inventory management system.

[0085] In this embodiment, when a user operation to specify a first storage location (first storage room) included in the first pattern R1 is accepted, the display control unit 523 displays second information I2 (see FIG. 15) that is more detailed than the first information I1 regarding the food material stored in the first storage location on the display screen 411. The "user operation to specify the first storage location" is, for example, tapping on the pattern indicating the first storage location in the first pattern R1, but is not limited thereto, and may be tapping on the display boxes D1 to D4 corresponding to each storage room 120, or may be specified by speaking through the smart speaker 200.

[0086] The display control unit 523 may change any one of the pattern, color, and size of the storage location in the first pattern R1 that corresponds to the tapped storage room 120. With this configuration, it becomes easy to understand that a tap has been made.

[0087] In this embodiment, when a user operation is received to specify a first storage location included in the first picture R1 for the first picture R1, a list of ingredients stored in the first storage location is displayed on the display screen 411. In this embodiment, when a user operation is received to specify a first storage location included in the first picture R1 for the first picture R1 while the image S3 is displayed on the display screen 411, the display control unit 523 switches the display on the display screen 411 to an image S5 including second information I2 (see FIG. 14) that is more detailed than the first information I1 regarding the ingredients stored in the first storage room. Image S5 is an example of a "second display."

[0088] In more detail, when a user operation to specify the refrigerator compartment 121 for the first pattern R1 is accepted (when the user taps on the pattern R1a or the first display box D1), the display control unit 523 displays an ingredient list of ingredients stored in the refrigerator compartment 121. Similarly, when a user operation to specify the vegetable compartment 123 for the first pattern R1 is accepted (when the user taps on the pattern R1b or the second display box D2), the display control unit 523 displays an ingredient list of ingredients stored in the vegetable compartment 123. When a user operation to specify the freezer compartments 125, 126 for the first pattern R1 is accepted (when the user taps on the pattern R1d, R1e or the third display box D3), the display control unit 523 displays an ingredient list of ingredients stored in the freezer compartments 125, 126. When a user operation to specify an external storage location for the first picture R1 is accepted (when the user taps the second picture R2 or the fourth display box D4), the display control unit 523 displays an ingredient list of ingredients stored outside the refrigerator 100. Each of these ingredient lists is an example of the "second information I2." The ingredient list will be described in detail later.

[0089] In this embodiment, the registration information management unit 522 and the display control unit 523 may receive different control instructions when a user's operation of the first mode is received for each storage chamber 120 (patterns R1a to R1e or first to fourth display boxes D1 to D4) included in the first pattern R1 and when a user's operation of the second mode is received for each storage chamber 120 (patterns R1a to R1e or first to fourth display boxes D1 to D4) included in the first pattern R1. For example, when a user's operation of the first mode is received for each storage chamber 120 included in the first pattern R1, the display control unit 523 displays the second information I2 about the ingredients stored in the selected storage chamber 120, for example, by screen transition. On the other hand, when a user's operation of the second mode is received for each storage chamber 120 included in the first pattern R1, the registration information management unit 522 and the display control unit 523 perform a predetermined process collectively for all ingredients stored in the selected storage chamber 120.

[0090] The "first mode operation" is, for example, a tap (single press) on the display screen 411. The "second mode operation" is, for example, a long press, double tap, strong press (in the case of a pressure-sensitive display), swipe, flick, etc. on the display screen 411. The "predetermined process" may be a process different from the process corresponding to the "first mode operation", but it is more preferable to set a collective operation common to the ingredients stored in the selected storage room 120. For example, it may be a process for deleting the registration of all ingredients stored in the selected storage room 120 at once, or a process for opening a reception window for collective deletion. By providing a collective deletion function, the contents of the refrigerator 100 can be easily organized. Furthermore, if the collective deletion function is associated with the pattern or display box of each storage room 120 included in the first pattern R1, operability is improved.

[0091] From another perspective, when the registration information management unit 522 and the display control unit 523 receive a first mode operation from the user for each storage room 120 included in the first image R1, they open the food management screen (registration screen / display screen) for the selected storage room 120, and if a second mode operation is performed for the same location, which is a long press, they may perform processing to delete all food registrations for that storage location at once.

[0092] The first icon K1 includes, for example, an arrow-shaped pattern and is arranged to the side of the first pattern R1. When the first icon K1 is operated (for example, tapped) by the user, the screen switches to one that displays a list of all ingredients in stock. This will be described in detail later.

[0093] The second to fourth icons K2 to K4 are arranged at the bottom of the image S3. When the second icon K2 is operated (e.g., tapped) by the user, the screen switches to a search screen on which ingredients in stock can be searched. When the third icon K3 is operated (e.g., tapped) by the user, the screen switches to a screen displaying a shopping list. The third icon K3 is an example of "display of an operation unit for switching to a screen displaying ingredients that the user is encouraged to purchase". This will be described later. When the fourth icon K4 is operated (e.g., tapped) by the user, the screen switches to a screen displaying a trash bin list. The trash bin list is a list of ingredients registered in the trash bin DB 603. The trash bin list is generated, for example, by the display control unit 523 referring to the trash bin DB 603.

[0094] In this embodiment, the second to fourth icons K2 to K4 are arranged below the first pattern R1. Alternatively, the second to fourth icons K2 to K4 may be arranged above the first pattern R1. For example, the third icon K3 may be arranged at the top right of the first pattern R1. The third icon K3 is an example of a "second operation unit." A screen including a shopping list that is displayed by operating the third icon K3 is an example of a "fourth display."

[0095] The door arrangement of the refrigerator 100 in the first pattern R1 and the patterns of the first to fourth display boxes D1 to D4 may be appropriately color-coded to make them easy to distinguish.

[0096] 13 shows an image S3A of a modified example of the top page of food management. In this modified example, the display control unit 523 displays a first pattern R1A instead of the first pattern R1. The first pattern R1A is a schematic illustration of the refrigerator 100 with the left and right refrigerator compartment doors open, allowing the inside of the refrigerator compartment 121 to be seen. The first pattern R1A includes a pattern R1aa indicating the chilled compartment 122.

[0097] For example, the display control unit 523 displays the image S3A (instead of the above-mentioned image S3) as an image of the top page of foodstuff management. In another example, the display control unit 523 displays the above-mentioned image S3 as the image of the top page of foodstuff management, and may switch to the image S3A of this modified example when the user operates (e.g., taps) the pattern R1a indicating the refrigerator compartment 121 or the first display box D1 in the image S3. In this case, the first pattern R1A is not limited to a pattern corresponding to the entire refrigerator 100, and may display an enlarged pattern of the refrigerator compartment 121 and the chilled compartment 122 (a pattern in which the other storage compartments 120 are omitted).

[0098] <6.3 Ingredients List> FIG. 14 is an image S4 displaying the all ingredients list L0. The all ingredients list L0 is a list that displays all ingredients registered in the inventory list DB 601 (all ingredients stored inside and outside the refrigerator 100). In the all ingredients list L0, an icon indicating the ingredient classification, the ingredient name, the purchase date, the best-before date (or expiry date), the quantity, and the unit are displayed for each ingredient. The ingredients are arranged in a predetermined order regardless of the storage location. The "predetermined order" is, for example, order by meal name (in alphabetical order), order by oldest (or newest) purchase date, or order by closest (or furthest) best-before date (expiry date). The user can change the above-mentioned predetermined order (rearrange the display order of ingredients) by operating (for example, tapping) the operation unit G0 included in the image S4. This is not limiting, and a function for rearranging ingredients by storage location according to the user's setting operation may also be provided.

[0099] When the user operates the first icon K1 of the image S3 (see FIG. 12), the display control unit 523 displays the image S4. That is, the display control unit 523 generates display data including the complete ingredient list L0 based on the inventory list DB 601 and transmits the display data to the terminal device 400.

[0100] FIG. 15 is an image S5 displaying an ingredient list for each storage compartment 120. For example, FIG. 15 is an image displaying ingredient list L2 for vegetable compartment 123. Ingredient list L2 for vegetable compartment 123 is a list that displays ingredients stored in vegetable compartment 123 selected from ingredients registered in inventory list DB 601. Ingredient list L2 displays an icon indicating the ingredient classification, ingredient name, purchase date, best-before date (or expiry date), quantity, and unit for each ingredient. The ingredients are arranged in a predetermined order, similar to all ingredient list L0. The user can change the predetermined order (rearrange the display order of ingredients) by operating (e.g., tapping) operation unit G0 included in image S5.

[0101] When the user operates (e.g., taps) the image R1b of the vegetable compartment 123 included in the first image R1 of the image S3 (see FIG. 12) or the second display box D2, the display control unit 523 displays the image S5. That is, the display control unit 523 generates display data including the ingredient list L2 of the vegetable compartment 123 based on the inventory list DB 601 and transmits the data to the terminal device 400. The same applies to the ingredient list of the refrigerator compartment 121 and the ingredient lists of the freezers 125, 126.

[0102] 16 is an image S5A of a modified example showing a list of ingredients. In this modified example, image S5A includes operation units O1, O3, and O4 for switching to another display. The other display to which the user can switch includes second information I2 about ingredients stored in another storage location instead of second information I2 about ingredients stored in the vegetable compartment 123.

[0103] For example, image S5A includes operation units O1, O2, O3, and O4 for switching between ingredient list L1 for refrigerator compartment 121, ingredient list L2 for vegetable compartment 123, ingredient list L3 for freezer compartments 125 and 126, and ingredient list L4 for other ingredients (storage outside refrigerator 100). When ingredient list L2 for vegetable compartment 123 is displayed, ingredient list L1 for refrigerator compartment 121, ingredient list L3 for freezer compartments 125 and 126, and ingredient list L4 for other ingredients are each an example of a "third display." The user operates operation units O1, O2, O3, and O4 to switch between the four ingredient lists L1, L2, L3, and L4. The display of operation units O1, O2, O3, and O4 is set in common in images in which ingredient lists for any storage compartment 120 are displayed.

[0104] <6.4 Adding new ingredients to the ingredients list> 17 shows image S6 displayed when registering a new ingredient in the ingredient list (inventory list DB 601). In this embodiment, when the user inputs an ingredient name, the registration information management unit 522 automatically inputs the ingredient classification, storage location, purchase date, best before date (or expiration date), ingredient classification icon, and unit. Each of the ingredient classification, storage location, purchase date, best before date, ingredient classification icon, and unit are examples of "accompanying information."

[0105] More specifically, based on the input ingredient name, the registration information management unit 522 automatically inputs the ingredient classification, default storage location, and unit registered in association with the ingredient name in the ingredient DB 605 as "ingredient classification," "storage location," and "unit," respectively. When the registration information management unit 522 receives an input from a user to register associated information along with the input of an ingredient name, the registration information management unit 522 prioritizes the user's input over the ingredient DB 605, and registers the ingredient in association with the associated information set by the user.

[0106] In this embodiment, if the user setting DB 606 contains information (e.g., a storage location whose settings have been changed) registered in correspondence with the input ingredient name, the registration information management unit 522 inputs the information in the user setting DB 606 in preference to the ingredient DB 605. That is, the registration information management unit 522 can refer to the ingredient DB 605, which indicates a general correspondence between ingredients and storage locations, and the user setting DB 606, which indicates a user-specific correspondence between ingredients and storage locations. If the user setting DB 606 exists, the registration information management unit 522 sets the storage location to be input based on the contents of the user setting DB 606, and if the user setting DB 606 does not exist, the registration information management unit 522 sets the storage location to be input based on the ingredient DB 605. Based on the ingredient classification obtained by referring to the ingredient DB, the registration information management unit 522 acquires information indicating an icon of the corresponding ingredient classification from the storage unit 530 and automatically registers it. The registration information management unit 522 automatically registers the date when the ingredient name was entered as the "purchase date and time" and automatically registers the date and time of the best-before date (or expiry date) estimated based on the purchase date and time. This configuration can reduce the effort required to register ingredients.

[0107] The registration information management unit 522 may automatically register incidental information based on the history DB 608 instead of the user setting DB 606 or in combination with the user setting DB 606. That is, if the history DB 608 of the target ingredient exists, the registration information management unit 522 may input the storage location based on the history DB 608, and if the history DB 608 of the target ingredient does not exist, the registration information management unit 522 may input the storage location based on the ingredient DB 605. In this way, when registering a specific ingredient, if the incidental information is input according to the history DB 608 rather than the ingredient DB 605, the condition may be that the frequency of registration of the ingredient is greater than or equal to a predetermined frequency. Of course, the frequency of registration of a specific ingredient can be obtained by referring to the history DB 608.

[0108] When a predetermined condition is satisfied for a certain ingredient, the registration information management unit 522 sets the state in which the first storage location is input as the associated information, and when the predetermined condition is satisfied, the registration information management unit 522 sets the state in which the second storage location is input as the associated information. The "predetermined condition" is a condition related to the degree of filling of the first storage location or the second storage location, or the amount of ingredient to be registered as inventory. For example, when "refrigerator" is associated with "pork" in the ingredient DB 605, if the filling rate of the refrigerator 121 is higher than a threshold, it is assumed that the user will stop putting the pork in the refrigerator 121 and put it in the freezer 126, and the "freezer" is registered as the associated information for the pork. In another example, when "refrigerator" is associated with "pork" in the ingredient DB 605, if the amount of pork is higher than a threshold, the "freezer" is registered as the associated information for the pork as being frozen.

[0109] The registration information management unit 522 may change the automatically input best-before date (or expiration date) for the same food ingredient depending on whether it is stored in the refrigerator compartment 121 or the freezer compartments 125 and 126. For example, the registration information management unit 522 may set the number of days until the best-before date (or expiration date) to a first number of days (e.g., 3 days) for the same food ingredient when it is stored in the refrigerator compartment 121, and set the number of days until the best-before date (or expiration date) to a second number of days (e.g., 1 week) longer than the first number of days for the same food ingredient when it is stored in the freezer compartments 125 and 126.

[0110] The registered information management unit 522 can accept a user operation to change the associated information after the associated information has been input based on the ingredient DB 605. When the registered information management unit 522 accepts such a user operation, it replaces the associated information automatically registered in the inventory list DB 601 with content based on the user operation.

[0111] In the example described above, when a user inputs an ingredient name to the terminal device 400, the registration information management unit 522 automatically inputs associated information based on the ingredient DB 605 and / or the user setting DB 606, even if no information other than the ingredient name is input from outside. Input of an ingredient name by a user to the terminal device 400 is an example of "input of information indicating an ingredient name from outside."

[0112] However, the "input of information indicating the name of an ingredient from outside" is not limited to manual input by the user. For example, the registration information management unit 522 may receive an image captured by the camera 170 of the refrigerator 100 or the camera 450 of the terminal device 400, identify an ingredient based on the received image, and automatically input the ingredient name, ingredient classification, storage room, purchase date, best-before date (or expiration date), ingredient classification icon, unit, etc. corresponding to the identified ingredient. In this case, the registration information management unit 522 may identify an ingredient from the received image based on, for example, a machine-learned model for ingredient image recognition. Furthermore, instead of the user manually inputting the ingredient name, the registration information management unit 522 may accept the input of the ingredient name by the user's speech to the smart speaker 200, and automatically input the ingredient classification, storage room, purchase date, best-before date (or expiration date), ingredient classification icon, unit, etc. corresponding to the received ingredient name.

[0113] From one point of view, the ingredient DB 605 may include a first DB in which ingredient names and ingredient classifications are associated with each other, and a second DB in which ingredient classifications and storage locations are associated with each other. In this case, the registration information management unit 522 may identify the ingredient classification corresponding to the ingredient based on the information indicating the ingredient name input from outside and the first DB, identify the storage location using the identified ingredient classification and the second DB, and automatically input the identified storage location as associated information.

[0114] FIG. 18 shows image S6A of a modified example for registering a new ingredient in the ingredient list. In this modified example, the characters indicating the content entered by the user and the characters indicating the content automatically entered by the registration information management unit 522 are displayed in different formats. For example, the characters indicating the content entered by the user are displayed in black, and the characters indicating the content automatically entered by the registration information management unit 522 are displayed in a format different from the black text (such as blue or gray text, shown by dashed lines in FIG. 18) as a provisional registration state. In this case, the registration information management unit 522 may set the contents to a permanent registration state based on the user's confirmation operation of the contents displayed in the different formats. According to this modified example, the user can easily distinguish and confirm the contents entered by the user and the automatically entered contents.

[0115] <6.4 Changes to the Ingredient List> Fig. 19 shows image S7 when changing (deleting or moving) the ingredient list. Fig. 19 shows an example of ingredient list L2 in the vegetable compartment 123. The user can select an ingredient by manipulating (e.g. tapping) the display of the ingredient to be selected in ingredient list L2. When an ingredient is selected, first to third icons K5 to K7 are displayed at the bottom of image S7.

[0116] The first icon K5 is an icon for moving the registration of the selected ingredient to another storage room. When the first icon K5 is operated (e.g., tapped) by the user, the display control unit 523 displays a display S8 for selecting a destination room, superimposed on the image S7. The registration information management unit 522 changes the storage location associated with the ingredient in the inventory list DB 601 based on the user's selection on the display S8. As a result, the selected ingredient is deleted from the ingredient list of the storage location before the move.

[0117] The second icon K6 is an icon for adding ingredients to a shopping list (shopping list DB 602) described later. When the second icon K6 is operated (e.g., tapped) by the user, the display control unit 523 adds the selected ingredient to the shopping list DB 602. In this case, the ingredient remains registered in the ingredient list.

[0118] The third icon K7 is an icon for moving the registration of the selected ingredient to the trash bin. When the third icon K7 is operated (e.g., tapped) by the user, the registration information management unit 522 deletes the selected ingredient from the stock list DB 601 and registers the deleted ingredient in the trash bin DB 603. As described above, in this embodiment, when an ingredient deleted from the stock list DB 601 is registered in the trash bin DB 603, the same ingredient is also registered in the candidate list DB 604.

[0119] <6.5 Trash List> 20 shows image S8 including trash list L5. Trash list L5 is a list that displays ingredients registered in trash DB 603. In trash list L5, an icon indicating the ingredient classification, ingredient name, purchase date, best-before date (or expiry date), quantity, and unit are displayed for each ingredient. Image S8 displays a display D21 that informs the user that ingredients moved to trash list L5 will be automatically deleted a predetermined number of days (e.g., three days) after being moved.

[0120] <6.6 Shopping Lists and Wish Lists> 21 shows an image S9 including a shopping list L6 and a candidate list L7. The shopping list L6 is a list that displays ingredients registered in the shopping list DB 602. In the shopping list L6, an icon indicating an ingredient classification, an ingredient name, a planned purchase quantity, and a unit are displayed for each ingredient. The display control unit 523 generates display data including the shopping list L6 based on the shopping list DB 602 and transmits the data to the terminal device 400.

[0121] On the other hand, the candidate list L7 is a list that displays ingredients registered in the candidate list DB 604. In the candidate list L7, an icon indicating an ingredient classification and an ingredient name are displayed for each ingredient. The display control unit 523 generates display data including the candidate list L7 based on the candidate list DB 604 and transmits the display data to the terminal device 400.

[0122] In this embodiment, the shopping list L6 and the candidate list L7 are displayed on the same screen in the display screen 411. For example, the shopping list L6 and the candidate list L7 are displayed side by side one above the other. The term "same screen" refers to a continuous screen in which the display contents are continuous and can be displayed without switching the entire display contents in response to an external operation. The term "same screen" is not limited to a case where the display contents are always displayed simultaneously, and also refers to a screen displayed in response to an operation (so-called scroll operation) in which the display contents change sequentially one part at a time without switching the entire display contents in response to a user's sliding operation on the screen. For example, when there are many ingredients registered in the shopping list K6 and the display screen 411 is occupied by the shopping list L6, the shopping list L6 may be scrolled downward to display the candidate list L7 following the shopping list L6. In other words, the term "same screen" also includes a case where only a part (for example, the bottom end) of the shopping list L6 and the candidate list L7 are displayed simultaneously. In this embodiment, the shopping list L6 and the candidate list L7 are displayed on the same screen of the display screen 411. For example, the shopping list L6 and the candidate list L7 are displayed side by side one above the other. "The same screen" means a screen that the user can view at the same time. However, "the same screen" is not limited to always being displayed at the same time. "The same screen" also includes a case where, for example, when there are many ingredients registered in the shopping list K6 and the display screen 411 is occupied by the shopping list L6, the candidate list L7 is displayed next to the shopping list L6 by scrolling the shopping list L6 downward. In other words, "the same screen" also includes a case where only a part of the shopping list L6 (for example, the bottom end) and the candidate list L7 are displayed at the same time.

[0123] For example, when the shopping list L6 and the candidate list L7 are displayed simultaneously, the registration information management unit 522 accepts a user operation to move the registration of an ingredient included in the candidate list L7 to the shopping list L6. That is, the user can move an ingredient displayed in the candidate list L7 to the shopping list L6.

[0124] FIG. 22 is a diagram showing another image S10 including the shopping list L6. The display control unit 523 can accept a user's operation to hide the candidate list L7 or to make the candidate list L7 less noticeable (for example, to make the color lighter) compared to the shopping list L6 on a screen that can display the shopping list L6 and the candidate list L7 side by side. In this embodiment, the candidate list L7 can be closed based on a user's operation. For example, when the display of the item indicating the candidate list L7 in the image S9 is operated (for example, tapped) by the user, the display control unit 523 switches the image S0 including both the shopping list L6 and the candidate list L7 to an image S10 that displays the shopping list L6 with the candidate list L7 closed. This allows the user to shop while checking only the shopping list L6, and can reduce the risk of buying the wrong item compared to when the candidate list L7 is displayed.

[0125] <7. Voice dialogue function> <7.1 Overall control configuration> Next, the voice dialogue function of the server system 500 of this embodiment will be described. In this embodiment, the information processing unit 520 of the server system 500 can register or delete multiple ingredients before one reception of the voice dialogue function is completed. In other words, the voice processing unit 524 can continuously receive multiple instructions contained in one set of user utterances (a series of utterances, a series of conversations). Note that starting the voice dialogue function (which may be expressed as a "voice recognition function" as described above) means moving to a state in which any utterance by the user can be received. In a state in which the voice dialogue function is started, the instructions contained in the contents of the user's utterance are executed, and responses are made to the user's utterance. When the voice dialogue function is terminated, the state in which the user's utterance is not received is entered, except for a trigger word, which is a cue to start the voice dialogue function. The period from starting the voice dialogue function once to terminating the voice dialogue function is called "one reception of the voice dialogue function".

[0126] More specifically, the voice processing unit 524 ends the voice dialogue function when a predetermined end condition is satisfied after a trigger word is spoken to the smart speaker 200 to activate the voice dialogue function. The control mode change unit 525 can select, based on a user's action, a first mode in which the voice dialogue function is ended under a first end condition, and a second mode in which the voice dialogue function is ended under a second end condition different from the first end condition.

[0127] The first mode is a control mode that accepts one instruction uttered by the user. In the first mode, the activation mode is terminated after one instruction is executed in one activation of the voice dialogue function. "Ending the activation mode" means, for example, that the server system 500 transmits to the smart speaker 200 a control signal that terminates the standby state, or a control signal indicating that the standby state does not need to be continued. Examples of controls that are set to be accepted in the first mode include controls related to the functions of the refrigerator 100. Examples of controls related to the functions of the refrigerator 100 include changing the set temperature range of each storage compartment 120, changing the control mode of the chilled compartment 122, "quick freezing", or "quick ice making". The first termination condition is that one control has been executed in response to the contents of the user's utterance, or that the user has not spoken for a certain period of time or more after the activation of the voice dialogue function (a so-called timeout, and this certain period of time corresponds to the "first standby time" in the present invention). In the first mode, in order to input multiple instructions, it is necessary to utter a trigger word and activate the voice dialogue function once each time an instruction is made.

[0128] The second mode is a control mode capable of accepting multiple instructions by the user's speech in one activation of the voice dialogue function. That is, the second mode is a control mode in which the activation state continues even if one instruction is accepted in one activation. In addition, the second mode has a function of retaining memory so that instructions for multiple processing targets can be executed even if multiple processing targets are included in one group of user's speech content. The "group of user's speech" here means a series of utterances continued by the user without a response from the smart speaker 200 or execution of control based on the speech content. In this embodiment, the instruction accepted in the second mode is an instruction related to food management. The instruction related to food management is, for example, registration, deletion, or stock confirmation for multiple food ingredients. The second end condition is that an instruction to end the voice dialogue function is accepted based on the user's action such as speech. In this embodiment, when the second mode is being executed, if the user does not speak for a predetermined time, a voice (voice for confirming termination) asking whether there is any additional instruction is output from the smart speaker 200, and if the user's response to the voice satisfies a predetermined condition, the second termination condition is considered to be satisfied. The second termination condition will be described in detail later. In this embodiment, the control mode change unit 525 executes the second mode to accept user instructions regarding the registration, deletion, and confirmation of ingredients in various DBs.

[0129] In this embodiment, the control mode change unit 525 executes the second mode based on the user performing a predetermined action. On the other hand, the control mode change unit 525 executes the first mode when the user does not perform the predetermined action. The "predetermined action" is, for example, the user making an utterance including a predetermined word, operating a predetermined operation unit (e.g., a predetermined button) provided on the smart speaker 200 (or the terminal device 400 or the refrigerator 100), or opening (and / or closing) the door 130 of the refrigerator 100.

[0130] The "predetermined words" are words related to ingredients. "Words related to ingredients" are not limited to the names of ingredients themselves, but include synonyms. "Predetermined words" are words that indicate the names of ingredients themselves, words that indicate ingredient classifications such as "vegetables" and "fruits", words related to ingredient management such as "ingredient management", "ingredient registration", "inventory management", "inventory registration", and "new purchase", or words that the user is asked to specify as consecutive skills.

[0131] In other words, when the user's speech includes words related to the functions of the refrigerator 100, the control mode change unit 525 executes the first mode, and when the user's speech includes words related to ingredients or synonyms of ingredients, the control mode change unit 525 executes the second mode.

[0132] In this embodiment, in the first mode, the voice processing unit 524 ends the active state when a first waiting time has elapsed while waiting for the user to speak, and in the second mode, the voice processing unit 524 continues the active state until a second waiting time longer than the first waiting time has elapsed while waiting for the user to speak. For example, in the first mode, the voice processing unit 524 ends the active state after waiting for the first waiting time after the user's voice stops, and in the second mode, the voice processing unit 524 ends the active state after waiting for the second waiting time longer than the first waiting time after the user's voice stops. The first waiting time is, for example, several seconds. The second waiting time is, for example, twice the first waiting time or more, for example, 30 seconds or more, for example, 1 minute or more. The user may open the door 130 of the refrigerator 100 and make an utterance for food management while checking the inside, so the second waiting time is set longer than the first waiting time.

[0133] From another point of view, in the first mode, the voice processing unit 524 executes a process according to the contents of the user's speech, waits for a third waiting time, and then transitions to the next process, and in the second mode, executes a process according to the contents of the user's speech, waits for a fourth waiting time longer than the third waiting time, and then transitions to the next process. For example, in the first mode, the voice processing unit 524 executes a process according to the contents of the user's speech, waits for a third waiting time, and then transitions to the next process, and in the second mode, may wait for a fourth waiting time longer than the third waiting time after the user's speech stops, and then transitions to the next process. The "next process" in the first mode is, for example, a process of informing by voice that the process based on the user's instruction has been completed (or started) (for example, output of a voice indicating "quick freezing has started"). The "next process" in the second mode may be a process of informing the user through voice that the process based on their instructions has been completed (for example, a process of outputting a voice indicating "Registration completed"), or a process of outputting a voice asking whether there are any additional instructions regarding ingredient management (a voice confirming completion, described below).

[0134] In the modified example, in the first mode, the voice processing unit 524 executes a process according to the user's speech and then proceeds to the next process without waiting time, and in the second mode, it waits for a predetermined waiting time after the user's speech stops and then proceeds to the next process. The "next process" in the first mode is, as described above, a process of notifying by voice that the process based on the user's instruction has been completed (or started), for example. The "next process" in the second mode may be, as described above, a process of notifying by voice that the process based on the user's instruction has been completed (for example, a process of outputting a voice indicating "registration is complete"), or may be a process of outputting a voice asking whether there are any additional instructions regarding the food management (a voice for confirming completion, which will be described later).

[0135] In this embodiment, when the second mode is executed, the voice processor 524 outputs a voice asking whether there are any additional instructions regarding food management after the user's voice stops, and ends the active state when the user's response to the voice output satisfies a predetermined condition. The "predetermined condition" is, for example, a case where the user responds by uttering an utterance indicating that there are no additional instructions, an utterance confirming the end of the voice dialogue function, or no utterance from the user for a certain period of time (a so-called timeout).

[0136] 7.2 Flow of Control 23 is a flowchart showing the flow of processing related to the voice dialogue function. In the following, when a predetermined word is included in the user's utterance, the second mode is executed.

[0137] First, when the user speaks, the information processing unit 250 of the smart speaker 200 determines whether the user's speech includes a trigger word (S101). When the user's speech does not include a trigger word (S101: NO), the information processing unit 250 of the smart speaker 200 continues the pause state. On the other hand, when the user's speech includes a trigger word (S101: YES), the information processing unit 250 of the smart speaker 200 activates the voice dialogue function of the smart speaker 200 (S102). As a result, the smart speaker 200 acquires voice data of the user's speech and transmits the acquired voice data to the server system 500.

[0138] The information processing unit 520 of the server system 500 (hereinafter simply referred to as the "information processing unit 520") analyzes the voice data received from the smart speaker 200 and determines whether or not the user's speech contains a predetermined word (e.g., a word related to an ingredient or a synonym of an ingredient) (S103).

[0139] If the user's utterance does not include a predetermined word (S103: NO), the information processing unit 520 accepts an instruction from the user in the first mode (S104) and executes a process corresponding to the accepted instruction (S105). For example, when the information processing unit 520 accepts an instruction related to a function of the refrigerator 100, the information processing unit 520 generates a control instruction for causing the refrigerator 100 to execute the function, and transmits the generated control instruction to the refrigerator 100.

[0140] Then, the information processing unit 520 receives a response from the refrigerator 100 to the transmitted control instruction, and if the response is normal, generates a voice indicating that the instruction received from the user has been executed (or started), and outputs the voice from the smart speaker 200. After that, the information processing unit 520 transmits to the smart speaker 200 a control signal to end the standby state, or a control signal indicating that the standby state does not need to be continued, thereby terminating the active state of the smart speaker 200 and transitioning the smart speaker 200 to a sleep state (S106).

[0141] On the other hand, when the user's utterance contains a predetermined word (S103: YES), the information processing unit 520 switches the control mode from the first mode to the second mode, accepts the user's instruction in the second mode (S111), and executes a process corresponding to the accepted instruction (S112). For example, the information processing unit 520 registers, deletes, and checks a plurality of ingredients in various DBs (e.g., inventory list DB 601) based on the user's instruction. For example, the information processing unit 520 executes sequential processing starting from an instruction whose content has been confirmed among one or more instructions from the user, without asking the user whether there are any additional instructions. Alternatively, the information processing unit 520 may ask the user whether there are any additional instructions, and after confirming that there are no additional instructions, execute the plurality of instructions that have been accepted up to that point all at once.

[0142] Next, the information processing unit 520 judges whether there is an end instruction from the user (S113). The end instruction is, for example, an utterance by the user indicating that there is no additional instruction, and is a word or phrase such as "end," "that's it," "that's all," or "thank you." Alternatively / in addition to this, the end instruction may be an operation of a predetermined operation unit (e.g., a predetermined button) provided on the smart speaker 200 (or the terminal device 400 or the refrigerator 100) (or a pressed operation unit being released), or the door 130 of the refrigerator 100 being closed, which was opened, or the like.

[0143] When there is an instruction to terminate from the user (S113: YES), the information processing unit 520 terminates the active state of the smart speaker 200 by sending a control signal to the smart speaker 200 to terminate the standby state or a control signal indicating that the standby state does not need to continue, and transitions the smart speaker 200 to a hibernation state (S106).

[0144] On the other hand, if there is no end instruction from the user (S113: NO), the information processing unit 520 causes the smart speaker 200 to output an end confirmation voice (e.g., a voice saying "Is there anything else?") asking the user whether there is any additional instruction (S114). Then, the information processing unit 520 determines whether there is an additional instruction uttered by the user (S115). In other words, even if one or more instructions are received from the user, if there is no end instruction from the user, the information processing unit 520 does not end the voice dialogue function and continues the standby state.

[0145] If there is an additional instruction from the user (S115: YES), the information processing unit 520 returns to S112 and executes processing corresponding to the additional instruction. On the other hand, if there is an additional instruction from the user (S115: NO), the information processing unit 520 returns to S113 and checks again whether there is an end instruction from the user. Then, if there is no end instruction from the user even after outputting the end confirmation voice one or more times (e.g., multiple times), the information processing unit 520 ends the voice dialogue function by timeout.

[0146] <7.3 Processing example> When a first ingredient and a first storage location are uttered in a relationship with each other and a second ingredient and a second storage location are uttered in a relationship with each other by the end of one reception of the voice dialogue function, the information processing unit 520 processes the first ingredient and the first storage location in association with each other and processes the second ingredient and the second storage location in association with each other. For example, when the user utters "Add 3 eggs to the refrigerator, add 1 filleted horse mackerel to the freezer, and remove 1 pepper from the vegetable compartment," the registration information management unit 522 processes the "refrigerator" as the storage location for "eggs," processes the "freezer" as the storage location for "horse mackerel," and processes the "vegetable compartment" as the storage location for "pepper."

[0147] When a user speaks about multiple ingredients by the end of one reception of the voice dialogue function, and a storage location is spoken before or after the speech about the multiple ingredients, the information processing unit 520 processes the multiple ingredients in association with the single storage location. For example, when the user speaks "Add 3 eggs and 1 filleted horse mackerel to the refrigerator," or "Add 3 eggs and 1 filleted horse mackerel to the refrigerator," the registration information management unit 522 processes the "refrigerator" as the storage location of the "eggs" and "horse mackerel" in association with the "refrigerator."

[0148] In this embodiment, if the name of the ingredient can be recognized based on the user's utterance, utterance regarding the storage location and / or quantity of the ingredient is not required. For example, if there is no utterance regarding the storage location or quantity, the information processing unit 520 may proceed with registration with those information left blank. Alternatively, the information processing unit 520 may automatically input the default storage location registered in the ingredient DB 605 as the storage location. Alternatively, the information processing unit 520 may output a voice from the smart speaker 200 asking the user about the missing information.

[0149] In this embodiment, the holding status of ingredients managed by the voice dialogue function can also be changed by inputting an input operation on the content of the display screen 411 displayed on the terminal device 400. When the holding status of ingredients is changed by an input operation on the content of the display screen 411, the information processing unit 520 manages the holding status of ingredients by the voice dialogue function based on the holding status after the change.

[0150] The management of the holding status of ingredients includes at least information on the name and quantity of the ingredients. In the voice dialogue function, when the information processing unit 520 receives the name of an ingredient based on the user's utterance but the quantity of the ingredient is not included in the utterance, the information processing unit 520 accepts the registration of the ingredient by treating it as if a predetermined quantity (e.g., one piece or one pack) based on a predetermined rule has been registered. In a modified example, in the voice dialogue function, when the information processing unit 520 receives the name of an ingredient based on the user's utterance but the quantity of the ingredient is not included in the utterance, the information processing unit 520 outputs a voice asking about the quantity of the ingredient, and if information indicating the quantity of the ingredient cannot be obtained based on the user's utterance, the information processing unit 520 does not accept the registration of the ingredient.

[0151] In this embodiment, when the information processing unit 520 notifies the user of the stock of ingredients, and the number of ingredients in stock is equal to or greater than a first predetermined number (e.g., 6 or more), it first outputs a voice indicating how many ingredients are in stock, then outputs a voice indicating the names of a second predetermined number of ingredients (e.g., 5 ingredients) that is less than the first predetermined number, and then generates and outputs a voice asking whether to read out the names of the remaining ingredients. For example, when there are 20 ingredients, the information processing unit 520 first outputs a voice saying "There are 20 ingredients in total," then outputs a voice indicating the names of the five ingredients with the highest priority in a predetermined order of priority (e.g., purchase date, best-before date, use-by date), and then outputs a voice asking the user whether to read out the names of the remaining ingredients. In other words, when many ingredients are registered, it is difficult to listen to all of them at once, so the information processing unit 520 speaks out a few ingredients at a time.

[0152] In this embodiment, when deleting an ingredient registered in the inventory list DB 601, if there are multiple ingredients with the same name, the registration information management unit 522 selects the registration to be deleted based on the purchase date, best-before date, or expiration date. For example, if there are multiple ingredients with the same name, the registration information management unit 522 deletes the registration of an ingredient with an older purchase date, an ingredient with an upcoming best-before date, or an ingredient with an upcoming expiration date. For example, if three tomatoes are registered on February 15th and two tomatoes are registered separately on February 16th, when the user utters "delete four tomatoes," three tomatoes on February 15th and one tomato on February 16th are deleted, for a total of four tomatoes. The method of deletion is not limited to this. For example, if a storage location is specified, such as "delete tomatoes from the vegetable compartment," it is preferable to delete only the ingredients in that storage location. Similarly, when checking the inventory of ingredients, if a storage location is specified by the user, the quantity of the ingredients may be returned only for the ingredients in that storage location. Furthermore, when deleting an ingredient, if the user does not utter the quantity, the ingredient itself may be deleted.

[0153] In this embodiment, the response generating unit 527 generates and outputs a voice prompting the user to add missing ingredients to the shopping list when there is no stock or the stock is below a predetermined amount, thereby eliminating the need to add ingredients to the shopping list.

[0154] An example of utterance is shown below. U indicates the user, and S indicates the smart speaker 200. (Example 1) U: "Hello XX (trigger word), open △△ (name of app)." S: "Yes, what is it?" U: "Food management" S: "Yes." U: "3 carrots registered." S: "I'll register three carrots." U: "3 cucumbers registered" S: "I'll register three cucumbers." U: "That's all." S: "Finish registering ingredients" (except for trigger words) Voice dialogue function disabled

[0155] (Example utterance 2) When at least the information on the ingredient name and the quantity of that ingredient is available, the ingredient registration is accepted, and if there is any missing information, the user is asked. U: "Hello XX (trigger word), open △△ (name of app)." S: "Yes." U: "Please register ingredients." S: "Yes." U: "3 tomatoes." S: "Three tomatoes." U: "Egg" (Predetermined time has passed) S: "How many eggs?" U: "10" S: "I registered 10 eggs." S: "Anything else?" U: "No." S: "Finish registering ingredients."

[0156] <7.3 Working Example> Example 1 The ingredient name is registered in association with the quantity of that ingredient, and even if the spoken content when instructing to register an ingredient does not include information about the quantity, the ingredient name is accepted for registration without inquiring about the quantity from the voice recognition function. For example, ingredients without quantity information are registered with a blank field. Alternatively, ingredients without quantity information are registered with a predetermined amount. In this case, the predetermined amount is registered as something that can be set by the user, or ingredients without quantity information are registered with the minimum unit amount.

[0157] Example 2 Once the ingredient registration is accepted via voice recognition, the accepted details are then read out. The accepted details are read out in a repeat format as each ingredient is registered. In this case, only the ingredient name is read out. When registering, if the quantity of an ingredient is unknown, a voice guide is provided to encourage the user to input it (assuming the system has both an ingredient registration function via voice recognition and an ingredient registration function via manual input).

[0158] Example 3 When registering ingredients, once the intention to register ingredients is received in advance by voice recognition, all information on the ingredient names recognized by the series of voice recognition functions thereafter is accepted as an instruction to register ingredients. For example, if the intention to register ingredients is expressed by verb V and the name of the ingredient to be registered is expressed by object O, when the user speaks "V (O1, O2, O3, O4)", the smart speaker 200 responds with "O1, O2, O3, O4 have been V'd."

[0159] On the other hand, if the user starts speaking the names of ingredients without any prior instruction to register the ingredients, the names of the ingredients are temporarily stored and the smart speaker 200 waits for (or asks for) subsequent instructions (registration / deletion / stock availability). For example, if the user speaks "O1, O2, O3, O4", the smart speaker 200 responds with "...O1, O2, O3, O4. Would you like to V1 these or V2 them?"

[0160] The above-described control unit 160 of the refrigerator 100, the information processing unit 250 of the smart speaker 200, the information processing unit 440 of the terminal device 400, and the information processing unit 520 of the server system 500 are each realized by a hardware processor, such as a central processing unit (CPU) or a graphics processing unit (GPU) mounted in each device, executing a program (software). However, all or part of the control unit 160, the information processing unit 250, the information processing unit 440, and the information processing unit 520 may be realized by hardware (including circuitry) such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), or may be realized by a combination of software and hardware.

[0161] Although the embodiment has been described above, the contents of the embodiment are not limited to the above examples. For example, Fig. 24 to Fig. 38 show modified examples of images displayed on the display screen 411 of the terminal device 400 (or the display screen 141 of the refrigerator 100). For example, Fig. 24 and Fig. 25 show examples in which the first information I1 is displayed in a different manner instead of the first to fourth display boxes D1 to D4. Fig. 25 also shows a pattern illustrating the handles of each door 130.

[0162] Figures 26 to 31 show modified examples of the first pattern R1. That is, the pattern R1 which shows the refrigerator 100 in a schematic manner may be changed according to the shape of the refrigerator 100, the door arrangement, etc. By adding the display boxes D1 to D4 shown in Figures 12 and 13 or similar displays to Figures 26 to 31, each storage compartment 120 and the food ingredient information of each storage compartment 120 can be displayed in a linked manner.

[0163] Figs. 32 to 38 show modified examples of the patterns of the first to fourth display boxes D1 to D4. That is, the first to fourth display boxes D1 to D4 are not limited to rectangular shapes, but may be round, elliptical, oval, or other shapes as appropriate. The positions of the first to fourth display boxes D1 to D4 are also not limited to beside the refrigerator pattern, but may be above, diagonally above, below, or diagonally below. For example, Fig. 35 shows an example in which a part of the display box is displayed partially overlapping the refrigerator pattern. The contents shown in Figs. 12, 13, and 24 to 38 can be used in appropriate combination with each other.

[0164] According to at least one of the embodiments described above, usability can be improved by having a processing unit that displays on a display screen a first display in which the picture of each storage location included in the first picture representing the refrigerator is associated with first information about the food ingredients stored in that storage location.

[0165] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention.

[0166] (Additional Note) [1] An information display device according to an embodiment includes a processing unit that displays on a display screen a first display in which a picture of each storage location included in a first picture representing a refrigerator is associated with first information about ingredients stored in the storage locations; It is an information display device equipped with the above. [2] In the information display device of [1] above, The first information includes information indicating the number of ingredients stored in each of the storage locations or the degree of filling of each of the storage locations. [3] In the information display device of [1] or [2] above, In the first display, the first information about the food material stored in each storage location is displayed beside a picture of the storage location. [4] In the information display device of [1] or [2] above, In the first display, the first information about the foodstuff stored in each storage location is displayed at the same height as the picture of that storage location. [5] In the information display device according to [1] to [4] above, When a user's operation specifying a first storage location included in the first pattern for the first pattern is accepted, the processing unit displays second information, which is more detailed than the first information, regarding the food material stored in the first storage location. [6] In the information display device according to [1] to [4] above, When the processing unit receives a user operation specifying a first storage location included in the first pattern for the first image while the first display is displayed on the display screen, it switches the display on the display screen to a second display including second information that is more detailed than the first information regarding the food ingredient stored in the first storage location. [7] In the information display device of [6] above, the second display includes a display of a first operation unit for switching to a third display; The third display includes the second information about ingredients stored in a second storage location of the refrigerator instead of the second information about ingredients stored in the first storage location. [8] In the information display device according to [1] to [7] above, The first display includes a second operation unit for switching to a fourth display in which ingredients are displayed to encourage the user to purchase, and the second operation unit is displayed below or above the first image. [9] In the information display device according to [1] to [8] above, In the first display, in addition to the first pattern and the first information being displayed in association with each other, a second pattern arranged outside the first pattern is displayed in association with outside information regarding food ingredients that are not stored in the refrigerator.

[10] In the information display device according to any one of [1] to [9] above, When a first mode operation by the user is accepted for a first storage location included in the first picture, the processing unit displays second information regarding the ingredients stored in the first storage location that is more detailed than the first information, and when a second mode operation by the user is accepted for the first storage location included in the first picture, the processing unit performs a predetermined process collectively on all ingredients stored in the first storage location. [Explanation of symbols]

[0167] 1...information processing system (foodstuff management system), 100...refrigerator, 200...smart speaker, 250...information processing unit, 400...terminal device, 440...information processing unit (processing unit), 500...server system, 520...information processing unit (processing unit).

Claims

1. On the computer, displaying first information related to ingredients stored in the refrigerator on a display screen; displaying on a display screen a special display including a first list and a second list showing ingredients that a user of the refrigerator is encouraged to purchase; Acquiring regular item registration information registered as a regular item; When the amount of an ingredient registered as the regular item based on the regular item registration information is reduced, the ingredient is displayed in a first list, and when the amount of an ingredient not registered as the regular item is reduced, the ingredient is displayed in a second list different from the first list. Displaying, in the first list, ingredients for which the user's intention to purchase has been confirmed from the second list; A program to achieve this.

2. The computer includes: Designating a predetermined ingredient as the stock item through an operation by the user; The program according to claim 1 for realizing the above.

3. The computer includes: When the foodstuff registered as a regular item is to be displayed in the special display, the foodstuff is displayed in a shopping list associated with the user regardless of the user's intention to purchase the foodstuff. The program according to claim 2 for realizing the above.

4. The computer includes: When an ingredient registered as stock is deleted or when the amount of the ingredient registered as stock is reduced and a predetermined condition is satisfied, the ingredient is displayed in the special display; The program according to any one of claims 1 to 3 for realizing the above.

5. The computer Displaying first information about food stored in the refrigerator on a display screen; displaying on a display screen a special display including a first list and a second list showing ingredients that a user of the refrigerator is encouraged to purchase; Obtain the registration information of regular supplies that are registered as regular supplies, When an amount of an ingredient registered as a regular item based on the regular item registration information is reduced, the ingredient is displayed in a first list, whereas when an amount of an ingredient not registered as a regular item is reduced, the ingredient is displayed in a second list different from the first list; An information display method for displaying, in the first list, ingredients for which the user's intention to purchase has been confirmed from the second list.

Images