Food Management System

Abstract

To provide a foodstuff management system capable of smoothly performing foodstuff registration using voice input in combination.SOLUTION: A foodstuff management system causes one or more computers to operate as: a start part which starts receiving voice input; a display output part which displays a voice-recognized character string on a display screen; an editing processing part which accepts editing of the character string displayed on the display screen as operated by a user and also displays the character string having been edited on the display screen when the character string is edited; a termination part which terminates the voice-recognized character string and then handles a subsequently voice-recognized character string as a new character string; and a registration processing part which adds the character string displayed on the display screen as a foodstuff name to a foodstuff list as operated by the user.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a foodstuff management system. [Background technology]

[0002] In recent years, advances in IT technology have enabled the management and control of home appliances using the voice input function of mobile information terminals. For example, in relation to refrigerators, Patent Document 1 proposes, with the aim of providing an information processing system that can improve usability in managing ingredients stored in a refrigerator, that "the information processing system has a processing unit that can register multiple types of ingredients before one reception of the voice dialogue function is completed. The processing unit can execute a first mode that ends the activated state when one instruction is received in one activation, and a second mode that continues the activated state even if one instruction is received in one activation, and executes the second mode for registering ingredients." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-128559 Summary of the Invention [Problem to be solved by the invention]

[0004] According to Patent Document 1, continuous voice input is possible for the mobile information terminal without having to use a wake-up word each time, and for unknown information such as the quantity associated with the food name, guidance is provided to encourage manual input.

[0005] However, with voice input, there is a possibility of unintended ingredient names being recognized due to voice recognition errors, and there is room for improvement in smoothly correcting these errors while registering ingredients. [Means for solving the problem]

[0006] In view of the above, the present invention provides an ingredient management system in which one or more computers are caused to operate as an initiation unit that starts accepting voice input, a display output unit that displays a voice-recognized character string on a display screen, an edit processing unit that accepts editing of the character string displayed on the display screen in response to a user's operation and, if edited, displays the edited character string on the display screen, a termination adding unit that adds a terminator to the voice-recognized character string and then treats the voice-recognized character string as a new character string, and a registration processing unit that adds the character string displayed on the display screen as an ingredient name to an ingredient list in response to a user's operation. The food ingredient management system further comprises any one of the following configurations (1) to (3): It was decided. (1) The display output unit displays two or more character strings, and the editing processing unit is capable of editing each of the character strings displayed by the display output unit. The registration processing unit adds each of the character strings displayed by the display output unit to an ingredient list as an ingredient name and is capable of referencing a master database that stores ingredient names and ingredient detail information corresponding to the ingredient names. When the registration processing unit operates, the display output unit searches the master database for ingredient names corresponding to each of the character strings displayed, and for a character string with a corresponding ingredient name, the ingredient name and the corresponding ingredient detail information can be added to the ingredient list as a set. If the search of the master database using the character string displayed by the display output unit as a search word results in the acquisition of multiple candidate ingredients that are exact matches, the ingredient detail information will not be set for the ingredient name even if the registration processing unit is operated. (2) The display output unit displays a character string and, after the operation of the registration processing unit, automatically operates the start unit and is capable of referring to a master database that stores ingredient names and ingredient detail information corresponding to the ingredient names. The display output unit searches the master database for an ingredient name corresponding to the character string displayed by the display output unit. If a corresponding ingredient name is found, the display output unit displays it as a candidate ingredient. When a selection of a candidate ingredient displayed on the display screen is accepted from a user, the registration processing unit adds a pair of the ingredient name of the candidate ingredient and the corresponding ingredient detail information to an ingredient list. When the display output unit searches the master database using the character string displayed as a search word and obtains multiple candidate ingredients that are exact matches, the ingredient detail information is not set for the ingredient name even if the registration processing unit is operated. (3) Depending on the presence of ingredients included in the ingredient list for which time has passed without an estimated consumption date being set, or the number of ingredients for which an estimated consumption date has not been set, an output is provided to encourage the setting of an estimated consumption date. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram showing an example of the overall configuration of a foodstuff management device according to an embodiment of the present invention. [Diagram 2] FIG. 4 is a diagram showing an example of the screen configuration of the portable information terminal 2 when voice input is performed. [Diagram 3] FIG. 13 shows an example of screen transition in method 1. [Figure 4] FIG. 13 is a diagram showing an example of screen transition when candidate information exists in method 1. [Diagram 5] FIG. 13 is a diagram showing an example of screen transition when candidate information does not exist in method 1. [Figure 6] FIG. 11 is a sequence diagram showing a series of processing steps among a user, a mobile information terminal, and a server in method 1. [Figure 7] FIG. 13 is a diagram showing an example of screen transition in method 2. [Figure 8] FIG. 13 shows an example of screen transition in method 2. [Figure 9] FIG. 13 shows an example of a screen transition when changing an ingredient name from a provisional list display. [Figure 10] FIG. 13 shows an example of screen transition when deleting an ingredient name from the provisional list display. [Figure 11]FIG. 13 is a sequence diagram showing a series of processing steps among a user, a mobile information terminal, and a server in method 2. [Figure 12] FIG. 2 is a diagram showing an example of the configuration of a master database and a concierge database. [Figure 13] FIG. 13 is a diagram showing a schematic flow when referring to a master database and a concierge database. [Figure 14] FIG. 13 is a diagram showing an example of a data structure of a concierge database. [Figure 15] A diagram showing the assumed storage chamber locations for each model and setting. [Figure 16] FIG. 13 is a diagram showing the screen transition when referring to a concierge database. [Figure 17] FIG. 13 is a sequence diagram showing a procedure for linking a master database and a concierge database. [Figure 18] A diagram showing the process when there are multiple ingredients that exactly match the search word. [Figure 19] A diagram showing how to handle the labeling of food ingredients that have passed their recommended consumption date. [Figure 20] FIG. 13 is a diagram showing how to handle an ingredient that has been deleted from the list of ingredients being stored. [Figure 21] FIG. 13 is a diagram showing an example of a top screen displayed immediately after starting an ingredient management function. [Figure 22] 13A to 13C are diagrams showing screen transitions relating to a reminder function for registering estimated consumption. [Figure 23] FIG. 13 shows screen transitions when the user can select a method for adding ingredients. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. In the embodiment, a foodstuff management device (foodstuff management system) will be described, but the present invention is not limited to being applied to foodstuffs, and can also be applied to any object. EXAMPLES

[0009] Fig. 1 is a diagram showing an example of the overall configuration of a foodstuff management device according to an embodiment of the present invention. The foodstuff management device includes a mobile information terminal 2 as a computer, and a server 3 as a computer that stores a master database 31, an ingredient list 32, and a concierge database 33. These are linked via a cloud or a network (hereinafter simply referred to as cloud 5) to perform computational processing of the computer, and generate a screen for the mobile information terminal 2 as appropriate. For example, the ingredient list 32 may be stored in the mobile information terminal 2. A refrigerator 4 as an example of a storage facility can be at least connectable to the cloud 5 if it is equipped with a camera, which will be described later.

[0010] As shown in FIG. 12, in the master database 31, a unique ingredient ID 311 and ingredient name 312 are stored in association with a search word 313 and reference ingredient information 310 (estimated consumption period 314, recommended storage location 315, ingredient category 316, and one or more standard image data 317, 318).

[0011] Furthermore, the ingredient list 32 that can be displayed on the mobile information terminal 2 is a list in which one or more ingredients are registered, and for each ingredient, an ingredient name 322 and ingredient details 320 (estimated consumption date 324, storage location 325, ingredient category 326, ingredient image 327) are linked and stored in the ingredient list 32. Please refer to Fig. 12 which shows an example of the display screen of the mobile information terminal 2 displaying the ingredient list 32. In addition, as will be described later, a link button 72 with the concierge database 33 can be displayed on the display screen of the ingredient details 320 shown in Fig. 12.

[0012] The ingredient management device creates and holds an ingredient list 32 of ingredients managed by the user, and sets and stores ingredient name 322 and ingredient detail information 320 for each ingredient registered in ingredient list 32. The ingredient name 322 and ingredient detail information 320 can be set by referring to ingredient name 312 and reference ingredient information 310 in master database 31, or can be set by user 1 via mobile information terminal 2.

[0013] In the mobile information terminal 2, a processing unit CPU, a RAM for temporarily storing data, a ROM for recording programs, a communication unit Tr, and other components are connected via a bus BUS. The processing unit CPU can perform various processes according to the programs recorded in the ROM, but Fig. 1 mainly describes a processing function specialized for voice input. The mobile information terminal 2 can be equipped with a microphone for receiving voice input.

[0014] The functions related to voice input include a start unit 21a that starts accepting voice input as a function of the voice input unit 21 related to voice input, and a continuation unit 21b that starts accepting voice input of a new character string without the need for operation by the user 1. In addition, the display output unit 22 displays the voice input character string on the display screen, an end adding unit 23a that adds an end to one character string as a function of the character conversion unit 23 that converts the input voice into a character string, an edit processing unit 24 that accepts editing of the character string displayed on the display screen in response to the operation of the user 1, and a registration processing unit 25 that registers ingredients corresponding to each displayed character string in the ingredient list 32. In addition, the individual functions are organically combined and executed, and the overall control via the cloud 5, including user requests, is managed by the central processing unit 26. The central processing unit 26 is not essential.

[0015] 2 shows an example of the screen configuration of the portable information terminal 2 during voice input. In this case, the display screen example is preferably configured to include at least a main screen 91 at the top, a character input screen 92 at the bottom, and a voice input status display section 93 and a character string display section 94 at the middle, with various menu buttons 96 being arranged at the bottom, for example.

[0016] On the main screen 91, information is successively changed and displayed in response to voice input or operational instructions from the user 1. The character input screen 92 is a character input screen using a keyboard, and enables the user 1 to manually input text (character strings). The voice input status display unit 93 uses, for example, a microphone symbol to distinguish between when voice input is enabled and when voice input is disabled. The character string display unit 94 displays a character string generated in response to character input from a keyboard using the character input screen 92 or voice input from a microphone, either as is or after converting it into kanji or the like.

[0017] In the above-described foodstuff management device, in the first embodiment of the present invention, voice input is performed by the mobile information terminal 2 according to the processing procedure shown in Fig. 3 or Fig. 7. This series of processing is performed based on instructions from the overall processing unit 26.

[0018] The processing procedure shown in Fig. 3 or 7 is executed by the foodstuff management device shown in Fig. 1. For example, text editing, which will be described later, is realized by the edit processing unit 24 accepting editing of a character string displayed on the display screen of the mobile information terminal 2 in response to an operation by the user 1. Continuous input by voice is realized by the continuation unit 21b of the voice input unit 21 starting to accept voice input of a new character string.

[0019] <Method 1: Registering one ingredient at a time> The process of registering ingredients in the ingredient list 32 using method 1 will be described. Fig. 3 is a diagram showing an example of screen transitions in method 1. Fig. 4 is a diagram showing an example of screen transitions in a case where candidate information exists in method 1. Fig. 5 is a diagram showing an example of screen transitions in a case where candidate information does not exist in method 1.

[0020] First, when voice input is started by operating the voice input status display unit 93 or inputting a wake-up word (S10), voice input is accepted and the recognized character string is displayed on the character string display unit 94. When the end of voice input is detected, for example, when a period of silence continues for one second or more or the voice input status display unit 93 is tapped (S11, Y), it is determined that the voice input (provisional input) of one ingredient name has ended and that this is the end of one character string (ingredient name). The character string that is voice recognized after the terminator is added can be treated as a new character string. If no registration instruction, which will be described later, is given between the time when the terminator is added and the time when new voice recognition is started, and there is only one candidate ingredient name found by searching the master database 31, the candidate ingredient name may be registered in the ingredient list 32, or the character string that was input immediately before may be discarded.

[0021] Next, the display of the voice input status display unit 93 is set to voice reception OFF, and reception of voice input is temporarily suspended. Also, at least up to this point, the edit processing unit 24 is enabled and a keyboard is displayed on the character input screen 92. In Figs. 4 and 5, the edit processing unit 24 is enabled before reception of voice input, but this may be done.

[0022] Next, the master database 31 is searched using the character string displayed in the character string display section 94 as the search word 313 (S12). If an ingredient (ingredient ID 311 or ingredient name 312) that matches the search word 313 is found (S13, Y), the ingredient name 312 is displayed as a candidate ingredient name, for example, on the main screen 91 (S14).

[0023] When the voice recognition is not working properly, the user can edit the character string displayed in the character string display unit 94 as the voice recognition result through the edit processing unit 24. If the character string is edited (S15, Y), the process returns to S12 again with the edited character string. This makes it easy to correct the voice recognition result.

[0024] When a candidate ingredient name displayed on the main screen 91 is selected by tapping or the like (S16, Y), it is determined that a registration instruction has been issued with the selected ingredient name 312. That is, the ingredient name 312 is added to the ingredient list 32 as an ingredient name 322, and the reference ingredient information 310 stored in the master database 31 is automatically set to the ingredient (S17). On the other hand, if a registration instruction is issued by operating, for example, the "OK" button on the keyboard when no candidate ingredient is found in the master database 31, the reference ingredient information 310 cannot be used, so the food details information 320 is not automatically set, and the ingredient name entered in the character string display unit 94 is added to the ingredient list 32. After adding one ingredient to the ingredient list 32 in this way, voice input is started again (S10). This start may be performed automatically as an operation of the continuation unit 21b, or may be performed by operating the voice input status display unit 93.

[0025] If there is no ingredient in the master database 31 that matches the search word 313 in S13, the candidate ingredients (ingredient name 312) are not displayed (S13N). In this case, although not shown in FIG. 3, the user may edit the character string (S15) and search the master database 31 again to find candidate ingredients (S12). Also, an ingredient name that is not stored in the master database 31 may be added to the ingredient list 32 as it is (the character string displayed in the character string display section 94). In this case, the ingredient is added to the ingredient list 32 as the ingredient name 322, but since there is no reference ingredient information 310 to refer to, the detailed ingredient information 320 is not automatically set.

[0026] Also, step S12 may be performed before the end of voice input.

[0027] Figures 4 and 5 show examples of screen transitions in each state in the first method, with Figure 4 showing an example of screen transitions when candidate information exists (S13, Y) and Figure 5 showing an example of screen transitions when candidate information does not exist (S13, N). Both show screen transitions starting from S10.

[0028] First, a description will be given with reference to Fig. 4. The screen display contents change in sequence from screen 101 to screen 105.

[0029] Screen 101 can be displayed immediately before S10 (the "Start" state in FIG. 3). Main screen 91 can display a portion of the ingredient list that lists ingredients that have been registered at this point. When voice input status display section 93 is tapped, the screen transitions to screen 102 (S10).

[0030] A keyboard is displayed as a character input screen 92 in a part of the screen 101, for example in the lower portion. This allows for both voice input and text editing.

[0031] Screen 102 is a screen during voice input, and voice input status display section 93 is highlighted to indicate that voice input is in progress. Character strings confirmed by voice input are displayed in character string display section 94. In this example, the user speaks "tomato," and the voice is recognized as intended.

[0032] Screen 103 is the screen after voice input (S11, Y). Based on the voice input, master database 31 is referenced and "tomato" and then "cherry tomato" are displayed on main screen 91 as candidates corresponding to the voice input (S14). If the voice recognition result is not good and a character string such as "Tometou" is displayed in character string display area 94, it can be edited using character input screen 92. Note that in this state, voice input from the microphone is stopped (S12), so a display indicating that voice input is disabled is displayed on voice input status display area 93. Also, while the candidate ingredient names are being displayed, the ingredient displays on ingredient list 32 are hidden. This makes it easy for the user to recognize that candidate ingredients are being displayed.

[0033] Screen 104 is a candidate confirmation screen, and allows a user to instruct (select) one of the ingredients displayed on main screen 91 by tapping, etc. In method 1, for example, tapping on a candidate ingredient name functions as registration instruction unit 95. Here, it is assumed that "tomato" has been selected and confirmed.

[0034] Screen 105 is the screen after the registration instruction. The selected ingredient "tomato" has been added to the ingredient list 32 on the main screen 91. In this case, the ingredient information for "tomato" is not added to the ingredient list 32 only with the ingredient name 322 "tomato", but the ingredient details 320 are automatically set by referencing the reference ingredient information 310 set in the master database 31. For example, based on the date 329 (see FIG. 12) of "tomato" being added to the ingredient list 32, the number of days set in the estimated consumption period 314 is referenced, and the date this number of days after the date of addition 329 is automatically set as the estimated consumption date 324 for "tomato". Also, for example, from the item with the ingredient name 312 "tomato" in the master database 31, the recommended storage location 315 "vegetable compartment" is referenced and set as storage location 325.

[0035] On screen 105, after "tomato" is added to the ingredient list, voice input is automatically activated (S10). The voice input status display unit 93 is highlighted to indicate that voice input is in progress. In this way, in method 1, which accepts instructions to register one ingredient at a time, acceptance of voice input is temporarily stopped after the end of the character string is added (S12). After edit processing unit 24 operates in response to instructions from user 1, and registration instruction unit 95 operates to add one ingredient to ingredient list 32, continuation unit 21b operates to start accepting voice input of a new character string. Input of the next ingredient can immediately begin.

[0036] Next, Fig. 5 will be described. The screen display contents change in sequence from screen 111 to screen 115. Note that screens 111 and 112 are basically the same as screens 101 and 102 in Fig. 4, so the description will not be repeated. However, it is assumed that screen 112 has been entered with "〇△□×", which is a character string that does not have a matching search word 313 registered in master database 31.

[0037] In screen 113, when the master database 31 is referenced using the character string in character string display section 94 as search word 313, no matching candidate ingredients are found (no ingredient names 312 linked to search word 313 are found), so no candidates are displayed on the main screen 91 (S13N).

[0038] Screen 114 shows a scene where the "Done" button as the registration instruction section 95 is selected by keyboard operation on character input screen 92 when no candidate ingredients have been found.

[0039] Screen 115 is the screen after the registration instruction has been given, and an ingredient has been added with the character string "〇△□×" that was entered on main screen 91 as ingredient name 322. However, because no corresponding candidate ingredient has been found in master database 31, food detail information 320 such as estimated consumption date 324 and storage location 325 cannot be automatically set. In other words, only the ingredient name 322 is registered, and ingredient detail information 320 is registered as unset (S17). Then, voice input acceptance begins again (S10).

[0040] In this way, if no candidate information exists, the image 327 of the ingredient and the estimated consumption date 324 are not automatically set. For the ingredient “〇△□×”, the user can select the ingredient from the ingredient list 32, edit it, and manually set the ingredient details information 320.

[0041] 6 is a sequence diagram showing a series of processing steps between the user 1, the mobile information terminal 2, and the server 3 in the first method. In processing step S201, the user 1 taps the microphone button (corresponding to the voice input status display unit 93 in FIG. 2), and the mobile information terminal 2, upon receiving this, starts the voice input function in processing step S202. When the user 1 inputs voice in processing step S203, the voice is converted into a character string and then the character string is displayed in processing step S204. In processing step S205, the character string after voice input can be edited by operating the keyboard.

[0042] The character string edited by voice input and / or keyboard operation is sent to the server 3 as search word 313 in processing step S206, and candidates are extracted by referring to the master database 31 in the server 3. In processing step S207, the candidate ingredient names are presented on the mobile information terminal 2. As is clear from the above example, the number of candidate ingredient names can be 0, 1, or 2 or more.

[0043] In processing step S208, user 1 checks the candidate ingredient names 312 displayed on mobile information terminal 2. If user 1 finds a suitable candidate ingredient name 312 among the displayed candidate ingredient names 312, he or she can select that candidate ingredient name 312 and give a registration instruction; if not, user 1 can edit the character string to change search word 313, or give a registration instruction to register the ingredient as it is in ingredient list 32 as it is, since it is not in master database 31. In this way, ingredient name 322 to be added to ingredient list 32 is confirmed.

[0044] The information on the ingredient name for which registration has been instructed is sent to the server 3 in processing step S209, and is added to the ingredient list 32 in the server 3 in processing step S210. At this time, the ingredient details information 320 can be automatically set by referring to the reference ingredient information 310 in the master database 31. The ingredient list 32 may be stored in the mobile information terminal 2.

[0045] In process step S211, the registered ingredient list 32 and ingredient detail information 320 are displayed on the mobile information terminal 2. In process step S202, the process moves to a voice input standby state to wait for the next voice input.

[0046] <Method 2 that allows you to register multiple ingredients at once> FIG. 7 is a diagram showing the processing procedure of the second method. First, when voice input is started by operating the voice input status display unit 93 or inputting a wake-up word (S20), voice character input is accepted. When the end of voice input is detected by, for example, a period of silence lasting for one second or more or by tapping the voice input status display unit 93 (S21, Y), it is determined that the voice input (provisional input) of one ingredient name has ended and that this is the end of one character string (ingredient name). That is, one character string is displayed as a provisional list in the character string display unit 94 (S22). In method 2, multiple ingredients can be kept in the provisional list. For this reason, the provisional list is displayed in the range of the main screen 91 where part of the ingredient list 32 was displayed in method 1.

[0047] Furthermore, in method 2, acceptance of voice input is resumed (S22). The following steps S23 and S25 are executed as interrupt processing. The acceptance of voice input may be resumed automatically as an operation of the continuation unit 21b, or may be resumed by operating the voice input status display unit 93.

[0048] During the reception of voice input (S22), if any of the character strings (ingredient name) registered in the provisional list is selected by tapping or the like (S23, Y), the edit processing unit 24 activates the selected character string and executes character string editing (S24). During the execution of character string editing, voice input is stopped.

[0049] Furthermore, when a registration instruction is given by, for example, tapping the registration button serving as the registration instruction unit 95 (S25, Y), the master database 31 is searched for each character string registered in the provisional list as search word 313. For each character string, if an ingredient matching the search word 313 is found, the corresponding ingredient name 312 is added to the ingredient list 32, and the reference ingredient information 310 is automatically set as the ingredient details 320 for that ingredient. If no matching ingredient is found, the ingredient details 320 are not automatically set, and the character string is added to the ingredient list 322 as the ingredient name 322. On the other hand, if a registration instruction is not given (S25, N), voice input acceptance continues (S20).

[0050] Figures 8, 9, and 10 show examples of screen transitions in each state in method 2. Figure 8 is an example of a screen transition in method 2. Figure 9 is an example of a screen transition when an ingredient name is changed from the provisional list display. Figure 10 is an example of a screen transition when an ingredient name is deleted from the provisional list display.

[0051] First, in Fig. 8, the screen display contents are sequentially changed from screen 121 to screen 125. However, this screen shows an example in which only the main screen 91 and the voice input status display section 93 are displayed. First, screen 121 can be displayed in the state immediately before S20. In this screen, for example, the main screen 91 may display information for explaining to user 1 how to perform the operations shown in Figs. 9 and 10. For example, "You can edit the ingredient name by tapping the ingredient name in the temporary list" or "You can delete the ingredient by swiping the ingredient name in the temporary list." Receiving voice input is started by tapping the voice input status display section 93 in screen 121 (S20).

[0052] A screen 122 is a screen during voice input, and the voice input status display portion 93 is highlighted to indicate that voice input is in progress. A character string input by voice is displayed on the main screen 91 as a character string display portion 94.

[0053] Screen 123 is a screen where voice input of one character string has been completed (S21, Y). In this example, the user speaks "carrot" and the voice is recognized as intended. After finishing speaking "carrot", voice input of one character string ends when silence continues for, for example, one second or when the voice input status display unit 93 is tapped, and the end of one character string is added. After the end is added, if there is no interruption such as editing the character string or a registration instruction, acceptance of voice input is started again (S20). Furthermore, if one or more character strings are input to the temporary list in the character string display unit 94, the registration instruction unit 95 (register button) is activated.

[0054] Screen 124 shows a case where the user similarly inputs "broccoli," "leeks," and "salmon" by combining voice input with appropriate character string editing (repeating S20 to S25, N).

[0055] When a registration instruction is given, for example by tapping the registration instruction unit 95 (S25, Y), each of the ingredient names (character strings) registered in the temporary list in the character string display unit 94 is used as a search word 313 to search the master database 31 and automatically set the ingredient detail information 320, as described in method 1.

[0056] Screen 125 shows the state after S26 is completed and transition to the list display screen of ingredient list 32. Screen 126 shows detailed display of each ingredient on the list display screen of ingredient list 32. Screen 126 displays the number of days remaining until estimated consumption date 324 (displayed as "1 day remaining") and image data 327 for an ingredient with ingredient name 322 "carrot". Also displayed superimposed on image data 327 are a vegetable icon 3251 indicating the vegetable compartment which is storage location 325, and an "N" icon 3252 indicating that the ingredient has been newly registered in ingredient list 32. In this way, various icons indicating information about the ingredient, such as storage location 325 or the fact that the ingredient has been newly registered, can be displayed superimposed on image data 327 in the list display screen.

[0057] In the example of Fig. 8, the character string "sake" is a polysemous word that can be the search word 313 of the food ingredient names "sake" and "salmon." In this example, one of these, "salmon," is automatically selected and registered in the food ingredient list 32. However, as will be described later, for polysemous words, an instruction from the registration instruction unit 95 may be used as a trigger, for example, to transition to a screen that allows the user to select which food ingredient in the master data 31 to select.

[0058] 9 shows an example of screen transitions during character string editing in method 2 (S23, S24). A tentative list of character string display section 94 of main screen 91 (tentatively displaying "carrot," "broccoli," "leeks," and "sake") and a registration instruction section 95 are displayed on screen 131. A microphone icon is displayed as voice input status display section 93. Here, it is assumed that "carrot" in this character string display section 94 is tapped to edit it.

[0059] In the next screen 132, the character input screen 92 is displayed and the edit processing unit 24 is put into operation. In addition, the character string display unit 94 displays the parts other than the selected character string "Ninjin" darkly. Using the keyboard on the character input screen 92, the character string "Ninjin" can be corrected to, for example, "Ninniku". In the screen 133 after correction, the character string edit in the character string display unit 94 is reflected.

[0060] 10 is an example of screen transitions for deleting an ingredient from the temporary list in the character string display section 94. When a predetermined operation, for example a swipe operation, is performed on the character string "carrot" in the character string display section 94 of the main screen 91, a "delete" button is activated as displayed on the next screen 142. When the "delete" button is tapped, the screen transitions to screen 143. In other words, "carrot" is deleted from the temporary list.

[0061] Fig. 11 is a sequence diagram showing a series of processing steps according to Fig. 7 between user 1, mobile information terminal 2, and server 3 in the second method. In processing step S221, user 1 taps on voice input status display section 93 to instruct acceptance of voice input, and mobile information terminal 2 starts accepting voice input in processing step S222 (S20). When user 1 inputs voice in processing step S223, the voice is converted into a character string and then displayed in processing step S224, and the end of the string is determined based on a silent period or the like (S21, Y), and one ingredient is added to the tentative list (S22). The system enters a voice input waiting mode again (S20).

[0062] In process step S225, when an instruction to correct the character string using the keyboard is given by the user 1 in process step S224, the input character string can be corrected or deleted by operating the keyboard. The voice input and character string correction input in process step S223 may be repeated multiple times, and process steps S224 and S225 are executed each time accordingly. In the figure, the symbols a, b, and c represent multiple processes. The reception of voice input (S20) may be temporarily stopped while the character string is being edited (S23, S24).

[0063] Then, in processing step S226, user 1 issues a registration instruction (S25, Y). This registration instruction is sent to server 3 together with the provisional list in processing step S227. In processing step S228, master database 31 in server 3 is referenced to search for each of the character strings in the provisional list as search word 313 to see if there are any matching ingredients. If there are any matching ingredients, in processing step S229, ingredient name 312, estimated consumption period 314, etc. are obtained from master database 31, linked as detailed ingredient information 320, and registered in ingredient list 32. When used in conjunction with a concierge function described later, ingredient ID 311 may also be obtained.

[0064] <Linkage with Concierge Database 33> The concierge database 33 is a database that can be created for a function independent of the food management function (food list 32 management function). For example, the concierge database 33 includes detailed information for each model of refrigerator produced by a certain manufacturer, such as what kind of storage compartments the refrigerator has and what temperature setting patterns the refrigerator has. The concierge database 33 can be created for the concierge function that can output advice corresponding to the model of refrigerator when storing a certain food. More specifically, if a certain model of refrigerator has only a refrigerator and a chilled compartment, the concierge database 33 is a function that can advise whether it is more suitable to store each food in the refrigerator or the chilled compartment. For example, advice can be given that the chilled compartment should be used for the food name "prosciutto" and the refrigerator should be used for the food name "milk". In addition, the location of the chilled compartment in the refrigerator of that model can be clearly indicated by using an icon that simulates the refrigerator. In addition, if the temperature setting of a certain storage compartment of the refrigerator of that model can be changed, for example, if the temperature setting of the refrigerator can be changed by the refrigeration temperature setting or the freezing temperature setting, advice can be given for each setting.

[0065] Master database 31 uses ingredient name 312 (or ingredient ID 311) as a unique key, and includes as data items such as estimated consumption period 314 and recommended storage location 315 at the level of storage compartments (refrigerator, freezer, vegetable compartment, etc.) of a typical refrigerator, making it suitable for general management of ingredients. Meanwhile, concierge database 33 also uses ingredient name 332 (or ingredient ID 331) as a unique key, but includes information on refrigerator model and settings as data items. For this reason, linking (integrating) master database 31, which is premised on general-level information on refrigerators, with concierge database 33, which includes detailed information on models and settings, can contribute to the creation of convenient functions.

[0066] In this embodiment, when searching for ingredients in the concierge database 33 using the ingredient management function, the ingredient name entered or set using the ingredient management function (a function that uses ingredient list 32, etc.) is automatically entered, thereby contributing to convenience.

[0067] FIG. 12 is a diagram showing an example of the configuration of the master database 31 and the usage concierge database 33. As shown in FIG.

[0068] The master database 31 includes an ingredient ID 311 that uniquely corresponds to the ingredient ID 331 stored in the usage concierge database 33, an ingredient name 312 that uniquely corresponds to the ingredient ID 311, a search word 313 that corresponds to the ingredient name 312, an estimated consumption period 314 which is an indicator corresponding to the expiration date etc. and is an estimate of consumption from the time of ingredient registration, a recommended storage location 315 assigned as being suitable for the ingredient indicated by the ingredient name, an ingredient category 316 assigned to correspond to the ingredient indicated by the ingredient name 312, standard image data 317 showing the appearance etc. of the ingredient indicated by the ingredient name 312, and standard image data 318 corresponding to the ingredient category 316.

[0069] As described above, the detailed ingredient information 320 displayed on the mobile information terminal 2 can be automatically set to information obtained by referring to the master database 31 using the search word 313. The detailed ingredient information 320 may include information that is not displayed on the mobile information terminal 2, and in this embodiment includes an ingredient ID that is the same as the ingredient ID 311 in the master database 31. Then, by tapping the link button 72 on the display screen of the detailed ingredient information 320, the concierge database 33 can be searched for and referenced as an ingredient corresponding to this ingredient ID.

[0070] 13 explains the above-mentioned process. First, in process step S31, the master database 31 is searched through the character string input of the search word 313 on the mobile information terminal 2, and the ingredient name 312 "carrot" is found and an instruction to register it is issued. In response to this, in process step S32, the master database 31 is referenced, and information on the estimated consumption period 314, recommended storage location 315, ingredient category 316, and standard image data 317, 318 corresponding to the ingredient name 312 "carrot" is read, and these are automatically set in the ingredient list 32 as detailed ingredient information 320 (S33).

[0071] Furthermore, suppose that user 1 wishes to refer to information other than the detailed ingredient information 320 using the concierge function. For example, in order to obtain advice on the appropriate storage location for the ingredient name 312 "carrot" in the model of refrigerator used by user 1, user 1 operates the link button 72 displayed on the display screen of the detailed ingredient information 320 for the ingredient name 312 "carrot". This causes a search of the concierge database 33 using the ingredient ID 311 (which is not particularly limited as long as the link can be established, and may be the ingredient name 312). As described above, the ingredient ID 311 in the master database 31 and the ingredient ID 331 in the usage concierge database 33 are unique, so that it is possible to search and refer to the corresponding ingredient. In this way, by specifying the ingredient in the ingredient management function and operating the link button 72, the concierge database 33 can be searched for this ingredient. This saves user 1 the trouble of activating the concierge function and inputting the ingredient name again.

[0072] 14 is a diagram showing an example of the data structure of the concierge database 33. As described above, the ingredient ID 331 is unique to the ingredient ID 311 in the master database 31. The search word 333 is used when the user uses the concierge function, separate from the ingredient management function. Here, since the search is performed from the ingredient ID 311 in the master database 31, it is not necessary to pay much attention to the search word.

[0073] The concierge database 33 has storage location / advice patterns 334 for each food ingredient, which are subdivided by refrigerator model and settings. For example, as illustrated in the storage location / advice pattern 334 in FIG. 14, a refrigerator of "model A" is assumed to have a so-called 6-door layout, which is common to the illustration AM shown in FIG. 15. The refrigerator of "model A" is a model that can have four settings, settings 1-4. Here, the settings 1-4 are different in the temperature settings of the two storage compartments (switchable compartments) shown in illustrations D and K, respectively, and the temperature settings of the storage compartments filled in the illustrations (D, K) are assumed to be setting 1 = (vegetables, frozen), setting 2 = (frozen, vegetables), setting 3 = (vegetables, vegetables), and setting 4 = (frozen, frozen). In addition, in any setting, the storage compartment shown in illustration A is assumed to have a refrigerated temperature setting.

[0074] For example, since the ingredient name "carrot" 332 in FIG. 14 is a vegetable, the switchable compartment with the vegetable temperature setting is recommended as the storage location. If there is no switchable compartment with the vegetable temperature setting, the refrigerator should be recommended. The locations of the switchable compartment for the vegetable temperature range in settings 1-4 correspond to illustrations D, K, L, and M, respectively. As the storage location / advice pattern 334, illustrations D, K, and L corresponding to the locations of the switchable compartment for the vegetable temperature range are set in settings 1, 2, and 3. On the other hand, since there is no switchable compartment for the vegetable temperature range in setting 4, illustration A, which indicates a storage compartment for the refrigerator temperature range, is set.

[0075] 16 is a diagram showing the screen transition when referring to the concierge database 33. Here, "tomato" is used instead of "carrot" as the ingredient to be searched for, but "carrot" would also work. First, screen 141 shows an example of the main screen 91 displayed on the mobile information terminal 2 after the reference ingredient information 310 has been reflected in processing step S33 in FIG. 13. Tapping the link button 72 ("View advice on storage location") on this screen transitions to screen 142.

[0076] Screen 142 is a screen for searching for recommended storage locations for ingredients by type, as an example of the concierge function. By tapping link button 72, ingredient name 322, which is set by referring to ingredient name 312 in master database 31, is set as a search keyword for the concierge function. Here, "tomato" is searched for in the concierge database 33. For ease of explanation, the ingredient name is used here, but if ingredient ID 311 in master database 31 is set when registering an ingredient in ingredient list 32, that ID may be used instead.

[0077] The concierge function can then refer to the model information of the refrigerator used by user 1 separately to display the location of the recommended storage location for the selected "tomatoes" on the main screen 91 for this model of refrigerator. In this case, if the selected storage location is "Setting 1" for the above model, illustration D will be displayed.

[0078] As part of the concierge function, the user can be redirected to a page for further advice on food storage. Screen 143 shows further advice for this model, and indicates that the recommended storage location is "when the switchable compartment is set to 'vegetables'."

[0079] Figure 17 is a sequence diagram showing the linking process procedure between the master database 31 and the concierge database 33. Following the process contents in Figure 6, in process step S212, the user 1 taps on a food for which he / she wishes to check details from the food list 32 displayed on the mobile information terminal 2 to display the ingredient details 320. In response to this, in process step S213, the ingredient details 320 and "View advice on storage location" (link button 72) are displayed on the mobile information terminal 2.

[0080] In processing step S214, the link button 72 is tapped. In processing step S215, the screen moves to a "concierge function" screen within the app, and in processing step S216, the model of the refrigerator 4 used by user 1, which has been acquired in advance, and the settings as of processing step S215 are acquired from the refrigerator 4. In processing step S217, the concierge database 33 is searched using the ingredient name 322 (or ingredient ID 321) corresponding to the ingredient detail information 320 as a key. For the matching ingredient, the storage location / advice pattern 334 in the concierge database 33 is further acquired using the information acquired in S216. In processing step S218, the acquired results are displayed on the mobile information terminal 2.

[0081] [Other features] Next, various related functions will be described.

[0082] <325 pre-defined storage options> In the above embodiment, the master database 31 has a recommended storage location 315 defined for each ingredient ID 311. However, depending on the model of the refrigerator used by the user 1, the recommended storage location 315 may not be provided. For this reason, a function may be provided for designating in advance a storage room that can be automatically set as the storage location 325 among the recommended storage locations 315.

[0083] Specifically, a function for registering storage compartments of the refrigerator used by user 1 is added. For example, it is assumed that the only storage compartments that user 1 can use are the "refrigerator" and the "freezer". In this case, for food items for which the "vegetable compartment" is set as the recommended storage location 315, for example, the "refrigerator", which has a similar temperature range, can be converted to the storage location 325 and automatically set, or the storage location 325 can be prevented from being automatically set.

[0084] <Selection and adjustment of automatically entered ingredient details 320> In the above embodiment, the estimated consumption period 314, the recommended storage location 315, the food ingredient category 316, and the standard image data 317, 318 were defined as items that were automatically set through a search of the master database 31. However, for example, there may be a user who does not wish to manage the estimated consumption period 314. For this reason, a function that allows the user to select the items that are automatically set may be provided.

[0085] In the above embodiment, one type of period is set for each ingredient in the master database 31 as the estimated consumption period 314 (for example, five days for "pork"). However, it is conceivable that some users may be sensitive to freshness, for example, they may use a refrigerator with excellent freshness-keeping capabilities, which allows them to keep ingredients fresh for a longer period than usual. For this reason, a function may be provided that allows the period automatically set as the estimated consumption period 314 to be adjusted to multiple levels, such as "short," "standard," and "long."

[0086] Also, food stores may sell food items at a discount because the expiration date is approaching. Since setting a normal estimated consumption period 314 for such food items may not be realistic, a check box or the like may be provided to register the food item as a so-called "discount item." If the food item is registered in the food item list 32 with this check box checked, a shorter estimated consumption period 314 than normal may be set. Food items with this check box registered may be displayed in a distinctive manner on the display screen of the food item list 32.

[0087] <Selection of which image data 317 or 318 in the master database 31 is to be reflected> In the above embodiment, two types of standard image data 317 and 318 are prepared as image data to be automatically set as detailed ingredient information 320. For example, by preparing two types of data with completely different atmospheres, photo data and illustration data, it is possible to assume that each user will use the data after determining their own implicit understanding, such as "photo data is for Family A to buy and use" and "illustration data is for Family B to buy and use." For this reason, the image data to be used as detailed ingredient information 320 may be selected manually each time, rather than automatically.

[0088] <Database structure that distinguishes food ingredient names 312 according to their preservation state> In the above embodiment, the master database 31 has food ingredient names 312 that do not distinguish the state of the food ingredient, such as "carrot" and "broccoli." However, for example, with regard to "shrimp," there may be different storage states, such as raw shrimp that is suitable for storage in a chilled compartment, and frozen shrimp that is sold frozen. For this reason, food ingredient names that distinguish the storage state, such as "shrimp (raw)" and "shrimp (frozen)," may be prepared in advance in the master database 31 or the concierge database 33 as food ingredient names 312. In this case, the database may be set so that the chilled compartment or refrigerator compartment is set for the former, and the freezer compartment is set for the latter, as the recommended storage location 315.

[0089] <Processing when there are multiple ingredients that exactly match search word 313> For example, when a homophoneically ambiguous character string is entered as the search word 313, two ingredient names 312 may correspond as an exact match. For example, the search word 313 "sake" (salmon) corresponds to two ingredient names 312, "sake" (sake) and "salmon" (salmon), and it is not possible to determine which one it is. In this case, the detailed food information 320 may be temporarily registered without being set, and then the information may be determined later.

[0090] 18 is a diagram showing the process when there are multiple ingredients that perfectly match the search word 313. First, screen 151 shows the display content when a temporary list is created, for example, in method 2. Screen 152 shows the display content when an ingredient name determined using the character string in the temporary list is reflected in ingredient list 32 in response to a registration instruction. Since there are ingredients "sake" and "salmon" that perfectly match the search word 313 "salmon," the ingredient is not determined and is added to ingredient list 32 as "salmon," treating it as an ingredient that is not in master database 31. This makes it possible to prevent incorrect ingredient names 322 and detailed ingredient information 320 from being set.

[0091] User 1 can add an appropriate ingredient by transitioning from screen 152 to the ingredient detail information 320 screen for "salmon" and editing the ingredient name 322 to, for example, "salmon." After "salmon" is entered in the ingredient name 322, the master database 31 is searched again and the ingredient detail information 320 is automatically set.

[0092] <Food management using refrigerator cameras> In the above embodiment, ingredients are added to the ingredient list using voice input or text editing using a keyboard, but it is also possible to add ingredients to the ingredient list using camera images, for example by attaching a camera to the refrigerator. The camera is included in the ingredient management device illustrated in FIG.

[0093] First, the ingredient name and actual storage location are registered and known in the ingredient details information 320 using the method described in the above embodiment. The actual storage location is not a recommended storage location obtained by referring to the master database 31, but a storage location in the refrigerator or the like used by user 1 that is appropriately modified by user 1. In this way, information that matches the actual storage can be obtained in advance.

[0094] Next, the camera captures the food in the storage compartment, and uses this image to perform image recognition of the food. A known method can be used for image recognition processing. First, it is recognized whether the food is a food or not, and the image is cut out for each food item. For example, when an image of a vegetable compartment is captured, the image is cut out for each food item, such as carrots or broccoli. At this point, image recognition can recognize the presence of food items, but does not recognize what each cut-out food item is.

[0095] Next, a process is performed to recognize the item of each cut ingredient. At this time, it is preferable to configure the system so that information regarding which storage compartment (refrigerator, freezer, vegetable compartment, etc.) the camera has captured is obtained. This can be achieved, for example, by having the user 1 input this information before capturing the image. Furthermore, the detailed ingredient information 320 of each ingredient list 32 is referenced to see what is stored in each storage compartment.

[0096] The accuracy of image recognition can be improved by using the storage room information of the locations where these cameras captured images and the storage location information in the ingredient list 32. That is, for example, if the ingredient list 32 contains information indicating that carrots are stored in the vegetable compartment, the image recognition function can weight the vegetable compartment so that carrots are likely to be there, and proceed with ingredient recognition.

[0097] Then, for each ingredient captured by the camera, it is determined whether it has already been registered in the ingredient list 32. For ingredients determined to be already registered, the image data captured by the camera can be registered as image data 327 of the ingredient details 320 for that ingredient in the ingredient list 32. In this way, image data in actual use can be used instead of standard image data 317, 318, making it possible to provide an image close to what user 1 remembers.

[0098] On the other hand, if an ingredient is determined to be not registered in the ingredient list 32, the process starts as adding a new ingredient. That is, the master database 31 is searched using the ingredient name obtained by image recognition as the search word 313, and detailed ingredient information 320 of the ingredient found is set. However, this does not apply to the recommended storage location. Also, the image captured by the camera is set as image data 327.

[0099] At the current state of the art, it is not necessarily easy to perform image recognition with high accuracy. For this reason, image data 327 of all ingredients captured by the camera may be set and presented to user 1, and user 1 may visually determine whether or not to add them to ingredient list 32. In this case, all ingredients may be added to ingredient list 32, and user 1 may delete unnecessary ingredients, or a new list may be created and presented as a list of candidates to be added.

[0100] <Processing of ingredients in the ingredient list 32 that have passed the estimated consumption date 324> Figure 19 is a diagram showing how to handle the display of ingredients that have passed their estimated consumption date. As shown in screen 161 of Figure 19, multiple types of lists can be defined as ingredient lists 32. For convenience, ingredient list 32 described in the above embodiment will be referred to as a "storage ingredient" list. This is the list shown on screen 161.

[0101] When one day has passed since the state of screen 161, the two ingredients at the top of the list on screen 161 (carrots and broccoli) will have passed the estimated consumption date 324 in their respective ingredient details information 320. If the "storage ingredients" list is displayed again in this state, a pop-up is displayed (screen 162) indicating that the ingredients whose estimated consumption date 324 has passed have been deleted from the "storage ingredients" list. These ingredients have also been deleted from the "storage ingredients" list (screen 163).

[0102] FIG. 20 is a diagram showing how ingredients that have been deleted from the list of ingredients in storage are handled. The ingredient management device defines, for example, a list of "recently deleted ingredients" in addition to the "ingredients in storage" list. These lists are displayed as tabs on the same display screen, and transitions between them can be made with one tap. An ingredient registered in the "ingredients in storage" list can be moved to the "recently deleted ingredients" list when the estimated consumption date 324 in the ingredient details information 320 has passed (screens 171, 172). An ingredient moved to the "recently deleted ingredients" list (screen 172) can be completely deleted, for example, 48 hours after being moved.

[0103] This prevents ingredients from accumulating in the "Ingredients in Storage" list, and even if User 1 neglects to update the "Ingredients in Storage" list, the list can be maintained to some extent in accordance with the actual stored items. On the other hand, by moving ingredients to the "Recently Deleted Ingredients" list, it is possible to discover from the "Recently Deleted Ingredients" list that there are ingredients that need to be consumed relatively soon because the estimated consumption date 324 has passed and they are still in storage.

[0104] The above-mentioned alarm function may be deactivated from the ingredient details information 320 of an ingredient that has been moved to the "recently deleted ingredients" list (screen 173). Also, the estimated consumption date 324 may be maintained as it was in the "storage ingredients" list.

[0105] <Top screen (home screen) of food management function> The following describes the top screen that is displayed when the food ingredient management device starts up the food ingredient management function (food ingredient management application) on the mobile information terminal 2. Fig. 21 is a diagram showing an example of the top screen displayed immediately after starting up the food ingredient management function. The top screen 181 has a function of displaying recommended ingredients, and in particular has a function of changing the recommended ingredients depending on whether or not there are ingredients in the food ingredient list 32 (the "food ingredients in storage" list) whose estimated consumption date 324 is approaching. Two screens 182, 183 included in the top screen 181 will be described next.

[0106] If there are no ingredients in the ingredient list 32 with an approximate consumption date 324 (for example, only ingredients that are in season in the next 3 days or more), ingredients that are in season today are displayed (recommendation display, screen 182). In the recommendation display, if there are ingredients in season in the ingredient list 32, the ingredient with the fewest days remaining until the estimated consumption date 324 may be displayed.

[0107] If the ingredient list 32 contains an ingredient whose estimated consumption date is today, a display is displayed suggesting that the ingredient be consumed (reminder display, screen 183). If the ingredient list 32 contains multiple ingredients with the same estimated consumption date, they may be displayed randomly, for example. The reminder display may be a push notification sent to the mobile information terminal 2, and a screen transition may be made to display the ingredient list 32 in response to this notification.

[0108] <Registration reminder for estimated consumption date 324> Fig. 22 shows screen transitions related to the reminder function for registering estimated consumption. The foodstuff management function of the foodstuff management device can be configured as an application that integrates multiple functions, such as the concierge function described above. For example, in this case or not, a menu for transitioning to the foodstuff management function can be displayed on the top screen 191, and by selecting this, transition can be made to the foodstuff management function screen (screens 192, 193).

[0109] Screen 192 shows the case where no ingredients have been registered in ingredient list 32. In this case, if an ingredient is registered and the estimated consumption date 324 is set in ingredient detail information 320, a message is displayed indicating that a notification (reminder) such as a recommendation of an ingredient to be used can be sent accordingly.

[0110] Screen 193 shows a case where an ingredient has already been registered in ingredient list 32. When the ingredient details information 320 of a registered ingredient has no estimated consumption date 324 set, a message to that effect is displayed, as shown in screen 193. Whether an ingredient has no estimated consumption date 324 set can be determined by whether the ingredient has no information such as the number of days remaining until estimated consumption in the list display of ingredient list 32 such as screen 193. Screen 183 displays "Estimated consumption remaining D days" (D is the actual number of days remaining) for an ingredient for which an estimated consumption date 324 has been set.

[0111] The message indicating that there is an ingredient for which the estimated consumption date 324 has not been set may be displayed when there is an ingredient for which the estimated consumption date 324 has not been registered for several days, when the number of unregistered ingredient items has exceeded a certain threshold, etc. Also, the user 1 may be allowed to select whether or not to display the message.

[0112] <Sorting of ingredient list 32 when user 1 owns multiple refrigerators> The detailed ingredient information 320 may further be able to distinguish "which refrigerator" is the storage location. In the above embodiment, under the tacit understanding that the user 1 uses only one refrigerator, the storage room in a certain refrigerator, such as "refrigerator" and "vegetable room", is prepared as the storage location 325 as a choice. However, it is also possible to set two types of information in more detail as the storage location 325. Specifically, the first type of information may be set to a storage facility such as "refrigerator 1", "refrigerator 2", or "warehouse", and the second type of information may be set to a storage location within the facility such as "refrigerator" or "vegetable room". The name of the first type of information may be defined by the user 1. The name of the second type of information may be hierarchically structured so that it changes depending on the setting of the first type of information. For example, it may be possible to define that when "refrigerator 1" is set, options such as "refrigerator" and "vegetable room" are displayed, and when "warehouse" is set, different options such as "first row from the bottom" and "second row from the bottom" are displayed.

[0113] The ingredient management device can be equipped with filtering and sorting functions so that the ingredient list 32 can be displayed for each of these storage locations 325. For example, the filtering function may be a function that allows only ingredients stored in "Refrigerator 1" to be displayed in the ingredient list. The filtering and sorting functions may include a function to display ingredients in order by storage location 325, as well as a function to filter or sort ingredients that have been recently registered in the ingredient list 32 (the "storage ingredients" list). For ingredients that have been recently registered, an icon indicating this may be displayed on the display screen of the ingredient list 32 (e.g., screen 125 in FIG. 8) or the display screen of the ingredient details information 320 (e.g., screen 320 in FIG. 12). As a specific example, the "N" icon 3252 illustrated in FIG. 8 may be displayed.

[0114] <Variations for adding ingredients to the ingredient list 32> FIG. 23 is a diagram showing the screen transition when user 1 can select which method to use to add ingredients. When an instruction is given to activate the "ingredients management" function on top screen 201, ingredients management top screen 202 is displayed. On ingredients management top screen 192, ingredients list 32 including ingredients that have already been registered is displayed along with ingredients registration method selection button 75, and operating this button transitions to screen 203. On screen 203, selection buttons for camera input, voice input, and text input are displayed as ingredients registration methods. By selecting any of these, any of ingredients input screens 203A, 203B, and 203C are displayed. Voice input is as in the above embodiment. In addition, text input using only the keyboard without using voice, and camera input starting from setting image data 327 are also prepared.

[0115] <Detection of food turnover rate> Although it differs depending on the user 1, it is assumed that there are foods such as eggs, dairy products, and beer that are consumed relatively daily and purchased frequently, with a high turnover rate. If such foods can be recognized, it is expected that notifications can be provided to prevent out-of-stocks, so an example of this will be described. Here, the camera described above is used.

[0116] For example, a method may be used to estimate, as a candidate for an ingredient with a high turnover rate, an ingredient that was added to the ingredient list 32 and consumed within a short period (e.g., within 24 hours) or an ingredient whose location or orientation in the storage location has changed by utilizing the image recognition function of the camera described above. Another method may be to display an image of a storage room in which ingredients are stored, captured by the camera, on the mobile information terminal 2, and have the user 1 select an ingredient whose turnover rate is to be tracked. For ingredients with a high turnover rate or ingredients designated by the user 1, it is preferable to perform number recognition in the image recognition of the camera. In order to facilitate the number recognition, when the number recognition function is turned on, the ingredient management device may output, for example, a message via the display screen of the mobile information terminal 2 requesting that the ingredients to be recognized be stored in areas close to each other in the storage room. In addition, a plurality of areas for storing ingredients to be recognized may be prepared in advance on the display screen of the mobile information terminal 2 for each ingredient, and the number recognition may be performed for each area selected by the user 1. That is, the camera is set up in advance to virtually divide the interior of a certain storage room into a plurality of areas so that it can be recognized. For example, if there are multiple shelves in a refrigerator, each shelf can be recognized as a different area.The user can then specify that a specific foodstuff, such as yogurt, should be placed in a specific area, such as a specific shelf.This makes it easier for the camera to recognize the yogurt placed in that area, and therefore easier to recognize the number of yogurts and the turnover rate.

[0117] The user can specify the shelf on which the yogurt is stored via the mobile information terminal 2 or the like.

[0118] On the other hand, ingredients that have not been consumed for a long time or ingredients that have not changed their storage location or orientation are estimated to be candidates for ingredients with a low turnover rate. For ingredients with a low turnover rate, there is a concern that the user 1 may have forgotten about them, so a notification may be sent at a time that is not tied to the estimated consumption date. [Explanation of symbols]

[0119] 1: User 2: Mobile information terminal 3: Server 4: Refrigerator (storage, goods storage) 5: Cloud (Network) 31: Master database 32: Ingredients list CPU: Arithmetic unit Tr: Communications Department Bus: 21: Audio input section 21a:Start part 21b:Continuous section 22: Display output section 23: Character conversion section 23a: Termination section 24: Editing section 25: Registration processing unit 26: General Processing Unit 91: Main screen 92: Text input screen 93: Audio input status display 94: Character string display section

Claims

1. On one or more computers, A start unit that starts accepting voice input; a display output unit that displays a character string that has been recognized by speech on a display screen; an edit processing unit that accepts editing of the character string displayed on the display screen in response to a user operation and, if edited, causes the edited character string to be displayed on the display screen; a termination adding unit that adds a termination to a character string obtained by speech recognition and then treats the character string obtained by speech recognition as a new character string; a registration processing unit that, in response to a user's operation, adds the character string displayed on the display screen to an ingredient list as an ingredient name; the display output unit displays two or more character strings, the editing processing unit is capable of editing each character string displayed by the display output unit, The registration processing unit is an ingredient management system that adds each character string displayed by the display output unit as an ingredient name to an ingredient list, A master database that stores ingredient names and ingredient detail information corresponding to the ingredient names can be referenced; During the operation of the registration processing unit, the master database is searched for an ingredient name corresponding to each character string displayed by the display output unit, and for a character string having a corresponding ingredient name, the ingredient name and the corresponding ingredient detailed information can be added as a pair to an ingredient list; In this ingredient management system, when the master database is searched using the character string displayed by the display output unit as a search word and multiple perfectly matching candidate ingredients are obtained, detailed ingredient information is not set for the ingredient name even if the registration processing unit is operated.

2. On one or more computers, A start unit that starts accepting voice input; a display output unit that displays a character string that has been recognized by speech on a display screen; an edit processing unit that accepts editing of the character string displayed on the display screen in response to a user operation and, if edited, causes the edited character string to be displayed on the display screen; a termination adding unit that adds a termination to a character string obtained by speech recognition and then treats the character string obtained by speech recognition as a new character string; a registration processing unit that, in response to a user's operation, adds the character string displayed on the display screen to an ingredient list as an ingredient name; The display output unit is a food ingredient management system that displays one character string, automatically operating the initiation unit after the operation of the registration processing unit; A master database that stores ingredient names and ingredient detail information corresponding to the ingredient names can be referenced; A search is made for an ingredient name corresponding to the character string displayed by the display output unit in the master database, and if there is a corresponding ingredient name, the display output unit is caused to display it as a candidate ingredient; When the selection of the candidate ingredient displayed on the display screen is received from the user, the registration processing unit adds the ingredient name of the candidate ingredient and the corresponding ingredient detail information as a pair to an ingredient list; In this ingredient management system, when the master database is searched using the character string displayed by the display output unit as a search word and multiple perfectly matching candidate ingredients are obtained, detailed ingredient information is not set for the ingredient name even if the registration processing unit is operated.

3. On one or more computers, A start unit that starts accepting voice input; a display output unit that displays a character string that has been recognized by speech on a display screen; an edit processing unit that accepts editing of the character string displayed on the display screen in response to a user operation and, if edited, causes the edited character string to be displayed on the display screen; a termination adding unit that adds a termination to a character string obtained by speech recognition and then treats the character string obtained by speech recognition as a new character string; a registration processing unit that, in response to a user's operation, adds the character string displayed on the display screen to an ingredient list as an ingredient name; The food ingredient management system outputs a message urging the user to set an estimated consumption date depending on the presence of food ingredients included in the food ingredient list for which time has passed without an estimated consumption date being set, or the number of food ingredients for which an estimated consumption date has not been set.

4. The foodstuff management system according to claim 3, the display output unit displays two or more character strings, the editing processing unit is capable of editing each character string displayed by the display output unit, The registration processing unit adds each of the character strings displayed by the display output unit as an ingredient name to an ingredient list.

5. The foodstuff management system according to claim 4, A master database that stores ingredient names and ingredient detail information corresponding to the ingredient names can be referenced; When the registration processing unit is operating, the ingredient management system searches the master database for ingredient names corresponding to each character string displayed by the display output unit, and for character strings that have a corresponding ingredient name, adds the ingredient name and corresponding ingredient details information as a pair to an ingredient list.

6. The foodstuff management system according to claim 3, the display output unit displays one character string, The food ingredient management system automatically operates the initiation unit after the registration processing unit operates.

7. The foodstuff management system according to claim 6, A master database that stores ingredient names and ingredient detail information corresponding to the ingredient names can be referenced; A search is made for an ingredient name corresponding to the character string displayed by the display output unit in the master database, and if there is a corresponding ingredient name, the display output unit is caused to display it as a candidate ingredient; When a selection of a candidate ingredient displayed on the display screen is accepted from a user, the registration processing unit adds the ingredient name of the candidate ingredient and the corresponding detailed ingredient information as a pair to an ingredient list.

8. The foodstuff management system according to any one of claims 1, 2, 5 and 7, A concierge database is provided separately from the master database, and can refer to ingredients corresponding to the ingredients in the master database by ingredient name or ingredient ID. By specifying a certain ingredient in the ingredient list, the ingredient name or ingredient ID of the specified ingredient can be automatically input as a search key in the concierge database.

9. The foodstuff management system according to any one of claims 1, 2, 5 and 7, The detailed information of ingredients includes a storage location of the storage facility, A food ingredient management system in which a user can specify in advance the type of storage room that can be set as the storage location.

10. The foodstuff management system according to any one of claims 1, 2, 5 and 7, The detailed information of ingredients includes an estimated consumption period of the ingredients, This foodstuff management system allows a user to specify in advance the length of the period to be set as the estimated consumption period.

11. The foodstuff management system according to any one of claims 1, 2, 5 and 7, The detailed ingredient information includes two or more types of standard image data of ingredients, The ingredient management system allows a user to specify the type of ingredient to be set as the standard image data when adding each ingredient to an ingredient list.

12. The foodstuff management system according to any one of claims 1, 2, 5 and 7, In this food ingredient management system, two or more food ingredient names indicating the same food ingredient but different storage conditions are prepared in the master database.

13. The foodstuff management system according to any one of claims 1, 2, 5 and 7, The food management system is equipped with a camera, An image recognition process is performed on the image captured by the camera to cut out each of the ingredients included in the image, For ingredients whose names are recognized as a result of image recognition, An image of the ingredient cut out from the image is set as the ingredient detail information of the ingredient, At least a part of the ingredient detail information corresponding to the ingredient name is acquired from the master database, and set as the ingredient detail information of the ingredient; The ingredient is added to the ingredient list.

14. The foodstuff management system according to any one of claims 1, 2, 5 and 7, The food management system is equipped with a camera, An image recognition process is performed on the image captured by the camera to cut out each of the ingredients included in the image, The ingredient management system displays ingredient images cut out for each ingredient as a result of image recognition, and asks the user whether or not each ingredient needs to be added to the ingredient list.

15. The foodstuff management system according to any one of claims 1 to 7, The ingredient list includes a list of ingredients currently being stored and a list of ingredients that have been deleted, In this ingredient management system, ingredients included in the stored ingredient list that have passed their estimated consumption date are moved to the deleted ingredient list.

16. The foodstuff management system according to any one of claims 1 to 7, In the food ingredient management system, if the food ingredient list includes an ingredient whose estimated consumption date is approaching, a reminder is displayed to recommend consumption of the ingredient, and if not, a recommendation display is performed to display ingredients that are in season.

17. The foodstuff management system according to any one of claims 1 to 7, To provide an ingredient management system that allows setting two or more storage locations for one ingredient as ingredient detail information.

18. The foodstuff management system according to any one of claims 1 to 7, The food ingredient management system includes a camera that is set to virtually divide a storage room into a plurality of areas and recognize the divided areas, The food ingredient management system receives a user's instruction to designate one of the plurality of areas to store a specified ingredient, and performs count recognition and / or tracking of turnover rate of the specified ingredient stored in the area.

Images