Storehouse management system, program of storehouse management device, and production method of storage management device

Abstract

To provide a program capable of suppressing occurrence of variation in photographic images.SOLUTION: A program causes a terminal 20 comprising a display screen and an imaging unit, to operate as a storehouse management device. The program causes the terminal to operate as: a storeroom selection part 966 for receiving types of managed storehouses, and a selection of types of storerooms to be photographing objects, from storerooms comprised in the storehouse; a photographing guide information reference part 968 for reading photographing guide information corresponding to the storerooms selected by the storeroom selection part; a photographing start part for starting an imaging function for making the imaging unit photograph an image; and a display processing part 970 for displaying the read photographing guide information in the display screen by overlapping the information on a scene inputted to the imaging unit.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a storage management system, a storage management device program, and a method for producing a storage management device. [Background technology]

[0002] Recently, users (consumers) have been managing the food inventory in their refrigerators. For example, by storing and managing food inventory in the memory element of a mobile device such as a smartphone, it is possible to prevent forgetting to buy food that you planned to buy, to prevent buying more food than you need, and to reduce the time spent shopping at the store.

[0003] Such a refrigerator management device is disclosed, for example, in Japanese Patent Application Laid-Open No. 2021-117572 (Patent Document 1). The refrigerator management device of Patent Document 1 includes a storage room designation unit that designates one of multiple storage rooms provided in the refrigerator based on a user's operation on an operation / display unit that inputs and outputs information or information supplied from the refrigerator, and a storage room image storage control unit that stores an image acquired from an image capturing unit that captures images in an image storage unit as a storage room image associated with the storage room designated by the storage room designation unit.

[0004] This means that even if the number of storage compartments, such as vegetable compartments, refrigerator compartments, chilled compartments, and freezer compartments, increases, the user can easily check which food is stored in which storage compartment by looking at an image of the inside of the storage compartment, allowing food to be managed appropriately. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-117572 Summary of the Invention [Problem to be solved by the invention]

[0006] In Patent Document 1, the interior of a fridge is photographed using a mobile terminal (hereinafter referred to as a smartphone), so the photographed images vary from one photograph to the next. For example, this varies in terms of the distance, height, and angle from the subject. In other words, it is expected that the photographs will be taken with different fields of view and angles depending on the photographing position and angle of the smartphone.

[0007] In programs that recognize, extract, and manage food from captured images, such as those using machine learning, variations in the field of view or angle of the images may prevent proper food recognition. Furthermore, when updating such programs, for example through machine learning, variations in the field of view or angle of view between images may make it difficult to perform efficient learning. Furthermore, there is a need for methods to reduce distortions in food images caused by camera shake during photography.

[0008] An object of the present invention is to provide a program that can suppress variations in captured images. [Means for solving the problem]

[0009] The present invention is a program that causes a terminal equipped with a display screen and a photographing unit to operate as a storage management device, and is characterized in that the program causes the terminal to operate as a storage room selection unit that accepts selection of the type of storage room to be managed and the type of storage room to be photographed from the storage rooms equipped in the storage room, a photographing guide information reference unit that reads out photographing guide information corresponding to the storage room selected by the storage room selection unit, a photographing activation unit that activates a photographing function that causes the photographing unit to photograph an image, and a display processing unit that displays the read out photographing guide information on the display screen, superimposed on the scenery input into the photographing unit. [Effects of the Invention]

[0010] According to the present invention, since the storage room can be photographed in accordance with the photographing guide information, it is possible to achieve the effect of suppressing variations in the photographed images. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a front view showing a state in which the heat-insulating door of a refrigerator to be managed by the present invention is open. [Figure 2] 1 is a block diagram showing a system configuration of a refrigerator control system to which the present invention is applied. [Figure 3] FIG. 10 is a flowchart illustrating a first image capturing sequence executed between a refrigerator and a smartphone. [Figure 4] 4 is an explanatory diagram illustrating the state of the display screen of the smartphone in the shooting sequence shown in FIG. 3. [Figure 5] 4 is an explanatory diagram illustrating the state of another screen of the smartphone in the shooting sequence shown in FIG. 3. FIG. [Figure 6] FIG. 10 is a flowchart illustrating a second image capturing sequence executed between the refrigerator and the smartphone. [Figure 7] FIG. 10 is a flowchart illustrating a third image capturing sequence executed between the refrigerator and the smartphone. [Figure 8] 10 is an explanatory diagram illustrating still another screen state of the smartphone in the shooting sequence shown in FIG. 3. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiment, and various modifications and application examples within the technical concept of the present invention are also included within its scope.

[0013] First, the configuration of a refrigerator as an example of a storage cabinet to which the present invention is applied will be described. Fig. 1 is a front view of a refrigerator according to an embodiment of the present invention. Note that Fig. 1 shows a state in which the front door is open.

[0014] As shown in FIG. 1, a refrigerator according to an embodiment of the present invention includes an insulating box body 1 and an insulating door 2 attached to the insulating box body 1. In this embodiment, the insulating door 2 is made up of multiple insulating doors 2A to 2F. That is, the insulating door 2 is used as a general term for the insulating doors 2A to 2F. The outer box of the insulating box body 1 includes a top panel 3A, a back panel (not shown), a bottom panel 3B, a left side panel 3C, and a right side panel 3D. The insulating doors 2A and 2B are supported by hinges 4 so as to be rotatable.

[0015] The refrigerator has, from top to bottom, storage compartments: a refrigerator compartment 5, an ice-making compartment 6, an upper freezer compartment 7, a lower freezer compartment 8, and a vegetable compartment 9. Insulated doors 2A to 2F are doors that open and close openings formed in the front of each storage compartment.

[0016] A chilled compartment 5A is formed below the refrigerator compartment 5. A plurality of shelves 5B for placing food items are arranged in the refrigerator compartment 5. Furthermore, storage pockets 5C are provided on the inside of the heat-insulating doors 2A and 2B.

[0017] The insulating doors 2A and 2B are pivotable insulating doors that pivot around hinges 4, and the insulating doors 2C to 2F are drawer-type insulating doors. When the drawer-type insulating doors 2C to 2F are pulled out, the containers that make up each chamber are pulled out together with the insulating door.

[0018] Insulated doors 2A and 2B are insulated doors for refrigerator compartment 5, insulated door 2C is an insulated door for ice-making compartment 6, insulated door 2D is an insulated door for upper freezer compartment 7, insulated door 2E is an insulated door for lower freezer compartment 8, and insulated door 2F is an insulated door for vegetable compartment 9. Here, ice-making compartment 6 can be said to be a freezer compartment arranged alongside upper freezer compartment 7.

[0019] The refrigerator 1 shown above is an example, and there are several other refrigerators with different storage compartments, insulated door shapes, etc. As will be described later, photography guide information for photographing the interior of the refrigerator corresponding to the type of refrigerator is stored in a server or the like. This will be described later.

[0020] Figure 2 is a block diagram showing the hardware configuration of a refrigerator management system that uses a smartphone equipped with a display screen and a camera. Note that although different reference numbers are used for the storage compartments of the refrigerator, they are essentially the same.

[0021] The refrigerator management device includes one or more refrigerators 10-1 to 10-M (hereinafter, sometimes collectively referred to as refrigerators 10) and one or more smartphones 20-1 to 20-N (refrigerator management devices, which include mobile terminals such as tablets in addition to smartphones).

[0022] The refrigerator 10 and the smartphone 20 can communicate with each other via a LAN 52, and can also communicate with each other via the LAN 52 and a WAN 54. The LAN 52 is, for example, a wireless LAN within a home, and the WAN 54 is, for example, the Internet. The LAN 52 and the WAN 54 form a so-called communication network.

[0023] The connected home appliance server 56 is connected to the WAN 54 and includes an image storage unit 912 and a food storage unit 914. The members' club server 58 is also connected to the WAN 54. Although not shown, it also includes an image storage unit and a food storage unit.

[0024] The connected home appliance server 56 has a function for pairing with a smartphone and managing information about IoT-enabled refrigerators. The members' club server 58 is a server that can be registered by any customer, regardless of whether the refrigerator is IoT-enabled or not. It has functions for registering purchased refrigerators and for easily purchasing replacement parts and accessing support pages. The connected home appliance server 56 and the members' club server 58 can be a common server. For example, a learning model is stored in the connected home appliance server 56 or a separate server (not shown). The learning model, for example, outputs the location and name of an item, such as food, in an image in response to an input image. The learning model can be trained using images captured by the refrigerator 10 or the smartphone 20. The learning models can be separated by refrigerator model, for example. In other words, although there are multiple types of refrigerator models to be managed by the refrigerator management system, only some of these models can be assigned to one learning model, and the remaining or other parts can be assigned to another learning model. In this way, it is preferable to prepare separate learning models for identical or similar storage compartments.

[0025] Refrigerator 10 includes control unit 11, memory unit 12, communication unit 13, operation and display unit 14, freezing and refrigerating unit 15, and housing 60. Freezing and refrigerating unit 15 includes a compressor, a heat exchanger, etc. Operation and display unit 14 includes an LED display, membrane switches, etc., and displays various states of refrigerator 10 to the user, and also enables the user to input various commands via the membrane switches, etc.

[0026] The housing 60 of the refrigerator 10 includes a refrigeration compartment 61, an upper freezer compartment 62, a lower freezer compartment 63, a vegetable compartment 64, a vacuum chilled compartment 65, and an ice-making compartment 66.

[0027] The storage unit 12 stores various types of information. The communication unit 13 communicates with the smartphone 20 and the like via the LAN 52. The control unit 11 includes hardware as a general computer, such as a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory), and the ROM stores control programs executed by the CPU, various types of data, and the like.

[0028] Here, refrigerator compartment 61 is an open-door refrigerator compartment (storage temperature: approximately 2 to 6°C), but it can also be set as a chilled compartment. Lower freezer compartment 63 and vegetable compartment 64 are drawer-type compartments, and the user can arbitrarily select their functions from a number of options on operation / display unit 14. Selectable function options include, for example, vegetable compartment (storage temperature: approximately 3 to 7°C), refrigerator compartment (same: 2 to 6°C), chilled compartment (same: -1 to 1°C), and freezer compartment (same: -20 to -18°C). Control unit 11 sets the air pressure of vacuum chilled compartment 65 to approximately 0.8 atmospheres. However, vacuum chilled compartment 65 can also be set to a non-vacuum temperature (approximately 1 atmosphere) by the user.

[0029] The smartphone 20 also includes a control unit 21, a storage unit 22, a communication unit 23, an operation / display unit 24, an image capture unit 26, and a microphone 27. The operation / display unit 24 includes a flat panel display and a touch sensor provided on its surface, and displays various information to the user and enables the user to input various commands. The communication unit 23 communicates with the refrigerator 10 and the like via the LAN 52 or the WAN 54.

[0030] Memory unit 22 stores various types of information. Note that the data stored in memory units 12 and 22 does not necessarily need to be stored inside refrigerator 10 or smartphone 20; a storage device for storing this data may be provided in a separate device such as a data server and made accessible via LAN 52 or WAN 54. Photography unit 26, which includes a camera, captures videos or still images based on commands from control unit 21. Microphone 27 collects audio.

[0031] The control unit 21 is equipped with general computer hardware such as a CPU, RAM, and ROM, and the ROM stores control programs executed by the CPU, various data, etc. In Fig. 2, the inside of the control unit 21 is shown as blocks, with functions realized by the control programs, etc.

[0032] That is, the control unit 21 includes a storage room designation unit 936 (storage room designation means), a storage room image storage control unit 938 (storage room image storage control means), a food designation unit 940, a location presentation unit 942, a weather information acquisition unit 944, a shopping index display control unit 946, a postal code input unit 948, a geographic information estimation unit 950 (geographic information acquisition unit), a food data management unit 952, a retailer communication unit 954, a recommended food screen display control unit 956, a recipe display screen control unit 958, an operation information screen display control unit 960, a cyclic screen selection unit 962, and an image analysis unit 964. Details of these units other than those related to this embodiment will not be described here.

[0033] As mentioned above, depending on the camera angle of the smartphone, it is expected that the food inside the storage unit will be photographed from above, below, left, right, and diagonal directions. In such cases, if there is variation in the field of view of the captured image, for example, if there is variation in the angle or area of ​​the storage unit wall or the angle of the food included in the image, issues will arise such as not being able to recognize the food from the captured image when recognizing the food and determining its type, etc. Furthermore, there is a possibility that the learning accuracy of the learning model that uses the captured image data to recognize food images will decrease.

[0034] In order to solve such problems, in this embodiment, the control unit 21 of the smartphone 20 shown in Figure 2 is newly equipped with a storage room selection unit 966 that accepts from the user the selection of a storage room to be photographed from multiple types of storage rooms, a photography guide information reference unit 968 that reads out photography guide information (e.g., a guide frame) corresponding to the storage room selected by the storage room selection unit, and a display processing unit 970 that displays the read photography guide information on the display screen so as to be superimposed on the photographed image input to the photography unit.

[0035] In other words, the server 56 (58) stores the type of refrigerator 10 and the shooting guide information, which is positioning information for reducing variations in the field of view and angle of shooting for each storage compartment of the refrigerator 10 corresponding to this type. The smartphone 20 requests the server 56 (58) to transmit the necessary shooting guide information, and upon receiving the shooting guide information from the server 56 (58), the smartphone 20 displays the shooting guide information on the display screen of the smartphone 20 and also displays the image captured by the shooting unit 26 superimposed on it.

[0036] These functions are substantially executed by calculations performed by a microcomputer constituting the control unit 21. Next, a specific photographing sequence of this embodiment will be described with reference to a flowchart.

[0037] <<When pairing the smartphone 20 with the refrigerator 10>> The first image capture sequence shown in Fig. 3 is a flowchart when the smartphone 20 and the refrigerator 10 are paired. In this case, the smartphone 20 communicates with the connected home appliance server 56. In Fig. 3, the term "app" used in the control steps means an application executed on the smartphone, and "smartphone" means a smartphone.

[0038] <Step S10> In step S10, an application for pairing the refrigerator 10 with the smartphone 20 is executed, and the smartphone 20 and the refrigerator 10 are paired using Bluetooth (registered trademark) or the like. This pairing is a typical pairing using the smartphone 20. Once pairing is established, the process proceeds to step S11.

[0039] <Step S11> In step S11, information about the paired refrigerator 10 (for example, model number, type, etc.) is used to obtain refrigerator information about the refrigerator 10 to be photographed from the connected home appliance server 56. The refrigerator information includes various types of information, and in this embodiment, it also includes photographing guide information.

[0040] The photographing guide information is positioning information that serves as a reference when photographing the interior of the refrigerator 10 with the camera (photographing unit) of the smartphone 20. This positioning information is formed by a hollow rectangular guide frame or linear guide lines. The photographing guide information corresponds to the information of the paired refrigerator 10 and also corresponds to the storage room selected in the storage room selection unit 966.

[0041] For example, suppose that the refrigerator compartment 5 is specified through the storage compartment selection unit 966. In this case, the smartphone 20 can refer to the positioning information determined for the refrigerator compartment 5 of the paired refrigerator 10. The positioning information includes a guide frame indicating the area in front of the chilled compartment 5A (the insulated door side), a guide frame indicating the area of ​​the storage shelf 5B provided in the refrigerator compartment, and a guide frame indicating the area of ​​the storage pocket 5C provided in the insulated doors 2A and 2B. In this way, the guide frames have a shape that imitates the predetermined area formed in the storage compartment.

[0042] Furthermore, the photography guide information may be created by the user. For example, a guide information creation mode for creating photography guide information may be defined, a test image may be taken of the target storage room, and the user may create photography guide information from this test image.

[0043] By using the above-described photographing guide information, it is easy to check whether the camera of the smartphone 20 is approximately aligned with a predetermined field of view and angle of view. For example, it is easy to check whether the camera is positioned so that it can directly face the storage compartment and photograph from diagonally above at a predetermined angle.

[0044] When refrigerator information including photography guide information for the paired refrigerator is acquired from connected home appliance server 56, the camera function (photography function) of smartphone 20 is activated, and the process proceeds to step S12.

[0045] If only one type of storage compartment is expected to be photographed based on the information on the paired refrigerator 10, it is not necessary to specify it from the storage compartment selection unit 966, and the processing can proceed assuming that the one type has been automatically specified. Also, if the processing proceeds without pairing, the user can input in advance the model information of the refrigerator 10 that will be photographed on a daily basis in the future, and this information can be considered as the information on the refrigerator 10 to be referenced in the future.

[0046] <Step S12> In step S12, the process waits for pressing of a capture button in a capture mode application of the smartphone 20. For example, the capture button is displayed in a part of the display screen 20D (see FIG. 4) of the smartphone 20. FIG. 4 shows the display screen (display) 20D of the smartphone 20.

[0047] In part of the display screen 20D, in this case the lower center of the display screen, a fridge interior image capture button 20B is displayed. A guide frame 20G as image capture guide information may not yet be displayed on the display screen of the user's smartphone 20. When this image capture button 20B is tapped, the process proceeds to step S13. The process may also proceed to step S13 without waiting for the tap.

[0048] That is, after step S11, the process may proceed immediately to step S13. Before proceeding to step S13, a sample image may be displayed as a sample of the field of view and angle of view to be photographed, as exemplified in Fig. 8. The sample image preferably includes the same type of guide frame 20G as that to be displayed on the display screen of smartphone 20 and structure 5G to be superimposed on this guide frame 20G, and therefore, preferably, multiple types are prepared in advance and selected according to the specified storage room information.

[0049] The sample image may be displayed before the photographing guide information step S14, preferably before step S13, and preferably after step S11. The sample image shows a situation in which the storage room structure 5G corresponding to the photographing guide information 20G overlaps with the photographing guide information 20G.

[0050] <Step S13> In step S13, based on the refrigerator information (here, the photography guide information), photography guide information is displayed on the display screen 20D. This photography guide information is acquired from the connected home appliance server 56 in step S11 and stored in the storage unit 22, for example.

[0051] This photography guide information can be specified by selecting a storage room to be photographed. This is a function of the storage room selection unit 966 that selects a storage room to be photographed. The photography guide information corresponding to the selected storage room is also read from the storage unit 22. This is a function of the photography guide information reference unit 968. Furthermore, the photography guide information is displayed on the display screen 20D of the smartphone 20. This is a function of the display processing unit 970.

[0052] 4 shows the display screen 20D in this state. A guide frame 20G, which is one piece of shooting guide information that serves as a reference for shooting, is displayed on the display screen 20D. In addition to the guide frame 20G, the scenery captured by the camera provided on the smartphone 20 is displayed in real time. In the following description, the shooting guide information is described as the guide frame 20G.

[0053] This guide frame 20G is modeled, for example, on the front side of the vacuum chilled compartment 5A of the target refrigerator 10. This guide frame 20G is positioned so that the camera of the smartphone 20 can face the storage compartment directly and take an image from diagonally above at a predetermined angle. However, the guide frame 20G can be set to any shape corresponding to the area to be photographed. The key point is that it is sufficient to encourage photography with a pre-determined field of view and angle of view.

[0054] Furthermore, the display screen 20D above this guide frame 20G displays a photographing guidance message 20M to the user, such as, "Please superimpose the area in front of the chilled compartment on the guide frame and take a photograph." After reading this photographing guidance message 20M, the user will align the guide frame 20G displayed on the display screen 20D of the smartphone 20 with the camera and photograph the area in front of the chilled compartment 5A.

[0055] While Fig. 4 shows an example in which the area in front of the chilled compartment 5A is set as the guide frame 20G, other areas can also be set as the guide frame. An example is shown in Fig. 5. The display screen 20D may display guide storage compartment information 20c1 that shows information about the name of the storage compartment corresponding to the currently displayed guide frame 20G (for example, the name of the storage compartment or an image showing the position of the storage compartment in the storage), and a guide change button 20c2 that is used to change the type of guide frame 20G.

[0056] Fig. 5 shows an example in which a guide frame 20G is configured to fit the shape of a storage shelf 5B provided in a freezer compartment 5. In Fig. 5, a guide frame 20G indicating the storage surface of the storage shelf 5B is displayed. Also, above the guide frame 20G, a shooting guide message 20M is displayed to the user, stating, "Please take a photo by superimposing the storage shelf on the guide frame."

[0057] A user who reads this photographing guidance message 20M will photograph the area of ​​the storage shelf 5B with the camera by aligning it with the guide frame 20G displayed on the display screen 20D of the smartphone 20. However, since the storage shelf 5B is a component whose installation position can be changed by the user, a structure that does not allow this is more preferable.

[0058] For this reason, in this embodiment, the area in front of chilled compartment 5A (floor of refrigerator compartment 5) is preferred, as the structure does not allow the user to change the installation position. Other examples of such a structure include a structure in which the refrigerator is integrally formed with the insulating box 1, a structure in which the refrigerator is fixed to the insulating box 1 with a tool such as a screw, and a structure in which the refrigerator is fitted with a claw.

[0059] It is assumed that food may be present within the area of ​​guide frame 20G, and in this case, a straight guide line can be set for each storage compartment. Therefore, photographing can be performed using this guide line as a reference. Also, for movable food storage areas, such as storage pocket 5C, a guide frame aligned with storage pocket 5C may be displayed.

[0060] Once the guide frame 20G is displayed on the display screen 20D of the smartphone 20, the process proceeds to step S14.

[0061] <Step S14> In step S14, the camera of smartphone 20 is used to capture an image by superimposing (overlapping) guide frame 20G shown in Fig. 4 on an actual image of the area in front of actual chilled compartment 5A that corresponds to it. Alternatively, the camera is used to capture an image by superimposing (overlapping) guide frame 20G shown in Fig. 5 on an actual image of actual storage shelf 5B that corresponds to it. This is a function of display processing unit 970.

[0062] This allows the camera of the smartphone 20 to face the storage compartment directly and take an image from diagonally above at a predetermined angle, making it possible to obtain a photographed image that allows the food to be properly confirmed and recognized.

[0063] Here, when the overlap of the guide frame 20G with the corresponding actual image of the area in front of the chilled compartment 5A is detected, the display on the display screen 20D can be changed, for example, by changing the color of the guide frame 20G or changing the display state of the guide frame 20G from flashing to lit (or vice versa), thereby informing the matching state of the guide frame 20G with the actual image.

[0064] Other outputs that can be perceived by the user's five senses may be used, such as a vibration function or audio output function of the smartphone 20. At this time, by tapping the inside fridge photo button 20B, an appropriate photographed image can be obtained.

[0065] In this way, in step S14, the inside of the storage compartment can be photographed according to guide frame 20G, which allows the user to easily check the stored food from the photographed image and also improves the accuracy of image recognition. Regarding image recognition, images photographed by smartphone 20 or the like can be provided to connected home appliance server 56, allowing the learning model to learn the images.

[0066] Here, smartphone 20 can continuously capture images of the inside of the storage compartment, and the captured images are linked to the storage compartment in which the image was captured and stored in memory unit 22. For example, the captured image is linked to the storage compartment selected in storage compartment selection unit 966. The date and time (date, time, etc.) when the image was captured is also stored. This makes it easy to check the food inventory at the time of capturing the image.

[0067] The storage unit 22 may use a RAM area or a flash ROM of the microcomputer of the smartphone 20. When the photographing of the inside of the storage compartment is completed, the process proceeds to step S15.

[0068] <Step S15> In step S15, the image captured by the smartphone 20 is transferred to the connected home appliance server 56 via the WAN 54 and stored in the image storage unit 912. The captured image is preferably linked to the refrigerator information (model, etc.) specified via the storage room selection unit 966 or the like, or the storage room information (type of storage room, such as refrigerator, freezer, or vegetable room) that was the subject of the image, as tag information or the like.

[0069] This makes it easier to classify the captured image to be used in a learning model. It is preferable that the superimposed capture guide not be included in the transferred image data, or that the capture guide be separable (deletable) from the image data. The captured image can be corrected when it is saved or before it is transferred to the connected home appliance server 56.

[0070] That is, the captured image can be corrected by determining the degree to which the captured image is tilted or slanted relative to the image (model image) assumed by guide frame 20G. For example, correction can be performed taking into consideration the difference in length and angle of the lines on the left and right of the area corresponding to guide frame 20G, the difference in length and offset from the center of the lines on the front and rear.

[0071] Here, the captured image may be stored in the smartphone 20, or may be deleted from the storage unit 22 since it is stored in the connected home appliance server 56 and can be obtained from the connected home appliance server 56 as needed. In this embodiment, the captured image is stored in the smartphone 20. When the captured image is transferred to the connected home appliance server 56, the process proceeds to step S16.

[0072] <Step S16> In step S16, a food list creation mode application is executed on the smartphone 20. The food list creation mode described below does not need to be executed immediately after step S15, and can be executed by the user as appropriate. In this embodiment, the food list creation mode is executed following the photography mode.

[0073] In step S16, the saved captured image is displayed on the display screen 20D of the smartphone 20. If the captured image is saved in the connected home appliance server 56, the captured image is acquired from the connected home appliance server 56 and displayed on the display screen 20D of the smartphone 20.

[0074] In this way, the photographed image is displayed on the display screen in order to individually specify the food items shown in the photographed image. The food items included in the photographed image have been recognized in advance, and the outline information of the food items has been detected. In other words, outline recognition processing that distinguishes between the area where the food appears and the other areas can be performed in advance. Additionally, if the name of the recognized food item can be detected, the food name can be associated with the food image and stored. Once the photographed image is displayed, the process proceeds to step S17.

[0075] <Step S17> In step S17, foods to be added to the list are selected on display screen 20D. Food selection is performed by touching an individual displayed food with a finger or by specifying an area that includes the food. The selected food may be displayed with an outline so that it is clear that it has been selected. Foods may be selected one by one and step S18 below executed, or multiple foods may be selected simultaneously and step S18 below executed.

[0076] <Step S18> In step S18, the food item designated in step S17 is added to the food item list. When step S18 ends, the series of processes is completed.

[0077] Thereafter, when using smartphone 20 to take a picture of food in a storage compartment, corresponding photography guide information can be displayed on display screen 20D by referring to the refrigerator information acquired in step S11 and the storage compartment information specified by storage compartment selection unit 966. The storage compartment selection unit 966 may require a new designation each time, or the previous designation may be maintained until it is changed.

[0078] In this embodiment, the storage compartment is photographed along the guide frame 20G, so the image of the food can be clearly grasped, thereby improving the accuracy of image recognition. The recognized food is then listed, for example, in a table format, and displayed on the display screen 20D.

[0079] The created food list is transferred to the connected home appliance server 56 via the WAN 54 and stored in the food storage unit 914.

[0080] In this way, in this embodiment, the photography guide information corresponding to the selected storage compartment is displayed on the display screen so that it can be superimposed on the photographed image input to the photography unit, so that in actual photography, an appropriate photographed image can be obtained by photographing the storage compartment in accordance with the photography guide information. Therefore, the user can easily check the stored food from the photographed image, and the recognition accuracy of the image recognition can be improved.

[0081] <<When the smartphone 20 and the refrigerator 10 are not paired (Case 1)>> The second photographing sequence when the smartphone 20 and the refrigerator 10 are not paired will be described with reference to the flowchart of FIG.

[0082] <Step S20> In step S20, the smartphone and refrigerator are not paired, but the smartphone can use an application to access the connected home appliance server 56 via the WAN 54 and send and receive data. An application for selecting a refrigerator to be photographed is executed on the smartphone, and the refrigerator to be photographed that is stored in the connected home appliance server 56 is selected. In this case, the storage room to be photographed is also selected, as described above. Once the refrigerator to be photographed and its storage room have been selected, the process proceeds to step S11.

[0083] Step S11 to Step S18 Since this is the same process as shown in FIG. 3, the explanation will be omitted.

[0084] In this manner, in this embodiment, the photographing guide information corresponding to the selected storage compartment is displayed on the display screen so as to be superimposed on the photographed image input to the photographing unit, so that in actual photographing, an appropriate photographed image can be obtained by photographing the storage compartment in accordance with the photographing guide information. Therefore, the user can easily check the stored food from the photographed image, and the accuracy of image recognition can be improved.

[0085] <<When the smartphone 20 and the refrigerator 10 are not paired (Case 2)>> Next, a third image capture sequence when the smartphone 20 and the refrigerator 10 are not paired will be described with reference to the flowchart in Fig. 7. Fig. 7 shows an example in which refrigerator information is acquired from the members' club server 58 without using the connected home appliance server 56. Here, the members' club server 58 stores refrigerator information including image capture guide information, as described in the flowchart above.

[0086] <Step S30> In step S20, an application for selecting a refrigerator to be photographed is executed, and a refrigerator to be photographed stored in the members' club server is selected. In this case, the storage room to be photographed is also selected, as described above. Once the refrigerator to be photographed and its storage room are selected, the process proceeds to step S11.

[0087] Step S11 to Step S18 3, so a description thereof will be omitted. However, since the members' club server 58 is used in this embodiment, the captured image in step S15 is saved in the members' club server 58, not in the connected home appliance server 56.

[0088] In this manner, in this embodiment, the photographing guide information corresponding to the selected storage compartment is displayed on the display screen so as to be superimposed on the photographed image input to the photographing unit, so that in actual photographing, an appropriate photographed image can be obtained by photographing the storage compartment in accordance with the photographing guide information. Therefore, the user can easily check the stored food from the photographed image, and the accuracy of image recognition can be improved.

[0089] As described above, the present invention is a program that causes a terminal equipped with a display screen and a photographing unit to operate as a storage facility management device, and this program is characterized by causing the terminal to operate as a storage facility selection unit that accepts selection of the type of storage facility to be managed and the type of storage facility to be photographed from the storage facilities equipped in this storage facility, a photographing guide information reference unit that reads out photographing guide information corresponding to the storage facility selected by the storage facility selection unit, a photographing activation unit that activates a photographing function that causes the photographing unit to photograph an image, and a display processing unit that displays the read out photographing guide information on the display screen, superimposed on the scenery input to the photographing unit.

[0090] This allows the storage room to be photographed in accordance with the photographing guide information, thereby providing the effect of suppressing variations in the photographed images.

[0091] The present invention is not limited to the above-described embodiments, but includes various modifications. The above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace other configurations with respect to the configuration of each embodiment. [Explanation of symbols]

[0092] 10...refrigerator, 20...mobile terminal (smartphone), 26...photography unit, 56...connected home appliance server, 58...members club server, 21...control unit, 966...storage room selection unit, 968...photography guide information reference unit, 970...display processing unit.

Claims

1. A program that causes a terminal having a display screen and a photographing unit to operate as a storage management device, The terminal, a storage room selection unit that accepts a selection of a storage room to be photographed from among storage rooms included in a storage facility under management; a photography guide information reference unit that reads out photography guide information corresponding to the storage room selected by the storage room selection unit; a photography activation unit that activates a photography function that causes the photography unit to photograph an image; A program that operates as a display processing unit that displays the read-out photography guide information on the display screen, superimposed on a scene input to the photography unit.

2. 2. The program according to claim 1, The terminal, A program characterized by displaying on the display screen, in accordance with the operation of the photography startup unit, the photography guide information corresponding to the information previously selected by the storage room selection unit, superimposed on the scenery input to the photography unit.

3. 2. The program according to claim 1, A program characterized by displaying on the display screen a sample image including the shooting guide information and a structure to be superimposed on the shooting guide information, according to the type of storage room selected in the storage room selection unit.

4. 2. The program according to claim 1, A program that causes the terminal to execute an output that can be perceived by a user in response to the photography guide information being superimposed on the corresponding structure of the storage room.

5. 2. The program according to claim 1, A program characterized by displaying on the display screen, in addition to the shooting guide information, information regarding the name of the storage room corresponding to the shooting guide information currently being displayed and / or a guide change button for changing the shooting guide information currently being displayed.

6. A storage management system including the program according to claim 1, A storage management system characterized in that images captured in accordance with the operation of the display processing unit are transmitted to an external device of the terminal and used to train a learning model that recognizes the food contained in the images.

7. The storage management system according to claim 6, A storage management system characterized in that the transmitted image is linked to information on the type of the storage facility and / or storage room corresponding to the image.

8. A production method for producing a storage management device used in a storage management system that manages food stored in a storage warehouse from a terminal having a display screen and a photographing unit, The terminal, a storage room selection unit that accepts a selection of a storage room to be photographed from among storage rooms included in a storage facility under management; a photography guide information reference unit that reads out photography guide information corresponding to the storage room selected by the storage room selection unit; a photography activation unit that activates a photography function that causes the photography unit to photograph an image; A production method characterized by including a step of providing a program that functions as a display processing unit that displays the read-out shooting guide information on the display screen, superimposed on the scenery input to the shooting unit.

9. A storage management system having a server, a storage connected to the server via a communication network, and a storage management device connected to the server and the storage via the communication network and including a display screen and a photographing unit, the server stores photographing guide information which is positioning information for each of the storage rooms of the storage facility; When the storage management device refers to the photography guide information, the storage management device displays the photography guide information on the display screen and also displays the scenery input to the photography unit in a superimposed manner. A storage management system characterized by:

Images