Program, imaging system, terminal, and control method of terminal

Abstract

To provide a program, an imaging system, a terminal, and a control method of a terminal, capable of improving convenience for a user in preparation for using a camera and performing management.SOLUTION: A program causes a processor of a terminal comprising a display part to function as a display control part for acquiring storage house information related to a configuration of a storage house having a storage room and a movable part moving when a stored object is taken in / from the storage room, and displaying on the display part a guide screen for guiding work for installing a photographing marker at a position corresponding to the movable part. The guide screen includes a guide screen indicating the installation position of the marker, and the display control part changes the guide screen by corresponding it to the configuration of the storage house indicated by the storage house information.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present disclosure relates to a program, an imaging system, a terminal device, and a control method for a terminal device. [Background technology]

[0002] Patent Document 1 discloses a technology in which a camera is installed on the top of a refrigerator, photographs an object when it detects passing through the opening of the refrigerator, and manages the food stored in the refrigerator based on the photographed result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-070476 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a program, an imaging system, a terminal device, and a method for controlling a terminal device that can improve user convenience in terms of preparation for management using a camera. [Means for solving the problem]

[0005] The program of the present disclosure causes a processor of a terminal device having a display unit to function as a display control unit that acquires storage information regarding the configuration of a storage unit having a storage chamber and a movable part that moves when storing items in and out of the storage chamber, and displays on the display unit a guidance screen that guides the user in placing a marker for photography in a position corresponding to the movable part, the guidance screen including a guidance image indicating the installation position of the marker, and the display control unit changes the guidance screen to correspond to the configuration of the storage unit indicated by the storage chamber information.

[0006] The photography system of the present disclosure comprises a terminal device with a display unit, and a photography device arranged on the top surface of a storage facility, the terminal device acquiring storage facility information regarding the configuration of the storage facility having a storage room and a movable part that moves when storing items in and out of the storage room, and a display control unit causing the display unit to display a guidance screen that guides the user in placing a marker for photography at a position corresponding to the movable part, the guidance screen including a guidance image indicating the installation position of the marker, and the display control unit changing the guidance screen to correspond to the configuration of the storage facility indicated by the storage facility information, and the photography device comprising: a photography unit that photographs an area including the storage room or the movable part, a marker detection unit that detects the marker from an image captured by the photography unit, and a notification unit that notifies the user that the marker has been detected by the marker detection unit.

[0007] The terminal device of the present disclosure includes a display unit, a display control unit that acquires storage information regarding the configuration of a storage facility having a display unit, a storage chamber, and a movable part that moves when stored items are placed in or removed from the storage chamber, and causes the display unit to display a guidance screen that guides the user in placing a marker for photography in a position corresponding to the movable part, the guidance screen including a guidance image indicating the placement position of the marker, and the display control unit changes the guidance screen to correspond to the configuration of the storage facility indicated by the storage facility information.

[0008] The control method of a terminal device in the present disclosure is a control method of a terminal device equipped with a display unit, and includes an acquisition step of acquiring storage facility information regarding the configuration of a storage facility having a storage chamber and a movable part that moves when storing items in and out of the storage chamber, and a display step of displaying a guidance screen on the display unit that guides the user in placing a marker for photography in a position corresponding to the movable part, wherein the guidance screen includes a guidance image that indicates the installation position of the marker, and the guidance screen displayed in the display step is changed to correspond to the configuration of the storage facility indicated by the storage facility information. Effect of the Invention

[0009] According to the program, the photography system, the terminal device, and the control method of the terminal device disclosed herein, the task of installing a marker used for photography by the camera in the storage is guided by a guidance screen corresponding to the configuration of the storage, thereby improving the user's convenience in preparing to manage using the camera. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing the configuration of a food management system according to an embodiment. [Diagram 2] Perspective view of a refrigerator [Diagram 3] Right side view of the imaging device [Figure 4] Front view of the refrigerator and the imaging device [Diagram 5] Right side view of the refrigerator and the imaging device [Figure 6] Plan view of the imaging device [Figure 7] Block diagram showing the configuration of the imaging device [Figure 8] An explanatory diagram showing the narrow-angle camera's shooting range and the marker positions [Figure 9] FIG. 1 is a perspective view of a main portion including a right door hinge member [Figure 10] Block diagram showing the configuration of a terminal device and a food management server [Figure 11] A diagram showing an example of the app UI configuration [Figure 12] A diagram showing an example of the app UI configuration [Figure 13] A diagram showing an example of the app UI configuration [Figure 14] A diagram showing an example of the app UI configuration [Figure 15] A diagram showing an example of the app UI configuration [Figure 16] FIG. 1 is a sequence diagram showing an example of an operation of the imaging system. [Figure 17] A flowchart showing an example of the operation of the imaging device. [Figure 18] 1 is a flowchart showing an example of an operation of a terminal device. [Figure 19] A diagram showing another example of the app UI configuration [Figure 20] A diagram showing another example of the app UI configuration [Figure 21] A diagram showing another example of the app UI configuration [Figure 22] A diagram showing another example of the app UI configuration [Diagram 23] A diagram showing another example of the app UI configuration DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (The knowledge and other information that formed the basis of this disclosure) At the time the inventors came up with the idea of ​​the present disclosure, there was a technology in place that involved installing a camera on top of a storage compartment, taking a picture when it detected an object passing through the opening of the storage compartment, and managing the food stored in the refrigerator based on the results of the picture taking. In order to use this type of technology, it is necessary to make appropriate preparations for the storage facility. For example, preparations include installing a camera in the storage facility. Also, for example, when using a marker as an indicator of position or direction in a captured image, it is necessary to install the marker in the storage facility. The inventors discovered that when a user performs these tasks, there is a problem in that it is a burden for the user to perform the tasks appropriately, and have come to constitute the subject matter of the present disclosure in order to solve this problem. In view of this, the present disclosure provides a program, an imaging system, a terminal device, and a control method for a terminal device that can improve user convenience in terms of preparation for management using a camera.

[0012] Hereinafter, the embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed explanations than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations of substantially the same configurations may be omitted. It should be noted that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0013] [1. Configuration] [1-1. Structure of food control system] FIG. 1 is a diagram showing the configuration of a food management system 1 according to an embodiment. The food management system 1 is a system that manages food stored in a refrigerator 2. As an example, the food management system 1 of this embodiment manages images of food stored in the refrigerator 2. The refrigerator 2 corresponds to an example of a "storage room."

[0014] In this embodiment, the food stored in refrigerator 2 includes the food itself, the container in which the food is stored, etc. The food may be marked with a predetermined marker. Food corresponds to an example of a "storage item."

[0015] The food management system 1 includes a refrigerator 2 . The refrigerator 2 includes a main box 20 with an opening at the front. The main box 20 is formed with a refrigerator compartment 21, an ice-making compartment 22, a fresh-freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25. These compartments are cooled with cold air. A revolving door 21A is provided at an opening 201 at the front of the refrigerator compartment 21. The door 21A opens and closes the opening 201. The door 21A corresponds to an example of a "movable part." Each of the refrigerator compartment 21, the freezer compartment 24, and the vegetable compartment 25 corresponds to an example of a "storage compartment" in the present disclosure, and the "storage compartment" may include the ice-making compartment 22 and the fresh-freezing compartment 23. The right door 21B and the left door 21C constituting the door 21A correspond to an example of a "door."

[0016] In the refrigerator 2 of this embodiment, the door 21A of the refrigeration chamber 21 is a door with a double door shape, that is, a so-called French door. The door 21A includes a rotating right door 21B and a rotating left door 21C. This is just one example, and the door 21A may be a single-leaf door in which one of the left and right ends is supported by the main box body 20 so as to be rotatable.

[0017] Drawers (drawers) 22A, 23A, 24A, and 25A capable of accommodating food are installed in each of ice making compartment 22, fresh freezing compartment 23, freezing compartment 24, and vegetable compartment 25. Each of drawers 22A to 25A has an open top when installed. In the following description, when drawers 22A, 23A, 24A, and 25A are referred to collectively without distinction, they are given the symbol "26A" and described as "drawer 26A." Drawer 26A corresponds to an example of a "movable part." In addition, when ice-making compartment 22, fresh-freezing compartment 23, freezer compartment 24, and vegetable compartment 25 are referred to collectively without distinction, they are given the symbol "26" and described as storage compartment 26.

[0018] The food management system 1 includes an imaging device 3. The imaging device 3 is provided on a top surface 20A of a main box 20 of a refrigerator 2. The top surface 20A of the main box 20 is made of metal. The imaging device 3 takes an image of the refrigerator 2.

[0019] Food management system 1 includes terminal device 4. Terminal device 4 is a portable computer such as a smartphone. An application program related to the management of food stored in refrigerator 2 is installed on terminal device 4, and communicates with food management server 5 using the functions of this application program. In the following description, this application program will be referred to as a “food management application” and will be given the reference number “411.” The configuration of the terminal device 4 including the food management application 411 will be described later with reference to FIG.

[0020] Food management system 1 includes food management server 5. Food management server 5 is a device that manages food stored in refrigerator 2. Food management server 5 stores images of food as part of food management. Food management server 5 is connected to network NW. In FIG. 1 and each of the figures described below, food management server 5 is represented by a single block, but this does not necessarily mean that food management server 5 is composed of a single device. Food management server 5 may be composed of one server computer, may be composed of multiple server computers, or may be a cloud server.

[0021] The photographing device 3 and the food management server 5, which constitute the food management system 1, and the terminal device 4 and the food management server 5 are connected to each other so as to be able to communicate data with each other. A user who uses the food management system 1 is indicated by the symbol P. In FIG. 1, a state in which user P is at home is indicated by a solid line, and a state in which user P is away from home H is indicated by a dotted line. When user P uses terminal device 4 while at home, terminal device 4 communicates with food management server 5 via communication device 6 or without communication device 6. For example, when user P uses terminal device 4 while away from home, terminal device 4 communicates with food management server 5 without via communication device 6.

[0022] The communication device 6 is connected to a network NW constituted by a public line network or the like, and communicates with the food management server 5 via the network NW. The communication device 6 functions as an interface device for connecting the refrigerator 2, the photographing device 3, and the terminal device 4 to the network NW. The communication device 6 establishes a local network in home H.

[0023] [1-2. Refrigerator configuration] FIG. 2 is a perspective view of the refrigerator 2 in an installed state. The X-axis, Y-axis, and Z-axis are illustrated in FIG. 2 and in each figure described later. The X-axis, Y-axis, and Z-axis are coordinate axes that are mutually perpendicular. The Z-axis is an axis that extends in the vertical direction, and the positive direction along the Z-axis is the upward direction. The size of the refrigerator 2 in the direction along the Z-axis corresponds to the height of the refrigerator 2. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis corresponds to the left-right direction of the refrigerator 2, and the positive direction of the X-axis indicates the rightward direction. The Y-axis corresponds to the front-rear direction of the refrigerator 2, and the positive direction of the Y-axis indicates the forward direction. The left-right direction can also be said to be the width direction of the refrigerator 2.

[0024] The refrigerator 2 comprises a main box 20, a door 21A, and a drawer 26A. The main box body 20 is a box-shaped member forming the main body of the refrigerator 2. The main box body 20 includes an outer box 202 that opens to the front, an inner box 203 that is housed inside the outer box 202 and opens to the front, and a heat insulating material filled in the space between the outer box 202 and the inner box 203. The outer box 202 is formed of, for example, a metal such as iron or a hard resin such as ABS. The upper surface of the outer box 202, i.e., the top surface 20A of the main box body 20, is formed of a metal. The inner box 203 is formed of, for example, a hard resin such as ABS. For example, a hard urethane foam is used as the heat insulating material. Note that the heat insulating material is not limited to this, and may be any material that has heat insulating properties and can be filled in the space between the outer box 202 and the inner box 203.

[0025] The internal space of the inner box 203 is divided into a plurality of spaces by partition members. In this embodiment, the internal space of the inner box 203 is divided into five spaces each capable of accommodating food. These five spaces function as a refrigerator compartment 21, an ice-making compartment 22, a fresh-freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25, respectively.

[0026] The right door 21B is rotatably supported by a pair of right door hinge members 204 relative to the main box body 20. Each of the pair of right door hinge members 204 is fixed to the main box body 20. One of the pair of right door hinge members 204 supports an upper end of the right door 21B, and the other supports a lower end of the right door 21B. Figure 2 illustrates the right door hinge member 204 supporting the upper end of the right door 21B.

[0027] The left door 21C is rotatably supported relative to the main box body 20 by a pair of left door hinge members 205. Each of the pair of left door hinge members 205 is fixed to the main box body 20. One of the pair of left door hinge members 205 supports an upper end of the left door 21C, and the other supports a lower end of the left door 21C. Figure 2 illustrates the left door hinge member 205 supporting the upper end of the left door 21C.

[0028] [1-3. Configuration of the imaging device] The photographing device 3 will be described with reference to FIGS. 3 to 6, the X-axis direction corresponds to the left-right direction of the imaging device 3, the Y-axis direction corresponds to the front-back direction of the imaging device 3, and the Z-axis direction corresponds to the up-down direction of the imaging device 3. The left-right direction of the imaging device 3 can be said to be the width direction of the imaging device 3.

[0029] Fig. 3 is a right side view of the camera device 3 attached to the refrigerator 2. Fig. 4 is a front view of the refrigerator 2 and the camera device 3 with the door 21A open. Fig. 5 is a right side view of the refrigerator 2 and the camera device 3. Fig. 5 shows the drawer 25A in the pulled-out state. Fig. 6 is a plan view of the camera device 3 attached to the refrigerator 2, and also shows the top surface 20A of the refrigerator 2.

[0030] 3, the photographing device 3 includes a main body 30 installed on the top surface 20A of the main box 20, and a photographing member 31 extending forward from the main body 30. The photographing member 31 is provided above the front of the main body 30.

[0031] The main body 30 is provided with a first LED (Light emitting diode) 33 (FIG. 7), a second LED 308, and a third LED 309. The first LED 33 turns on and off depending on the power state of the photographing device 3, etc. The second LED 308 and the third LED 309 are installed on the bottom surface 31A of the photographing member 31. The second LED 308 and the third LED 309 illuminate the refrigerator 2 from above. The second LED 308 and the third LED 309 turn on under the control of the photographing control unit 354 (FIG. 7) while photographing is being performed by the wide-angle camera 301 and the narrow-angle camera 302 described below. This illuminates the photographing ranges of the wide-angle camera 301 and the narrow-angle camera 302. The user P can know whether the photographing device 3 is performing photographing based on whether the second LED 308 and the third LED 309 are turned on or not.

[0032] The main body 30 includes a restricting member 34. The restricting member 34 is a plate-like member extending downward from the bottom surface 30A of the main body 30. The restricting member 34 extends in the width direction of the main body 30 (the width direction of the photographing device 3) on the front surface of the main body 30, and has a mark 35 at the center of the main body 30 in the width direction. The mark 35 indicates the center of the photographing device 3 in the width direction.

[0033] When installing the image capturing device 3 on the top surface 20A, the position of the image capturing device 31 in the front-rear direction of the refrigerator 2 is restricted by abutting the restricting member 34 against the front end edge 20B of the main box body 20. This allows the image capturing device 3 to be installed at an appropriate position in the front-rear direction. Since the restricting member 34 extends downward from the bottom surface 30A, it is preferable that the restricting member 34 does not interfere with opening and closing of the door 21A. Specifically, it is preferable that the length of the restricting member 34 is shorter than the length from the top surface 20A to a door packing provided approximately parallel to the top surface 20A at the upper inner surface of the door 21A.

[0034] The imaging device 3 includes a wide-angle camera 301 and a narrow-angle camera 302. The wide-angle camera 301 and the narrow-angle camera 302 are installed at the front end of the bottom surface 31A of the imaging member 31. When the imaging device 3 is attached to the refrigerator 2 and the door 21A is in a closed state, the wide-angle camera 301 and the narrow-angle camera 302 are located in front of the door 21A. The wide-angle camera 301 and the narrow-angle camera 302 are digital cameras that capture images of the storage compartment 26 of the refrigerator 2. The wide-angle camera 301 and the narrow-angle camera 302 correspond to an example of a "imaging section".

[0035] With the photographing device 3 attached to the refrigerator 2, the wide-angle camera 301 photographs the refrigeration compartment 21 from the front and above the refrigerator 2. The wide-angle camera 301 photographs a wider photographing range (angle of view) than the narrow-angle camera 302. For example, the photographing range of the wide-angle camera 301 includes the range A1 shown in FIG. 4 when viewed from the front of the refrigerator 2. With the right door 21B and the left door 21C open to their maximum opening angles, the range A1 includes the door pocket 21E of the right door 21B, the door pocket 21E of the left door 21C, and the opening 201 of the refrigeration compartment 21. Also, for example, the photographing range of the wide-angle camera 301 includes the range A3 shown in FIG. 5 when viewed from the right side of the refrigerator 2. With the right door 21B and the left door 21C open, the range A3 includes from the front end of the right door 21B and the left door 21C to the front end of the shelf 21F in the front-rear direction of the refrigerator 2. Moreover, range A3 includes opening 201 of refrigerating compartment 21 in the vertical direction of refrigerator 2.

[0036] Narrow-angle camera 302 captures an image in a relatively narrower shooting range (angle of view) than wide-angle camera 301. Specifically, the angle of view of narrow-angle camera 302 in the left-right direction, i.e., the X-axis direction, and the angle of view in the front-back direction, i.e., the Y-axis direction, are both smaller than those of wide-angle camera 301.

[0037] With the imaging device 3 attached to the refrigerator 2, narrow-angle camera 302 images drawer 26A from above and in front of refrigerator 2. For example, the imaging range of narrow-angle camera 302 includes range A2 shown in FIG. 4 when viewed from the front of refrigerator 2. Range A2 includes drawer 26A. Also, for example, the imaging range of narrow-angle camera 302 includes range A4 shown in FIG. 5 when viewed from the right side of refrigerator 2. Range A4 is a range that includes drawer 26A when it is pulled out to the furthest position in the front-to-rear direction. In this way, narrow-angle camera 302 can image the inside of drawer 23A from above and in front of drawer 23A when drawer 23A is pulled out, and the same applies to drawers 24A and 25A. The imaging range of narrow-angle camera 302 may be a range that includes drawers provided in refrigeration chamber 21.

[0038] 6, the image capturing device 3 includes a first distance measuring sensor 303, a second distance measuring sensor 304, and a thermopile 305. The first distance measuring sensor 303 and the second distance measuring sensor 304 are disposed on the bottom surface 30A so as to face downward.

[0039] The first distance measuring sensor 303 is a sensor that detects the distance between the first distance measuring sensor 303 and the right door 21B. The second distance measuring sensor 304 is a sensor that detects the distance between the second distance measuring sensor 304 and the left door 21C. The distance between the first distance measuring sensor 303 and the right door 21B differs depending on whether the right door 21B is open or closed. Therefore, the first distance measuring sensor 303 can detect that the right door 21B has been opened. Similarly, the distance between the second distance measuring sensor 304 and the left door 21C differs depending on whether the left door 21C is open or closed. Therefore, the second distance measuring sensor 304 can detect that the left door 21C has been opened. The first distance measuring sensor 303 and the second distance measuring sensor 304 are, for example, reflective optical distance measuring sensors having a light source that emits infrared light and an optical sensor that detects the amount of reflected light of the light emitted by the light source. As the first distance measuring sensor 303 and the second distance measuring sensor 304, it is of course possible to use other types of sensors, such as ultrasonic distance measuring sensors.

[0040] Thermopile 305 is disposed at the front end of bottom surface 31A. The detection range of thermopile 305 is below bottom surface 31A. More specifically, the detection range of thermopile 305 is below bottom surface 31A and includes the front surface of refrigerator 2 and the area in front of it. Therefore, drawers 23A, 24A, 25A are within the detection range of thermopile 305 when pulled out. Thermopile 305 functions as a radiation thermometer and outputs a detection value indicating the detected temperature.

[0041] When there is no obstacle between the thermopile 305 and the floor within the detection range of the thermopile 305, the thermopile 305 detects the temperature of the floor in front of the refrigerator 2. The floor temperature detected in this case is room temperature, for example, 20°C. For example, when the drawer 23A is pulled out, the thermopile 305 detects the temperature of the drawer 23A. Since the fresh freezing compartment 23 is lower than room temperature, the temperature detected by the thermopile 305 changes before and after the drawer 23A is pulled out. In this way, when the drawer 23A is pulled out, for example, the image capture device 3 can detect that the drawer 23A has been pulled out based on the change in temperature detected by the thermopile 305. By using the thermopile 305, it is also possible to detect that either the drawer 24A or the drawer 25A has been opened. In this case, the photographing device 3 may take advantage of the temperature difference between the fresh freezing compartment 23, the freezer compartment 24, and the vegetable compartment 25, and identify which of the drawers 23A, 24A, and 25A has been pulled out based on the detection results of the thermopile 305.

[0042] The image capturing device 3 can also detect the temperature of an object located in front of the refrigerator 2 using the thermopile 305. For example, when the body of the user P is within the detection range of the thermopile 305, the thermopile 305 detects a temperature higher than the room temperature. In this case, the image capturing device 3 may detect that at least a part of the body of the user P is in front of the refrigerator 2 using the thermopile 305.

[0043] The photographing device 3 includes a buzzer 306. The buzzer 306 is housed in the main body 30, and sounds when an audio notification is to be given to the user P. The buzzer 306 corresponds to an example of a "notification unit".

[0044] Image capture device 3 includes illuminance sensor 307. Illuminance sensor 307 is installed on bottom surface 31A, and the detection range of illuminance sensor 307 is below bottom surface 31A and in front of refrigerator 2. It is desirable that the detection range of illuminance sensor 307 includes at least a part of the shooting range of wide-angle camera 301 and at least a part of the shooting range of narrow-angle camera 302. The detection result of illuminance sensor 307 is used for adjusting the exposure of wide-angle camera 301 and narrow-angle camera 302, etc.

[0045] As shown in FIG. 6, the photographing device 3 has a first control board 311, a second control board 312, and a wireless communication unit 313, which are housed in the main body 30. In the main body 30, the first control board 311 is disposed behind the second control board 312 and the wireless communication unit 313. The first control board 311 controls various sensors included in the photographing device 3 and the power supply of the photographing device 3. The second control board 312 controls the wide-angle camera 301, the narrow-angle camera 302, and the wireless communication unit 313. The wireless communication unit 313 is a wireless communication device that includes hardware such as an antenna and a wireless communication circuit, and transmits and receives wireless signals. The wireless communication unit 313 executes communication according to a wireless communication method such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0046] FIG. 7 is a block diagram showing the configuration of the photographing device 3. The first control board 311 is connected to the first LED 33, the second LED 308, the third LED 309, the first distance measuring sensor 303, the second distance measuring sensor 304, the thermopile 305, the buzzer 306, and the illuminance sensor 307. The second control board 312 is connected to the wide-angle camera 301, the narrow-angle camera 302, and the wireless communication unit 313.

[0047] A first communication module 340 is mounted on the first control board 311. A second communication module 370 is mounted on the second control board 312. The first communication module 340 and the second communication module 370 each have a communication circuit including a connector for connecting a cable, and a transmitter and a receiver for transmitting and receiving signals via the cable, and may also include other peripheral circuits. The first communication module 340 and the second communication module 370 are connected to each other by a communication cable, and perform inter-board signal transmission between the first control board 311 and the second control board 312.

[0048] The first control board 311 includes a first processor 320 and a first memory 330. The first processor 320 is configured with a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The first memory 330 is a storage device that stores programs and data executed by the first processor 320. The first memory 330 is configured with a non-volatile storage device such as a ROM (Read Only Memory). The first memory 330 is configured with a ROM (Read Only Memory) or a RAM (Random Access Memory). The first memory 330 may include a volatile storage device that configures a work area of ​​the first processor 320. The first memory 330 stores data processed by the first processor 320 and a control program 331 executed by the first processor 320.

[0049] The first processor 320 reads and executes the control program 331 to control each unit connected to the first control board 311. Specifically, the first processor 320 controls turning on and off the first LED 33, the second LED 308, and the third LED 309. The first processor 320 causes the first distance measurement sensor 303, the second distance measurement sensor 304, the thermopile 305, and the illuminance sensor 307 to perform detection and acquire detection values. The first processor 320 transmits the detection values ​​of each sensor to the second control board 312 via the first communication module 340. The first processor 320 also causes the buzzer 306 to sound in response to the detection values ​​of the sensors, the processing results of the second control board 312, or other conditions.

[0050] The second control board 312 includes a second processor 350 and a second memory 360. The second processor 350 is configured with a CPU, an MPU, or the like. The second memory 360 is a storage device that stores programs and data executed by the second processor 350. The second memory 360 is configured with a non-volatile storage device such as a ROM. The second memory 360 is configured with a ROM or a RAM, for example. The second memory 360 may include a volatile storage device that constitutes a work area of ​​the second processor 350. The second memory 360 stores data processed by the second processor 350, a control program 361 executed by the second processor 350, and a photographing device ID (Identification) 362. The photographing device ID 362 is identification information of the photographing device 3. For example, the photographing device ID 362 allows the food management server 5 to uniquely identify one photographing device 3 from multiple photographing devices 3.

[0051] The second processor 350 reads and executes the control program 361, thereby functioning as a first communication control unit 351, a second communication control unit 352, an open / close determination unit 353, an imaging control unit 354, a marker detection unit 355, an image determination unit 356, and a generation unit 357. In addition, the second processor 350 causes the second memory 360 to function as a state storage unit 363.

[0052] The first communication control unit 351 controls the wireless communication unit 313 to execute data communication with the food management server 5. The second communication control unit 352 controls the second communication module 370 to execute communication with the first control board 311. The second communication control unit 352 receives detection values ​​of various sensors from the first control board 311 via the second communication module 370.

[0053] The open / close determination unit 353 determines whether the right door 21B is open or closed based on the detection value of the first distance measurement sensor 303. When the detection value of the first distance measurement sensor 303 is different from the detection value corresponding to the closed state, the open / close determination unit 353 determines that the right door 21B is open. When the detection value of the first distance measurement sensor 303 is the detection value corresponding to the closed state, the open / close determination unit 353 determines that the right door 21B is closed. Similarly to the right door 21B, the open / close determination unit 353 determines whether the left door 21C is open or closed based on the detection value of the second distance measurement sensor 304.

[0054] Open / close determination unit 353 determines whether drawer 26A is open or closed based on the detection value of thermopile 305. Open / close determination unit 353 determines that drawer 26A has changed from the closed state to the open state, for example, when the temperature indicated by the detection value of thermopile 305 rises and the rise in temperature within a predetermined time is equal to or greater than a first threshold value TH1. First threshold value TH1 is, for example, 5°C. Open / close determination unit 353 also determines that drawer 26A has changed from the closed state to the open state, for example, when the temperature indicated by the detection value of thermopile 305 falls and the fall in temperature within a predetermined time is equal to or greater than a second threshold value TH2. Second threshold value TH2 is, for example, 5°C.

[0055] Open / close determination unit 353 determines that drawer 26A is closed, for example, when the temperature detected by thermopile 305 remains within a predetermined range for a predetermined period of time or longer. The predetermined temperature range in this case is determined according to the floor temperature, and is, for example, between 15°C and 25°C. The predetermined period is, for example, 5 seconds.

[0056] The shooting control unit 354 controls the start and stop of shooting by the wide-angle camera 301 and the start and stop of shooting by the narrow-angle camera 302 . For example, when the open / close determination unit 353 determines that the door 21A or the drawer 26A is in the open state, the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start photographing. Here, the second communication control unit 352 transmits lighting request information to the first control board 311 when the photography control unit 354 causes at least one of the wide-angle camera 301 and the narrow-angle camera 302 to start photographing. The lighting request information is information requesting lighting of the second LED 308 and the third LED 309. When the first processor 320 receives the lighting request information, it turns on the second LED 308 and the third LED 309. The lighting of the second LED 308 and the third LED 309 notifies the user P that photographing has started.

[0057] When the opening / closing determination unit 353 determines that the door 21A and the drawer 26A are in the closed state, the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end photography. Here, the second communication control unit 352 transmits turn-off request information to the first control board 311 when the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end photography. The turn-off request information is information requesting turning off the second LED 308 and the third LED 309. When the first processor 320 receives the turn-off request information, it turns off the second LED 308 and the third LED 309. When the second LED 308 and the third LED 309 are turned off, the user P is notified that photography has ended.

[0058] In this embodiment, the shooting control unit 354 causes wide-angle camera 301 and narrow-angle camera 302 to shoot moving images (video). The shooting control unit 354 acquires video data shot by wide-angle camera 301, and stores the acquired video data in second memory 360 as first recorded data DV1. The shooting control unit 354 acquires video data shot by narrow-angle camera 302, and stores the acquired video data in second memory 360 as second recorded data DV2.

[0059] The shooting control unit 354 generates still images from the video data captured by the wide-angle camera 301 and the narrow-angle camera 302. Specifically, the shooting control unit 354 acquires the first recording data DV1, extracts still images constituting the frames of the first recording data DV1, and stores the extracted still image data as first captured data 363A in the state storage unit 363. Similarly, the shooting control unit 354 acquires the second recording data DV2, extracts still images constituting the frames of the second recording data DV2, and stores the extracted still image data as second captured data 363B in the state storage unit 363.

[0060] When analyzing the images, the imaging control unit 354, and the marker detection unit 355, image determination unit 356, and generation unit 357 described below detect images of markers included in the first imaging data 363A and the second imaging data 363B. Here, the marker will be described.

[0061] FIG. 8 is an explanatory diagram showing the shooting range of narrow-angle camera 302 and the positions of markers, and is a right side view of refrigerator 2 and imaging device 3.

[0062] In this embodiment, a refrigerator compartment marker MK1 is provided on the right door hinge member 204 supporting the right door 21B, and on the left door hinge member 205 supporting the left door 21C. Also, a first marker MK41 is provided on the drawer 24A, a second marker MK42 is provided on the storage case 24B, a third marker MK51 is provided on the drawer 25A, and a fourth marker MK52 is provided on the storage case 25B. When the refrigerator compartment marker MK1, the first marker MK41, the second marker MK42, the third marker MK51, and the fourth marker MK52 are collectively referred to without distinction, they will be described as markers MK.

[0063] Storage case 24B is a drawer-shaped member disposed at the top of freezer compartment 24. Storage case 24B is slidable inside drawer 24A, and can be pulled out to the front of refrigerator 2 when drawer 24A is pulled out. Food can be stored in freezer compartment 24, divided into the bottom of drawer 24A and storage case 24B. Storage case 25B is a drawer-shaped member disposed at the top of vegetable compartment 25. Storage case 25B is slidable inside drawer 25A, and can be pulled out to the front of refrigerator 2 when drawer 25A is pulled out. Food can be stored in vegetable compartment 25, divided into the bottom of drawer 25A and storage case 25B. Storage cases 24B and 25B are an example of a "movable part" like drawers 24A and 25A. That is, freezer compartment 24 has two movable parts, drawer 24A and storage case 24B, and vegetable compartment 25 has two movable parts, drawer 25A and storage case 25B.

[0064] The marker MK has an appearance that can be distinguished from other subjects in the images captured by the wide-angle camera 301 and the narrow-angle camera 302. In this embodiment, the marker MK is a geometric figure including a rectangle. The marker MK is painted in two or more colors with different brightness so that it can be clearly identified even in a monochrome image, and is composed of, for example, white and black, or white and a primary color. The marker MK is composed of a geometric figure printed or drawn on a base material. Examples of the base material include a synthetic resin or metal member having a flat surface to which the marker MK can be added, a sticker, paper, or a synthetic resin sheet.

[0065] The marker MK is fixedly installed at a position corresponding to a movable part of the refrigerator 2. There is no limitation on the method of fixing the marker MK. For example, the marker MK is installed by attaching a base material of the marker MK to the movable part or a member supporting the movable part.

[0066] It is preferable that each marker MK provided corresponding to each movable part has a different appearance. For example, the refrigerator compartment marker MK1, the first marker MK41, the second marker MK42, the third marker MK51, and the fourth marker MK52 each have a different appearance. In this case, by extracting the image of the marker MK from the captured image, it is possible to identify the movable part that is open.

[0067] 9 is a perspective view of a main portion including the right door hinge member 204, and shows an installation state of the refrigerator compartment marker MK1. The refrigerator compartment marker MK1 is disposed on the lower right door hinge member 204 of the pair of right door hinge members 204 supporting the right door 21B.

[0068] The right door hinge member 204 is a strong member fixed to the main box body 20 of the refrigerator 2. The hinge cover AS is a member having a shape that allows it to be fitted from above into the right door hinge member 204, and has a size and shape that allows the hinge cover AS not to interfere with the right door 21B when the hinge cover AS is attached to the right door hinge member 204. The upper surface of the hinge cover AS is flat, and a refrigerator compartment marker MK1 is printed on this flat surface, or a sticker with the refrigerator compartment marker MK1 is affixed to it.

[0069] With right door 21B open, refrigerator compartment marker MK1 is exposed in the shooting range of wide-angle camera 301 and in the shooting range of narrow-angle camera 302. Therefore, with right door 21B open, refrigerator compartment marker MK1 can be photographed by wide-angle camera 301 and narrow-angle camera 302. That is, the image photographed by wide-angle camera 301 and the image photographed by narrow-angle camera 302 include the image of refrigerator compartment marker MK1.

[0070] Although not shown, a hinge cover (not shown) formed symmetrically to the hinge cover AS is attached to the lower left door hinge member 205 of the pair of left door hinge members 205 supporting the left door 21C. A refrigerator compartment marker MK1 is attached to this hinge cover. Therefore, when the wide-angle camera 301 or the narrow-angle camera 302 captures an image with the left door 21C open, the captured image includes an image of the refrigerator compartment marker MK1.

[0071] Returning to FIG. 8, a first marker MK41 is placed on the upper surface of the front end of drawer 24A. A second marker MK42 is placed on the upper surface of the front end of storage case 24B. Therefore, when drawer 24A is pulled out, the first marker MK41 is exposed, and the image captured by wide-angle camera 301 and the image captured by narrow-angle camera 302 include the image of the first marker MK41. Similarly, when storage case 24B is pulled out, the second marker MK42 is exposed, and the image captured by wide-angle camera 301 and the image captured by narrow-angle camera 302 include the image of the second marker MK42.

[0072] In drawer 25A, a third marker MK51 is placed on the upper surface at the front end of drawer 25A. Also, a fourth marker MK52 is placed on the upper surface at the front end of storage case 25B. Therefore, when drawer 25A is pulled out, the third marker MK51 is exposed, and the image captured by wide-angle camera 301 and the image captured by narrow-angle camera 302 include the image of the third marker MK51. Similarly, when storage case 25B is pulled out, the fourth marker MK52 is exposed, and the image captured by wide-angle camera 301 and the image captured by narrow-angle camera 302 include the image of the fourth marker MK52.

[0073] As shown in Fig. 8, it is preferable that the third marker MK51 falls within the imaging range of the narrow-angle camera 302 when the drawer 25A is pulled out to its fullest extent. In other words, it is preferable that the position and imaging range of the narrow-angle camera 302 are configured so that the third marker MK51 falls within the imaging range of the narrow-angle camera 302 when the drawer 25A is pulled out to its fullest extent. Since the imaging range of the wide-angle camera 301 is wider than that of the narrow-angle camera 302, the third marker MK51 is included in the imaging range of the wide-angle camera 301. In this case, regardless of the amount by which the drawer 25A is pulled out, the third marker MK51 can be imaged by the wide-angle camera 301 and the narrow-angle camera 302 when the drawer 25A is pulled out. In this configuration, the fourth marker MK52 can also be imaged by the wide-angle camera 301 and the narrow-angle camera 302.

[0074] Similarly, it is preferable that the first marker MK41 falls within the imaging range of narrow-angle camera 302 when drawer 24A is pulled out to its fullest extent. In other words, it is preferable that the position and imaging range of narrow-angle camera 302 are configured so that the first marker MK41 falls within the imaging range of narrow-angle camera 302 when drawer 24A is pulled out to its fullest extent. Since the imaging range of wide-angle camera 301 is wider than that of narrow-angle camera 302, the first marker MK41 is included in the imaging range of wide-angle camera 301. In this case, regardless of the amount by which drawer 24A is pulled out, the first marker MK41 can be imaged by wide-angle camera 301 and narrow-angle camera 302 when drawer 24A is pulled out. In this configuration, the second marker MK42 can also be imaged by wide-angle camera 301 and narrow-angle camera 302.

[0075] Returning to FIG. 7, the marker detection unit 355 determines whether or not an image of the marker MK can be detected for each of the first captured data 363A and the second captured data 363B.

[0076] The image determination unit 356 determines the type of the image of the marker MK for the image determined by the marker detection unit 355 that the marker MK can be detected. That is, the image determination unit 356 determines whether the detected marker MK is the refrigerator compartment marker MK1, the first marker MK41, the second marker MK42, the third marker MK51, or the fourth marker MK52. The image determination unit 356 detects the outer edge of the image of the marker MK from the first photographed data 363A or the second photographed data 363B, extracts the image of the detected marker MK, and identifies which marker MK the extracted image is by pattern matching.

[0077] The image determination unit 356 determines whether the image in which the marker MK has been determined includes an image of food. The image determination unit 356 determines whether the image includes an image of food based on, for example, the position of the marker MK detected from the image. For example, when the position of the image of the third marker MK51 in the captured image of the drawer 25A is close to the refrigerator 2, the image determination unit 356 determines that the drawer 25A is not fully opened and therefore does not include an image of food. Also, for example, when the position of the image of the third marker MK51 in the captured image of the drawer 25A is away from the refrigerator 2, the image determination unit 356 determines that the drawer 25A is fully opened and therefore includes an image of food.

[0078] The generating unit 357 extracts an image of food from the captured image determined by the image determining unit 356 to include an image of food. The generating unit 357 generates management data D1 including data related to the food stored in the refrigerator 2 based on the extracted image of the food, and stores the management data D1 in the second memory 360. The management data D1 includes data related to the food and data indicating the storage room in which the food is stored. The management data D1 may include data indicating the captured image from which the data was extracted. The data indicating the storage room in which the food is stored is data specifying any one of the refrigerator compartment 21, the drawer 24A, the storage case 24B, the drawer 25A, and the storage case 25B. The data indicating the captured image includes, for example, data indicating whether the captured image is the first captured data 363A or the second captured data 363B, and data indicating the capture date and time.

[0079] In this way, image capture device 3 generates management data D1 related to the food stored in refrigerator compartment 21 based on first recorded data DV1 captured by wide-angle camera 301 and second recorded data DV2 captured by narrow-angle camera 302. Management data D1 is transmitted to food management server 5, for example, using wireless communication unit 313. In this way, food management system 1 can manage the food stored in refrigerator 2.

[0080] The photographing device 3 uses the marker MK to identify whether the photographed movable part is the door 21A, the drawers 24A, 25A, or the storage cases 24B, 25B. The photographing device 3 also extracts the image of the marker MK from the photographed image to determine whether the opening angle of the door 21A and the amount of the drawers 24A, 25A and the storage cases 24B, 25B pulled out are sufficient or not. To make these determinations, it is necessary that the marker MK is placed at an appropriate position in the refrigerator 2.

[0081] The photography system 7 has a function of providing guidance via the terminal device 4 when the user P performs the task of placing the marker MK in the refrigerator 2, and this guidance supports the task of the user P. The function of providing guidance regarding the task of placing the marker MK will be described below.

[0082] [1-4. Food management server configuration] 10 is a block diagram showing the configuration of the terminal device 4 and the food management server 5. First, the configuration of the food management server 5 will be described.

[0083] The food management server 5 includes a server control device 50 and a server communication unit 51.

[0084] The server control device 50 includes a server processor 500 such as a CPU or an MPU, a server memory 510, and an interface circuit to which other devices and sensors are connected.

[0085] Server memory 510 is a memory that stores programs and data. Server memory 510 has a non-volatile storage area, and stores programs executed by server processor 500 and data processed by server processor 500. Specifically, server memory 510 stores control program 511 and refrigerator data 512. Server memory 510 also has a volatile storage area, and may constitute a work area of ​​server processor 500. Server memory 510 is constituted by, for example, a ROM or a RAM.

[0086] Refrigerator data 512 includes data related to refrigerator 2 in which camera device 3 is installed. For example, refrigerator data 512 includes first recorded video data DV1, second recorded video data DV2, first captured data 363A, second captured data 363B, etc. captured by camera device 3. Refrigerator data 512 may also include data related to food detected by camera device 3. In other words, refrigerator data 512 may include management data D1.

[0087] The server communication unit 51 includes hardware such as a communication circuit that complies with a predetermined communication standard, and communicates with devices connected to the network NW under the control of the server control device 50.

[0088] The server processor 500 reads and executes a control program 511 stored in a server memory 510 , thereby functioning as a server communication control unit 501 and a server processing unit 502 .

[0089] The server communication control unit 501 communicates with the image capturing device 3 and the terminal device 4 via the server communication unit 51. The server processing unit 502 performs various information processes with the image capturing device 3 and the terminal device 4 as clients. For example, the server communication control unit 501 receives the first recorded data DV1, the second recorded data DV2, the first captured data 363A, the second captured data 363B, the management data D1, etc. from the image capturing device 3.

[0090] Server processing unit 502 refers to the photographing device ID 362 included in the data received by server communication control unit 501, and generates refrigerator data 512 including the photographing device ID 362. Server processing unit 502 stores the generated refrigerator data 512 in server memory 510. If refrigerator data 512 including the referred photographing device ID 362 is already stored in server memory 510, server processing unit 502 may calculate a difference between the newly generated refrigerator data 512 and the refrigerator data 512 stored in server memory 510, and may generate new data regarding a change in the inventory amount of ingredients stored in refrigerator 2. In this case, the data regarding the change in the inventory amount of ingredients may be included in refrigerator data 512.

[0091] [1-5. Terminal device configuration] As shown in FIG. 10, the terminal device 4 includes a terminal control device 40, a terminal communication unit 41, and a touch panel .

[0092] The terminal control device 40 includes a terminal processor 400 such as a CPU or an MPU, a terminal memory 410, and an interface circuit to which other devices and sensors are connected.

[0093] The terminal memory 410 is a memory that stores programs and data. The terminal memory 410 has a non-volatile storage area, and stores programs executed by the terminal processor 400 and data processed by the terminal processor 400. Specifically, the terminal memory 410 stores a food management application 411. The terminal memory 410 may also have a volatile storage area and constitute a work area of ​​the terminal processor 400. The terminal memory 410 is composed of, for example, a ROM or a RAM. The terminal memory 410 stores an account ID that uniquely identifies a user P of the food management application 411. The terminal processor 400 is an example of a "processor", and the food management application 411 is an example of a "program".

[0094] The terminal communication unit 41 includes hardware such as a communication circuit that complies with a predetermined communication standard, and communicates with the food management server 5 under the control of the terminal processor 400 . The touch panel 42 includes a display panel and a touch sensor that is provided over or integral with the display panel. The touch panel 42 corresponds to an example of a "display unit."

[0095] The terminal processor 400 reads out and executes a food management application 411 stored in the terminal memory 410, thereby functioning as a terminal communication control unit 401, a display control unit 402, and a reception unit 403.

[0096] The terminal communication control unit 401 communicates with the food management server 5 via the terminal communication unit 41. The display control unit 402 causes the touch panel 42 to display a user interface related to management of food stored in the refrigerator 2 and initial settings related to the refrigerator 2. In the following description, this user interface is referred to as an "application UI (User Interface)" and is denoted by the reference number "430." The reception unit 403 receives various inputs from the user P via the application UI 430 .

[0097] [1-6. Example of app UI configuration] Here, a description will be given of the configuration of the application UI 430. Fig. 11 to Fig. 15 show an application UI 430A as a first configuration example of the application UI 430. In each of Fig. 11 to Fig. 15, the upward direction in the figure is indicated by the symbol "UP", the downward direction in the figure is indicated by the symbol "DW", the leftward direction in the figure is indicated by the symbol "L", and the rightward direction in the figure is indicated by the symbol "R". The up-down direction in Fig. 11 to Fig. 15 is a direction approximately parallel to the side of the display surface of the touch panel 42 when the display surface of the touch panel 42 is viewed. Also, the left-right direction in Fig. 11 to Fig. 15 is a direction approximately parallel to the side perpendicular to the side of the display surface of the touch panel 42 when the display surface of the touch panel 42 is viewed.

[0098] The display control unit 402 displays a plurality of guide screens G in a switchable manner on the application UI 430A. The guide screen G is a screen that guides the user P on the content and procedure of the work of placing a marker MK on the refrigerator 2. The display control unit 402 sequentially switches and displays the plurality of guide screens G. When the reception unit 403 receives an operation of the user P while the guide screen G is being displayed, the display control unit 402 switches the guide screen G according to the input of the user P. The guide screens G include guide screens G101 to G104, G111 to G114, G121 to G124, G131 to G135, and G181 to G183. These screens will be described in order. In the following description, when there is no need to distinguish between the guides 111 to G114, G121 to G124, G131 to G135, and G181 to G183, they will be referred to as guide screens G. Note that the guide screens G are not limited to still images. The guidance screen G may be displayed, for example, as a moving image (animation).

[0099] In the following description, the image of the storage room 26 displayed on the application UI 430A is referred to as a storage room image GP, and the image of the marker MK is referred to as a marker image MP. As specific examples of the storage room image GP, storage room images GP101, GP102, GP111, GP112, GP121, and GP122 are shown in each of Figs. 11, 13 to 15, and when there is no need to distinguish between them, they are referred to as storage room images GP. As specific examples of the marker image MP, marker images MP101, MP102, MP111, MP112, MP121, and MP122 are shown in Figs. 11, 13 to 15, and when there is no need to distinguish between them, they are referred to as marker images MP.

[0100] There are no limitations on the specific aspects of the images, messages, and other display objects arranged on the guidance screens G101 to G104, G111 to G114, G121 to G124, G131 to G135, and G181 to G183 described below. For example, the size of the images, the type and size of the font of the characters, and the size and form (portrait or landscape, etc.) of each guidance screen G can be appropriately changed depending on the specifications of the touch panel 42 that displays the application UI 430AA and the usage form of the terminal device 4, or other reasons.

[0101] Guidance screens G101 to G104, G111 to G114, G121 to G124, G131 to G135, and G181 to G183 are screens that are displayed as the app UI 430A when markers MK are placed on the refrigerator 2 shown in Fig. 2. The photography system 7 provides the app UI 430A corresponding to the number of doors and drawers of the refrigerator 2, the number of storage compartments in the refrigerator 2, and the number of markers MK to be placed on the refrigerator 2. Specifically, the app UI 430A includes guidance screens G in the number of storage compartments 26 in which the markers MK are placed plus the refrigeration compartments 21, and the terminal device 4 switches between these guidance screens G to display them on the app UI 430A.

[0102] Fig. 11 is a diagram showing an example of guidance screens G101 to G104 and their transitions. In Fig. 11, Fig. 11(A) shows guidance screen G101, Fig. 11(B) shows guidance screen G102, Fig. 11(C) shows guidance screen G103, and Fig. 11(D) shows guidance screen G104.

[0103] The guidance screens G101, G102, G103, and G104 constitute a combination of one set of guidance screens G that guides the user in placing the marker MK in the refrigerator compartment 21.

[0104] When the display of the application UI430A starts, a guide screen G101 that provides guidance on placing the marker MK in the refrigerator compartment 21 is displayed on the touch panel 42 before a guide screen G that provides guidance on attaching the marker MK to the freezer compartment 24 and the vegetable compartment 25. Therefore, the marker MK is placed in the refrigerator compartment 21 before the freezer compartment 24 and the vegetable compartment 25.

[0105] The guidance screen G101 includes a message M101 and a button B101. The message M101 includes text and images that explain the outline of the procedure for attaching the markers MK to the refrigerator compartment 21, the freezer compartment 24, and the vegetable compartment 25. The button B101 is an operation unit that accepts an operation by the user P, and is a button that instructs transition to the guidance screen G. When the button B101 is operated, the display of the application UI 430A transitions to the guidance screen G102.

[0106] The guidance screen G102 is displayed as a pop-up so as to overlap the grayed-out guidance screen G101, for example, as shown in FIG. 11. The guidance screen G102 includes a message M102 and buttons B102, B103, and B104. The message M102 includes character strings and images that provide information such as the packaging state of the marker MK, the appearance of the refrigerator marker MK1, and the orientation of the refrigerator marker MK1 when attached. The buttons B102, B103, and B104 are operation units that accept operations by the user P. The button B102 accepts an operation to instruct the display of the application UI430A to be returned to the guidance screen G101. Note that the button B102 may accept an operation to instruct the display of the application UI430A to be displayed again as the guidance screen G102. In this case, the page reload of the guidance screen G102 is executed by operating the button B102, and the guidance screen G102 is displayed again on the application UI430. The button B103 accepts an operation to instruct a transition from the guidance screen G102 to the next guidance screen G. The button B104 accepts an operation to instruct the start of an automatic check function.

[0107] The automatic check function is a function that, after the user P places the marker MK on the refrigerator 2, checks whether the marker MK can be appropriately photographed by the photographing device 3 using the photographing device 3. The automatic check function corresponds to the processing of step SA3 (FIG. 16) described below. The automatic check function checks whether the marker MK is shown in an appropriate position in the image photographed by the photographing device 3. In other words, by using the automatic check function, it can be determined whether the relative position of the photographing device 3 and the marker MK is appropriate.

[0108] When the button B103 on the guidance screen G102 is operated, the display of the application UI 430A transitions to the guidance screen G103. The guidance screen G103 includes a storage room image GP101, a marker image MP101, a message M103, and buttons B104, B105, and B106.

[0109] The message M103 is a character string prompting the user P to open the door 21A and place the refrigerator compartment marker MK1, and is placed at the top of the guidance screen G103.

[0110] The storage compartment image GP101 is an image showing the position where the refrigerator compartment marker MK1 is to be installed, and is an image showing the door 21A as a movable part. The storage compartment image GP101 is an image showing the installation position of the refrigerator compartment marker MK1 on an image showing the door 21A and the inside of the refrigerator compartment 21 with the door 21A open. The image of the refrigerator compartment 21 included in the storage compartment image GP101 may be an image captured in advance by the imaging device 3, or may be an image prepared in advance to match the shape of the door 21A of the refrigerator 2.

[0111] Marker image MP101 is an image of refrigerator compartment marker MK1 installed corresponding to refrigerator compartment 101. Marker image MP101 is an image in which the appearance of refrigerator compartment marker MK1, i.e., the shape and pattern of refrigerator compartment marker MK1, can be identified, and is, for example, a photographed image of refrigerator compartment marker MK1. Marker image MP101 is preferably an image showing a state in which refrigerator compartment marker MK1 is installed on right door hinge member 204. In this case, user P can know more specifically the installation position and installation state of refrigerator compartment marker MK1.

[0112] The refrigerator compartment marker MK1 is placed on each of the right door hinge member 204 and the left door hinge member 205. The guidance screen G103 provides guidance on placing the marker MK1 on the right door hinge member 204. For this reason, the storage compartment image GP101 shows the right door hinge member 204 as the placement position of the marker MK1, and the marker image MP101 is an image of the marker MK1 placed on the right door hinge member 204.

[0113] The button B104 is also used for the guidance screen G102. The buttons B105 and B106 are operation units that accept operations by the user P. The button B105 accepts an operation to instruct the display of the application UI 430A to return to the guidance screen G102. The button B106 accepts an operation to instruct the transition from the guidance screen G103 to the next guidance screen G.

[0114] When the button B106 on the guidance screen G103 is operated, the display of the application UI 430A transitions to the guidance screen G104. The guidance screen G104 includes a storage room image GP102, a marker image MP102, a message M104, and buttons B104, B107, and B108.

[0115] The message M104 is a character string that prompts the user P to operate the button B104 to execute the automatic check function after installing the refrigerator compartment marker MK1.

[0116] The storage room image GP102 and the marker image MP102 guide the installation of the marker MK1 on the left door hinge member 205. For this reason, the storage room image GP101 indicates the left door hinge member 205 as the installation position of the marker MK1, and the marker image MP102 is an image of the marker MK1 installed on the left door hinge member 205. Except for these points, the marker image MP102 is similar to the marker image MP101, and the storage room image GP102 is similar to the storage room image GP101.

[0117] The button B104 is also used for the guidance screen G102. The buttons B107 and B108 are operation units that accept operations by the user P. The button B107 accepts an operation to instruct the display of the app UI 430A to be returned to the guidance screen G103. The button B108 accepts an operation to instruct the display of the app UI 430A to be returned to the guidance screen G101. Note that the button B108 may also accept an operation to instruct the display of the app UI 430A to be returned to the guidance screen G102 and to redisplay the guidance screen G102.

[0118] Fig. 12 is a diagram showing an example of guide screens G181, G182, and G183, and their transitions. In Fig. 12, Fig. 12(A) shows guide screen G181, Fig. 12(B) shows guide screen G182, and Fig. 12(C) shows guide screen G183.

[0119] The guide screens G181, G182, and G183 are screens that are displayed on the application UI430A after the automatic check function is executed. When the automatic check function determines that the position of the marker MK relative to the imaging device 3 is appropriate, a guide screen G181 is displayed on the touch panel 42. The guide screen G181 is displayed as a pop-up, for example, superimposed on the guide screen G104 or another guide screen G. The guide screen G181 includes a message M181 and a button B181. The message M181 includes a character string or an image that notifies the user P that the automatic check function has been completed normally. In other words, the message M181 notifies the user P that it has been confirmed that the position of the marker MK is appropriate.

[0120] The button B181 is an operation unit that accepts an operation by the user P indicating that the user P has understood the contents of the message M. When the button B181 is operated, the display of the app UI 430A transitions to the next guidance screen G. For example, when the guidance screen G181 is displayed after the marker MK is placed in the refrigerator compartment 21 and the button B181 is operated, the guidance screen G that guides the user to place the marker MK in the freezer compartment 24 or the vegetable compartment 25 is displayed on the app UI 430A.

[0121] The guidance screens G182 and G183 are displayed when the automatic check function determines that the position of the marker MK is not appropriate. The guidance screen G182 includes a message M182 and buttons B182 and B183. The message M182 includes a character string that prompts the user P to check the attachment position and attachment state of the marker MK and to check the installation position of the photographing device 3. The button B182 is an operation unit that accepts an operation to instruct the user P to move from the guidance screen G182 to the next guidance screen G. When the button B182 is operated, the display of the application UI 430A moves to the guidance screen G that guides the user to install the marker MK. For example, when the button B182 is operated, the display of the application UI 430A moves from the guidance screen G182 to the guidance screen G102. Note that when the button B182 is operated, the display of the application UI 430A may move, for example, from the guidance screen G182 to the guidance screen G101. The button B183 accepts an operation to instruct the user P to omit the automatic check. When the button B183 is operated, the display of the application UI 430A transitions to a guide screen G183.

[0122] The guidance screen G183 includes a message M183 and buttons B184 and B185. The message M183 includes a character string that explains to the user P about omitting the automatic check. The buttons B184 and B185 accept an operation by the user P to instruct whether or not to accept the contents of the message M183. When the button B184 is operated, the display of the application UI430A returns to the guidance screen G182. Note that when the button B184 is operated, the display of the application UI430A may shift to the guidance screen G102. When the button B185 is operated, the display of the application UI430A shifts to a guidance screen G (for example, a guidance screen G111 described later) that provides guidance on placing the next marker MK.

[0123] Fig. 13 is a diagram showing an example of guidance screens G111 to G114 and their transitions. In Fig. 13, Fig. 13(A) shows guidance screen G111, Fig. 13(B) shows guidance screen G112, Fig. 13(C) shows guidance screen G113, and Fig. 13(D) shows guidance screen G114.

[0124] The guidance screens G111, G112, G113, and G114 constitute a combination of one set of guidance screens G that guides the user in placing the marker MK in the vegetable compartment 25.

[0125] The guidance screen G111 includes a message M111 and a button B111. The message M111 includes a character string and an image that explain the outline of the procedure for attaching the marker MK to the vegetable compartment 25. The button B111 is an operation unit that accepts an operation by the user P, and is a button that instructs transition to the guidance screen G. When the button B111 is operated, the display of the application UI 430A transitions to the guidance screen G112.

[0126] The guidance screen G112 is displayed as a pop-up so as to overlap the grayed-out guidance screen G111, as shown in FIG. 13, for example. The guidance screen G112 includes a message M112 and buttons B112, B113, and B114. The message M112 includes a character string or an image that provides information on the packaging state, appearance, and orientation when attached of the third marker MK51 and the fourth marker MK52 installed in the vegetable compartment 25. The buttons B112, B113, and B114 are operation units that accept operations by the user P. The button B112 accepts an operation to instruct the display of the app UI430A to be returned to the guidance screen G111. The button B112 may also accept an operation to instruct the display of the app UI430A to be displayed again as the guidance screen G112. In this case, the page reload of the guidance screen G112 is executed by operating the button B112, and the guidance screen G112 is displayed again on the app UI430. The button B113 accepts an operation to instruct transition from the guidance screen G112 to the next guidance screen G. The button B114, like the button B104 (FIG. 11), accepts an operation to instruct start of an automatic check function. That is, the automatic check function executed in response to the operation of the button B114 is the same as the function executed in response to the operation of the button B104.

[0127] When the button B113 is operated, the display of the application UI 430A transitions to a guidance screen G113. The guidance screen G113 includes a storage room image GP111, a marker image MP111, a message M113, and buttons B114, B115, and B116.

[0128] The message M113 is a character string that prompts the user P to open the drawer 25A and place the third marker MK51, and is placed at the top of the guidance screen G113.

[0129] The storage room image GP111 is an image showing the position where the third marker MK51 is placed, and is an image showing the drawer 25A as a movable part. The storage room image GP111 is an image showing the placement position of the third marker MK51 on an image showing the drawer 25A and the inside of the vegetable room 25 with the drawer 25A open. The image of the vegetable room 25 included in the storage room image GP111 may be an image photographed in advance by the photographing device 3, or may be an image prepared in advance to match the shape of the drawer 25A.

[0130] The marker image MP111 is an image of the third marker MK51. The marker image MP111 is an image that allows the appearance of the third marker MK51, that is, the shape and pattern of the third marker MK51 to be identified, and is, for example, a photographed image of the third marker MK51. It is preferable that the marker image MP111 is an image showing a state in which the third marker MK51 is placed in the drawer 25A. In this case, the user P can know the placement position and placement state of the third marker MK51 more specifically.

[0131] The button B114 is also used for the guidance screen G112. The buttons B115 and B116 are operation units that accept operations by the user P. The button B115 accepts an operation to instruct the display of the application UI 430A to return to the guidance screen G112. The button B116 accepts an operation to instruct the transition from the guidance screen G113 to the next guidance screen G.

[0132] When the button B116 on the guidance screen G113 is operated, the display of the application UI 430A transitions to a guidance screen G114. The guidance screen G114 includes a storage room image GP112, a marker image MP112, a message M113, and buttons B114, B117, and B118.

[0133] The message M113 is a character string that prompts the user P to open the storage case 25B and place the fourth marker MK52, and is placed at the top of the guidance screen G114.

[0134] The storage room image GP112 is an image showing the position where the fourth marker MK52 is placed, and is an image showing the storage case 25B as a movable part. The storage room image GP112 is an image showing the placement position of the fourth marker MK52 in an image showing the storage case 25B and the inside of the vegetable compartment 25 with the storage case 25B open. The image of the vegetable compartment 25 included in the storage room image GP112 may be an image photographed in advance by the photographing device 3, or may be an image prepared in advance to match the shape of the storage case 25B.

[0135] The marker image MP112 is an image of the fourth marker MK52. The marker image MP112 is an image that allows the appearance of the fourth marker MK52, that is, the shape and pattern of the fourth marker MK52 to be identified, and is, for example, a photographed image of the fourth marker MK52. It is preferable that the marker image MP112 is an image that shows a state in which the fourth marker MK52 is placed in the storage case 25B. In this case, the user P can know the placement position and placement state of the fourth marker MK52 more specifically.

[0136] The button B114 is common to the guidance screen G114. The buttons B117 and B118 are operation units that accept operations by the user P. The button B117 accepts an operation to instruct the display of the app UI 430A to be returned to the guidance screen G113. The button B118 accepts an operation to instruct the display of the app UI 430A to be returned to the guidance screen G111. Note that the button B118 may accept an operation to instruct the display of the app UI 430A to be returned to the guidance screen G112 and to redisplay the guidance screen G112.

[0137] When the button B114 is operated while the guidance screens G112, G113, and G114 are displayed, an automatic check function is executed. Depending on the result of the determination in this automatic check function, the guidance screen G181 or the guidance screen G182 shown in FIG. 12 is displayed.

[0138] Fig. 14 is a diagram showing an example of guidance screens G121 to G124 and their transitions. In Fig. 14, Fig. 14(A) shows guidance screen G121, Fig. 14(B) shows guidance screen G122, Fig. 14(C) shows guidance screen G123, and Fig. 14(D) shows guidance screen G124.

[0139] The guidance screens G121, G122, G123, and G124 constitute a combination of one set of guidance screens G that guides the user in placing the marker MK in the freezer compartment 24.

[0140] The guidance screen G121 includes a message M121 and a button B121. The message M121 includes a character string and an image that explain the outline of the procedure for attaching the marker MK to the freezer compartment 24. The button B121 is an operation unit that accepts an operation by the user P, and is a button that instructs transition to the guidance screen G. When the button B121 is operated, the display of the application UI 430A transitions to the guidance screen G122.

[0141] The guidance screen G122 is displayed as a pop-up so as to overlap the grayed-out guidance screen G121, for example, as shown in FIG. 14. The guidance screen G122 includes a message M122 and buttons B122, B123, and B124. The message M122 includes a character string or an image that provides information on the packaging state, appearance, and orientation when attached of the first marker MK41 and the second marker MK42 installed in the freezer compartment 24. The buttons B122, B123, and B124 are operation units that accept operations by the user P. The button B122 accepts an operation to instruct the display of the application UI430A to be returned to the guidance screen G121. The button B122 may also accept an operation to instruct the display of the application UI430A to be displayed again as the guidance screen G122. In this case, the page reload of the guidance screen G122 is executed by operating the button B122, and the guidance screen G122 is displayed again on the application UI430. The button B123 accepts an operation to instruct transition from the guidance screen G122 to the next guidance screen G. The button B124, like the button B104 (FIG. 11), accepts an operation to instruct start of an automatic check function. That is, the automatic check function executed in response to the operation of the button B124 is the same as the function executed in response to the operation of the button B104.

[0142] When the button B123 is operated, the display of the application UI 430A transitions to a guidance screen G123. The guidance screen G123 includes a storage room image GP121, a marker image MP121, and buttons B124, B125, and B126.

[0143] The storage room image GP121 is an image showing the position where the first marker MK41 is placed, and is an image showing the drawer 24A as a movable part. The storage room image GP121 is an image showing the placement position of the first marker MK41 on an image showing the drawer 24A and the inside of the freezer room 24 with the drawer 24A open. The image of the freezer room 24 included in the storage room image GP121 may be an image photographed in advance by the photographing device 3, or may be an image prepared in advance to match the shape of the drawer 24A.

[0144] The marker image MP121 is an image of the first marker MK41. The marker image MP121 is an image that allows the appearance of the first marker MK41, that is, the shape and pattern of the first marker MK41 to be identified, and is, for example, a photographed image of the first marker MK41. It is preferable that the marker image MP121 is an image that shows the state in which the first marker MK41 is placed in the drawer 24A. In this case, the user P can know the placement position and placement state of the first marker MK41 more specifically.

[0145] The button B124 is also used for the guidance screen G122. The buttons B125 and B126 are operation units that accept operations by the user P. The button B125 accepts an operation to instruct the display of the application UI 430A to return to the guidance screen G122. The button B126 accepts an operation to instruct the transition from the guidance screen G123 to the next guidance screen G.

[0146] When the button B126 on the guidance screen G123 is operated, the display of the application UI 430A transitions to a guidance screen G124. The guidance screen G124 includes a storage room image GP122, a marker image MP122, and buttons B124, B127, and B128.

[0147] The storage room image GP122 is an image showing the position where the second marker MK42 is placed, and is an image showing the storage case 24B as a movable part. The storage room image GP122 is an image showing the placement position of the second marker MK42 on an image showing the storage case 24B and the inside of the freezer room 24 with the storage case 24B open. The image of the freezer room 24 included in the storage room image GP122 may be an image photographed in advance by the photographing device 3, or may be an image prepared in advance to match the shape of the storage case 24B.

[0148] The marker image MP122 is an image of the second marker MK42. The marker image MP122 is an image that allows the appearance of the second marker MK42, i.e., the shape and pattern of the second marker MK42, to be identified, and is, for example, a photographed image of the second marker MK42. It is preferable that the marker image MP122 is an image that shows the state in which the second marker MK42 is placed in the storage case 24B. In this case, the user P can know the placement position and placement state of the second marker MK42 more specifically.

[0149] The button B124 is common to the guidance screen G124. The buttons B127 and B128 are operation units that accept operations by the user P. The button B127 accepts an operation to instruct the display of the app UI 430A to be returned to the guidance screen G123. The button B128 accepts an operation to instruct the display of the app UI 430A to be returned to the guidance screen G121. Note that the button B128 may accept an operation to instruct the display of the app UI 430A to be returned to the guidance screen G122 and to redisplay the guidance screen G122.

[0150] When the button B124 is operated while the guidance screens G122, G123, and G124 are displayed, an automatic check function is executed. Depending on the result of the determination in this automatic check function, the guidance screen G181 or the guidance screen G182 shown in FIG. 12 is displayed.

[0151] Fig. 15 is a diagram showing an example of guidance screens G131 to G135 and their transitions. In Fig. 15, Fig. 15(A) shows guidance screen G131, Fig. 15(B) shows guidance screen G132, Fig. 15(C) shows guidance screen G133, Fig. 15(D) shows guidance screen G134, and Fig. 15(E) shows guidance screen G135.

[0152] The guidance screens G131, G132, G133, G134, and G135 are screens that are displayed after the markers MK have been placed in all storage rooms 26 that are targets for placing the markers MK. That is, these are screens for the user P to check the placement status of the markers MK after the user P places the markers MK in accordance with the guidance screens G101-104, G111-114, and G121-G124.

[0153] The guidance screen G131 includes a message M131 and a button B131. The message M131 includes a character string and an image explaining that the markers MK placed in the refrigerator compartment 21, the freezer compartment 24, and the storage compartment 26 are photographed by the photographing device 3. The button B131 is an operation unit that accepts an operation by the user P, and is a button that instructs transition to the guidance screen G. When the button B131 is operated, the display of the app UI 430A transitions to the guidance screen G132.

[0154] The guidance screen G132 is displayed as a pop-up so as to overlap the grayed-out guidance screen G131, for example, as shown in FIG. The guidance screen G122 includes a message M132 and buttons B132, B133, and B134. The message M132 includes a character string and an image that explains the work to be performed by the user P to photograph the refrigerator compartment 21. Specifically, the message M132 guides the user P to open the door 21A to photograph the refrigerator compartment 21.

[0155] The buttons B132, B133, and B134 are operation units that accept operations by the user P. The button B132 accepts an operation to instruct the display of the app UI 430A to be returned to the guidance screen G131. The button B132 may also accept an operation to instruct the display of the app UI 430A to be displayed again as the guidance screen G132. In this case, the operation of the button B132 causes a page reload of the guidance screen G132 to be executed, and the guidance screen G132 is displayed again on the app UI 430. The button B134 accepts an operation to instruct the transition from the guidance screen G132 to the next guidance screen G. The button B133 accepts an operation to instruct the transition to the guidance screen G135. The button B133 is commonly arranged on the guidance screen G133 and the guidance screen G134.

[0156] When the button B134 is operated, the display of the app UI 430A switches to the guidance screen G133. The guidance screen G133 includes a message M133 and buttons B133, B135, and B136. The message M133 includes a character string and an image explaining the work to be performed by the user P to photograph the vegetable compartment 25. The buttons B135 and B136 are operation units that accept operations by the user P. The button B135 accepts an operation to instruct the display of the app UI 430A to return to the guidance screen G132. The button B136 accepts an operation to instruct the transition from the guidance screen G133 to the next guidance screen G.

[0157] When the button B136 is operated, the display of the app UI 430A switches to the guidance screen G134. The guidance screen G134 includes a message M134 and buttons B133, B137, and B138. The message M134 includes a character string and an image that explain the work that the user P will perform to photograph the freezer compartment 24. The buttons B137 and B138 are operation units that accept operations by the user P. The button B137 accepts an operation to instruct the display of the app UI 430A to return to the guidance screen G133. The button B138 accepts an operation to instruct the display of the app UI 430A to return to the guidance screen G131.

[0158] When the button B133 is operated while any of the guidance screens G132, G133, and G134 is displayed, the display of the application UI 430A is switched to the guidance screen G135. The guidance screen G135 includes a storage room image GP131 and buttons B141, B142, B143, and B144. The storage room image GP131 is a photographed image taken by the photographing device 3. For example, when the button B133 or the button B134 is operated while the guidance screen G132 is displayed, the content of this operation is transmitted from the terminal device 4 to the photographing device 3 via the food management server 5, and the photographing device 3 performs photographing. The photographed image taken by the photographing device 3 is transmitted to the food management server 5 and stored in the food management server 5 in association with the photographing device 3. The food management server 5 transmits the photographed image of the photographing device 3 to the terminal device 4, and causes the photographed image to be displayed on the touch panel 42 at the timing of displaying the guidance screen G135. Similarly, when the button B133 or the button B136 is operated while the guidance screen G133 is displayed, and when the button B133 or the button B138 is operated while the guidance screen G134 is displayed, the imaging device 3 performs imaging.

[0159] Therefore, the photographing device 3 photographs the refrigerator compartment 21 when the guidance screen G132 is displayed on the touch panel 42, photographs the vegetable compartment 25 when the guidance screen G133 is displayed, and photographs the freezer compartment 24 when the guidance screen G134 is displayed. When photographing the refrigerator compartment 21, the photographing device 3 performs photographing using the wide-angle camera 301 and the narrow-angle camera 302, and transmits the photographed image to the food management server 5. The image photographed by the wide-angle camera 301 is an image of the range in which the right door hinge member 204 and the left door hinge member 205 move. The image photographed by the narrow-angle camera 302 is an image of the refrigerator compartment 21 viewed from above the front. Note that photographing by the photographing device 3 may be performed when the guidance screens G132, G133, and G134 are displayed.

[0160] The user P can switch between the images of the refrigerator compartment 21 captured by the wide-angle camera 301, the image of the refrigerator compartment 21 captured by the narrow-angle camera 302, the image of the freezer compartment 24, and the image of the vegetable compartment 25 on the guidance screen G135. The buttons B141, B142, B143, and B144 receive an operation to instruct switching of the storage compartment image GP131. When the button B141 is operated, the image of the refrigerator compartment 21 captured by the wide-angle camera 301 is displayed as the storage compartment image GP131. When the button B142 is operated, the image of the refrigerator compartment 21 captured by the narrow-angle camera 302 is displayed as the storage compartment image GP131. When the button B143 is operated, the storage compartment image GP131 is switched to the image of the vegetable compartment 25, and when the button B144 is operated, the storage compartment image GP131 is switched to the image of the freezer compartment 24.

[0161] The guidance screen G135 displays an image captured by the photographing device 3 after the marker MK is placed, thereby enabling the user P to check the installation state of the marker MK that he or she has placed. The guidance screen G135 is an example of a confirmation screen. For example, if the user P determines that the installation state of the marker MK is not appropriate after looking at the confirmation screen, the user P can redo the installation of the marker MK. Note that, if an image cannot be displayed on the guidance screen G135, the user P may be notified that the photographing device 3 has not been able to capture the image.

[0162] In this manner, the app UI 430A switches between and displays guidance screens G101-G104, guidance screens G111-G114, and guidance screens G121-G124 that provide guidance on placing the markers MK in the storage compartments 26. The number of these guidance screens G corresponds to the number of storage compartments 26 that the refrigerator 2 has.

[0163] The guidance screens G101 to G104 which guide the user P on the installation of the marker MK in the refrigerator compartment 21 include storage compartment images GP101 and GP102 corresponding to the left and right doors 21A which are movable parts. Similarly, the guidance screens G111 to G114 include storage compartment images GP111 and GP112 which respectively correspond to the drawer 25A and the storage case 25B which are movable parts of the vegetable compartment 25. The guidance screens G121 to G124 include storage compartment images GP121 and GP122 which respectively correspond to the drawer 24A and the storage case 24B which are movable parts of the freezer compartment 24. Therefore, the storage compartment images GP which correspond to the number of movable parts can guide the user P on the installation of the marker MK to each movable part.

[0164] In the examples shown in Figs. 11 to 15, the guide screens G101 to G104 for guiding the placement of the refrigerator compartment marker MK1 are displayed before the guide screens G111 to G114 and G121 to G124 for guiding the placement of the marker MK in the freezer compartment 24 and the vegetable compartment 25. That is, the user P is guided to place the marker MK in the refrigerator compartment 21 first among the refrigerator compartment 21 and the multiple storage compartments 26. As described above, the refrigerator compartment marker MK1 is placed in at least one of the right door hinge member 204 and the left door hinge member 205, and therefore the placement position of the refrigerator compartment marker MK1 is determined with high accuracy. For example, the possibility that the user P places the refrigerator compartment marker MK1 in a position significantly deviated from the right door hinge member 204 is very low.

[0165] For this reason, the refrigerator compartment marker MK1 can be used as a reference for the installation position of the photographing device 3. Specifically, it is possible to determine whether the installation position of the photographing device 3 with respect to the refrigerator 2 is appropriate or not, based on whether the position of the refrigerator compartment marker MK1 is appropriate within the shooting ranges of the wide-angle camera 301 and the narrow-angle camera 302. Therefore, by guiding the user P to first install the refrigerator compartment marker MK1 for the refrigerator compartment 21 among multiple storage compartments 26 including the refrigerator compartment 21, there is an advantage that the appropriateness of the installation position of the photographing device 3 can be determined before installing the other markers MK.

[0166] The storage room image GP corresponds to an example of a "movable part image." The combination of the storage room image GP and the marker image MP corresponds to an example of a "guide image." For example, the combination of the storage room image GP101 and the marker image MP101, the combination of the storage room image GP102 and the marker image MP102, the combination of the storage room image GP111 and the marker image MP1112, etc. all correspond to examples of the "guide image."

[0167] The automatic check function is a function that should be executed after the user P places the marker MK. Therefore, when the user P operates the button B104, it means that the user P has completed placing the marker MK. Therefore, the operation of the button B104 corresponds to the user P inputting that the placement of the marker MK has been completed, and the button B104 corresponds to an example of an "input section."

[0168] [1-7. Operation] The operation of the photographing system 7 will now be described. Fig. 16 is a sequence diagram showing an example of the operation of the photography system 7, and shows the operation related to the output of the application UI 430. In Fig. 16, a flowchart FA shows the operation of the photography device 3, and steps SA1 to SA4 are executed by the first processor 320 and the second processor 350. A flowchart FB shows the operation of the terminal device 4, and steps SB1 to SB6 are executed by the terminal processor 400. A flowchart FC shows the operation of the food management server 5, and steps SC1 to SC4 are executed by the server processor 500.

[0169] Prior to the operation, communication settings are made in the photographing device 3 (step SA1). The communication settings are settings for the photographing device 3 to communicate with the food management server 5 via the communication device 6, and include, for example, settings for wireless communication between the photographing device 3 and the communication device 6. After the communication settings are made in step SA1, the photographing device 3 becomes capable of communicating with the food management server 5.

[0170] The terminal device 4 accepts input of device information (step SB1). The device information is information necessary for the food management server 5 to identify the configuration of the refrigerator 2, and specifically, is information about the refrigerator 2 that is the target of the installation work of the marker MK. For example, the device information is information indicating the model number of the refrigerator 2, the type of refrigerator 2, a supporting symbol or number given to the refrigerator 2, the number of movable parts of the refrigerator 2, etc. The terminal device 4 transmits the device information accepted in step SB1 to the food management server 5 (step SB2).

[0171] The food management server 5 receives and acquires the device information transmitted by the terminal device 4 (step SC1). The food management server 5 generates configuration information based on the acquired device information and transmits it to the terminal device 4 and the image capturing device 3 (step SC2). The configuration information includes information for identifying the shape of the movable parts of the refrigerator 2, the number of movable parts, the number of markers MK to be installed on the refrigerator 2, the movable parts on which the markers MK are installed, and the appearance of each marker MK. In this embodiment, the configuration information includes, for example, information indicating that the movable parts of the refrigerator 2 are the door 21A, the drawers 22A, 23A, 24A, 25A, and the storage cases 24B and 25B, and information indicating that the markers MK are installed on the door 21A, the drawers 24A, 25A, and the storage cases 24B and 25B. In addition, for example, the configuration information includes information indicating that the refrigerator compartment marker MK1 is installed on the right door hinge member 204 and the left door hinge member 205, and information specifying the type or appearance of each marker MK. The configuration information corresponds to an example of "storage information".

[0172] The photographing device 3 acquires the configuration information transmitted by the food management server 5 (step SA2). The terminal device 4 receives the configuration information transmitted by the food management server 5 and stores it in the terminal memory 410 (step SB3). The terminal device 4 generates the application UI 430 according to the received configuration information (step SB4). In step SB4, the terminal device 4 generates data of the guide screen G to be displayed on the application UI 430 according to the contents of the configuration information. For example, the terminal device 4 refers to the configuration information and generates data of the guide screen G so as to correspond to information such as the shape of the movable parts of the refrigerator 2, the number of movable parts, and the movable parts on which the markers MK are to be placed. The data of the guide screen G may be data corresponding to a specific model of the refrigerator 2, or general-purpose data that is not limited to the corresponding refrigerator 2 may be used. Moreover, the data of the guide screen G generated in step SB4 may not include the storage room image GP. In step SB4, the photographing system 7 can display the guide screen G suitable for the refrigerator 2 on which the user P is to perform work by generating the guide screen G displayed on the application UI 430 in accordance with the configuration information.

[0173] The terminal device 4 executes a guidance process using the application UI 430 generated in step SB4 (step SB5). The guidance process will be described in detail later with reference to FIG. On the other hand, after receiving the configuration information, the photographing device 3 executes a marker detection process (step SA3). As a result, the marker detection process of the photographing device 3 is executed in parallel with the guidance process of the terminal device 4. The details of the marker detection process will be described later with reference to FIG.

[0174] After completing the notification process in step SB5, the terminal device 4 notifies the food management server 5 of the completion of the process (step SB6).

[0175] When the food management server 5 receives a notification of completion of processing from the terminal device 4 (step SC3), it instructs the photographing device 3 to end marker detection (step SC4). The photographing device 3 determines whether or not an instruction to end marker detection has been received from the food management server 5 (step SA4), and if the instruction to end marker detection has not been received (step SA4: NO), it continues the processing of step SA3. If the instruction to end marker detection has been received (step SA4: YES), the photographing device 3 ends this processing.

[0176] 17 is a flowchart showing an example of the operation of the photographing device 3, and shows the marker detection process of step SA3 (FIG. 16) in detail. Steps SD1 to SD8 of the flowchart FD are executed by the first processor 320 and the second processor 350.

[0177] The image capturing device 3 acquires at least one of an image captured by the wide-angle camera 301 and an image captured by the narrow-angle camera 302 (step SD1). In step SD1, the photographing device 3 may acquire at least one of the first recording data DV1 and the second recording data DV2 stored in the second memory 360. Also, the photographing device 3 may acquire first photographed data 363A generated from the first recording data DV1 and second photographed data 363B generated from the second recording data DV2. In step SD1, when the photographing device 3 acquires an image stored in the second memory 360 in step SD1, the photographing device 3 acquires the image with the most recent photographed date and time.

[0178] The photographing device 3 determines whether or not the image of the marker MK is included in the photographed image acquired in step SD1 (step SD2). If the image of the marker MK is not included in the photographed image (step SD2: NO), the photographing device 3 returns to step SD1. If the image of the marker MK is included in the photographed image (step SD2: YES), the photographing device 3 determines whether or not the position of the image of the marker MK in the photographed image is within a preset appropriate range (step SD3). If the position of the image of the marker MK in the photographed image is within the appropriate range (step SD3: YES), the photographing device 3 proceeds to step SD8 described below.

[0179] If the position of the image of the marker MK in the photographed image is not within the appropriate range (step SD3: NO), the photographing device 3 performs a misalignment correction process on the photographed image so that the position of the image of the marker MK falls within the appropriate range (step SD4). Here, the photographing device 3 determines whether the correction amount in the misalignment correction process exceeds the correction allowable amount (step SD5). For example, in the misalignment correction process, the photographing device 3 trims the photographed image so that the image of the marker MK is located within the appropriate range in the photographed image. In this case, the distance between the center of the trimmed image and the center of the photographed image can be referred to as the correction amount. For example, the correction allowable amount is a threshold value of the correction amount at which the trimmed image becomes an image that is not suitable for detecting ingredients.

[0180] If the correction amount in the misalignment correction process does not exceed the correction allowable amount (step SD5: NO), the photographing device 3 proceeds to step SD8, which will be described later.

[0181] If the correction amount in the deviation correction process exceeds the correction allowable amount (step SD5: YES), the photographing device 3 issues a notification regarding the position of the marker MK (step SD6). The notification method is, for example, outputting a warning sound by the buzzer 306.

[0182] Furthermore, the photographing device 3 transmits an installation position adjustment request to the food management server 5 (step SD7), and returns to step SD1. Although not shown in the sequence diagram of FIG. 16, the installation position adjustment request transmitted by the photographing device 3 is received by the food management server 5, and is further transmitted from the food management server 5 to the terminal device 4. When the installation position adjustment request is transmitted to the terminal device 4, the application UI 430 displayed on the touch panel 42 displays a display requesting the user P to check the installation position of the photographing device 3 or that the position of the marker MK is not appropriate. For example, a guide screen G182 (FIG. 12) of the application UI 430A is displayed on the touch panel 42.

[0183] In step SD8, the photographing device 3 transmits confirmation information to the food management server 5 (step SD8). Although not shown in the sequence diagram of FIG. 16, the confirmation information transmitted by the photographing device 3 is received by the food management server 5, and is further transmitted from the food management server 5 to the terminal device 4. By transmitting the confirmation information to the terminal device 4, the application UI 430 displayed on the touch panel 42 displays a display notifying the user P that the position of the marker MK is appropriate. For example, the guide screen G181 (FIG. 12) of the application UI 430A is displayed on the touch panel 42. In step SD8, the photographing device 3 may notify the user P by the buzzer 306. The sound emitted by the buzzer 306 in step SD8 is a sound notifying the user P that the work of the user P is appropriate, and can be called, for example, a confirmation sound. The above-mentioned caution sound and the confirmation sound can be sounds having different ringing patterns of the buzzer 306. For example, the caution sound is preferably a sound that reminds the user P that some kind of error has occurred.

[0184] After transmitting the confirmation information, the photographing device 3 ends this process. Here, the photographing device 3 may return to step SD1. Furthermore, when an instruction to take a photograph is given by the application UI 430 being displayed on the terminal device 4, the photographing device 3 may perform photographing using at least one of the wide-angle camera 301 and the narrow-angle camera 302, and transmit the photographed image data to the food management server 5. For example, while the touch panel 42 is displaying one of the guide screens G132 to G134 of the application UI 430A, the photographing device 3 is instructed to take a photograph in response to the operation of the button B133.

[0185] 18 is a flowchart showing an example of the operation of the terminal device 4, and shows the guidance process of step SB5 (FIG. 16) in detail. Steps SE1 to SE10 of the flowchart FE are executed by the terminal processor 400.

[0186] The terminal device 4 displays the guidance screen G related to the refrigerator compartment 21 on the touch panel 42 (step SE1). While the guidance screen G is displayed, the terminal device 4 determines whether or not there is a switching instruction for the displayed guidance screen G (step SE2). The switching instruction is, for example, an operation of various buttons arranged on the guidance screen G. When there is no switching instruction (step SE2: NO), the terminal device 4 executes the determination of step SE2 at a predetermined time interval. Here, when a predetermined time has elapsed with no switching instruction, the terminal device 4 may transition the display of the app UI 430 to the next guidance screen G.

[0187] When the switching instruction is issued (step SE2: YES), the terminal device 4 judges whether or not an installation position adjustment request has been received from the food management server 5 (step SE3). The installation position adjustment request is a request that the photographing device 3 transmits to the food management server 5 in step SD8 (FIG. 17), and is transmitted to the terminal device 4 via the food management server 5. When the photographing device 3 transmits the installation position adjustment request, the relative position of the photographing device 3 with respect to the refrigerator compartment marker MK1 is not appropriate, so it is necessary to redo the installation of the photographing device 3 or the refrigerator compartment marker MK1. Therefore, when the terminal device 4 receives the installation position adjustment request (step SE3: YES), it guides the user P to redo the installation (step SE10) and ends this process. In step SE10, the terminal device 4 may execute, for example, output of a screen that guides the user P to install the photographing device 3, such as the guidance screen G182 in FIG. 12.

[0188] If an installation position adjustment request has not been received (step SE3: NO), the terminal device 4 switches the display of the application UI 430 to the next guidance screen G (step SE4). After switching and displaying the guidance screen G, the terminal device 4 determines whether or not there is a switching instruction for the guidance screen G (step SE5). The switching instruction is, for example, an operation of various buttons arranged on the guidance screen G. In a state where there is no switching instruction (step SE5: NO), the terminal device 4 executes the determination of step SE5 at a predetermined time interval.

[0189] If a switching instruction has been issued (step SE5: YES), the terminal device 4 determines whether or not an installation value adjustment request transmitted from the photographing device 3 has been received (step SE6).

[0190] When the terminal device 4 receives the installation position adjustment request (step SE6: YES), the terminal device 4 proceeds to step SE10. If the installation position adjustment request has not been received (step SE6: NO), the terminal device 4 judges whether or not the process for all storage compartments has been completed (step SE7). "All storage compartments" refers to storage compartments in the refrigerator 2 that are targets for placing markers MK, and in this embodiment, these are the refrigerator compartment 21, the freezer compartment 24, and the vegetable compartment 25. The judgment in step SE7 may be replaced with a judgment as to whether or not the process for all installation guide screens SG has been completed.

[0191] If the process for all the storage compartments is not completed (step SE7: NO), the terminal device 4 returns to step SE4 and switches the display to the next installation guide screen SG. If the process for all the storage compartments is completed (step SE7: YES), the terminal device 4 acquires still image data captured by the photographing device 3 (step SE8). The still image data acquired in step SE8 is, for example, data of a captured image captured in response to the operation of the button B133 while the guidance screens G132 to G134 are displayed. This data includes, for example, a captured image of the refrigerator compartment 21 with the door 21A open, a captured image of the freezer compartment 24 with the storage case 24B pulled out, and a captured image of the freezer compartment 24 with the drawer 24A pulled out. The data also includes a captured image of the vegetable compartment 25 with the storage case 25B pulled out, and a captured image of the vegetable compartment 25 with the drawer 25A pulled out.

[0192] The terminal device 4 generates a confirmation screen using the still image data acquired in step SE8, and displays it on the touch panel 42 (step SE9). The confirmation screen is, for example, the guide screen G135 (FIG. 15). After that, the terminal device 4 ends this process.

[0193] In the operations shown in Figs. 16-18, the terminal device 4 may be configured to provide a general-purpose app UI 430 in addition to the app UI 430 corresponding to the configuration information. This configuration is applicable, for example, in the operations of Fig. 16, when the food management server 5 cannot generate configuration information corresponding to the device information in step SC1, and the number of storage compartments and the configuration of the movable parts of the refrigerator 2 to which the marker MK is to be placed are unknown. In this case, the terminal device 4 uses the app UI 430 corresponding to the configuration of a typical refrigerator. For example, a typical refrigerator is a refrigerator having a refrigerator compartment, a freezer compartment, and a vegetable compartment, and having a refrigerator compartment door configured as a single door or a French door. The terminal device 4 displays a guide screen G for placing the marker MK in each storage compartment of the typical refrigerator on the touch panel 42. In this case, the terminal device 4 can support the user P in placing the marker MK in a refrigerator whose configuration is unknown. For example, the data of the general-purpose guide screen G may be stored in the terminal memory 410 in advance by the terminal device 4, or the food management server 5 may transmit the data of the general-purpose guide screen G to the terminal device 4 instead of the configuration information in step SC2.

[0194] The timing of performing the operations shown in Figs. 16 to 18 is not limited to when the marker MK is placed. These operations can properly position the marker MK in the shooting range of the camera 3, so it is also useful to perform them after the marker MK is placed. Since the movable part of the refrigerator 2 is opened and closed on a daily basis, the marker MK may move little by little as the movable part is opened and closed. For this reason, for example, after the marker MK is placed on the refrigerator 2, the terminal device 4 may notify the user P to check the position of the marker MK every time a predetermined period of time has elapsed. In this case, the application UI430 may be displayed by the terminal device 4 when the user P performs an input operation in response to the notification. Specifically, the start guide screen SG1 and the guide screen G may be displayed in this order to prompt the user P to check the position of the marker MK.

[0195] [1-8. Another example of app UI configuration] 19 to 23, a description will be given of an application UI 430B which is a second configuration example of the application UI 430. The application UI 430B is a user interface displayed on the photography system 7, similar to the application UI 430A.

[0196] In each of Fig. 19 to Fig. 23, the upward direction in the figure is indicated by the symbol "UP", the downward direction in the figure is indicated by the symbol "DW", the leftward direction in the figure is indicated by the symbol "L", and the rightward direction in the figure is indicated by the symbol "R". The up-down direction in Fig. 19 to Fig. 23 is a direction approximately parallel to the side of the display surface of the touch panel 42 when the display surface of the touch panel 42 is viewed. Also, the left-right direction in Fig. 19 to Fig. 23 is a direction approximately parallel to the side perpendicular to the side of the display surface of the touch panel 42 when the display surface of the touch panel 42 is viewed.

[0197] The display control unit 402 switchably displays multiple guidance screens G on the app UI 430B. The guidance screens G are screens that guide the user P on the content and procedures of the work of placing a marker MK on the refrigerator 2. The display control unit 402 switches between the multiple guidance screens G in order. When the reception unit 403 receives an input from the user P while the guidance screen G is being displayed, the display control unit 402 switches the guidance screen G in response to the input from the user P.

[0198] The application UI 430B includes a start guidance screen SG1, an installation guidance screen SG2, SG3, SG4, and a confirmation screen SG5 as examples of the guidance screen G. These screens will be described in order. In the following description, the installation guidance screens SG2, SG3, SG4, and the confirmation screen SG5 will be collectively referred to as the installation guidance screen SG when there is no need to distinguish between them. Furthermore, the start guidance screen SG1, the installation guidance screen SG, and the confirmation screen SG5 will be collectively referred to as the guidance screen G.

[0199] The storage room images GP21, GP31, GP32, GP41, and GP42 are other examples of the storage room image GP, and are referred to as the storage room image GP when there is no need to distinguish between them. The marker images MP21, MP31, MP32, MP41, and MP42 are other examples of the marker image MP, and are referred to as the marker image MP when there is no need to distinguish between them. Furthermore, the input units CH31, CH32, CH41, and CH42 are referred to as the input unit CH when there is no need to distinguish between them.

[0200] There are no limitations on the specific aspects of images, input sections, messages, etc., arranged on the start guide screen SG1, installation guide screens SG2, SG3, SG4, and confirmation screen SG5 described below. For example, the size of the image, the shape of the input section and the input method, the font type and size of the characters, and whether each guide screen G is vertically or horizontally long can be appropriately changed depending on the specifications of the touch panel 42 that displays the application UI 430B and the usage mode of the terminal device 4, or other reasons.

[0201] [1-8-1. Configuration of the start guide screen] FIG. 19 is a diagram showing an example of the start guidance screen SG1.

[0202] The start guidance screen SG1 is a screen that notifies the user P that guidance for the installation work of the marker MK will begin in the app UI 430B. The start guidance screen SG1 includes a message M11 and a button B11. The message M11 is a character string that notifies the user P that guidance for installation of the marker MK will begin. The button B11 is an operation unit that instructs switching of the app UI 430B, and when the user P touches the position of the button B11, the app UI 430B is switched to the installation guidance screen SG2 shown in FIG. 20.

[0203] The installation guide screens SG2, SG3, and SG4 are screens that are displayed on the app UI430B when a marker MK is installed in the refrigerator 2. The photography system 7 provides the app UI430B corresponding to the number of doors and drawers of the refrigerator compartment 21, the number of storage chambers 26 as storage chambers included in the refrigerator compartment 21, and the number of markers MK to be installed corresponding to the refrigerator compartments 21 and the storage chambers 26. Specifically, the app UI430B includes installation guide screens SG in the number of storage chambers 26 in which the markers MK are installed plus the refrigerator compartments 21, and the terminal device 4 switches between these installation guide screens SG to display them on the app UI430B.

[0204] [1-8-2. Configuration of the installation guide screen] FIG. 20 is a diagram showing an example of the installation guide screen SG2. The installation guide screen SG2 is a screen that guides the user P to install a refrigerator compartment marker MK1 corresponding to the refrigerator compartment 21. The installation guide screen SG2 includes messages M21, M22, a storage compartment image GP21, a marker image MP21, and buttons B21, B22.

[0205] The message M21 is a character string prompting the user P to open the door 21A and install the refrigerator compartment marker MK1, and is placed at the top of the installation guide screen SG2.

[0206] The storage compartment image GP21 is an image showing the position where the refrigerator compartment marker MK1 is to be installed, and is an image showing the door 21A as a movable part. The storage compartment image GP21 is an image obtained by combining an image showing the door 21A and the inside of the refrigerator compartment 21 with the door 21A open with a position display image GG2 showing the installation position of the refrigerator compartment marker MK1. The image of the refrigerator compartment 21 included in the storage compartment image GP21 may be an image photographed in advance by the photographing device 3, or may be an image prepared in advance to match the shape of the door 21A of the refrigerator 2.

[0207] The marker image MP21 is an image of the refrigerator compartment marker MK1 installed corresponding to the refrigerator compartment 21. The marker image MP21 is an image in which the appearance of the refrigerator compartment marker MK1, i.e., the shape and pattern of the refrigerator compartment marker MK1, can be identified, for example, a photographed image of the refrigerator compartment marker MK1. Furthermore, it is preferable that the marker image MP21 is an image showing a state in which the refrigerator compartment marker MK1 is installed on the right door hinge member 204. The user P can know the installation position and installation state of the refrigerator compartment marker MK1 by the position display image GG2 included in the storage compartment image GP21 and the marker image MP21, and can easily install the refrigerator compartment marker MK1.

[0208] While the installation guide screen SG2 is displayed, the photographing device 3 detects the refrigerator compartment marker MK1 installed in an appropriate position from the image captured by the wide-angle camera 301 or narrow-angle camera 302. When the photographing device 3 detects the refrigerator compartment marker MK1, it outputs a notification sound from the buzzer 306, for example, in step SD8 (FIG. 17). The notification sound from the photographing device 3 indicates that the task of installing the refrigerator compartment marker MK1 has been completed. The message M22 is a character string asking the user P whether or not he or she has confirmed that the photographing device 3 has emitted the notification sound.

[0209] The buttons B21 and B22 are operation units operated by the user P. The button B21 is an operation unit for the user P to input that the photographing device 3 has emitted a notification sound. When the user P touches the position of the button B21, the app UI 430B ends the display of the installation guide screen SG2 and switches the display to the next installation guide screen SG. In other words, the button B21 is an operation unit for both inputting that the user P has confirmed the notification sound of the photographing device 3 and issuing an instruction to switch the app UI 430B. The button B21 corresponds to an example of an "input unit".

[0210] The button B22 is an operation unit for the user P to input that he / she cannot confirm the notification sound of the photographing device 3. When the button B22 is operated, a screen (not shown) explaining the reason why the notification sound is not output is displayed on the application UI 430B. This screen is switched from the installation guide screen SG2, or is temporarily displayed as a pop-up superimposed on the installation guide screen SG2.

[0211] Fig. 21 is a diagram showing an example of the installation guide screen SG3, and Fig. 22 is a diagram showing an example of the installation guide screen SG4. The installation guide screen SG3 is a guide screen G that provides guidance for installation of the first marker MK41 and the second marker MK42 corresponding to the freezer compartment 24, and the installation guide screen SG4 is a guide screen G that provides guidance for installation of the third marker MK51 and the fourth marker MK52 corresponding to the vegetable compartment 25.

[0212] The installation guide screen SG3 includes messages M31, M32, M33, storage room images GP31, GP32, marker images MP31, MP32, and buttons B31, B32. The message M32 is arranged side by side in input section CH31, and the message M33 is arranged side by side in input section CH32. The input sections CH31 and CH32 are check boxes that accept input by touch operation of the user P.

[0213] The message M31 is a character string prompting the user P to open the opening and closing part of the freezer compartment 24 and place the marker MK, and is placed at the top of the installation guide screen SG3.

[0214] The installation guide screen SG3 guides the user in installing a marker MK in each of the drawer 24A and the storage case 24B. The storage room image GP31, the marker image MP31, the message M32, and the input unit CH31 guide the user in installing a second marker MK42 in the storage case 24B. The storage room image GP32, the marker image MP32, the message M33, and the input unit CH32 guide the user in installing a first marker MK41 in the drawer 24A.

[0215] Since the storage case 24B is located above the drawer 24A, it is efficient to first attach the second marker MK42 to the storage case 24B and then attach the first marker MK41 to the drawer 24A when installing the markers MK. Therefore, the storage room image GP31, the marker image MP31, the message M32, and the input section CH31 are arranged at the top of the installation guide screen SG3. And, below them, the storage room image GP32, the marker image MP32, the message M33, and the input section CH32 are arranged.

[0216] That is, the installation guide screen SG3 includes the same number of storage compartment images GP31, GP32 as the number of movable parts of the freezer compartment 24, and the storage compartment images GP31, GP32 are displayed on the installation guide screen SG3 in accordance with the positions of the corresponding movable parts in the refrigerator 2.

[0217] The images of the inside of the freezer compartment 24 included in the storage compartment images GP31 and GP32 may be images captured in advance by the photographing device 3, or may be images prepared in advance to match the shapes of the drawer 24A and storage case 24B in the refrigerator 2.

[0218] The storage room image GP31 is an image of the storage case 24B as a movable part, and indicates the position where the second marker MK42 is placed in the storage case 24B. The storage room image GP31 is an image obtained by combining an image showing the inside of the freezer chamber 24 with the storage case 24B pulled out and a position display image GG31 showing the installation position of the second marker MK42. The storage room image GP32 is an image of the drawer 24A as a movable part, and indicates the position where the first marker MK41 is placed in the drawer 24A. The storage room image GP32 is an image obtained by combining an image showing the inside of the freezer chamber 24 with the drawer 24A pulled out and a position display image GG32 showing the installation position of the first marker MK41.

[0219] The marker image MP31 is an image in which the shape and pattern of the external appearance of the second marker MK42 can be identified, for example, a photographed image of the second marker MK42. The marker image MP32 is an image in which the shape and pattern of the external appearance of the first marker MK41 can be identified, for example, a photographed image of the first marker MK41.

[0220] While the installation guide screen SG3 is displayed, if the photographing device 3 detects the first marker MK41 or the second marker MK42 installed in an appropriate position in the image captured by the wide-angle camera 301 or the narrow-angle camera 302, the photographing device 3 outputs a notification sound via the buzzer 306. The messages M32 and M33 are character strings asking the user P whether or not he or she has confirmed that the camera 3 has emitted a notification sound. When the user P confirms the notification sound of the camera 3 after placing the second marker MK42, the user P makes an input to the input unit CH31, and when the user P confirms the notification sound of the camera 3 after placing the first marker MK41, the user P makes an input to the input unit CH32.

[0221] The buttons B31 and B32 are operation units operated by the user P. The button B31 is a button for instructing switching of the display from the installation guide screen SG3 to the next screen. The button B32 is an operation unit for inputting that the user P cannot confirm the notification sound of the photographing device 3. When the button B32 is operated, the same screen as when the button B22 is operated is displayed on the app UI 430B.

[0222] 22, the installation guide screen SG4 includes messages M41, M42, M43, storage room images GP41, GP42, marker images MP41, MP42, and buttons B41, B42. The message M42 is arranged side by side in input section CH41, and the message M43 is arranged side by side in input section CH42. The input sections CH41, CH42 are check boxes that accept input by touch operation of the user P.

[0223] The message M41 is a character string prompting the user P to open the opening and closing part of the vegetable compartment 25 and place the marker MK, and is placed at the top of the installation guide screen SG4.

[0224] The installation guide screen SG4 guides the user in installing a marker MK in each of the drawer 25A and the storage case 25B. The storage room image GP41, the marker image MP41, the message M42, and the input unit CH41 guide the user in installing a fourth marker MK52 in the storage case 25B. The storage room image GP42, the marker image MP42, the message M43, and the input unit CH42 guide the user in installing a third marker MK51 in the drawer 25A.

[0225] Since the storage case 25B is located above the drawer 25A, the storage room image GP41, the marker image MP41, the message M42, and the input section CH41 are arranged at the top of the installation guide screen SG4. Below them, the storage room image GP42, the marker image MP42, the message M43, and the input section CH42 are arranged.

[0226] The images of the interior of the vegetable compartment 25 included in the storage compartment images GP41 and GP42 may be images captured in advance by the photographing device 3, or may be images prepared in advance to match the shapes of the drawer 25A and storage case 25B in the refrigerator 2.

[0227] The storage room image GP41 is an image of the storage case 25B as a movable part, and shows the position where the fourth marker MK52 is placed in the storage case 25B. The storage room image GP41 is an image obtained by superimposing a position display image GG41 showing the placement position of the fourth marker MK52 on an image showing the inside of the vegetable room 25 when the storage case 25B is pulled out. The storage room image GP42 is an image of the drawer 25A as a movable part, and shows the position where the third marker MK51 is placed in the drawer 25A. The storage room image GP42 is an image obtained by superimposing a position display image GG42 showing the placement position of the third marker MK51 on an image showing the inside of the vegetable room 25 when the drawer 25A is pulled out.

[0228] The marker image MP41 is an image in which the shape and pattern of the external appearance of the fourth marker MK52 can be identified, for example, a photographed image of the fourth marker MK52. The marker image MP42 is an image in which the shape and pattern of the external appearance of the third marker MK51 can be identified, for example, a photographed image of the third marker MK51.

[0229] While the installation guide screen SG4 is displayed, if the photographing device 3 detects the third marker MK51 or the fourth marker MK52 installed in an appropriate position in the image captured by the wide-angle camera 301 or the narrow-angle camera 302, the photographing device 3 outputs a notification sound via the buzzer 306. The messages M42 and M43 are character strings asking the user P whether or not he or she confirmed that the camera 3 has emitted a notification sound. If the user P confirms the notification sound of the camera 3 after placing the fourth marker MK52, the user P makes an input to the input unit CH41, and if the user P confirms the notification sound of the camera 3 after placing the third marker MK51, the user P makes an input to the input unit CH42.

[0230] The buttons B41 and B42 are operation units operated by the user P. The button B41 is a button for instructing switching of the display from the installation guide screen SG4 to the next screen. The button B42 is an operation unit for inputting that the user P cannot confirm the notification sound of the photographing device 3. When the button B42 is operated, the application UI 430B displays the same screen as when the button B22 is operated.

[0231] The display order of the installation guide screens SG2, SG3, and SG4 can be changed as appropriate, but it is desirable that the installation guide screen SG2 is displayed first. Since the refrigerator compartment marker MK1 is installed on at least one of the right door hinge member 204 and the left door hinge member 205, there is a low possibility that the installation position will be shifted. Therefore, based on the position of the refrigerator compartment marker MK1 within the shooting ranges of the wide-angle camera 301 and the narrow-angle camera 302, it can be determined whether the installation position of the photographing device 3 in the refrigerator 2 is appropriate or not. In other words, the refrigerator compartment marker MK1 can be used to align the position of the photographing device 3. Therefore, by installing the refrigerator compartment marker MK1 first and determining the position of the refrigerator compartment marker MK1 within the shooting ranges of the wide-angle camera 301 and the narrow-angle camera 302, it can be determined whether the installation position of the photographing device 3 is appropriate or not before installing other markers MK.

[0232] 23 is a diagram showing an example of the confirmation screen SG5. The confirmation screen SG5 is a screen for allowing the user P to confirm that installation of all markers MK has been completed in the refrigerator 2. The confirmation screen SG5 includes a message M51, storage room images GP51, GP52, GP53, GP54, input sections CH51, CH52, CH53, CH54, and buttons B51, B52, B53.

[0233] The storage room image GP51 is a photographed image of the interior of the refrigerator room 21 taken by the wide-angle camera 301, and the storage room image GP52 is an image generated from the photographed image of the interior of the refrigerator room 21 and the inside of the door 21A taken by the wide-angle camera 301. The storage room image GP51 is an image of the entire view of the refrigerator room 21 seen from the front and above, and the storage room image GP52 is an image of a range including the installation position of the refrigerator room marker MK1. The door 21A described in this embodiment is a French door, and two refrigerator room markers MK1 corresponding to the door 21A are installed. Therefore, the storage room image GP52 is in a form in which an image showing the refrigerator room marker MK1 installed on the right door hinge member 204 and an image showing the refrigerator room marker MK1 installed on the left door hinge member 205 are arranged side by side. By looking at the storage room image GP52, the user P can confirm how the two refrigerator room markers MK1 are photographed by the photographing device 3.

[0234] The storage room image GP53 is an image generated from a captured image of the inside of the freezer room 24 captured by the narrow-angle camera 302. In Fig. 23, as an example, the storage room image GP53 shows an image showing the inside of the drawer 24A and the first marker MK41, and an image showing the inside of the storage case 24B and the second marker MK42. By looking at the storage room image GP53, the user P can confirm how the markers MK placed on the drawer 24A and the storage case 24B are captured by the imaging device 3.

[0235] The storage room image GP54 is an image generated from a captured image of the inside of the vegetable room 25 captured by the narrow-angle camera 302. In Fig. 23, as an example, the storage room image GP54 shows an image of the inside of the drawer 25A and the third marker MK51, and an image of the inside of the storage case 25B and the fourth marker MK52. By looking at the storage room image GP54, the user P can confirm how the markers MK placed on the drawer 25A and the storage case 25B are captured by the photographing device 3.

[0236] The storage room images GP51, GP52, GP53, and GP54 arranged on the confirmation screen SG5 are generated from images captured by the wide-angle camera 301 and the narrow-angle camera 302 after the installation work of the markers MK is started. For example, the storage room images GP51 and GP52 are generated from the first captured data 363A captured by the wide-angle camera 301 after the user P installs the refrigerator room marker MK1 according to the installation guide screen SG2. Similarly, the storage room image GP53 is generated from the second captured data 363B captured by the narrow-angle camera 302 after the user P installs the first marker MK41 and the second marker MK42 according to the installation guide screen SG3. The storage room image GP54 is generated from the second captured data 363B captured by the narrow-angle camera 302 after the user P installs the third marker MK51 and the fourth marker MK52 according to the installation guide screen SG4. For example, when transmitting the confirmation information in step SD8 (FIG. 17), the photographing device 3 may perform photographing using the wide-angle camera 301 and the narrow-angle camera 302, and transmit the photographed image data and the confirmation information.

[0237] Therefore, the application UI 430B can inform the user P of the state in which the marker MK placed by the user P is actually photographed by the photographing device 3 through the confirmation screen SG5. For example, when the user P sees the confirmation screen SG5 and determines that the placement state of the marker MK is not appropriate, the user P can redo the placement of the marker MK.

[0238] The message M51 is a character string that prompts the user P to check the installation status of the marker MK by looking at the confirmation screen SG5, and is placed at the top of the confirmation screen SG5. Input section CH51 is a checkbox for inputting that user P has viewed storage room image GP51. Input section CH52 is a checkbox for inputting that user P has viewed storage room image GP52, and input section CH53 is a checkbox for inputting that user P has viewed storage room image GP53. Input section CH54 is a checkbox for inputting that user P has viewed storage room image GP54.

[0239] The buttons B51, B52, and B53 are operation units operated by the user P. The button B51 is a button for instructing to end the display of the confirmation screen SG5. That is, the button B51 is a button for instructing to complete the installation work of the marker MK. The button B52 is a button for instructing to redo the installation of the marker MK. When the terminal device 4 accepts the operation of the button B52, the app UI 430B returns to the display of the start guide screen SG1. It is to be noted that it may also return to the display of the installation guide screen SG related to the storage room corresponding to the redo of the installation of the marker MK. The button B53 is a button for the user P to request an explanation about redoing the installation of the marker MK. When the button B53 is operated, a screen (not shown) or the like that explains the case where the installation of the marker MK needs to be redo is displayed on the app UI 430B. This screen or the like is switched from the confirmation screen SG5, or is temporarily superimposed on the confirmation screen SG5 and popped up.

[0240] Of the images included in the app UI430B, the storage room images GP21, GP31, GP32, GP41, and GP42 correspond to an example of a "movable part image." The combination of the storage room image GP and the marker image MP corresponds to an example of a "guide image." For example, the combinations of the storage room image GP21 and the marker image MP21, the storage room image GP31 and the marker image MP31, the storage room image GP32 and the marker image MP32, the storage room image GP41 and the marker image MP41, and the storage room image GP42 and the marker image MP42 correspond to an example of a "guide image."

[0241] The display order in which the start guide screen SG1 is displayed first in the application UI 430B, then the installation guide screen SG, and then the confirmation screen SG5 is displayed is an example. For example, the start guide screen SG1 may be the same screen as the installation guide screen SG2. Also, while multiple installation guide screens SG are switched and displayed, a screen different from the installation guide screen SG may be displayed. For example, while multiple installation guide screens SG are switched and displayed, the confirmation screen SG5 may be displayed multiple times, or the display of the confirmation screen SG5 and the installation guide screen SG may be transitioned to each other.

[0242] The start guide screen SG1, the installation guide screen SG, and the confirmation screen SG5 may be enlarged and displayed on the entire touch panel 42, or may be displayed on a part of the touch panel 42. For example, the display area of ​​the touch panel 42 may be provided with an area for switching and displaying the installation guide screen SG, and an area for displaying the confirmation screen SG5 alongside the installation guide screen SG. In this case, the confirmation screen SG5 may be continuously displayed while the installation guide screen SG is being switched and displayed. Here, among the storage room images GP51, GP52, GP53, and GP54 displayed on the confirmation screen SG5, the images of the movable parts for which the installation of the marker MK has not been completed may be displayed blank.

[0243] [2. Effects, etc.] The food management application 411 causes the terminal processor 400 of the terminal device 4 equipped with the touch panel 42 to function as the display control unit 402. The display control unit 402 acquires configuration information on the configuration of the refrigerator 2 having the refrigerator compartment 21, the freezer compartment 24, and the vegetable compartment 25 as storage compartments, and the movable parts that move when storing and removing items from the storage compartments. The movable parts include, for example, the door 21A, the drawers 24A, 25A, and the storage cases 24B, 25B. The display control unit 402 causes the touch panel 42 to display a guidance screen G that guides the user in placing a marker MK for photography at a position corresponding to the movable part. The guidance screen G includes a guidance image that indicates the placement position of the marker MK. The display control unit 402 changes the guidance screen G to correspond to the configuration of the refrigerator 2 indicated by the configuration information.

[0244] According to this, the guide screen G including the guide image is displayed on the touch panel 42 to guide the user P in placing the markers MK corresponding to the movable parts provided in the storage compartments of the refrigerator 2. This allows the user P to easily place the markers MK while looking at the touch panel 42, and effectively supports the user P's work. This improves the user's convenience in preparing to manage stored items such as ingredients using the imaging device 3.

[0245] The guide image displayed on the touch panel 42 includes a storage room image GP indicating the movable part, and a marker image MP indicating a marker MK corresponding to the movable part image. According to this, the user P can perform the work more easily while looking at the image of the movable part on which the marker MK is to be placed and the image of the marker MK. Therefore, the work of the user P can be more effectively supported, and the user's convenience can be further improved.

[0246] When markers MK are installed on multiple movable parts of refrigerator 2 and the markers MK installed on the multiple movable parts have different appearances, the guide image displayed by display control unit 402 includes a storage room image GP indicating one movable part selected from the multiple movable parts, and a marker image MP indicating the appearance of the marker MK corresponding to the selected movable part. According to this, since the storage room image GP and the marker image MP are displayed as guide images on the guide screen G, the work of placing the markers MK can be more clearly guided to the user P. Therefore, the work of the user P can be more effectively supported, and the convenience for the user can be further improved.

[0247] When markers MK are placed on multiple movable parts of the refrigerator 2, the display control unit 402 displays a guide screen G for each storage compartment. This allows the user P to be guided in a more easily understandable manner in the task of placing the markers MK in the refrigerator 2 having multiple storage compartments. This makes it possible to more effectively support the task performed by the user P and further improve user convenience.

[0248] When a plurality of movable parts are provided in one storage room, the guidance screen G displayed by the display control unit 402 includes a plurality of guidance images corresponding to the respective movable parts. According to this, the contents of the work of placing markers MK corresponding to a plurality of movable parts provided in one storage room can be displayed together on one guidance screen G. Therefore, the work of the user P can be supported more effectively, and the user's convenience can be further improved.

[0249] The display control unit 402 switches and displays the guide screens G in a predetermined order, the number of which corresponds to the number of storage compartments of the refrigerator 2, based on the configuration information. According to this, the user P can be guided about the work of placing markers MK corresponding to a plurality of storage rooms by switching the guidance screen G. This makes it possible to more effectively support the work of the user P and further improve the convenience for the user.

[0250] The guidance screen G includes a guidance image and an input section for inputting that installation of the marker MK has been completed. The display control section 402 switches the guidance screen G on the condition that an input has been made to the input section. According to this, on the condition that the user P has completed placing the marker MK corresponding to one movable part, the display is switched to the guidance screen G for the next movable part. This allows the user P to more reliably perform the task of placing markers MK corresponding to multiple movable parts, and effectively prevents operational errors. This makes it possible to more effectively support the task of the user P, and further improve the convenience for the user P.

[0251] When the movable parts include a door 21A provided at the opening of the refrigerator compartment 21 and drawers 24A, 25A pulled out from the openings of the freezer compartment 24 and the vegetable compartment 25, the display control unit 402 displays the guidance screen G corresponding to the door 21A before the guidance screen G corresponding to the drawers 24A, 25A. This allows the user P to place the marker MK on the door 21A before the markers MK on the drawers 24A and 25A. Therefore, by utilizing the fact that the marker MK on the door 21A is unlikely to be misaligned, the appropriateness of the installation position of the photographing device 3 relative to the refrigerator 2 can be confirmed before placing a large number of markers MK. This prevents unnecessary work from occurring due to an inappropriate installation position of the photographing device 3, and further improves convenience for the user P.

[0252] The display control unit 402 displays a general-purpose guide screen G when the configuration of the storage compartment in the refrigerator 2 cannot be specified based on the configuration information. According to this, the user P can be assisted in placing markers MK even in a refrigerator whose number and shape of storage compartments are unknown, thereby improving convenience for the user P.

[0253] The display control unit 402 outputs a notification prompting the execution of processing related to the guidance screen G at a predetermined timing after the guidance screen G is output. According to this, after the marker MK is placed in the refrigerator 2, the user P is prompted to correct or check the placement position of the marker MK, and the task of correcting or checking the placement position of the marker MK can be supported.

[0254] The photographing system 7 includes a terminal device 4 equipped with a touch panel 42, and a photographing device 3 arranged on the top surface of the refrigerator 2. The terminal device 4 includes a display control unit 402. The display control unit 402 acquires configuration information related to the configuration of the refrigerator 2 having a storage compartment and a movable part that moves when storing items in and out of the storage compartment. The display control unit 402 displays a guide screen G on the touch panel 42, which guides the user to set a marker MK for photographing at a position corresponding to the movable part. The guide screen G includes a guide image showing the installation position of the marker MK. The display control unit 402 changes the guide screen G in accordance with the configuration of the refrigerator 2 indicated by the configuration information. The photographing device 3 includes a wide-angle camera 301 and a narrow-angle camera 302 that photograph an area including the storage compartment or the movable part. The photographing device 3 includes a marker detection unit 355 that detects the marker MK from images captured by the wide-angle camera 301 and the narrow-angle camera 302, and a buzzer 306 that notifies the user that the marker MK has been detected by the marker detection unit 355. According to this, the terminal device 4 displays the guide screen G including the guide image to guide the user P in placing the markers MK in correspondence with the movable parts provided in the storage compartments of the refrigerator 2. This allows the user P to easily place the markers MK while looking at the touch panel 42, and effectively supports the user P's work. This improves the user's convenience in preparing to manage stored items such as ingredients using the imaging device 3.

[0255] The terminal device 4 includes a touch panel 42 and a display control unit 402. The display control unit 402 acquires configuration information relating to the configuration of the refrigerator 2 having a storage compartment and a movable part that moves when storing items in and out of the storage compartment. The display control unit 402 causes the touch panel 42 to display a guidance screen G that guides the user in placing a marker MK for photography in a position corresponding to the movable part. The guidance screen G includes a guidance image that indicates the placement position of the marker MK. The display control unit 402 changes the guidance screen G to correspond to the configuration of the refrigerator 2 indicated by the configuration information. According to this, the terminal device 4 displays the guide screen G including the guide image to guide the user P in placing the markers MK in correspondence with the movable parts provided in the storage compartments of the refrigerator 2. This allows the user P to easily place the markers MK while looking at the touch panel 42, and effectively supports the user P's work. This improves the user's convenience in preparing to manage stored items such as ingredients using the imaging device 3.

[0256] The control method of the terminal device 4 includes an acquisition step of acquiring configuration information related to the configuration of the refrigerator 2 having a storage compartment and a movable part that moves when storing items in and out of the storage compartment. The control method includes a display step of displaying on the touch panel 42 a guidance screen G that provides guidance for placing a marker MK for photography in a position corresponding to the movable part. The guidance screen G includes a guidance image that shows the placement position of the marker MK. The control method changes the guidance screen G displayed in the display step to correspond to the configuration of the refrigerator 2 indicated by the configuration information. According to this, the terminal device 4 displays the guide screen G including the guide image to guide the user P in placing the markers MK in correspondence with the movable parts provided in the storage compartments of the refrigerator 2. This allows the user P to easily place the markers MK while looking at the touch panel 42, and effectively supports the user P's work. This improves the user's convenience in preparing to manage stored items such as ingredients using the imaging device 3.

[0257] (Other embodiments) As described above, the above embodiment has been described as an example disclosed in this application. However, the technology in this disclosure is not limited to this, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above embodiment to form a new embodiment. Therefore, other embodiments will be exemplified below.

[0258] In the above-described embodiment, the app UIs 430A and 430B displayed on the touch panel 42 may be enlarged and displayed on the entire touch panel 42, or may be displayed on a part of the touch panel 42. Furthermore, the guide screen G and the installation guide screen SG constituting the app UIs 430A and 430B may be portrait-oriented or landscape-oriented screens, and may be configured so that their shapes can be switched according to the orientation of the touch panel 42.

[0259] In the above-described embodiment, the refrigerator 2 is configured such that the vegetable compartment 25 is provided below the freezer compartment 24, but the freezer compartment 24 may also be provided below the vegetable compartment 25, and the relative positions of the other storage compartments are not limited.

[0260] In the above-described embodiment, an example was described in which markers MK are not installed in the ice making compartment 22 and the freezer compartment 24, but it may also be possible to install markers MK corresponding to the drawer 22A of the ice making compartment 22 and the drawer 23A of the fresh freezing compartment 23.

[0261] In the above-described embodiment, the terminal device 4 is a smartphone or a tablet PC. In other embodiments, the terminal device 4 may be a desktop or laptop PC.

[0262] In the above-described embodiment, an example has been described in which the terminal device 4 generates and displays various screens, such as the guide screen G, constituting the app UI 430, based on data transmitted from the food management server 5 to the terminal device 4. The present disclosure is not limited to this, and for example, the food management server 5 may generate data for a screen constituting the app UI 430, and the food management server 5 may transmit the generated screen data to the terminal device 4. In this case, the terminal device 4 displays a screen constituting the app UI 430 on the touch panel 42 based on the screen data received from the food management server 5.

[0263] In the above-described embodiment, an application program that can be installed in the terminal device 4 is exemplified as the "program", but the "program" may be a software program that already exists in the terminal device 4, such as a browser.

[0264] The "programs" in the above-described embodiments can be non-temporarily recorded on a recording medium that is readable by the terminal processor 400, for example. The recording medium can be a magnetic or optical recording medium or a semiconductor memory device. In addition, these programs can be stored in a server device or the like, and the terminal device 4 can download the programs from the server device to realize the above-described operations.

[0265] The server processor 500 and the terminal processor 400 may be configured with a single processor or multiple processors. These processors may be hardware programmed to realize the corresponding functional units. That is, these processors may be configured with, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0266] The configuration of the image capture device 3 shown in Fig. 7 and the configurations of the terminal device 4 and food management server 5 shown in Fig. 10 are merely examples, and the specific implementation form is not particularly limited. In other words, it is not necessarily necessary to implement hardware corresponding to each part individually, and it is also possible to implement a configuration in which one processor executes a program to realize the functions of each part. Also, some of the functions realized by software in the above-mentioned embodiment may be hardware, or some of the functions realized by hardware may be software.

[0267] The step units of the operations shown in Figures 16 to 18 are divided according to the main processing contents in order to make the operations easier to understand, and the manner in which the processing units are divided or the names of the processing units do not limit the operations. The operations may be divided into more step units according to the processing contents. Furthermore, one step unit may be divided so as to include more processing. Furthermore, the order of the steps may be changed as appropriate within the scope of the purpose of this disclosure.

[0268] It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, substitutions, additions, omissions, and the like can be made within the scope of the claims or their equivalents.

[0269] (Additional Note) The above description of the embodiments discloses the following techniques.

[0270] (Technology 1) A program that causes a processor of a terminal device having a display unit to function as a display control unit that acquires storage information regarding the configuration of a storage unit having a storage chamber and a movable part that moves when storing items in and out of the storage chamber, and causes the display unit to display a guidance screen that guides the user in placing a marker for photography in a position corresponding to the movable part, the guidance screen including a guidance image that indicates the placement position of the marker, and the display control unit changes the guidance screen to correspond to the configuration of the storage unit indicated by the storage chamber information. According to this, by displaying a guide screen on the display unit, the user can be guided in the task of placing markers corresponding to the movable parts provided in the storage room of the storage facility, and the task by the user can be effectively supported. Therefore, the user's convenience can be improved in terms of preparation for managing stored items such as food ingredients using the imaging device.

[0271] (Technology 2) The program described in Technology 1, wherein the guide image includes a movable part image indicating the movable part, and a marker image indicating the marker corresponding to the movable part image. This allows the user to perform work more easily while looking at the image of the movable part and the image of the marker, thereby further improving user convenience.

[0272] (Technology 3) In the case where the markers are installed on a plurality of movable parts of the storage facility, and the markers installed on the plurality of movable parts have different appearances from each other, the guide image displayed by the display control unit includes the movable part image showing one movable part selected from the plurality of movable parts, and the marker image showing the appearance of the marker corresponding to the selected movable part, in the program described in Technology 2. According to this, by displaying the storage room image and the marker image as guide images on the guide screen, the work of placing the marker can be more clearly guided. Therefore, the work by the user can be more effectively supported, and the user's convenience can be further improved.

[0273] (Technology 4) The program according to any one of Technology 1 to Technology 3, wherein the display control unit displays the guidance screen for each storage room when the markers are installed on multiple movable parts of the storage facility. This allows the user to be guided more clearly through the task of placing markers in a storehouse having multiple storage compartments, thereby more effectively supporting the user's task and further improving user convenience.

[0274] (Technology 5) The program described in Technology 4, wherein when a plurality of movable parts are provided in the storage room, the guidance screen displayed by the display control unit includes a plurality of guidance images corresponding to each of the plurality of movable parts. According to this, the contents of the work of placing markers corresponding to the multiple movable parts provided in one storage room can be displayed together on one guidance screen, which can more effectively support the user's work and further improve user convenience.

[0275] (Technology 6) The program described in Technology 4 or Technology 5, in which the display control unit switches and displays, in a predetermined order, a number of the guidance screens corresponding to the number of storage rooms in which the markers are to be installed among the storage rooms possessed by the storage unit based on the storage unit information. This allows the user to be guided through the process of placing markers in multiple storage compartments by switching between different guide screens, which can more effectively support the user's work and further improve user convenience.

[0276] (Technology 7) The guidance screen includes the guidance image and an input unit for inputting that installation of the marker has been completed, and the display control unit switches the guidance screen on condition that input has been made to the input unit. The program described in Technology 6. According to this, when the user completes the installation of the marker corresponding to one movable part, the display is switched to a guidance screen related to the next movable part. This allows the user to more reliably perform the task of installing markers corresponding to multiple movable parts, and effectively reduces operational errors.

[0277] (Technology 8) The program described in Technology 6 or Technology 7, wherein when the movable part includes a door provided at the opening of the storage room and a drawer pulled out from the opening of the storage room, the display control unit displays the guidance screen corresponding to the door before the guidance screen corresponding to the drawer. According to this, the user is made to place the marker corresponding to the door before the marker corresponding to the drawer, and the appropriateness of the installation position of the image capturing device relative to the storage unit can be confirmed before placing the marker for the drawer. This prevents unnecessary work from occurring due to an inappropriate installation position of the image capturing device, and further improves user convenience.

[0278] (Technology 9) A program described in any one of Technology 1 to Technology 8, wherein the display control unit displays a general-purpose guidance screen when the configuration of the storage room in the storage facility cannot be identified based on the storage facility information. This allows the user to be assisted in placing markers even in a storehouse where the number and shape of storage compartments are unknown, thereby improving user convenience.

[0279] (Technology 10) The program described in any one of Technology 1 to Technology 9, wherein the display control unit outputs a notification prompting the execution of processing related to the guidance screen at a predetermined timing after outputting the guidance screen. According to this, after the marker is placed in the storage facility, the user is prompted to correct or confirm the placement position of the marker, and the task of correcting or confirming the placement position of the marker can be assisted.

[0280] (Technology 11) An photography system comprising: a terminal device having a display unit; and a photographing device arranged on an upper surface of a storage facility, the terminal device acquiring storage facility information relating to a configuration of the storage facility having a storage room and a movable part that moves when storing items in and out of the storage room; and a display control unit causing the display unit to display a guidance screen that guides the user in placing a marker for photography in a position corresponding to the movable part, the guidance screen including a guidance image indicating the installation position of the marker, and the display control unit changing the guidance screen in accordance with the configuration of the storage facility indicated by the storage facility information, the photographing device comprising: a photographing unit that photographs an area including the storage room or the movable part, a marker detection unit that detects the marker from an image photographed by the photographing unit, and a notification unit that notifies the user that the marker has been detected by the marker detection unit. This provides the same effect as the program of Technology 1.

[0281] (Technology 12) A terminal device comprising: a display unit; and a display control unit that acquires storage information regarding the configuration of a storage unit having a storage chamber and a movable part that moves when storing items in and out of the storage chamber; and causes the display unit to display a guidance screen that guides the user in placing a marker for photography in a position corresponding to the movable part, wherein the guidance screen includes a guidance image that indicates the placement position of the marker; and the display control unit changes the guidance screen to correspond to the configuration of the storage unit indicated by the storage chamber information. This provides the same effect as the program of Technology 1.

[0282] (Technology 13) A control method for a terminal device equipped with a display unit, the control method for a terminal device including: an acquisition step of acquiring storage facility information regarding the configuration of a storage facility having a storage room and a movable part that moves when storing items in and out of the storage room; and a display step of displaying a guidance screen on the display unit that guides users in placing a marker for photography in a position corresponding to the movable part, the guidance screen including a guidance image that indicates the installation position of the marker, and the guidance screen displayed in the display step is changed to correspond to the configuration of the storage facility indicated by the storage facility information. This provides the same effect as the program of Technology 1. [Industrial Applicability]

[0283] As described above, the program, the photography system, the terminal device, and the control method for the terminal device according to the present invention can be used to assist in the preparation work for managing ingredients and the like using a camera. [Explanation of symbols]

[0284] 1. Food Control System 2 Refrigerator (storage) 3. Imaging Equipment 4 Terminal Equipment 5. Food Management Server 6. Communications Equipment 7. Shooting System 20 Main box body 20A Top 20B Front edge 21 Refrigerator (storage room) 21A Door (moving part) 21B Right door (movable part) 21C Left door (movable part) 22 Ice Room 23 Fresh freezing room 24 Freezer (storage room) 24A drawer (movable part) 24B Storage case (movable part) 25 Vegetable room (storage room) 25A drawer (movable part) 25B Storage case (movable part) 26 Storage Room 26A drawer 30 Main unit 31 Photography materials 33 1st LED 34 Regulatory components 35 marks 40 Terminal control device 41 Terminal communication unit 42 Touch panel (display) 50 Server control device 51 Server Communication Department 201 Aperture 202 Outer box 203 Inner box 204 Right door hinge material 205 Left door hinge material 301 Wide-angle camera (photography unit) 302 Narrow-angle camera (photography unit) 303 First distance sensor 304 Second distance sensor 305 Thermopile 306 Buzzer (notification unit) 307 Illuminance sensor 308 2nd LED 309 3rd LED 311 First control board 312 Second control board 313 Radio Communication Department 320 First Processor 330 First Memory 331 Control Program 340 First communication module 350 Second Processor 351 First communication control unit 352 Second communication control section 353 Open / Close Judgment Unit 354 Shooting control section 355 Marker detection unit 356 Image Judgment Unit 357 Generation part 360 Second Memory 361 Control Program 362 Imaging Device ID 363 Status Memory Unit 363A First Shooting Data 363B 2nd Shooting Data 370 Second Communication Module 400 Terminal Processor (Processor) 401 Terminal communication control unit 403 Reception 410 Terminal Memory 411 Food management app (program) 430 App UI 500 Server Processors 501 Server communication control unit 502 Server processing unit 510 Server Memory 511 Control Program 512 Refrigerator Data AS hinge cover CH Input Section D1 Management data DV1 1st recording data DV2 Second recording data DV G Information screen GP Storage Room Image MK Marker MP marker image NW Network SG1 Start screen SG installation guide screen SG5 Confirmation Screen

Claims

1. A processor of a terminal device having a display unit, Acquire storage information regarding a configuration of a storage having a storage chamber and a movable part that moves when an item is put into or taken out of the storage chamber; a display control unit that causes a guidance screen that guides the user to place a marker for photography at a position corresponding to the movable unit to be displayed on the display unit; the guidance screen includes a guidance image indicating an installation position of the marker, The display control unit changes the guidance screen in accordance with a configuration of the storage unit indicated by the storage unit information. program.

2. The program according to claim 1 , wherein the guide image includes a movable part image indicating the movable part, and a marker image indicating the marker corresponding to the movable part image.

3. The program of claim 2, wherein when the markers are installed on a plurality of movable parts of the storage facility and the markers installed on the plurality of movable parts have different appearances from each other, the guidance image displayed by the display control unit includes the movable part image showing one movable part selected from the plurality of movable parts, and the marker image showing the appearance of the marker corresponding to the selected movable part.

4. The program according to claim 1 , wherein the display control unit displays the guidance screen for each of the storage compartments when the markers are placed on a plurality of the movable parts of the storage facility.

5. The program according to claim 4 , wherein when a plurality of the movable parts are provided in one of the storage chambers, the guidance screen displayed by the display control unit includes a plurality of the guidance images corresponding to each of the plurality of the movable parts.

6. The program according to claim 4 , wherein the display control unit switches and displays, in a predetermined order, a number of the guidance screens corresponding to the number of storage rooms in which the markers are to be placed among the storage rooms in the storage room, based on the storage room information.

7. the guidance screen includes the guidance image and an input unit for inputting that installation of the marker has been completed, The program according to claim 6 , wherein the display control unit switches the guidance screen on condition that an input is made to the input unit.

8. The program according to claim 6 or claim 7, wherein when the movable part includes a door provided at an opening of the storage chamber and a drawer pulled out from the opening of the storage chamber, the display control unit displays the guidance screen corresponding to the door before the guidance screen corresponding to the drawer.

9. The program according to claim 1 , wherein the display control unit causes a general-purpose guide screen to be displayed when a configuration of the storage room in the storage cannot be identified based on the storage information.

10. The program according to claim 1 , wherein the display control unit outputs a notification prompting execution of a process related to the guidance screen at a predetermined timing after outputting the guidance screen.

11. A terminal device having a display unit and a photographing device arranged on an upper surface of the storage facility, The terminal device a display control unit that acquires storage information relating to a configuration of a storage unit having a storage chamber and a movable part that moves when an item is put into or taken out of the storage chamber, and causes the display unit to display a guidance screen that guides the user in placing a marker for photography at a position corresponding to the movable part; the guidance screen includes a guidance image indicating an installation position of the marker, The display control unit changes the guidance screen in accordance with a configuration of the storage unit indicated by the storage unit information, The imaging device is an imaging unit that images an area including the storage chamber or the movable unit; a marker detection unit that detects the marker from an image captured by the imaging unit; a notification unit that notifies that the marker has been detected by the marker detection unit.

12. A display unit; a display control unit that acquires storage information relating to a configuration of a storage unit having a storage chamber and a movable part that moves when a stored item is put into or taken out of the storage chamber, and causes the display unit to display a guidance screen that guides the user in placing a marker for photography at a position corresponding to the movable part; the guidance screen includes a guidance image indicating an installation position of the marker, The display control unit changes the guidance screen in accordance with a configuration of the storage unit indicated by the storage unit information.

13. A method for controlling a terminal device having a display unit, comprising: An acquisition step of acquiring storage information relating to a configuration of a storage having a storage chamber and a movable part that moves when an item is put into or taken out of the storage chamber; a display step of displaying on the display unit a guidance screen that guides the user in placing a marker for photography at a position corresponding to the movable part, the guidance screen includes a guidance image indicating an installation position of the marker, A control method for a terminal device, the control method comprising: changing the guidance screen displayed in the display step in accordance with a configuration of the repository indicated by the repository information.

Images