Image capture system, image capture method, control program, and marker

Abstract

An image capture system according to the present disclosure properly determines whether a captured image contains an image of an item stored in a drawer or contains an image of an item stored in a storage case.　An image capture system 7 comprises: a narrow-angle camera 302 which is disposed on a top face 20A of a refrigerator 2 and which captures images of the refrigerator 2, the refrigerator 2 comprising a vegetable compartment 25 having a drawer 25A with an opening formed in the top face; a first marker MK51 disposed on the drawer 25A, the drawer 25A having a storage case 25B configured to be pulled out in an upper section of the drawer 25A; a second marker MK52 disposed on the storage case 25B; an image capture control unit 354 that captures an image of the vegetable compartment 25 with the narrow-angle camera 302 and generates a captured image PB2; and a determination unit 355C that determines, on the basis of the forms of the first marker MK51 and the second marker MK52 in the captured image PB2, whether the captured image PB2 contains an image of foodstuff stored in the drawer 25A or contains an image of foodstuff stored in the storage case 25B.

Description

Photographing system, photographing method, control program, and marker

[0001] The present disclosure relates to an imaging system, an imaging method, a control program, and a marker.

[0002] Patent Document 1 discloses a technology in which a camera is installed on top of a refrigerator, a photograph is taken when an object passing through the opening of the refrigerator is detected, and the food stored in the refrigerator is managed based on the photographed results.

[0003] Japanese Patent Application Laid-Open No. 2019-070476

[0004] The present disclosure provides an imaging system, imaging method, control program, and marker for accurately managing food stored in a refrigerator.

[0005] The photography system of the present disclosure photographs a storage facility having a storage compartment with a drawer having an opening formed on its top surface, and includes a camera arranged on the top surface of the storage facility, a first marker arranged on the drawer, the drawer having a storage case configured to be able to be pulled out in its upper section, and a second marker arranged on the storage case, a photography control unit that photographs the storage compartment with the camera and generates a photographed image, and a determination unit that determines whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the first marker and the second marker in the photographed image.

[0006] Furthermore, the imaging method disclosed herein is an imaging method for an imaging system that photographs a storage facility that includes a storage compartment with a drawer having an opening formed on its top surface, the imaging method comprising: a camera arranged on the top surface of the storage facility; a first marker arranged on the drawer; the drawer having a storage case configured to be able to be pulled out in an upper section of the drawer; and a second marker arranged on the storage case, the imaging method including: an imaging control step of photographing the storage compartment with the camera and generating a photographed image; and a determination step of determining whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the first marker and the second marker in the photographed image.

[0007] In addition, the control program of the present disclosure is a control program for an photography system that photographs a storage cabinet having a storage room with a drawer having an opening formed on its top surface, the control program comprising: a camera placed on the top surface of the storage cabinet; a first marker placed on the drawer; the drawer having a storage case configured to be able to be pulled out in its upper section, a second marker placed on the storage case; and a processor that controls the camera, and causes the processor to function as an photography control unit that photographs the storage room with the camera and generates a photographed image, and a determination unit that determines whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the first marker and the second marker in the photographed image.

[0008] Another photography system in the present disclosure photographs a storage facility having a storage compartment with a drawer having an opening formed on its top surface, and comprises: a camera arranged on the top surface of the storage facility; a third marker arranged on the drawer; the drawer having a storage case configured to be able to be pulled out in an upper section of the drawer; the storage case having an opening / closing body that opens and closes the opening of the storage case; the opening / closing body having two plate-shaped opening / closing members arranged to be able to slide freely in the left-right direction of the storage case; two fourth markers arranged on each of the two plate-shaped opening / closing members; a photography control unit that photographs the storage facility with the camera and generates a photographed image; and a determination unit that determines whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the third marker and the two fourth markers in the photographed image.

[0009] Another imaging method in the present disclosure is an imaging method for an imaging system that photographs a storage cabinet comprising a storage compartment with a drawer having an opening formed on its top surface, the imaging method comprising: a camera arranged on the top surface of the storage cabinet; a third marker arranged on the drawer; the drawer having a storage case configured to be able to be pulled out in an upper section of the drawer, the storage case having an opening / closing body that opens and closes the opening of the storage case, the opening / closing body having two plate-shaped opening / closing members arranged to be able to slide in the left-right direction of the storage case; and two fourth markers arranged on each of the two plate-shaped opening / closing members, the method comprising: an imaging control step of photographing the storage compartment with the camera and generating a captured image; and a determination step of determining whether the captured image includes an image of the drawer or an image of the storage case based on the appearance of the third marker and the two fourth markers in the captured image.

[0010] Another control program in the present disclosure is a control program for an imaging system that photographs a storage cabinet that includes a storage compartment with a drawer that has an opening on its top surface, the control program including a camera arranged on the top surface of the storage cabinet, a third marker arranged on the drawer, the drawer having a storage case that can be pulled out in an upper section of the drawer, the storage case having an opening / closing body that opens and closes the opening of the storage case, the opening / closing body having two plate-shaped opening / closing members that are arranged to slide freely in the left-right direction of the storage case, two fourth markers arranged on each of the two plate-shaped opening / closing members, and a processor that controls the camera, and causes the processor to function as an imaging control unit that photographs the storage compartment with the camera and generates a captured image, and a determination unit that determines whether the captured image includes an image of the drawer or an image of the storage case based on the appearance of the third marker and the two fourth markers in the captured image. This specification includes all of the contents of Japanese Patent Application No. 2023-109875, filed on July 4, 2023.

[0011] The photography system, photography method, control program, and marker disclosed herein include a camera disposed on the top surface of a storage compartment, in which a first marker is placed on a drawer and a second marker is placed on a storage case configured to be able to be pulled out and attached to the upper portion of the drawer, photographing the storage compartment having a drawer with an opening formed on its top surface, generating a photographed image, and determining whether the photographed image includes an image of the drawer or the storage case based on the appearance of the first marker and the second marker in the photographed image. This allows for accurate determination of whether the photographed image includes an image of the drawer or the storage case. This allows for accurate management of food stored in the refrigerator.

[0012] 1 is a diagram showing the configuration of a food management system in embodiment 1. FIG. 2 is a perspective view of a refrigerator in embodiment 1. FIG. 3 is a side view of a camera in embodiment 1 as seen from the right. FIG. 4 is a perspective view of a restricting member abutting the front edge of a main box body in embodiment 1 as seen from the front left. FIG. 5 is a front view of a refrigerator and a camera in embodiment 1 as seen from the front. FIG. 6 is a side view of a refrigerator and a camera in embodiment 1 as seen from the right. FIG. 7 is a plan view of a camera attached to a refrigerator in embodiment 1 as seen from above. FIG. 8 is a block diagram showing the configuration of a camera in embodiment 1. FIG. 9 is a diagram showing an example of a shooting state of a narrow-angle camera in embodiment 1. FIG. 10 is a diagram showing an example of a marker placed in a refrigerator compartment in embodiment 1. FIG. 11 is a diagram showing an example of a hinge cover in embodiment 1. FIG. 12 is a plan view showing an example of a marker placed in a vegetable compartment in embodiment 1. FIG. 13 is a plan view showing another example of a marker placed in a vegetable compartment in embodiment 1. FIG. 14 is a diagram showing an example of the configuration of a marker in embodiment 1. FIG. 15 is a diagram showing a terminal device and a food management server in embodiment 1. 24 is a flowchart showing the second captured image selection process of the imaging device in embodiment 2. FIG. 25 is a block diagram showing the control configuration in embodiment 3. FIG. 26 is an explanatory diagram showing an example of calculation of the amount of movement in embodiment 3. FIG. 27 is a flowchart showing the operation of embodiment 3. FIG. 28 is a perspective view from above showing the attachment state of a marker for a refrigerator compartment in embodiment 4.36 is a plan view showing a case marker in embodiment 4; FIG. 37 is a bottom view showing a case marker in embodiment 4; FIG. 38 is a side view showing a case marker in embodiment 4; FIG. 39 is a plan view showing a case marker attached to a case with one slide plate attached in embodiment 4; FIG. 40 is a plan view showing a case marker attached to a case with two slide plates attached in embodiment 4; FIG. 41 is a side cross-sectional view showing a case marker attached to embodiment 4;

[0013] (Knowledge and other information that forms the basis of the present disclosure) At the time the inventors conceived the present disclosure, there was a technology in which a camera was installed on top of a refrigerator, which captured an image when it detected an object passing through the refrigerator's opening, and managed the food stored in the refrigerator based on the captured image. Meanwhile, drawers such as the vegetable compartment and freezer compartment may have a removable storage case located in the upper section. In such cases, the inventors discovered a problem of determining whether a captured image includes an image of the drawer or the storage case, and the subject matter of the present disclosure was formed to solve this problem. Therefore, the present disclosure provides a photography system, photography method, control program, and marker that properly determine whether a captured image includes an image of the drawer or the storage case.

[0014] Hereinafter, embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed description than necessary will be omitted. For example, detailed description of already well-known matters or redundant description of substantially the same configuration will be omitted. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0015] (Embodiment 1) [1-1. Configuration] [1-1-1. Configuration of food management system] Figure 1 is a diagram showing the configuration of a food management system 1 in embodiment 1. Food management system 1 is a system that manages food stored in refrigerator 2. As an example of food management, food management system 1 of this embodiment manages images of food stored in refrigerator 2. Refrigerator 2 corresponds to an example of a "storage room."

[0016] In this embodiment, the food stored in refrigerator 2 includes the food itself and the containers that store the food. The food may be marked with a predetermined marker. The food corresponds to an example of an "item."

[0017] The food management system 1 includes a refrigerator 2. The refrigerator 2 includes a main box 20 with an open front. The main box 20 is formed with a refrigerator compartment 21, an ice-making compartment 22, a fresh-freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25. These storage compartments are cooled with cold air. A revolving door 21A is installed in an opening 201 on the front of the refrigerator compartment 21. The door 21A opens and closes the opening 201. The door 21A includes a revolving right door 21B and a revolving left door 21C.

[0018] The ice making compartment 22, the fresh freezing compartment 23, the freezer compartment 24, and the vegetable compartment 25 each include a drawer 22A, a drawer 23A, a drawer 24A, and a drawer 25A, respectively, capable of storing food. The top of each of the drawers 22A to 25A is open when installed. In the following description, when the drawers 22A, 23A, 24A, and 25A are referred to collectively without distinction, they will be denoted with the symbol "26A" and referred to as "drawer 26A." Furthermore, when the ice making compartment 22, the fresh freezing compartment 23, the freezer compartment 24, and the vegetable compartment 25 are referred to collectively without distinction, they will be denoted with the symbol "26" and referred to as storage compartment 26. The ice making compartment 22, the fresh freezing compartment 23, the freezer compartment 24, and the vegetable compartment 25 each correspond to an example of a "storage compartment" in the present disclosure. In the first embodiment, a case will be described in which the “storage compartment” of the present disclosure is a vegetable compartment 25.

[0019] The food management system 1 includes a camera 3. The camera 3 is provided on a top surface 20A of a main box 20 of a refrigerator 2. The top surface 20A of the main box 20 is made of metal. The camera 3 photographs the refrigerator 2. The top surface 20A corresponds to an example of the "top surface" in this disclosure.

[0020] Food management system 1 includes terminal device 4. Terminal device 4 is a portable computer such as a smartphone. An application program related to the management of food stored in refrigerator 2 is installed on terminal device 4, and terminal device 4 communicates with food management server 5 using the functions of this application program. In the following description, this application program will be referred to as the "food management app" and will be assigned the reference number "411."

[0021] 1 , a user P who is at home is indicated by a solid line, and a user P who has left home H is indicated by a dotted line. When used by user P who is at home, terminal device 4 communicates with food management server 5 via communication device 6 or without via communication device 6. When used by user P who has left home H and cannot establish a communication connection with communication device 6, terminal device 4 communicates with food management server 5 without via communication device 6.

[0022] The communication device 6 is connected to a network NW configured by a public line network or the like, and communicates with the food management server 5 via the network NW. The communication device 6 functions as an interface device for connecting the refrigerator 2, the image capturing device 3, and the terminal device 4 to the network NW. The communication device 6 establishes a local network in the home H.

[0023] The food management system 1 includes a food management server 5. The food management server 5 is a device that manages the food stored in the refrigerator 2. The food management server 5 stores images of the food as part of its food management. The food management server 5 is connected to a network NW. Note that although the food management server 5 is represented by a single block in FIG. 1, this does not necessarily mean that the food management server 5 is composed of a single device.

[0024] [1-1-2. Refrigerator Configuration] Figure 2 is a perspective view of the refrigerator 2 in an installed state. Figure 2 illustrates the X-axis, Y-axis, and Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to one another. The Z-axis indicates the up-down direction and corresponds to the height direction of the refrigerator 2. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction. The left-right direction corresponds to the width direction of the refrigerator 2. The Y-axis indicates the front-to-back direction. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the front direction. The positive direction of the Z-axis indicates the upward direction. The X-axis, Y-axis, and Z-axis are also illustrated in Figures 3 to 7, 9 to 13, and 20 to 23.

[0025] The refrigerator 2 includes a main box body 20, a door 21A, and a drawer 26A. The main box body 20 is a box-shaped member that forms the main body of the refrigerator 2. The main box body 20 includes an outer box 202 that is open at the front, an inner box 203 that is housed inside the outer box 202 and is also open at the front, and a thermal insulating material that fills the space between the outer box 202 and the inner box 203. The outer box 202 is formed of, for example, a metal such as iron or a hard resin such as ABS. The top surface of the outer box 202, i.e., the top surface 20A of the main box 20, is formed of metal. The inner box 203 is formed of, for example, a hard resin such as ABS. The thermal insulating material is, for example, hard urethane foam. However, the thermal insulating material is not limited to this and may be any material that has thermal insulating properties and can be filled in the space between the outer box 202 and the inner box 203.

[0026] The internal space of the inner box 203 is divided into a plurality of spaces by partition members. In this embodiment, the internal space of the inner box 203 is divided into five spaces, and the five spaces function as a refrigerator compartment 21, an ice-making compartment 22, a fresh-freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25, respectively.

[0027] The right door 21B is rotatably supported by a right door hinge member 204. The right door hinge member 204 is a hinge member installed on the top surface 20A of the main box body 20. The left door 21C is rotatably supported by a left door hinge member 205. The left door hinge member 205 is a hinge member installed on the top surface 20A of the main box body 20.

[0028] A second wireless communication module 206 is installed in the right door hinge member 204. The second wireless communication module 206 is a module that receives control data for controlling the refrigerator 2 from an external device and transmits predetermined data to the external device.

[0029] [1-1-3. Configuration of the Imaging Device] The imaging device 3 will be described with reference to Figs. 3 to 7. Figs. 3 to 7 illustrate the same X, Y, and Z axes as those shown in Fig. 2. The X-axis direction corresponds to the width direction of the imaging device 3 and the left-right direction of the imaging device 3. The Y-axis direction corresponds to the front-rear direction of the imaging device 3. The Z-axis direction corresponds to the up-down direction of the imaging device 3.

[0030] Fig. 3 is a side view of the camera device 3 attached to the refrigerator 2, seen from the right. Fig. 4 is a perspective view of the restricting member 34 abutting against the front edge 20B of the main box body 20, seen from the left front.

[0031] 3 and 4 , the photographing device 3 includes a main body 30 installed on the top surface 20A of the main box 20, and a photographing member 31 extending forward from the main body 30. The photographing member 31 is provided above the front of the main body 30.

[0032] The main body 30 has a power button 32 on the front surface for turning on and off the power of the photographing device 3. The power button 32 is provided with a first LED (Light emitting diode) 33 for notifying predetermined information.

[0033] Furthermore, main body 30 has, on the front surface, a shooting start button 32A for starting shooting with imaging device 3. When user P presses shooting start button 32A, shooting control unit 354 shown in Fig. 8 causes wide-angle camera 301 and narrow-angle camera 302 to start shooting. When shooting start button 32A is pressed, recording control unit 355 shown in Fig. 8 starts recording the results of shooting with wide-angle camera 301 and narrow-angle camera 302.

[0034] Note that user P presses shooting start button 32A when wide-angle camera 301 and narrow-angle camera 302 have not started shooting even though door 21A or drawer 26A is in the open state. When causing wide-angle camera 301 and narrow-angle camera 302 to start shooting, shooting control unit 354 turns on second LED 308 and third LED 309. Therefore, user P can recognize that wide-angle camera 301 and narrow-angle camera 302 have not started shooting because second LED 308 and third LED 309 are not turned on.

[0035] The main body 30 includes a regulating member 34. The regulating member 34 abuts against the front edge 20B of the main box body 20 to regulate the position of the imaging member 31 in the front-to-rear direction of the refrigerator 2. The regulating member 34 is a plate-like member extending downward from the bottom surface 30A of the main body 30. The regulating member 34 extends in the width direction of the main body 30 (the width direction of the imaging device 3) on the front surface of the main body 30, and has a mark 35 at the center in the width direction of the main body 30. The mark 35 indicates the center in the width direction of the imaging device 3. The length of the regulating member 34 extending from the bottom surface 30A is shorter than the length from the top surface 20A of the main box body 20 to a door gasket that is provided at the upper part of the inner surface of the door 21A and is approximately parallel to the top surface 20A.

[0036] FIG. 5 is a front view of the refrigerator 2 and the camera device 3 attached to the refrigerator 2. The refrigerator 2 shown in FIG. 5 has its door 21A in an open position. FIG. 6 is a side view of the refrigerator 2 and the camera device 3 attached to the refrigerator 2, seen from the right. The refrigerator 2 shown in FIG. 6 has its drawer 25A in an open position, i.e., pulled out. FIG. 7 is a plan view of the camera device 3 attached to the refrigerator 2, seen from above. The configuration of the camera device 3 will be described with reference to FIGS. 5 to 7.

[0037] The image capturing device 3 includes a wide-angle camera 301 and a narrow-angle camera 302. The wide-angle camera 301 and the narrow-angle camera 302 are installed at the front end of the bottom surface 31A of the image capturing member 31. When the image capturing device 3 is attached to the refrigerator 2 and the door 21A is closed, the wide-angle camera 301 and the narrow-angle camera 302 are located in front of the door 21A.

[0038] Wide-angle camera 301 has a wider angle than narrow-angle camera 302. When imaging device 3 is attached to refrigerator 2, wide-angle camera 301 captures images of refrigeration compartment 21 from above and in front of refrigerator 2. For example, the imaging range of wide-angle camera 301 includes range A1 shown in FIG. 5 when viewed from the front of refrigerator 2. Range A1 is a range that includes door pockets 21E of right door 21B and left door 21C and opening 201 of refrigeration compartment 21 when right door 21B and left door 21C are fully open. Furthermore, the imaging range of wide-angle camera 301 includes range A3 shown in FIG. 6 when viewed from the right of refrigerator 2. Range A3 is a range that includes, in the front-to-back direction, the front ends of right door 21B and left door 21C in the open state to the front end of shelf 21F, and also includes opening 201 of refrigeration compartment 21 in the up-down direction.

[0039] Narrow-angle camera 302 has a narrower angle than wide-angle camera 301. In other words, the imaging angle of narrow-angle camera 302 in the left-right direction, i.e., the X-axis direction, is smaller than the imaging angle of wide-angle camera 301 in the left-right direction, i.e., the X-axis direction. Also, the imaging angle of narrow-angle camera 302 in the front-to-back direction, i.e., the Y-axis direction, is smaller than the imaging angle of wide-angle camera 301 in the front-to-back direction, i.e., the Y-axis direction. When imaging device 3 is attached to refrigerator 2, narrow-angle camera 302 captures images of drawer 26A from above and in the front of refrigerator 2. For example, the imaging range of narrow-angle camera 302 includes range A2 shown in FIG. 5 when viewed from the front of refrigerator 2. Range A2 is the range that includes drawer 26A when viewed from the front of refrigerator 2. Also, for example, the imaging range of narrow-angle camera 302 includes range A4 shown in FIG. 6 when viewed from the right of refrigerator 2. Range A4 is the range that includes drawer 26A when it is pulled out to the farthest position in the front-to-back direction. The imaging range of narrow-angle camera 302 may be a range that includes a drawer provided in refrigerator compartment 21. Narrow-angle camera 302 corresponds to an example of a "camera" in the present disclosure.

[0040] As shown in FIGS. 5 to 7 , the narrow-angle camera 302 constitutes part of the photography system 7 in Embodiment 1. The photography system 7 in Embodiment 1 includes the narrow-angle camera 302, a first marker MK51, a second marker MK52, a photography control unit 354, and a determination unit 355C. In other words, the photography system 7 includes the refrigerator 2 equipped with the first marker MK51 and the second marker MK52, and the photography device 3 equipped with the narrow-angle camera 302, the photography control unit 354, and the determination unit 355C. The first marker MK51 and the second marker MK52 will be described with reference to FIGS. 9 and 12 to 14 . The photography control unit 354 and the determination unit 355C will be described with reference to FIG. 8 .

[0041] The photographing device 3 includes a first distance measurement sensor 303 and a second distance measurement sensor 304. The first distance measurement sensor 303 and the second distance measurement sensor 304 are installed on the main body 30. The first distance measurement sensor 303 is a sensor that detects the opening and closing of the right door 21B by detecting the distance between the right door 21B and the photographing device 3. For example, the first distance measurement sensor 303 is an infrared sensor. The first distance measurement sensor 303 detects the distance between the right door 21B and the photographing device 3 and outputs a detection value indicating the detected distance. The first distance measurement sensor 303 detects the distance by, for example, a change in the optical axis. The second distance measurement sensor 304 is a sensor that detects the opening and closing of the left door 21C by detecting the distance between the left door 21C and the photographing device 3. For example, the second distance measurement sensor 304 is an infrared sensor. The second distance measurement sensor 304 detects the distance between the left door 21C and the photographing device 3. For example, the second distance measurement sensor 304 is an infrared sensor. The second distance measurement sensor 304 detects the distance between the left door 21C and the photographing device 3. Outputs a detection value indicating the detected distance. The second distance measurement sensor 304 detects the distance by, for example, a change in the optical axis.

[0042] The imaging device 3 includes a thermopile 305. The thermopile 305 is a sensor that detects whether the drawer 26A is open or closed. The thermopile 305 is installed at the front end of the bottom surface 31A of the imaging member 31. The thermopile 305 outputs a detection value indicating the detected temperature. The thermopile 305 will be further described with reference to FIG. 9.

[0043] The thermopile 305 is disposed on the upper surface of the main box 20 of the refrigerator 2. In other words, as described with reference to FIGS. 5 to 7, the thermopile 305 is installed at the front end of the bottom surface 31A of the imaging member 31 that constitutes the imaging device 3. The thermopile 305 also detects the temperature of an object disposed on the front side of the refrigerator 2.

[0044] The thermopile 305 detects the temperature of an object placed within a detection range of, for example, a detection angle θ1. The detection range extends vertically along the front surface of the refrigerator 2. The front surface of the refrigerator 2 includes, for example, the front surface of the door 21A in the closed state and the front surface of the drawer 26A in the closed state. The detection range extends generally parallel to the Z axis. The detection angle θ1 is, for example, 10 degrees. The thermopile 305 is installed at the front end of the bottom surface 31A of the imaging member 31 so that its detection range is tilted forward, i.e., in the positive direction of the Y axis. The thermopile 305 is installed at the front end of the bottom surface 31A of the imaging member 31 so that its detection range is tilted in the negative direction of the X axis so that the detection range of the thermopile 305 includes the center position of the refrigerator 2 in the left-right direction. In other words, the thermopile 305 is preferably installed so that it includes the center position of the refrigerator 2 in the left-right direction. When no object is placed between the thermopile 305 and the floor surface within the detection range of the thermopile 305, the thermopile 305 detects the temperature of a partial area of ​​the floor surface. The temperature of the area is, for example, 20°C.

[0045] When the hand of the user P is placed between the thermopile 305 and the floor surface within the detection range of the thermopile 305, the thermopile 305 detects the temperature of a part of the upper surface of the hand (here, the back of the hand). The temperature of the part is, for example, 36°C.

[0046] When the state transitions from one in which no object is placed within the detection range of thermopile 305 to one in which user P's hand is placed within the detection range of thermopile 305, the detection value of thermopile 305 indicates a temperature rise of, for example, 16°C (=36°C-20°C). Since the detection value of thermopile 305 indicates a temperature rise equal to or greater than first threshold value TH1, open / close determination unit 353 determines that drawer 26A has changed from a closed state to an open state. Furthermore, since open / close determination unit 353 determines that drawer 26A has entered the open state, imaging control unit 354 causes wide-angle camera 301 and narrow-angle camera 302 to start capturing images.

[0047] When the bottom surface of drawer 25A of vegetable compartment 25 is located between thermopile 305 and the floor within the detection range of thermopile 305, thermopile 305 detects the temperature of area BA, which is a part of the bottom surface of drawer 25A. The temperature of area BA is, for example, 5°C.

[0048] For example, if the state transitions from one in which no object is located within the detection range of thermopile 305 to one in which the bottom surface of drawer 25A is located within the detection range of thermopile 305, the detection value of thermopile 305 indicates a temperature drop of, for example, 15°C (=20°C-5°C). Since the detection value of thermopile 305 indicates a temperature drop equal to or greater than second threshold value TH2, open / close determination unit 353 determines that drawer 26A has changed from a closed state to an open state. Note that second threshold value TH2 is, for example, 10°C. Furthermore, since open / close determination unit 353 determines that drawer 26A has entered the open state, imaging control unit 354 causes wide-angle camera 301 and narrow-angle camera 302 to start capturing images.

[0049] The image capture device 3 is equipped with a buzzer 306. The buzzer 306 alerts the user to changes in the state of the refrigerator 2. For example, when at least one of the right door 21B and the left door 21C is set to a predetermined opening / closing angle θ, the buzzer 306 outputs a buzzer sound. The predetermined opening / closing angle θ is, for example, 80 to 100 degrees. Whether the predetermined opening / closing angle θ is reached is determined based on the captured image PA1 captured by the narrow-angle camera 302. The captured image PA1 includes an image of a marker MK1. The captured image PA1 is stored in the state memory unit 364. For example, when the right door 21B is set from a closed state to the predetermined opening / closing angle θ, the buzzer 306 outputs a buzzer sound. Furthermore, for example, when the right door 21B is set from the predetermined opening / closing angle θ to the closed state, the buzzer 306 outputs a buzzer sound. The marker MK1 will be further described with reference to FIGS. 10 and 11 .

[0050] Furthermore, for example, when the first marker MK51 placed in the drawer 25A changes from a state where it is not in the predetermined position to a state where it is in the predetermined position, the buzzer 306 outputs a buzzer sound. Whether the first marker MK51 is in the predetermined position is determined based on the captured image PA2 captured by the narrow-angle camera 302. The captured image PA2 includes an image of the first marker MK51. The captured image PA2 is stored in the state memory unit 364. Furthermore, for example, when the first marker MK51 placed in the drawer 25A changes from a state where it is in the predetermined position to a state where it is not in the predetermined position, the buzzer 306 outputs a buzzer sound. The first marker MK51 will be described further with reference to Figures 12 to 14. Furthermore, when there is no need to distinguish between the marker MK1, the first marker MK51, and the second marker MK52, they will be referred to as marker MK. In other words, the markers MK include a marker MK1, a first marker MK51, and a second marker MK52. The second marker MK52 will be further described with reference to FIG.

[0051] The photographing device 3 includes an illuminance sensor 307. The illuminance sensor 307 is a sensor for adjusting the exposure of the wide-angle camera 301 and the narrow-angle camera 302. The illuminance sensor 307 outputs a detected value. The illuminance sensor 307 is installed at the front end of the photographing member 31.

[0052] The photographing device 3 is equipped with a second LED 308 and a third LED 309. The second LED 308 is installed at the right front end of the bottom surface 31A of the photographing member 31. The third LED 309 is installed at the left front end of the bottom surface 31A of the photographing member 31. The second LED 308 and the third LED 309 illuminate the refrigerator 2 to which the photographing device 3 is attached from above.

[0053] As shown in FIG. 7, the image capturing device 3 includes a first control board 311 and a second control board 312 .

[0054] The first control board 311 is a board that controls various sensors equipped in the imaging device 3 and the power supply of the imaging device 3. The first control board 311 is installed inside the imaging device 3. The power button 32, the imaging start button 32A, the first LED 33, the second LED 308, the third LED 309, the first distance measurement sensor 303, the second distance measurement sensor 304, the thermopile 305, the buzzer 306, and the illuminance sensor 307 are connected to the first control board 311. The first control board 311 controls these devices. The first control board 311 is installed behind the second control board 312 and the first wireless communication module 313. The first control board 311 may be configured as a board that controls the various sensors equipped in the imaging device 3 and a board that controls the power supply of the imaging device 3.

[0055] The second control board 312 is a board that controls the wide-angle camera 301 and the narrow-angle camera 302, and also controls the first wireless communication module 313. The second control board 312 is installed inside the imaging device 3. The wide-angle camera 301, the narrow-angle camera 302, and the first wireless communication module 313 are connected to the second control board 312. The second control board 312 controls these devices. The second control board 312 is installed in front of and alongside the first control board 311.

[0056] The first wireless communication module 313 is a device equipped with hardware such as an antenna and a wireless communication circuit. The first wireless communication module 313 is installed inside the photographing member 31 to the left of the second control board 312. The first wireless communication module 313 is installed in the photographing device 3 so as to be located forward of the opening 201 of the main box 20 when the photographing device 3 is attached to the refrigerator 2. More specifically, the first wireless communication module 313 is installed forward of the restricting member 34 in the photographing device 3.

[0057] As shown in FIG. 7, when the image capturing device 3 is attached to the refrigerator 2, a metal plate 314 is placed between the first wireless communication module 313 and the second wireless communication module 206 in the left-right direction.

[0058] Fig. 8 is a block diagram showing the configuration of the imaging device 3. As shown in Fig. 8, the first control board 311 includes a first processor 320 such as a CPU (Central Processing Unit), a first memory 330, a first communication module 340, and an interface circuit to which other devices and sensors are connected.

[0059] The first memory 330 is a storage device that stores programs and data executed by the first processor 320. The first memory 330 is configured by a non-volatile storage device such as a read-only memory (ROM). The first memory 330 may also include a volatile storage device that configures a work area for the first processor 320, such as a random access memory (RAM). The first memory 330 stores data processed by the first processor 320 and a control program 331 executed by the first processor 320.

[0060] The first communication module 340 includes hardware such as a communication circuit and a connector that conforms to a predetermined wired communication standard, and communicates with the second control board 312 under the control of the first processor 320 .

[0061] The first processor 320 reads and executes the control program 331 to turn on and off the power of the imaging device 3 in response to the operation of the power button 32. The first processor 320 also reads and executes the control program 331 to turn on and off the first LED 33, the second LED 308, and the third LED 309. The first processor 320 also reads and executes the control program 331 to transmit detection values ​​output by the sensors connected to the first control board 311 to the second control board 312 via the first communication module 340.

[0062] The second control board 312 includes a second processor 350 such as a CPU, a second memory 360, a second communication module 370, and an interface circuit to which other devices and sensors are connected. The second processor 350 corresponds to an example of a "processor."

[0063] The second memory 360 is a storage device that stores programs and data executed by the second processor 350. The second memory 360 is configured with a non-volatile storage device such as a ROM. The second memory 360 may also include a volatile storage device that configures a work area for the second processor 350, such as a RAM. The second memory 360 stores data processed by the second processor 350, a control program 361 executed by the second processor 350, and an image capture device ID (Identification) 362. The image capture device ID 362 is information that uniquely identifies the image capture device 3.

[0064] The second communication module 370 includes hardware such as a communication circuit and a connector that conforms to a predetermined wired communication standard, and communicates with the first control board 311 under the control of the second processor 350 .

[0065] The second processor 350 reads and executes the control program 361 stored in the second memory 360, thereby functioning as a first communication control unit 351, a second communication control unit 352, an open / close determination unit 353, an imaging control unit 354, a recording control unit 355, a first determination unit 355A, a second determination unit 355B, a determination unit 355C, and a generation unit 356. The second processor 350 also reads and executes the control program 361 stored in the second memory 360, thereby causing the second memory 360 to function as a state storage unit 364. The control program 361 corresponds to an example of a "control program" in the present disclosure.

[0066] The first communication control unit 351 communicates with the food management server 5 via the first wireless communication module 313 .

[0067] The second communication control unit 352 communicates with the first control board 311 via the second communication module 370. The second communication control unit 352 receives detection values ​​of various sensors connected to the first control board 311 from the first control board 311.

[0068] The open / close determination unit 353 determines whether the right door 21B is open or closed based on the detection value of the first distance measurement sensor 303 received by the second communication control unit 352. The open / close determination unit 353 determines that the right door 21B is open when the second communication control unit 352 receives a detection value different from the detection value corresponding to the closed state. For example, the open / close determination unit 353 determines that the right door 21B is open when the detection value of the first distance measurement sensor 303 is equal to or greater than a predetermined value. On the other hand, the open / close determination unit 353 determines that the right door 21B is closed when the second communication control unit 352 receives a detection value corresponding to the closed state. For example, the open / close determination unit 353 determines that the right door 21B is closed when the detection value of the second distance measurement sensor 304 is less than the predetermined value.

[0069] The open / close determining unit 353 determines whether the left door 21C is open or closed based on the detection value of the second distance measuring sensor 304 by the second communication control unit 352, in the same manner as the determination for the right door 21B.

[0070] The open / close determination unit 353 determines whether the drawer 26A is open or closed based on the detection value of the thermopile 305 received by the second communication control unit 352. For example, when the second communication control unit 352 receives a detection value of the thermopile 305 indicating a temperature increase equal to or greater than a first threshold value TH1, the open / close determination unit 353 determines that the drawer 26A has changed from the closed state to the open state. The first threshold value TH1 is, for example, 5°C. Furthermore, when the second communication control unit 352 receives a detection value of the thermopile 305 indicating a temperature decrease equal to or greater than a second threshold value TH2, the open / close determination unit 353 determines that the drawer 26A has changed from the closed state to the open state. The second threshold value TH2 is, for example, 5°C.

[0071] Furthermore, the open / close determination unit 353 determines that the drawer 26A is closed, for example, when the second communication control unit 352 receives a detection value from the thermopile 305 indicating that a temperature within a predetermined range has continued for a predetermined period of time or longer. The predetermined temperature range is, for example, 15°C or higher and 30°C or lower. The predetermined period is, for example, 5 seconds. The predetermined temperature range corresponds to the temperature of the floor.

[0072] When the open / close determination unit 353 determines that the door 21A or the drawer 26A has entered the open state, the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start photographing. When the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start photographing, the second communication control unit 352 transmits lighting request information to the first control board 311. The lighting request information is information requesting lighting of the second LED 308 and the third LED 309. Upon receiving the lighting request information, the first processor 320 lights up the second LED 308 and the third LED 309 to notify the user P that photographing has started. Furthermore, when the open / close determination unit 353 determines that the door 21A or the drawer 26A has entered the open state, the first processor 320 may also light up the second LED 308 and the third LED 309.

[0073] Furthermore, when the opening / closing determination unit 353 determines that the door 21A and the drawer 26A are closed, the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end photography. When the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end photography, the second communication control unit 352 transmits light-off request information to the first control board 311. The light-off request information is information requesting that the second LED 308 and the third LED 309 be turned off. Upon receiving the light-off request information, the first processor 320 turns off the second LED 308 and the third LED 309 and notifies the user P that photography has ended. Furthermore, when the opening / closing determination unit 353 determines that the door 21A or the drawer 26A is closed, the first processor 320 may turn off the second LED 308 and the third LED 309.

[0074] Furthermore, when the open / close determination unit 353 determines that the door 21A or the drawer 26A has entered the open state, the photography control unit 354 may cause the wide-angle camera 301 and the narrow-angle camera 302 to start photographing, and then may cause the wide-angle camera 301 and the narrow-angle camera 302 to stop photographing after a predetermined time (e.g., one minute) has elapsed. Furthermore, when the open / close determination unit 353 determines that the door 21A or the drawer 26A has entered the open state, the photography control unit 354 may cause the wide-angle camera 301 and the narrow-angle camera 302 to start photographing, and then may extend the predetermined time if the open / close determination unit 353 determines that the door 21A or the drawer 26A has entered the open state. For example, when the open / close determination unit 353 determines that the door 21A has entered the open state, the photography control unit 354 may cause the wide-angle camera 301 and the narrow-angle camera 302 to start photographing, and then may extend the predetermined time if the open / close determination unit 353 determines that the drawer 26A has entered the open state. In this case, a captured image of the drawer 26A can be reliably generated.

[0075] After the imaging control unit 354 causes the narrow-angle camera 302 to start imaging, the recording control unit 355 determines whether the door 21A is at a predetermined opening / closing angle θ based on the image PA1 captured by the narrow-angle camera 302. The predetermined opening / closing angle θ is, for example, 80 degrees to 100 degrees. If the recording control unit 355 determines that the door 21A is at the predetermined opening / closing angle θ, the recording control unit 355 starts recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302. That is, the recording control unit 355 starts recording the image data 363 in the second memory 360. The image data 363 includes first image data DV1 representing the image captured by the wide-angle camera 301 and second image data DV2 representing the image captured by the narrow-angle camera 302. In the following description, the state in which the door 21A is at the predetermined opening / closing angle θ may be referred to as "satisfying the recording start condition." The "recording start condition" is a condition under which the recording control unit 355 starts recording the results of image capture by the wide-angle camera 301 and the narrow-angle camera 302. After the image capture control unit 354 causes the narrow-angle camera 302 to start image capture, the recording control unit 355 may start recording the results of image capture by the wide-angle camera 301 and the narrow-angle camera 302 when the narrow-angle camera 302 is able to capture an image of the first marker MK1.

[0076] Furthermore, when the determination of the recording control unit 355 changes from a state in which the door 21A is not at the predetermined opening / closing angle θ to a state in which the door 21A is at the predetermined opening / closing angle θ, the second communication control unit 352 transmits information requesting the output of a buzzer sound to the first control board 311. Furthermore, when the determination of the recording control unit 355 changes from a state in which the door 21A is at the predetermined opening / closing angle θ to a state in which the door 21A is not at the predetermined opening / closing angle θ, the second communication control unit 352 transmits information requesting the output of a buzzer sound to the first control board 311. The information requesting the output of a buzzer sound is information requesting the buzzer 306 to output a buzzer sound. Upon receiving the information requesting the output of a buzzer sound, the first processor 320 causes the buzzer 306 to output a buzzer sound, thereby notifying the user P that the door 21A is at the predetermined opening / closing angle θ.

[0077] After the imaging control unit 354 causes the narrow-angle camera 302 to start imaging, the recording control unit 355 determines whether the drawer 26A is at a predetermined drawer level based on the image PA2 captured by the narrow-angle camera 302. The predetermined drawer level is, for example, 80% to 100%. If the recording control unit 355 determines that the drawer 26A is at the predetermined drawer level, the recording control unit 355 starts recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302. That is, the recording control unit 355 starts recording the image data 363 in the second memory 360. The image data 363 includes first image data DV1 representing the image captured by the wide-angle camera 301 and second image data DV2 representing the image captured by the narrow-angle camera 302. In the following description, the state in which the drawer 26A is at the predetermined drawer level may be referred to as "satisfying the recording start condition." The recording start condition is that door 21A is at a predetermined opening / closing angle θ or drawer 26A is pulled out to a predetermined extent. After imaging control unit 354 causes narrow-angle camera 302 to start imaging, recording control unit 355 may start recording the results of imaging by wide-angle camera 301 and narrow-angle camera 302 if narrow-angle camera 302 can capture an image of second marker MK2.

[0078] Furthermore, when the determination of the recording control unit 355 changes from a state in which drawer 26A is not at the predetermined drawer amount to a state in which drawer 26A is at the predetermined drawer amount, the second communication control unit 352 transmits information requesting the output of a buzzer sound to the first control board 311. Furthermore, when the determination of the recording control unit 355 changes from a state in which drawer 26A is at the predetermined drawer amount to a state in which drawer 26A is not at the predetermined drawer amount, the second communication control unit 352 transmits information requesting the output of a buzzer sound to the first control board 311. The information requesting the output of a buzzer sound is information requesting the buzzer 306 to output a buzzer sound. When the first processor 320 receives the information requesting the output of a buzzer sound, it causes the buzzer 306 to output a buzzer sound, thereby notifying the user P that the marker MK is at the predetermined position.

[0079] When the open / close determination unit 353 determines that the door 21A and the drawer 26A are closed, the recording control unit 355 ends recording of the image capture results of the wide-angle camera 301 and the narrow-angle camera 302. Note that the recording control unit 355 may end recording of the image capture results of the wide-angle camera 301 and the narrow-angle camera 302 a predetermined time (e.g., one minute) after the narrow-angle camera 302 is no longer able to capture an image of the second marker MK2. Furthermore, the imaging control unit 354 ends imaging by the wide-angle camera 301 and the narrow-angle camera 302 after the recording control unit 355 has ended recording of the image capture results of the wide-angle camera 301 and the narrow-angle camera 302.

[0080] In the first embodiment, when recording control unit 355 determines that the recording start condition is satisfied, recording control unit 355 starts recording the images captured by wide-angle camera 301 and narrow-angle camera 302, but the present disclosure is not limited to this. For example, when open / close determination unit 353 determines that door 21A or drawer 26A has entered the open state, recording control unit 355 may start recording the images captured by wide-angle camera 301 and narrow-angle camera 302. In this case, recording of images captured by narrow-angle camera 302 can be started at the appropriate timing with simple processing.

[0081] The state storage unit 364 stores a captured image PA1 captured by the narrow-angle camera 302 when the opening / closing angle of the door 21A is a predetermined opening / closing angle θ. The captured image PA1 includes an image of a marker MK1. The predetermined opening / closing angle θ is, for example, 80 degrees to 100 degrees. The marker MK1 will be described with reference to FIGS. 10 and 11.

[0082] The state memory unit 364 stores a captured image PA2 captured by the narrow-angle camera 302 when the drawer 26A is at a predetermined drawer amount. The captured image PA2 includes, for example, an image of the first marker MK51 and an image of the second marker MK52. When the drawer amount is zero, this indicates that the drawer 26A is closed. The larger the predetermined drawer amount, the more widely the drawer 26A is open. A drawer amount of 100% indicates that the drawer 26A is fully open. The predetermined drawer amount is, for example, 80% to 100%.

[0083] First marker MK51 is located at the top end of the front panel that is located at the front end of drawer 26A of storage compartment 26, and the state of image capture by narrow-angle camera 302 changes depending on the amount that drawer 26A is pulled out. Note that the amount that drawer 26A is pulled out refers to the amount that drawer 26A is pulled out relative to storage compartment 26. In embodiment 1, a case will be described where storage compartment 26 is vegetable compartment 25 and drawer 26A is drawer 25A. The first marker MK51 will be further described with reference to FIGS. 12 to 14.

[0084] Drawer 25A has a storage case 25B configured to be able to be pulled out at the upper part of drawer 25A. A second marker MK52 is arranged on the upper surface of the front side of storage case 25B. Storage case 25B and second marker MK52 will be further described with reference to Figures 12 to 14.

[0085] The first marker MK51 includes a first outer frame image MF1 and a first identification image MD1. The first identification image MD1 is formed inside the first outer frame image MF1 and indicates first identification information. The first identification information is, for example, information that identifies the drawer 25A of the vegetable compartment 25. The second marker MK52 includes a second outer frame image MF2 and a second identification image MD2. The second identification image MD2 is formed inside the second outer frame image MF2 and indicates second identification information. The second identification information is, for example, information that identifies the storage case 25B placed in the drawer 25A of the vegetable compartment 25.

[0086] The first determination unit 355A determines whether or not the first outer frame image MF1 and the second outer frame image MF2 can be detected for each of the captured images PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363.

[0087] The second determination unit 355B determines whether or not the first identification image MD1 and the second identification image MD2 can be detected for each of the captured images PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363.

[0088] In the first embodiment, as described with reference to FIG. 14 , the first outer frame image MF1 is easier to detect than the first identification image MD1. Therefore, if the first determination unit 355A determines that the first outer frame image MF1 cannot be detected, the second determination unit 355B determines that the first identification image MD1 cannot be detected. Similarly, the second outer frame image MF2 is easier to detect than the second identification image MD2. Therefore, if the first determination unit 355A determines that the second outer frame image MF2 cannot be detected, the second determination unit 355B determines that the second identification image MD2 cannot be detected.

[0089] The determination unit 355C determines whether the captured image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B, based on the appearance of the first marker MK51 and the second marker MK52 in each captured image PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363. The image of food stored in drawer 25A corresponds to an example of an "image of a drawer." The image of food stored in storage case 25B corresponds to an example of an "image of a storage case."

[0090] In this embodiment, a case will be described in which the "drawer image" is an image of food stored in drawer 25A, but the present disclosure is not limited to this. The "drawer image" may be any image of drawer 25A. For example, if no food is stored in drawer 25A, the "drawer image" is an image of drawer 25A.

[0091] In this embodiment, a case will be described in which the "image of the storage case" is an image of food stored in storage case 25B, but the present disclosure is not limited to this. The "image of the storage case" may be an image of storage case 25B. For example, if no food is stored in storage case 25B, the "image of the storage case" is an image of storage case 25B.

[0092] For example, the determination unit 355C determines whether the captured image PB2 includes an image of food stored in the drawer 25A or an image of food stored in the storage case 25B based on the determination results of the first determination unit 355A and the second determination unit 355B.

[0093] For example, if the first judgment unit 355A determines that each of the first outer frame image MF1 and the second outer frame image MF2 can be detected, and the second judgment unit 355B determines that each of the first identification image MD1 and the second identification image MD2 can be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the storage case 25B.

[0094] Furthermore, for example, if the first judgment unit 355A determines that the first outer frame image MF1 can be detected, and the first judgment unit 355A determines that the second outer frame image MF2 cannot be detected, and the second judgment unit 355B determines that the first identification image MD1 can be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 25A.

[0095] Furthermore, for example, if first determination unit 355A determines that first outer frame image MF1 cannot be detected, determination unit 355C determines that captured image PB2 does not include images of food stored in drawer 25A and food stored in storage case 25B. First determination unit 355A, second determination unit 355B, and determination unit 355C will be further described with reference to FIG. 19 .

[0096] In the first embodiment, the determination unit 355C determines whether the captured image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B based on the determination results of the first determination unit 355A and the second determination unit 355B, but the present disclosure is not limited to this. The determination unit 355C may determine whether the captured image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B based on at least one of the determination results of the first determination unit 355A and the determination results of the second determination unit 355B.

[0097] The generation unit 356 selects a first captured image P1 from a captured image PB1 included in the moving image indicated by the first video recording data DV1 of the video recording data 363, based on the captured image PA1. The first captured image P1 is an image captured when the door 21A is at a predetermined opening / closing angle θ. The generation unit 356 also selects a second captured image P2 from a captured image PB2 included in the moving image indicated by the second video recording data DV2 of the video recording data 363, based on the determination result of the determination unit 355C. The second captured image P2 is an image captured when the drawer 26A is at a predetermined drawer extension. If the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 25A, the generation unit 356 selects the captured image PB2 as the second captured image P2 including an image of food stored in the drawer 25A. If the determination unit 355C determines that the captured image PB2 includes an image of food stored in the storage case 25B, the generation unit 356 selects the captured image PB2 as the second captured image P2 that includes an image of the food stored in the storage case 25B.

[0098] The generation unit 356 then generates management data D1 from the first captured image P1 and the second captured image P2. The management data D1 is data to be uploaded to the food management server 5 and is data for managing the foods stored in the refrigerator 2. The first communication control unit 351 transmits the management data D1 generated by the generation unit 356 to the food management server 5. In this embodiment, a case where the image capture device 3 includes the generation unit 356 will be described, but this is not limiting. For example, the food management server 5 may also include the generation unit 356. In this case, the management data D1 includes the first recorded data DV1 and the second recorded data DV2.

[0099] Note that in the first embodiment, the generation unit 356 selects the first captured image P1 based on the captured image PA1 from the captured image PB1 included in the moving image indicated by the first recorded data DV1 of the recorded data 363, but the present disclosure is not limited to this. The generation unit 356 may select, from the captured images PB1 included in the moving image indicated by the first recorded data DV1 of the recorded data 363, the captured image PB1 that least reflects the user P based on the captured image PA1 as the first captured image P1. In this case, the generation unit 356 further selects the first captured image P1 from the first captured images P1 selected based on the captured image PA1 as follows. That is, the generation unit 356 recognizes the image of the user P by a method such as pattern matching, and selects the captured image PB1 with the smallest area of ​​the image of the recognized user P as the first captured image P1.

[0100] Similarly, in the first embodiment, the generation unit 356 selects the second captured image P2 from the captured image PB2 included in the moving image indicated by the second recorded data DV2 of the recorded data 363 based on the determination result of the determination unit 355C, but this is not limited to this. The generation unit 356 may select, from the captured images PB2 included in the moving image indicated by the second recorded data DV2 of the recorded data 363, the captured image PB2 that least includes the user P based on the determination result of the determination unit 355C as the second captured image P2. The processing of the generation unit 356 will be further described with reference to FIG. 17 .

[0101] [1-1-4. Shooting State of Narrow-Angle Camera] Next, the shooting state of narrow-angle camera 302 will be described with reference to Fig. 9. Fig. 9 illustrates the same X-axis, Y-axis, and Z-axis as those shown in Fig. 2. The X-axis direction corresponds to the width direction of refrigerator 2 and the left-right direction of refrigerator 2. The Y-axis direction corresponds to the front-rear direction of refrigerator 2. The Z-axis direction corresponds to the up-down direction of refrigerator 2.

[0102] Figure 9 is a diagram showing an example of the state of imaging by narrow-angle camera 302 in embodiment 1. As shown in Figure 9, narrow-angle camera 302 is placed on the top surface of main box 20 of refrigerator 2. In other words, as described with reference to Figures 5 to 7, narrow-angle camera 302 is installed at the front end of bottom surface 31A of imaging member 31 that constitutes imaging device 3. Also, as shown in Figure 9, narrow-angle camera 302 images drawer 26A from above the front of refrigerator 2.

[0103] As shown in FIG. 9 , narrow-angle camera 302 captures an image of an object placed within a capture range at a capture angle θ2. The capture range is a range along the front of refrigerator 2. The capture range extends generally parallel to the Z axis. Capture angle θ2 is, for example, 25 degrees. Narrow-angle camera 302 captures an image of food stored inside drawer 26A. Drawer 26A refers to drawers 22A, 23A, 24A, and 25A. Drawers 22A, 23A, 24A, and 25A capable of storing food are installed in ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively. FIG. 9 illustrates a case where drawer 26A is drawer 25A located in vegetable compartment 25.

[0104] As shown by the solid line in FIG. 9, when drawer 25A is pulled out to a position where first marker MK51 is not blocked by drawer 24A, narrow-angle camera 302 can photograph first marker MK51.

[0105] In FIG. 9 , drawer 25A is at its maximum open position. In other words, in FIG. 9 , drawer 25A is at its maximum open position. Also, as shown in FIG. 9 , when drawer 25A is at its maximum open position, first marker MK51 is located at a predetermined position. Therefore, for example, when drawer 25A changes from a closed state to an open state and transitions to the maximum open position shown in FIG. 9 , recording control unit 355 starts recording the images captured by wide-angle camera 301 and narrow-angle camera 302, and second communication control unit 352 transmits buzzer sound output request information to first control board 311. Upon receiving the buzzer sound output request information, first processor 320 causes buzzer 306 to output a buzzer sound, thereby notifying user P that first marker MK51 is at a predetermined position. When the first marker MK51 or the second marker MK2 is detected, the recording control unit 355 may start recording the results of the images captured by the wide-angle camera 301 and the narrow-angle camera 302.

[0106] 9 illustrates the case where drawer 26A is drawer 25A arranged in vegetable compartment 25, but the present disclosure is not limited to this. Drawer 26A may be any of drawers 22A, 23A, 24A, and 25A arranged in ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively.

[0107] As shown by the dashed lines in Figure 9, the drawer 26A of the vegetable compartment 25 has a storage case 25B. The storage case 25B is located in the upper section of the drawer 26A and is configured to be removable. The second marker MK52 is located on the upper surface of the front side of the storage case 25B. The storage case 25B and the second marker MK52 will be described further with reference to Figures 12 to 14.

[0108] 9, the marker MK1 is placed in the refrigerator compartment 21. The narrow-angle camera 302 photographs the marker MK1. The marker MK1 will be further described with reference to FIGS. 10 and 11.

[0109] [1-1-5. Markers placed in refrigerator compartment] The marker MK1 placed in refrigerator compartment 21 will be described with reference to Figs. 10 and 11. Figs. 10 and 11 illustrate the same X-axis, Y-axis, and Z-axis as those shown in Fig. 2. The X-axis direction corresponds to the width direction of refrigerator 2 and the left-right direction of refrigerator 2. The Y-axis direction corresponds to the front-rear direction of refrigerator 2. The Z-axis direction corresponds to the up-down direction of refrigerator 2.

[0110] Fig. 10 is a diagram showing an example of a marker MK1 placed in the refrigerator compartment 21 in the first embodiment. As shown in Fig. 10, the marker MK1 is placed on the hinge cover AS. The left view of Fig. 10 shows a state in which the hinge cover AS is not attached, and the right view of Fig. 10 shows a state in which the hinge cover AS is attached. In the left view of Fig. 10, the hinge cover AS is not attached, so the right door hinge member 204 is exposed. The right door hinge member 204 axially supports the right door 21B so that it can rotate.

[0111] The hinge cover AS is formed in an L-shape in a plan view, and has a photographing surface PF2 formed on its upper surface. A marker MK1 containing identification information indicating the refrigeration compartment 21 is formed on the photographing surface PF2. For example, a sticker with the marker MK1 formed on it is affixed to the photographing surface PF2. The marker MK1 will be further described with reference to FIG. 14 .

[0112] Figure 11 is a perspective view of the hinge cover AS. As shown in Figure 11, a notch AS1 is formed in the hinge cover AS. The notch AS1 fits into the upper part of the frame of the drawer 23A (e.g., the right door hinge member 204). The hinge cover AS is made of, for example, resin. The hinge cover AS may be the same color as the main box body 20 and the hinge, or may be a different color. The hinge cover AS is made of, for example, white.

[0113] The state storage unit 364 stores the captured image PA1 including the marker MK1. The generation unit 356 selects, based on the captured image PA1 stored in the state storage unit 364, a first captured image P1 when the right door 21B is at a predetermined opening / closing angle θ from the captured image PB1 included in the moving image indicated by the first recorded data DV1 of the recorded data 363.

[0114] 10 and 11, the case where the marker MK1 is placed on the right door 21B of the refrigerator compartment 21 has been described, but the present disclosure is not limited to this. The marker MK1 may also be placed on the left door 21C of the refrigerator compartment 21, for example.

[0115] [1-1-6. Markers placed in the vegetable compartment] Next, a first marker MK51 and a second marker MK52 placed in the vegetable compartment 25 will be described with reference to Figs. 12 and 13. Figs. 12 and 13 illustrate the same X-axis, Y-axis, and Z-axis as the X-axis, Y-axis, and Z-axis shown in Fig. 2. The X-axis direction corresponds to the width direction of the refrigerator 2 and the left-right direction of the refrigerator 2. The Y-axis direction corresponds to the front-rear direction of the refrigerator 2. The Z-axis direction corresponds to the up-down direction of the refrigerator 2.

[0116] Fig. 12 is a plan view showing an example of a first marker MK51 placed in the vegetable compartment 25. In Fig. 12, the drawer 25A is pulled out to the maximum opening position. Also, the storage case 25B is placed in the vegetable compartment 25 formed in the main box body 20. Therefore, the storage case 25B is not shown in Fig. 12.

[0117] 12, drawer 25A includes front panel 251A, first storage chamber 252A, partition member 253A, and second storage chamber 254A. Front panel 251A, first storage chamber 252A, partition member 253A, and second storage chamber 254A are arranged in this order from the front side (positive direction of the Y axis) to the rear side (negative direction of the Y axis).

[0118] The front panel 251A is disposed at the front end of the drawer 25A of the vegetable compartment 25. The front panel 251A is formed in a rectangular plate shape. The front surface of the front panel 251A forms part of the front surface of the refrigerator 2. The back surface of the front panel 251A forms the front surface of the first storage compartment 252A. A first marker MK51 is disposed on the upper surface of the front panel 251A. The first marker MK51 is disposed, for example, at the right end of the upper surface of the front panel 251A. Furthermore, for example, the width of the first marker MK1 may be disposed to match the width of the front panel 251A. This allows the area of ​​the front panel 251A to be effectively utilized, and improves the recognition accuracy of the first marker MK51. The first marker MK51 will be further described with reference to FIG. 14 .

[0119] The first storage compartment 252A is disposed on the front side of the drawer 25A of the vegetable compartment 25. Food items such as vegetables are stored in the first storage compartment 252A.

[0120] The partition member 253A divides the drawer 25A into a first storage chamber 252A and a second storage chamber 254A. The partition member 253A is formed in the shape of a rectangular plate and is disposed approximately parallel to the XZ plane. The front surface of the partition member 253A functions as the back surface of the first storage chamber 252A, and the back surface of the partition member 253A functions as the front surface of the second storage chamber 254A.

[0121] The second storage compartment 254A is disposed on the rear side of the drawer 25A of the vegetable compartment 25. Food items such as vegetables are stored in the second storage compartment 254A. The upper surface of the side portion of the second storage compartment 254A functions as a rail when the storage case 25B is slid forward and backward. In other words, the storage case 25B is configured to be movable forward and backward by sliding along the upper surface of the side portion of the second storage compartment 254A. Note that, as shown in FIG. 12 , when the storage case 25B is disposed in the vegetable compartment 25 formed in the main box 20, the upper surface of the side portion of the vegetable compartment 25 functions as a rail.

[0122] Fig. 13 is a plan view showing an example of a first marker MK51 and a second marker MK52 arranged in the vegetable compartment 25. In Fig. 13, the drawer 25A is pulled out to the maximum open position. Also, the storage case 25B is pulled out to the maximum open position. When the storage case 25B is pulled out to the maximum open position, as shown in Fig. 13, the front surface of the storage case 25B is located, for example, directly above the partition member 253A shown in Fig. 12.

[0123] The storage case 25B includes a front member 251B, a third storage chamber 252B, and a pair of sliding members 253B. The front member 251B is disposed at the front end of the third storage chamber 252B. A second marker MK52 is disposed on the upper surface of the front member 251B. The second marker MK52 is disposed, for example, at the right end of the upper surface of the front member 251B. Alternatively, for example, the second marker MK52 may be configured to fit into the front member 251B. This allows for effective use of the area of ​​the front member 251B and improves the recognition accuracy of the second marker MK52. The second marker MK52 will be further described with reference to FIG. 14 .

[0124] The third storage chamber 252B is formed in the storage case 25B. Food items such as vegetables are stored in the third storage chamber 252B.

[0125] The pair of sliding members 253B are formed to protrude outward in the left-right direction from the upper ends of the side members that constitute the third storage chamber 252B. Each of the pair of sliding members 253B is a plate-shaped member that extends in the front-rear direction. The sliding member 253B arranged on the right side of the third storage chamber 252B is formed to protrude rightward from the upper end of the right side member that constitutes the third storage chamber 252B. The sliding member 253B arranged on the left side of the third storage chamber 252B is formed to protrude leftward from the upper end of the left side member that constitutes the third storage chamber 252B. The pair of sliding members 253B are formed integrally with the third storage chamber 252B and slide on rails that are arranged on the upper surfaces of the side portions of the second storage chamber 254A and the vegetable compartment 25 when the storage case 25B is moved in the front-rear direction.

[0126] [1-1-7. Marker Configuration] Next, the configuration of the marker MK will be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of the configuration of the marker MK in embodiment 1. The marker MK includes a marker MK1, a first marker MK51, and a second marker MK52. The marker MK1 is placed in the refrigerator compartment 21. The first marker MK51 is placed in the drawer 25A of the vegetable compartment 25. The second marker MK52 is formed on the storage case 25B of the vegetable compartment 25.

[0127] As shown in FIG. 14 , the marker MK includes a rectangular outer frame image MF and an identification image MD formed within the outer frame image MF. The identification image MD corresponds to identification information indicating each of the refrigeration compartment 21 and the vegetable compartment 25 in the refrigerator 2. The outer frame image MF is formed, for example, in black. For example, the identification image MD of the marker MK1 corresponds to identification information indicating the refrigeration compartment 21. For example, the identification image MD of the first marker MK51 corresponds to identification information indicating the drawer 25A of the vegetable compartment 25. For example, the identification image MD of the second marker MK52 corresponds to identification information indicating the storage case 25B of the vegetable compartment 25. The outer frame image MF of the first marker MK51 corresponds to an example of the first outer frame image MF1. The outer frame image MF of the second marker MK52 corresponds to an example of the second outer frame image MF2. The identification image MD of the first marker MK51 corresponds to an example of the first identification image MD1. The identification image MD of the second marker MK52 corresponds to an example of a second identification image MD2.

[0128] The identification image MD is composed of a first rectangular image PD1 in a first color and a second rectangular image PD2 in a second color different from the first color, and the arrangement pattern of the first rectangular image PD1 and the second rectangular image PD2 indicates the identification information. The first color is, for example, white, and the second color is, for example, black. The size of the first rectangular image PD1 is the same as the size of the second rectangular image PD2, and each of the first rectangular image PD1 and the second rectangular image PD2 is, for example, square. In other words, the width WP1 of the first rectangular image PD1 and the width WP2 of the second rectangular image PD2 are the same as the height HP of the first rectangular image PD1 and the second rectangular image PD2.

[0129] The thickness HF of the outer frame image MF in the short side direction (vertical direction) and the thickness WF of the outer frame image MF in the long side direction (horizontal direction) are, for example, the same as the width WP1 of the first rectangular image PD1, the width WP2 of the second rectangular image PD2, and the height HP of the first rectangular image PD1 and the height HP of the second rectangular image PD2. The thickness HF of the outer frame image MF in the short side direction (vertical direction) and the thickness WF of the outer frame image MF in the long side direction (horizontal direction) are determined based on the resolution of the narrow-angle camera 302 and the distance between the narrow-angle camera 302 and the marker MK when the drawer 25A is pulled out a predetermined amount (e.g., 100%). Furthermore, the thickness HF of the outer frame image MF in the short side direction (vertical direction) and the thickness WF of the outer frame image MF in the long side direction (horizontal direction) are determined based on the imaging range or angle of view of the narrow-angle camera 302. For example, the higher the resolution of the narrow-angle camera 302, the smaller the thickness HF of the outer frame image MF in the long side direction and the thickness WF of the outer frame image MF in the short side direction can be. Furthermore, the longer the distance between the narrow-angle camera 302 and the marker MK when the drawer 25A is pulled out to a predetermined extent (e.g., 100%), the larger the thickness HF of the outer frame image MF in the long side direction and the thickness WF of the outer frame image MF need to be. Furthermore, the larger the imaging range or angle of view of the narrow-angle camera 302, the larger the thickness HF of the outer frame image MF in the long side direction and the thickness WF of the outer frame image MF need to be. For the marker MK shown in FIG. 14 , the thickness HF of the outer frame image MF in the long side direction and the thickness WF of the outer frame image MF in the short side direction are, for example, 6 mm.

[0130] The first rectangular images PD1 and the second rectangular images PD2 are arranged along the longitudinal direction (here, the left-right direction) of the outer frame image MF. In FIG. 14 , five first rectangular images PD1 and four second rectangular images PD2 are arranged as the identification image MD. The first rectangular image PD1 represents, for example, "0," and the second rectangular image PD2 represents, for example, "1." That is, the identification image MD represents a nine-digit binary number. The identification image MD is composed of an information image MDA and a code image MDB. The information image MDA is composed of five rectangular images PD, and the code image MDB is composed of four rectangular images PD. The rectangular image PD is either the first rectangular image PD1 or the second rectangular image PD2. The information image MDA corresponds to, for example, identification information indicating the drawer 25A. The code image MDB represents a code for error correction of the identification information indicated by the information image MDA. The information image MDA is composed of five rectangular images PD and therefore represents a five-digit binary number. That is, the information image MDA represents 2 to the power of 5, or 32 types of identification information. For example, the information image MDA shown in FIG. 14 represents a five-digit binary number, "01010."

[0131] The sizes of the first rectangular image PD1 and the second rectangular image PD2 are determined based on the resolution of the narrow-angle camera 302 and the distance between the narrow-angle camera 302 and the marker MK at a predetermined extension amount (e.g., 80% to 100%) of the drawer 25A. The sizes of the first rectangular image PD1 and the second rectangular image PD2 are also determined based on the imaging range or angle of view of the narrow-angle camera 302. For example, the higher the resolution of the narrow-angle camera 302, the smaller the sizes of the first rectangular image PD1 and the second rectangular image PD2 can be. Furthermore, the longer the distance between the narrow-angle camera 302 and the marker MK at a predetermined extension amount (e.g., 80% to 100%) of the drawer 25A, the larger the sizes of the first rectangular image PD1 and the second rectangular image PD2 must be. Furthermore, the larger the imaging range or angle of view of the narrow-angle camera 302, the larger the sizes of the first rectangular image PD1 and the second rectangular image PD2 must be. In the marker MK shown in FIG. 14, the size of the first rectangular image PD1 and the size of the second rectangular image PD2 are squares with sides of 6 mm.

[0132] 14 illustrates a case where the outer frame image MF is formed in black, but is not limited to this. For example, the outer frame image MF may be formed in blue. Also, a case where the first color is white and the second color is black is illustrated, but is not limited to this. For example, the first color may be yellow and the second color may be blue.

[0133] As described with reference to Fig. 13, a first marker MK51 is placed on the upper surface of the front panel 251A of the drawer 25A. Also, as shown in Fig. 13, a second marker MK52 is placed on the upper surface of the front member 251B of the storage case 25B. Each of the first marker MK51 and the second marker MK52 includes a rectangular outer frame image MF and an identification image MD formed inside the outer frame image MF.

[0134] Therefore, the determination unit 355C can determine whether the captured image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B based on the determination results of the first determination unit 355A and the second determination unit 355B. Note that the first determination unit 355A determines whether or not the first outer frame image MF1 and the second outer frame image MF2 can be detected for each captured image PB2 included in the moving image indicated by the second record data DV2 of the record data 363. Furthermore, the second determination unit 355B determines whether or not the first identification image MD1 and the second identification image MD2 can be detected for each captured image PB2 included in the moving image indicated by the second record data DV2 of the record data 363.

[0135] For example, when the captured image PB2 corresponds to the state shown in Fig. 12, the first determination unit 355A determines that the first outer frame image MF1 can be detected but that the second outer frame image MF2 cannot be detected. The second determination unit 355B determines that the first identification image MD1 can be detected but that the second identification image MD2 cannot be detected. The determination unit 355C then determines that the captured image PB2 includes an image of food stored in the drawer 25A.

[0136] 13, the first determination unit 355A determines that the first outer frame image MF1 and the second outer frame image MF2 can be detected. The second determination unit 355B determines that the first identification image MD1 and the second identification image MD2 can be detected. The determination unit 355C then determines that the captured image PB2 includes an image of food stored in the storage case 25B.

[0137] Furthermore, in order to enable the first determination unit 355A to easily detect the first outer frame image MF1 and the second outer frame image MF2 from each of the captured images PB2 included in the moving image indicated by the second video recording data DV2, it is preferable that the colors of the first outer frame image MF1 and the second outer frame image MF2 differ from the color of the inner surface of the vegetable compartment 25 and the color of the food stored in the vegetable compartment 25. Therefore, the color of the outer frame image MF is preferably, for example, black, blue, or the like.

[0138] 15 is a block diagram showing the configuration of the terminal device 4 and the food management server 5. The food management server 5 includes a server processor 50, which is a processor such as a CPU, a server memory 51, and a server communication module 52. The server processor 50 is connected to the server memory 51 and the server communication module 52.

[0139] The server memory 51 is a storage device that stores programs and data executed by the server processor 50. The server memory 51 is configured with a non-volatile storage device. The server memory 51 may also include a non-volatile storage device that forms a work area for the server processor 50. The server memory 51 stores data processed by the server processor 50, a control program 511 executed by the server processor 50, and a food management database (DB) 512. The food management DB 512 has one record for each image capture device 3. This one record includes the image capture device ID 362, image data of the first captured image P1, and image data of the second captured image P2.

[0140] The server communication module 52 includes a communication circuit and connectors that comply with a predetermined communication standard, and communicates with the refrigerator 2, the image capturing device 3, and the terminal device 4 via the network NW.

[0141] Server processor 50 functions as a server communication control unit 501 and a server processing unit 502 by reading and executing control program 511 stored in server memory 51. Server communication control unit 501 communicates with refrigerator 2, image capture device 3, and terminal device 4 via server communication module 52. Server processing unit 502 performs various information processing using image capture device 3 and terminal device 4 as clients. Server processing unit 502 may also use refrigerator 2 as a client.

[0142] [1-1-9. Configuration of Terminal Device] Next, we will explain the configuration of the terminal device 4. The terminal device 4 includes a terminal processor 40, which is a processor such as a CPU, a terminal memory 41, a terminal communication module 42, and a touch panel 43.

[0143] The terminal memory 41 is a storage device that non-volatilely stores programs and data executed by the terminal processor 40. The terminal memory 41 is configured by a non-volatile storage device. The terminal memory 41 may also include RAM that forms a work area for the terminal processor 40. The terminal memory 41 stores data processed by the terminal processor 40, control programs executed by the terminal processor 40, and a food management application 411.

[0144] The terminal communication module 42 includes a communication circuit, a connector, etc. that conform to a predetermined communication standard, and communicates with the food management server 5 under the control of the terminal processor 40. The touch panel 43 includes a display panel and a touch sensor that is provided on top of or integral with the display panel.

[0145] The terminal processor 40 functions as an application execution unit 400. The application execution unit 400 communicates with the food management server 5 via the terminal communication module 42. The application execution unit 400 displays a user interface related to the management of food stored in the refrigerator 2 on the touch panel 43. In the following description, this user interface is referred to as an "application UI." In this embodiment, the display control unit 402 displays the application UI to display images of food stored in the refrigerator 2, etc. The application execution unit 400 receives various inputs from the user P via the application UI.

[0146] [1-2. Operation] Next, the operation of each part of the food management system 1 will be described.

[0147] 16 is a flowchart FA showing the operation of the image capturing device 3. [1-2-1. Operation Related to Recording] First, the operation related to recording of the image capturing device 3 will be described.

[0148] The open / close determination unit 353 determines whether the door 21A or the drawer 26A is in the open state (step SA1). The door 21A includes the right door 21B and the left door 21C. The open / close determination unit 353 determines whether the right door 21B is in the open state or the closed state based on the detection value of the first distance measurement sensor 303. The open / close determination unit 353 also determines whether the left door 21C is in the open state or the closed state based on the detection value of the second distance measurement sensor 304. The open / close determination unit 353 also determines whether the drawer 26A is in the open state or the closed state based on the detection value of the thermopile 305. The drawer 26A refers to the drawers 22A, 23A, 24A, and 25A. Drawers 22A, 23A, 24A, and 25A are arranged in ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively.

[0149] If the open / close determining unit 353 determines that the door 21A or the drawer 26A is not in the open state (step SA1: NO), the second processor 350 returns the process to step SA1.

[0150] If the open / close determination unit 353 determines that the door 21A or the drawer 26A is in the open state (step SA1: YES), the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start photographing (step SA2).

[0151] Next, the recording control unit 355 determines whether the recording start condition is met (step SA3). As described with reference to Fig. 8, the recording start condition is that the door 21A is at a predetermined opening / closing angle θ or that the drawer 26A is pulled out to a predetermined extent. If the recording control unit 355 determines that the recording start condition is not met (step SA3: NO), the determination in step SA3 is performed again.

[0152] If the recording control unit 355 determines that the recording start condition is met (step SA3: YES), the recording control unit 355 starts recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302 (step SA4). Note that the recording control unit 355 may start recording only the images captured by the wide-angle camera 301 if it has captured the first marker MK1. Alternatively, the recording control unit 355 may start recording only the images captured by the narrow-angle camera 302 if it has captured the second marker MK2.

[0153] Next, the open / close determination unit 353 determines whether the door 21A and the drawer 26A are closed (step SA5). If the open / close determination unit 353 determines that the door 21A and the drawer 26A are not closed (step SA5: NO), the open / close determination unit 353 performs the determination in step SA5 again.

[0154] On the other hand, if it is determined that door 21A and drawer 26A are closed (step SA5: YES), recording control unit 355 ends recording of the captured image (step SA6). As a result, recording data 363 is generated in second memory 360.

[0155] Next, the shooting control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end shooting (step SA7).

[0156] Step SA2 in the flowchart FA of FIG. 16 corresponds to an example of a "photography control step."

[0157] [1-2-2. Operations Related to Transmission of Management Data] Next, operations related to transmission of management data D1 will be described. Fig. 17 is a flowchart showing the operations of the camera 3 and the food management server 5. In Fig. 17, flowchart FB shows the operations of the camera 3, and flowchart FC shows the operations of the food management server 5.

[0158] The generation unit 356 determines whether or not the recording data 363 is stored in the second memory 360 (step SB1). If the generation unit 356 determines that the recording data 363 is not stored (step SB1: NO), the generation unit 356 performs the determination of step SB1 again.

[0159] On the other hand, if the generation unit 356 determines that the recording data 363 has been stored (step SB1: YES), it selects the first captured image P1 from the captured image PB1 included in the moving image indicated by the first recording data DV1 of the recording data 363 stored in the second memory 360 based on the captured image PA1 stored in the state memory unit 364 (step SB2).

[0160] Step SB2 will be described in detail below. The first captured image P1 is an image captured when the right door 21B or the left door 21C is open at a predetermined opening / closing angle θ (for example, 80 degrees to 100 degrees).

[0161] The generation unit 356 first determines whether or not an outer frame image MF can be detected from each captured image PB1 included in the moving image indicated by the first recorded data DV1 of the recorded data 363. If the outer frame image MF can be detected from the captured image PB1, the generation unit 356 then determines whether or not identification information indicated by an identification image MD inside the detected outer frame image MF can be read. Next, if the identification information indicated by the identification image MD can be read and the read identification information matches the identification information indicated by the marker MK1 included in the captured image PA1 stored in the state storage unit 364, the generation unit 356 selects the captured image PB1 as the first captured image P1.

[0162] Returning to the explanation of the flowchart FB, the generation unit 356 selects the second captured image P2 from each captured image PB2 included in the moving image indicated by the second recorded image data DV2 of the stored recorded image data 363, based on the second captured image PA2 stored in the state storage unit 364 (step SB3). Step SB3 will be explained with reference to the flowchart FF shown in FIG.

[0163] The generation unit 356 determines whether or not at least one of the first captured image P1 and the second captured image P2 has been selected (step SB4).

[0164] If the generation unit 356 determines that the first captured image P1 and the second captured image P2 could not be selected (step SB4: NO), the generation unit 356 performs the process of step SB7.

[0165] On the other hand, if the generation unit 356 determines that at least one of the first captured image P1 and the second captured image P2 has been selected (step SB4: YES), the generation unit 356 generates management data D1 (step SB5).

[0166] If the first captured image P1 and the second captured image P2 are successfully selected, the generation unit 356 generates management data D1 including image data of the first captured image P1, image data of the second captured image P2, and the photographing device ID 362. If the first captured image P1 is successfully selected but the second captured image P2 is not successfully selected, the generation unit 356 generates management data D1 including image data of the first captured image P1 and the photographing device ID 362. If the second captured image P2 is successfully selected but the first captured image P1 is not successfully selected, the generation unit 356 generates management data D1 including image data of the second captured image P2 and the photographing device ID 362.

[0167] Next, the first communication control unit 351 transmits the management data D1 generated by the generation unit 356 to the food management server 5 (step SB6).

[0168] Next, the generation unit 356 erases the recording data 363 that has been processed in steps SB2 and after from the second memory 360 (step SB7).

[0169] As shown in flowchart FC, the server communication control unit 501 determines whether or not it has received management data D1 from the image capturing device 3 (step SC1). If the server communication control unit 501 determines that it has received management data D1 (step SC1: YES), the server processing unit 502 updates the food management DB 512 based on the management data D1 received by the server communication control unit 501 (step SC2).

[0170] Step SC2 will now be described in detail. The server processing unit 502 identifies a record in the food management DB 512 that includes the photographing device ID 362 included in the management data D1, and updates the image data of the first photographed image P1 in the identified record to the image data of the first photographed image P1 included in the management data D1. The server processing unit 502 identifies a record in the food management DB 512 that includes the photographing device ID 362 included in the management data D1, and updates the image data of the second photographed image P2 in the identified record to the image data of the second photographed image P2 included in the management data D1.

[0171] [1-2-3. Operation Related to the App UI] Next, the operation of the food management system 1 related to the app UI will be described. Fig. 18 is a flowchart showing the operation of the terminal device 4 and the food management server 5. In Fig. 18, flowchart FD shows the operation of the terminal device 4, and flowchart FE shows the operation of the food management server 5.

[0172] As shown in flowchart FD, the application execution unit 400 determines whether to request image data stored in the food management DB 512 from the food management server 5 (step SD1). For example, the terminal communication control unit 401 makes a positive determination in step SD1 when the food management application 411 is launched.

[0173] If the application execution unit 400 determines that image data stored in the food management DB 512 is to be requested (step SD1: YES), the application execution unit 400 transmits request data requesting the image data to the food management server 5 (step SD2). The request data includes the camera ID 362 stored in the terminal memory 41. The terminal device 4 acquires the camera ID 362 at a predetermined timing.

[0174] As shown in flowchart FE, the server communication control unit 501 receives the request data (step SE1). Next, the server processing unit 502 obtains image data corresponding to the image capture device ID 362 included in the request data received by the server communication control unit 501 from the food management DB 512 (step SE2). Next, the server communication control unit 501 transmits the image data obtained by the server processing unit 502 to the terminal device 4 as a response to the request data (step SE3).

[0175] Referring to flowchart FD, the application execution unit 400 receives image data from the food management server 5 (step SD3). Next, the application execution unit 400 displays an application UI including an image represented by the image data received in step SD3 on the touch panel 43 (step SD4).

[0176] [1-2-4. Second Captured Image Selection Process] Next, the "second captured image selection process" will be described with reference to FIG. 19 . The "second captured image selection process" is a process in which the generation unit 356 selects the second captured image P2 based on the determination result of the determination unit 355C. The determination unit 355C determines whether the captured image PB2 includes an image of food stored in the drawer 25A or an image of food stored in the storage case 25B based on the determination results of the first determination unit 355A and the second determination unit 355B. In FIG. 19 , a flowchart FF shows the operation of the image capture device 3.

[0177] As shown in flowchart FF, the generation unit 356 reads a captured image PB2 included in the moving image indicated by the second recorded data DV2 of the recorded data 363 (step SF1). Next, the first determination unit 355A determines whether a first outer frame image MF1 can be detected from the read captured image PB2 (step SF2). If the first determination unit 355A determines that the first outer frame image MF1 cannot be detected (step SF2: NO), the process proceeds to step SF9. If the first determination unit 355A determines that the first outer frame image MF1 can be detected (step SF2: YES), the first determination unit 355A determines whether a second outer frame image MF2 can be detected from the read captured image PB2 (step SF3). If the first determination unit 355A determines that the second outer frame image MF2 cannot be detected (step SF3: NO), the process proceeds to step SF7. If the first determination unit 355A determines that the second outer frame image MF2 can be detected (step SF3: YES), the second determination unit 355B determines whether the first identification image MD1 can be detected for the read-out captured image PB2 (step SF4). Note that the second determination unit 355B determines that the first identification image MD1 can be detected if the second determination unit 355B can detect first identification information corresponding to the first identification image MD1 for the captured image PB2 and if the detected first identification information matches the identification information indicated by the first marker MK51 included in the captured image PA2 stored in the state storage unit 364.

[0178] If the second determination unit 355B determines that the first identification image MD1 cannot be detected (step SF4: NO), the process proceeds to step SF9. If the second determination unit 355B determines that the first identification image MD1 can be detected (step SF4: YES), the second determination unit 355B determines whether the second identification image MD2 can be detected for the read-out photographed image PB2 (step SF5). Note that the second determination unit 355B determines that the second identification image MD2 can be detected if it can detect second identification information corresponding to the second identification image MD2 for the photographed image PB2 and if the detected second identification information matches the identification information indicated by the second marker MK52 included in the photographed image PA2 stored in the state storage unit 364. If the second determination unit 355B determines that the second identification image MD2 cannot be detected (step SF5: NO), the process proceeds to step SF9. If the second determination unit 355B determines that the second identification image MD2 can be detected (step SF5: YES), the process proceeds to step SF6. Then, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the storage case 25B, and the generation unit 356 selects the captured image PB2 as the second captured image P2 that includes an image of food stored in the storage case 25B (step SF6). Then, the process proceeds to step SF9.

[0179] If the first determination unit 355A determines that the second outer frame image MF2 cannot be detected (step SF3: NO), the second determination unit 355B determines whether the first identification image MD1 can be detected in the read captured image PB2 (step SF7). If the second determination unit 355B determines that the first identification image MD1 cannot be detected (step SF7: NO), the process proceeds to step SF9. If the second determination unit 355B determines that the first identification image MD1 can be detected (step SF7: YES), the process proceeds to step SF8. Then, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 25A, and the generation unit 356 selects the captured image PB2 as the second captured image P2 including an image of food stored in the drawer 25A (step SF8). Then, the process proceeds to step SF9.

[0180] Next, the generation unit 356 determines whether all captured images PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363 have been read (step SF9). If the generation unit 356 determines that all captured images PB2 have been read (step SF9: YES), the process returns to step SB4 in FIG. 17. If the generation unit 356 determines that not all captured images PB2 have been read (step SF9: NO), the process returns to step SF1. The generation unit 356 reads out any captured images PB2 that have not been read out from among the captured images PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363 (step SF1). Then, the process from step SF2 onwards is executed again.

[0181] Steps SF6 and SF8 in the flowchart FF of FIG. 19 correspond to an example of a "determining step."

[0182] As explained with reference to Figure 19, the generation unit 356 selects a second captured image P2 including an image of food stored in the storage case 25B and a second captured image P2 including an image of food stored in the drawer 25A based on the determination result of the determination unit 355C.

[0183] In the first embodiment, the case where the image capturing device 3 is configured as a separate entity from the refrigerator 2 has been described, but the present disclosure is not limited to this. The image capturing device 3 may be configured as an integral part of the refrigerator 2. In other words, the image capturing device 3 may be configured as a part of the refrigerator 2.

[0184] In the first embodiment, the determination unit 355C determines whether the captured image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B based on the determination results of the first determination unit 355A and the second determination unit 355B. However, the present disclosure is not limited to this. It is sufficient that the determination unit 355C determines whether the captured image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B based on at least one of the determination results of the first determination unit 355A and the determination results of the second determination unit 355B. The determination unit 355C may also determine whether the captured image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B based on the determination result of the first determination unit 355A. For example, if the first determination unit 355A determines that the first outer frame image MF1 can be detected but determines that the second outer frame image MF2 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 25A. Furthermore, if the first determination unit 355A determines that both the first outer frame image MF1 and the second outer frame image MF2 can be detected, the determination unit 355C may determine whether the captured image PB2 includes an image of food stored in the drawer 25A or an image of food stored in the storage case 25B based on the determination result of the second determination unit 355B. For example, if the second determination unit 355B determines that the first identification image MD1 can be detected but determines that the second identification image MD2 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 25A. Also, for example, if the second judgment unit 355B determines that the first identification image MD1 and the second identification image MD2 can be detected, the determination unit 355C determines that the image includes an image of food stored in the storage case 25B.

[0185] 1 to 19 , photography system 7 in embodiment 1 photographs refrigerator 2 equipped with vegetable compartment 25 having drawer 25A with an opening formed on the top surface thereof, and includes narrow-angle camera 302 arranged on top surface 20A of refrigerator 2, first marker MK51 arranged on drawer 25A, drawer 25A having storage case 25B configured to be able to be pulled out in an upper section of drawer 25A, and second marker MK52 arranged on storage case 25B, photography control unit 354 that photographs vegetable compartment 25 using narrow-angle camera 302 and generates photographed image PB2, and determination unit 355C that determines whether photographed image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B based on the appearance of first marker MK51 and second marker MK52 in photographed image PB2.

[0186] According to this, based on the appearance of the first marker MK51 and the second marker MK52 in the photographed image PB2, it is determined whether the photographed image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B. Therefore, it is possible to properly determine whether the photographed image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B. Therefore, the food stored in refrigerator 2 can be managed with high accuracy.

[0187] In the photography system 7, the first marker MK51 is placed on the upper surface of the front side of the drawer 25A, and the second marker MK52 is placed on the upper surface of the front side of the storage case 25B.

[0188] In this way, first marker MK51 is placed on the upper surface of the front side of drawer 25A, and second marker MK52 is placed on the upper surface of the front side of storage case 25B, so narrow-angle camera 302 can easily capture first marker MK51 and second marker MK52. Therefore, it is possible to properly determine whether captured image PB2 includes an image of food stored in drawer 25A or in storage case 25B. Therefore, food stored in refrigerator 2 can be managed with high accuracy.

[0189] In addition, in the photography system 7, the first marker MK51 includes a first outer frame image MF1, and the second marker MK52 includes a second outer frame image MF2, and is provided with a first judgment unit 355A that judges whether or not each of the first outer frame image MF1 and the second outer frame image MF2 can be detected from the photographed image PB2, and when the first judgment unit 355A judges that each of the first outer frame image MF1 and the second outer frame image MF2 can be detected, the determination unit 255C determines that the photographed image PB2 includes an image of food stored in the storage case 25B, and when the first judgment unit 355A determines that the first outer frame image MF1 can be detected but the second outer frame image MF2 cannot be detected, the determination unit 355C determines that the photographed image PB2 includes an image of food stored in the drawer 25A.

[0190] According to this, when the first determination unit 355A determines that both the first outer frame image MF1 and the second outer frame image MF2 can be detected, the determination unit 255C determines that the captured image PB2 includes an image of food stored in the storage case 25B. When the first determination unit 355A determines that the first outer frame image MF1 can be detected but the first determination unit 355A determines that the second outer frame image MF2 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 25A. This makes it possible to accurately determine whether the captured image PB2 includes an image of food stored in the drawer 25A or in the storage case 25B. This allows for accurate management of the food stored in the refrigerator 2.

[0191] Furthermore, in the photographing system 7, the first marker MK51 is formed inside the first outer frame image MF1 and includes a first identification image MD1 indicating the first identification information, the second marker MK52 is formed inside the second outer frame image MF2 and includes a second identification image MD2 indicating the second identification information, and a second determination unit 355B is provided that determines whether or not each of the first identification image MD1 and the second identification image MD2 can be detected from the photographed image PB2, and the first identification image MD1 and If the second judgment unit 355B determines that each of the second identification images MD2 can be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the storage case 25B, and if the second judgment unit 355B determines that the first identification image MD1 can be detected but the second identification image MD2 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 25A.

[0192] According to this, when the second determination unit 355B determines that both the first identification image MD1 and the second identification image MD2 can be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the storage case 25B. When the second determination unit 355B determines that the first identification image MD1 can be detected but the second identification image MD2 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 25A. Therefore, it is possible to properly determine whether the captured image PB2 includes an image of food stored in the drawer 25A or in the storage case 25B. This allows for accurate management of the food stored in the refrigerator 2.

[0193] Furthermore, the photographing method in embodiment 1 is a photographing method of photographing system 7, in which photographing system 7 photographs refrigerator 2 equipped with vegetable compartment 25 having drawer 25A with an opening formed on the top surface thereof, and includes narrow-angle camera 302 arranged on top surface 20A of refrigerator 2, first marker MK51 arranged on drawer 25A, drawer 25A having storage case 25B configured to be able to be pulled out in an upper portion of drawer 25A, and second marker MK52 arranged on storage case 25B, and includes a photographing control step of photographing vegetable compartment 25 using narrow-angle camera 302 to generate photographed image PB2, and a determination step of determining whether photographed image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B, based on the appearance of first marker MK51 and second marker MK52 in photographed image PB2.

[0194] As a result, the photographing method according to the first embodiment has the same effect as the photographing system 7 according to the first embodiment.

[0195] Furthermore, control program 361 in embodiment 1 is a control program 361 for photography system 7 that photographs refrigerator 2 equipped with vegetable compartment 25 having drawer 25A with an opening formed on the top surface thereof, and that includes narrow-angle camera 302 arranged on top surface 20A of refrigerator 2, first marker MK51 arranged on drawer 25A, drawer 25A having storage case 25B configured to be able to be pulled out in the upper part of drawer 25A, second marker MK52 arranged on storage case 25B, and second processor 350 that controls narrow-angle camera 302. Control program 361 causes second processor 350 to function as photography control unit 354 that photographs vegetable compartment 25 using narrow-angle camera 302 and generates photographed image PB2, and as determination unit 355C that determines whether photographed image PB2 includes an image of food stored in drawer 25A or an image of food stored in storage case 25B based on the appearance of first marker MK51 and second marker MK52 in photographed image PB2.

[0196] According to this, the control program 361 in the first embodiment achieves the same effects as the imaging system 7 in the first embodiment.

[0197] (Embodiment 2) [2-1. Configuration] The following explanation will mainly focus on the differences from embodiment 1, which was described with reference to Figures 1 to 19. The configuration of food management system 1 shown in Figure 1 and the configuration of refrigerator 2 shown in Figure 2 are the same as those in embodiment 1. Furthermore, the configuration of imaging device 3 shown in Figures 3 and 4 is the same as that in embodiment 1. In embodiment 2, a case will be described in which the "storage compartment" of the present disclosure is freezer compartment 24.

[0198] As shown in FIGS. 5 to 7 , narrow-angle camera 302 constitutes part of photography system 7 in Embodiment 2. Photography system 7 in Embodiment 2 includes narrow-angle camera 302, third marker MK41, fourth marker MK42, photography control unit 354, and determination unit 355C. In other words, photography system 7 includes refrigerator 2 equipped with third marker MK41 and fourth marker MK42, and photography device 3 equipped with narrow-angle camera 302, photography control unit 354, and determination unit 355C. The third marker MK41 and fourth marker MK42 will be described with reference to FIGS. 9 , 14 , and 20 to 23 . The determination unit 355C will be described with reference to FIG. 8 .

[0199] As shown in FIGS. 5-7 , the photographing device 3 includes a buzzer 306. For example, when the state of the third marker MK41 placed in the drawer 24A changes from not being in the predetermined position to being in the predetermined position, the buzzer 306 outputs a buzzer sound. Whether the third marker MK41 is in the predetermined position is determined based on the photographed image PA2 captured by the narrow-angle camera 302. The photographed image PA2 includes an image of the third marker MK41. The photographed image PA2 is stored in the state memory unit 364. Furthermore, for example, when the state of the third marker MK41 placed in the drawer 24A changes from being in the predetermined position to not being in the predetermined position, the buzzer 306 outputs a buzzer sound. Furthermore, when there is no need to distinguish between the marker MK1, the third marker MK41, and the fourth marker MK42, they will be referred to as markers MK. In other words, the markers MK include a marker MK1, a third marker MK41, and a fourth marker MK42. The third marker MK41 and the fourth marker MK42 will be further described with reference to Figures 9, 14, and 20-23.

[0200] 8 , the second processor 350 reads and executes a control program 361 stored in the second memory 360, thereby functioning as a first communication control unit 351, a second communication control unit 352, an open / close determination unit 353, an imaging control unit 354, a recording control unit 355, a first determination unit 355A, a second determination unit 355B, a determination unit 355C, and a generation unit 356. The second processor 350 also reads and executes the control program 361 stored in the second memory 360, thereby causing the second memory 360 to function as a state storage unit 364. The first communication control unit 351, the second communication control unit 352, the open / close determination unit 353, the imaging control unit 354, and the recording control unit 355 are the same as those in the first embodiment.

[0201] The state memory unit 364 stores a captured image PA2 captured by the narrow-angle camera 302 when the drawer 24A is at a predetermined drawer extension. The captured image PA2 includes, for example, an image of the third marker MK41 and an image of the fourth marker MK42. When the drawer extension is zero, this indicates that the drawer 24A is closed. The larger the predetermined drawer extension, the more widely the drawer 24A is open. A drawer extension of 100% indicates that the drawer 24A is fully open. The predetermined drawer extension is, for example, 80% to 100%.

[0202] The third marker MK41 is located at the top end of the front panel that is located at the front end of the drawer 24A of the freezer compartment 24, and the state of the image captured by the narrow-angle camera 302 changes depending on the amount that the drawer 24A is pulled out. Note that the amount that the drawer 24A is pulled out refers to the amount that the drawer 24A is pulled out of the freezer compartment 24.

[0203] Drawer 24A has storage case 24B, which is configured to be able to be pulled out, in the upper section of drawer 24A. Storage case 24B has an opening / closing body that opens and closes the opening of storage case 24B. The opening / closing body has two plate-shaped opening / closing members 241B and 242B that are arranged to be able to slide left and right on storage case 24B. A fourth marker MK42 is arranged on the top surface of opening / closing member 241B and opening / closing member 242B.

[0204] In the second embodiment, a case where storage case 24B has opening / closing members 241B and 242B is described, but the present disclosure is not limited thereto. Storage case 24B may have one opening / closing member (e.g., opening / closing member 241B) but not the other opening / closing member (e.g., opening / closing member 242B). In this case, a fourth marker MK42 is placed on the top surface of opening / closing member 241B. In this case, by detecting the position of the fourth marker MK42, it is possible to determine whether captured image PB2 includes an image of food stored in the space on the right side of storage case 24B or an image of food stored in the space on the left side of storage case 24B.

[0205] The third marker MK41 includes a third outer frame image MF3 and a third identification image MD3. The third identification image MD3 is formed inside the third outer frame image MF3 and indicates third identification information. The third identification information is information for identifying the drawer 24A of the freezer compartment 24. The fourth marker MK42 includes a fourth outer frame image MF4 and a fourth identification image MD4. The fourth identification image MD4 is formed inside the fourth outer frame image MF4 and indicates fourth identification information. The fourth identification information is information for identifying the storage case 24B placed in the drawer 24A of the freezer compartment 24. The third marker MK41 and the fourth marker MK42 will be further described with reference to FIGS. 9, 14, and 20-23.

[0206] The first judgment unit 355A judges whether or not the third outer frame image MF3 and each of the two fourth outer frame images MF4 can be detected for each of the captured images PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363.

[0207] The second judgment unit 355B judges whether or not the third identification image MD3 and each of the two fourth identification images MD4 can be detected for each of the captured images PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363.

[0208] In the second embodiment, as described with reference to FIG. 14 , the third outer frame image MF3 is easier to detect than the third identification image MD3. Therefore, if the first determination unit 355A determines that the third outer frame image MF3 cannot be detected, the second determination unit 355B determines that the third identification image MD3 cannot be detected. Similarly, the fourth outer frame image MF4 is easier to detect than the fourth identification image MD4. Therefore, if the first determination unit 355A determines that the fourth outer frame image MF4 cannot be detected, the second determination unit 355B determines that the fourth identification image MD4 cannot be detected.

[0209] The determination unit 355C determines whether the captured image PB2 includes an image of food stored in the drawer 24A or an image of food stored in the storage case 24B based on the appearance of the third marker MK41 and the two fourth markers MK42 in each of the captured images PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363.

[0210] For example, the determination unit 355C determines whether the captured image PB2 includes an image of food stored in the drawer 24A or an image of food stored in the storage case 24B based on the determination results of the first determination unit 355A and the second determination unit 355B.

[0211] For example, if the first judgment unit 355A determines that the third outer frame image MF3 and one of the two fourth outer frame images MF4 can be detected, and the first judgment unit 355A determines that the other of the two fourth outer frame images MF4 cannot be detected, and the second judgment unit 355B determines that the third identification image MD3 and one of the fourth identification images MD4 can be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the storage case 24B.

[0212] Furthermore, for example, if the first judgment unit 355A determines that the third outer frame image MF3 can be detected, and the first judgment unit 355A determines that the two fourth outer frame images MF4 cannot be detected, and the second judgment unit 355B determines that the third identification image MD3 can be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 24A.

[0213] Furthermore, for example, if first determination unit 355A determines that third outer frame image MF3 cannot be detected, determination unit 355C determines that captured image PB2 does not include images of food stored in drawer 24A and food stored in storage case 24B. First determination unit 355A, second determination unit 355B, and determination unit 355C will be further described with reference to FIG. 24 .

[0214] In the second embodiment, the determination unit 355C determines whether the captured image PB2 includes an image of food stored in drawer 24A or an image of food stored in storage case 24B based on the determination results of the first determination unit 355A and the second determination unit 355B, but the present disclosure is not limited to this. The determination unit 355C may determine whether the captured image PB2 includes an image of food stored in drawer 24A or an image of food stored in storage case 24B based on at least one of the determination results of the first determination unit 355A and the determination results of the second determination unit 355B.

[0215] The generation unit 356 selects a second captured image P2 from the captured image PB2 included in the moving image indicated by the second video recording data DV2 of the video recording data 363, based on the determination result of the determination unit 355C. The second captured image P2 is an image captured when the drawer 24A is pulled out to a predetermined amount. If the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 24A, the generation unit 356 selects the captured image PB2 as the second captured image P2 including an image of food stored in the drawer 24A. If the determination unit 355C determines that the captured image PB2 includes an image of food stored in the storage case 24B, the generation unit 356 selects the captured image PB2 as the second captured image P2 including an image of food stored in the storage case 24B.

[0216] When drawer 24A transitions from a closed state to an open state and reaches the maximum open position, recording control unit 355 starts recording the images captured by wide-angle camera 301 and narrow-angle camera 302, and second communication control unit 352 transmits information requesting the output of a buzzer sound to first control board 311. Upon receiving the information requesting the output of a buzzer sound, first processor 320 causes buzzer 306 to output a buzzer sound, thereby notifying user P that third marker MK41 is in a predetermined position.

[0217] As indicated by the dashed lines in FIG. 9 , the drawer 24A of the freezer compartment 24 has a storage case 24B. The storage case 24B is located in the upper section of the drawer 24A and is configured to be removable. The storage case 24B has an opening / closing body that opens and closes the opening of the storage case 24B. The opening / closing body has two plate-shaped opening / closing members 241B and 242B that are slidably disposed in the left-right direction of the storage case 24B. A fourth marker MK42 is disposed on the upper surfaces of the opening / closing members 241B and 242B. The fourth marker MK42 is disposed on the upper surfaces of the front sides of the opening / closing members 241B and 242B. The storage case 42B, the opening / closing members 241B and 242B, and the second marker MK42 will be further described with reference to FIGS. 20 to 23 .

[0218] Next, the configuration of the markers MK will be described with reference to FIG. 14 . The markers MK include a third marker MK41 and a fourth marker MK42. The third marker MK41 is placed on the drawer 24A of the freezer compartment 24. The fourth marker MK42 is placed on each of the opening / closing members 241B, 242B that open and close the opening of the storage case 24B of the drawer 24A of the freezer compartment 24. The identification image MD of the third marker MK41 corresponds to identification information indicating the drawer 24A of the freezer compartment 24. The identification image MD of the fourth marker MK42 corresponds to identification information indicating the storage case 24B of the freezer compartment 24. The outer frame image MF of the third marker MK41 corresponds to an example of the third outer frame image MF3. The outer frame image MF of the fourth marker MK42 corresponds to an example of the fourth outer frame image MF4. The identification image MD of the third marker MK41 corresponds to an example of the third identification image MD3. The identification image MD of the fourth marker MK42 corresponds to an example of a fourth identification image MD4.

[0219] [2-2. Markers Placed in Freezer Compartment] Next, the third marker MK41 and the fourth marker MK42 placed in the freezer compartment 24 will be described with reference to Figs. 20 to 23. Figs. 20 to 23 illustrate the same X-axis, Y-axis, and Z-axis as the X-axis, Y-axis, and Z-axis shown in Fig. 2. The X-axis direction corresponds to the width direction of the refrigerator 2 and the left-right direction of the refrigerator 2. The Y-axis direction corresponds to the front-rear direction of the refrigerator 2. The Z-axis direction corresponds to the up-down direction of the refrigerator 2.

[0220] Fig. 20 is a plan view showing an example of a third marker MK41 placed in freezer compartment 24. In Fig. 20, drawer 24A is pulled out to the maximum opening position. Also, storage case 24B is placed in freezer compartment 24 formed in main box body 20. Therefore, storage case 24B is not shown in Fig. 20.

[0221] 20, drawer 24A includes front panel 241A, first storage chamber 242A, partition member 243A, and second storage chamber 244A. Front panel 241A, first storage chamber 242A, partition member 243A, and second storage chamber 244A are arranged in this order from the front side (positive direction of the Y axis) to the rear side (negative direction of the Y axis).

[0222] The front panel 241A is disposed at the front end of the drawer 24A of the freezer compartment 24. The front panel 241A is formed in a rectangular plate shape. The front surface of the front panel 241A forms part of the front surface of the refrigerator 2. The back surface of the front panel 241A forms the front surface of the first storage compartment 242A. A third marker MK41 is disposed on the top surface of the front panel 241A. The third marker MK41 is disposed, for example, at the right end of the top surface of the front panel 241A.

[0223] The first storage compartment 242A is disposed on the front side of the drawer 24A of the freezer compartment 24. Food items such as vegetables are stored in the first storage compartment 242A.

[0224] The partition member 243A divides the drawer 24A into a first storage chamber 242A and a second storage chamber 244A. The partition member 243A is formed in the shape of a rectangular plate and is disposed approximately parallel to the XZ plane. The front surface of the partition member 243A functions as the back surface of the first storage chamber 242A, and the back surface of the partition member 243A functions as the front surface of the second storage chamber 244A.

[0225] The second storage compartment 244A is disposed on the rear side of the drawer 24A of the freezer compartment 24. Food items such as vegetables are stored in the second storage compartment 244A. The upper surfaces of the side portions of the second storage compartment 244A function as rails when the storage case 24B slides forward and backward. In other words, the storage case 24B is configured to be movable forward and backward by sliding along the upper surfaces of the side portions of the second storage compartment 244A. Note that, as shown in FIG. 20 , when the storage case 24B is disposed in the freezer compartment 24 formed in the main box 20, the upper surfaces of the side portions of the freezer compartment 24 function as rails.

[0226] 20, the first determination unit 355A determines that the third outer frame image MF3 can be detected, but determines that the two fourth outer frame images MF4 cannot be detected. The second determination unit 355B determines that the third identification image MD3 can be detected, but determines that the two fourth identification images MD4 cannot be detected. The determination unit 355C then determines that the captured image PB2 includes an image of food stored in the drawer 24A.

[0227] Fig. 21 is a plan view showing an example of a third marker MK41 and a fourth marker MK42 arranged in the freezer compartment 24. In Fig. 21, the drawer 24A is pulled out to its maximum open position. Also, the storage case 24B is pulled out to its maximum open position. When the storage case 24B is pulled out to its maximum open position, as shown in Fig. 21, the front surface of the storage case 25B is located, for example, directly above the first storage chamber 242A. In other words, when the storage case 24B is pulled out to its maximum open position, the front surface of the storage case 25B is located, for example, between the partition member 253A and the front panel 241A shown in Fig. 20 in the Y-axis direction.

[0228] The storage case 24B has two plate-shaped opening / closing members 241B and 242B that open and close the opening on the top surface of the storage case 24B. Each of the two plate-shaped opening / closing members 241B and 242B is arranged to be slidable in the left-right direction of the storage case 24B. The opening / closing member 241B covers the left half of the opening on the top surface of the storage case 24B. The opening / closing member 242B covers the right half of the opening on the top surface of the storage case 24B. The opening / closing member 242B is arranged to be slidable in the left-right direction of the storage case 24B along the top surface of the opening / closing member 241B. The opening / closing member 241B slides in the left-right direction on rails formed on the upper ends of the front member 243B and the back member 246B of the storage case 24B. The opening / closing member 242B slides left and right on rails formed on the upper ends of a front member 243B of the storage case 24B and a back member 246B of the storage case 24B. The front member 243B and the back member 246B will be described with reference to FIG.

[0229] The fourth marker MK421 is placed at the front end of the upper surface of the opening / closing member 241B. For example, the fourth marker MK421 is placed near the left end at the front end of the upper surface of the opening / closing member 241B. The fourth marker MK422 is placed at the front end of the upper surface of the opening / closing member 242B. For example, the fourth marker MK422 is placed near the right end at the front end of the upper surface of the opening / closing member 242B. The fourth marker MK421 and the fourth marker MK422 correspond to the two fourth markers MK42.

[0230] 21, the first determination unit 355A determines that the third outer frame image MF3 and two fourth outer frame images MF4 can be detected. The second determination unit 355B determines that the third identification image MD3 and two fourth outer frame images MF4 can be detected. The determination unit 355C then determines that the captured image PB2 does not include images of food stored in drawer 24A or storage case 24B.

[0231] Fig. 22 is a plan view showing an example of a third marker MK41 and a fourth marker MK42 arranged in the freezer compartment 24. In Fig. 22, the drawer 24A is pulled out to the maximum open position. Also, the storage case 24B is pulled out to the maximum open position. In Fig. 22, the opening / closing member 241B and the opening / closing member 242B are arranged on the right side of the storage case 24B. In this case, the left half of the third storage chamber 244B of the storage case 24B is open.

[0232] As shown in FIG. 22 , the storage case 24B includes a front member 243B, a third storage chamber 244B, a pair of sliding members 245B, and a rear member 246B. The front member 243B forms the front surface of the third storage chamber 244B. Rails are formed on the upper surface of the front member 243B to slidably support the opening / closing members 241B and 242B. The rails extend in the left-right direction. The rear member 246B forms the back surface of the third storage chamber 244B. Rails are formed on the upper surface of the rear member 246B to slidably support the opening / closing members 241B and 242B. The rails extend in the left-right direction. That is, the opening / closing members 241B and 242B slide on the rails formed on the upper surfaces of the front member 243B and the rear member 246B.

[0233] The third storage chamber 244B is formed in the storage case 24B. Food items such as vegetables are stored in the third storage chamber 244B.

[0234] The pair of sliding members 245B are formed to protrude outward in the left-right direction from the upper ends of the side members that constitute the third storage chamber 244B. Each of the pair of sliding members 245B is a plate-shaped member that extends in the front-rear direction. The sliding member 245B arranged on the right side of the third storage chamber 244B is formed to protrude rightward from the upper end of the right side member that constitutes the third storage chamber 244B. The sliding member 245B arranged on the left side of the third storage chamber 244B is formed to protrude leftward from the upper end of the left side member that constitutes the third storage chamber 244B. The pair of sliding members 245B are formed integrally with the third storage chamber 244B and slide on rails that are arranged on the upper surfaces of the side portions of the second storage chamber 244A and the freezer compartment 24 when the storage case 24B is moved in the front-rear direction.

[0235] 22 , the first determination unit 355A determines that the third outer frame image MF3 and one of the two fourth outer frame images MF4 can be detected, but determines that the other of the two fourth outer frame images MF4 cannot be detected. Note that one of the two fourth outer frame images MF4 is the fourth outer frame image MF4 of the fourth marker MK422, and the other of the two fourth outer frame images MF4 is the fourth outer frame image MF4 of the fourth marker MK421. Furthermore, the second determination unit 355B determines that the third identification image MD3 and one of the two fourth identification images MD4 can be detected, but determines that the other of the two fourth identification images MD4 cannot be detected. Note that one of the two fourth identification images MD4 is the fourth identification image MD4 of the fourth marker MK422, and the other of the two fourth identification images MD4 is the fourth identification image MD4 of the fourth marker MK421. The determination unit 355C then determines that the photographed image PB2 includes an image of food stored in the storage case 24B. If the photographed image PB2 corresponds to FIG. 22 , the determination unit 355C may determine that the photographed image PB2 includes an image of the left side area of ​​the image of the storage case 24B.

[0236] Fig. 23 is a plan view showing an example of a third marker MK41 and a fourth marker MK42 arranged in the freezer compartment 24. In Fig. 23, the drawer 24A is pulled out to the maximum open position. Also, the storage case 24B is pulled out to the maximum open position. In Fig. 23, the opening / closing member 241B and the opening / closing member 242B are arranged on the left side of the storage case 24B. In this case, the right half of the third storage chamber 244B of the storage case 24B is open.

[0237] 23 , the first determination unit 355A determines that the third outer frame image MF3 and one of the two fourth outer frame images MF4 can be detected, but determines that the other of the two fourth outer frame images MF4 cannot be detected. Note that one of the two fourth outer frame images MF4 is the fourth outer frame image MF4 of the fourth marker MK422, and the other of the two fourth outer frame images MF4 is the fourth outer frame image MF4 of the fourth marker MK421. Furthermore, the second determination unit 355B determines that the third identification image MD3 and one of the two fourth identification images MD4 can be detected, but determines that the other of the two fourth identification images MD4 cannot be detected. Note that one of the two fourth identification images MD4 is the fourth identification image MD4 of the fourth marker MK422, and the other of the two fourth identification images MD4 is the fourth identification image MD4 of the fourth marker MK421. The determination unit 355C then determines that the photographed image PB2 includes an image of food stored in the storage case 24B. If the photographed image PB2 corresponds to FIG. 23 , the determination unit 355C may determine that the photographed image PB2 includes an image of the right side area of ​​the image of the storage case 24B.

[0238] The configurations of the terminal device 4 and the food management server 5 shown in Fig. 15 and the flowchart FA showing the operation of the image capturing device 3 shown in Fig. 16 are the same as those in the first embodiment. The flowchart FB shown in Fig. 17 is the same as that in the first embodiment except for step SB3. Step SB3 of the flowchart FB shown in Fig. 17 will be described with reference to the flowchart FG shown in Fig. 24. In other words, in the first embodiment, the flowchart FF shown in Fig. 19 is executed in step SB3 of the flowchart FB shown in Fig. 17, whereas in the second embodiment, the flowchart FG shown in Fig. 24 is executed in step SB3 of the flowchart FB shown in Fig. 17. The flowchart FC shown in Fig. 17 and the flowcharts FD and FE shown in Fig. 18 are the same as those in the first embodiment.

[0239] 2-3. Second Captured Image Selection Process Next, the "second captured image selection process" will be described with reference to FIG. 24. The "second captured image selection process" is a process in which the generation unit 356 selects a second captured image P2 based on the determination result of the determination unit 355C. The determination unit 355C determines whether the captured image PB2 includes an image of food stored in the drawer 24A or an image of food stored in the storage case 24B based on the determination result of the first determination unit 355A and the determination result of the second determination unit 355B. In FIG. 24, a flowchart FG shows the operation of the image capture device 3.

[0240] As shown in flowchart FG, the generation unit 356 reads the captured image PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363 (step SG1). Next, the first determination unit 355A determines whether a third outer frame image MF3 can be detected from the read captured image PB2 (step SG2). If the first determination unit 355A determines that the third outer frame image MF3 cannot be detected (step SG2: NO), the process proceeds to step SG10. If the first determination unit 355A determines that the third outer frame image MF3 can be detected (step SG2: YES), the first determination unit 355A determines whether at least one fourth outer frame image MF4 can be detected from the read captured image PB2 (step SG3). If the first determination unit 355A determines that the fourth outer frame image MF4 cannot be detected (step SG3: NO), the process proceeds to step SG8. If the first judgment unit 355A determines that at least one fourth outer frame image MF4 can be detected (step SG3: YES), the second judgment unit 355B determines whether two fourth outer frame images MF4 can be detected for the read-out captured image PB2 (step SG4).

[0241] If the second determination unit 355B determines that two fourth outer frame images MF4 can be detected (step SG4: YES), the process proceeds to step SG10. If the second determination unit 355B determines that two fourth outer frame images MF4 cannot be detected (step SG4: NO), the second determination unit 355B determines whether the third identification image MD3 can be detected (step SG5). Note that the second determination unit 355B determines that the third identification image MD3 can be detected if it can detect third identification information corresponding to the third identification image MD3 in the captured image PB2 and if the detected third identification information matches the identification information indicated by the third marker MK41 included in the captured image PA2 stored in the state memory unit 364.

[0242] If the second determination unit 355B determines that the third identification image MD3 cannot be detected (step SG5: NO), the process proceeds to step SG10. If the second determination unit 355B determines that the third identification image MD3 can be detected (step SG5: YES), the second determination unit 355B determines whether the fourth identification image MD4 can be detected for the read-out photographed image PB2 (step SG6). Note that the second determination unit 355B determines that the fourth identification image MD4 can be detected if it can detect fourth identification information corresponding to the fourth identification image MD4 for the photographed image PB2 and if the detected fourth identification information matches the identification information indicated by the fourth marker MK42 included in the photographed image PA2 stored in the state storage unit 364. If the second determination unit 355B determines that the fourth identification image MD4 cannot be detected (step SG6: NO), the process proceeds to step SG10. If the second determination unit 355B determines that the fourth identification image MD4 can be detected (step SG6: YES), the process proceeds to step SG7. Then, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the storage case 25B, and the generation unit 356 selects the captured image PB2 as the second captured image P2 that includes an image of food stored in the storage case 25B (step SG7). Then, the process proceeds to step SG10.

[0243] If the first determination unit 355A determines that the fourth outer frame image MF4 cannot be detected (step SG3: NO), the second determination unit 355B determines whether the third identification image MD3 can be detected in the read-out captured image PB2 (step SG8). If the second determination unit 355B determines that the third identification image MD3 cannot be detected (step SG8: NO), the process proceeds to step SG10. If the second determination unit 355B determines that the third identification image MD3 can be detected (step SG8: YES), the process proceeds to step SG9. Then, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 25A, and the generation unit 356 selects the captured image PB2 as the second captured image P2 including an image of food stored in the drawer 25A (step SG9). Then, the process proceeds to step SG10.

[0244] Next, the generation unit 356 determines whether all captured images PB2 included in the moving image indicated by the second recorded data DV2 of the recorded data 363 have been read (step SG10). If the generation unit 356 determines that all captured images PB2 have been read (step SG10: YES), the process returns to step SB4 in FIG. 17. If the generation unit 356 determines that not all captured images PB2 have been read (step SG10: NO), the process returns to step SG1. The generation unit 356 reads out any captured images PB2 that have not been read out from among the captured images PB2 included in the moving image indicated by the second recorded data DV2 of the recorded data 363 (step SG1). Then, the process from step SG2 onwards is executed again.

[0245] Steps SG7 and SG9 in the flowchart FG of FIG. 24 correspond to an example of a "determining step."

[0246] As explained with reference to Figure 24, the generation unit 356 selects a second captured image P2 including an image of food stored in the storage case 25B and a second captured image P2 including an image of food stored in the drawer 25A based on the determination result of the determination unit 355C.

[0247] In the second embodiment, the case where the image capturing device 3 is configured as a separate entity from the refrigerator 2 has been described, but the present disclosure is not limited to this. The image capturing device 3 may be configured as an integral part of the refrigerator 2. In other words, the image capturing device 3 may be configured as a part of the refrigerator 2.

[0248] In the second embodiment, the determination unit 355C determines whether the captured image PB2 includes an image of food stored in drawer 24A or an image of food stored in storage case 24B based on the determination results of the first determination unit 355A and the second determination unit 355B. However, the present disclosure is not limited to this. It is sufficient that the determination unit 355C determines whether the captured image PB2 includes an image of food stored in drawer 24A or an image of food stored in storage case 24B based on at least one of the determination results of the first determination unit 355A and the determination results of the second determination unit 355B. The determination unit 355C may also determine whether the captured image PB2 includes an image of food stored in drawer 24A or an image of food stored in storage case 24B based on the determination result of the first determination unit 355A. For example, if the first determination unit 355A determines that the third outer frame image MF3 can be detected but determines that the two fourth outer frame images MF4 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in drawer 25A. Furthermore, if the first determination unit 355A determines that the third outer frame image MF3 and one of the two fourth outer frame images MF4 can be detected but determines that the other of the two fourth outer frame images MF4 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in drawer 24A or storage case 24B based on the determination result of the second determination unit 355B. For example, if the second determination unit 355B determines that the third identification image MD3 can be detected but determines that the two fourth identification images MD4 cannot be detected, the determination unit 355C determines that the image includes an image of a food item stored in the drawer 25A. Also, if the second determination unit 355B determines that the third identification image MD3 and one of the two fourth identification images MD4 can be detected but determines that the other of the two fourth identification images MD4 cannot be detected, the determination unit 355C determines that the image includes an image of a food item stored in the storage case 24B.

[0249] 1 to 11, 14, 15 to 18, and 20 to 24, photography system 7 in embodiment 2 photographs refrigerator 2 equipped with freezer compartment 24 having drawer 24A with an opening formed on the top surface thereof, and photographs the refrigerator 2 equipped with narrow-angle camera 302 arranged on top surface 20A of refrigerator 2, third marker MK41 arranged on drawer 24A, drawer 24A having storage case 24B arranged to be able to be drawn out at the upper part of drawer 24A, storage case 24B having an opening / closing body that opens and closes the opening of storage case 24B, and the opening / closing body is arranged on the left side of storage case 24B. The freezer has two plate-shaped opening / closing members 241B, 242B that are arranged to slide freely to the right, and two fourth markers MK42 that are arranged on each of the two plate-shaped opening / closing members 241B, 242B; an imaging control unit 354 that photographs the freezer compartment 24 using a narrow-angle camera 302 and generates a photographed image PB2; and a determination unit 255C that determines whether the photographed image PB2 includes an image of food stored in the drawer 24A or an image of food stored in the storage case 24B based on the appearance of the third marker MK41 and the two fourth markers MK42 in the photographed image PB2.

[0250] According to this, based on the appearance of the third marker MK41 and the two fourth markers MK42 in the photographed image PB2, it is determined whether the photographed image PB2 includes an image of food stored in drawer 24A or an image of food stored in storage case 24B. Therefore, it is possible to properly determine whether the photographed image PB2 includes an image of food stored in drawer 24A or an image of food stored in storage case 24B. Therefore, the food stored in refrigerator 2 can be managed with high accuracy.

[0251] In addition, in the photography system 7, the third marker MK41 is placed on the upper surface of the front side of the drawer 25A, and the two fourth markers MK42 are placed on the upper surface of the front side of each of the two plate-shaped opening / closing members 241B, 242B.

[0252] According to this, the third marker MK41 is placed on the upper surface of the front side of the drawer 25A, and the two fourth markers MK42 are placed on the upper surface of the front side of each of the two plate-shaped opening / closing members 241B, 242B. This makes it possible to properly determine whether the captured image PB2 includes an image of food stored in the drawer 24A or in the storage case 24B. This allows for accurate management of the food stored in the refrigerator.

[0253] Furthermore, in the photographing system 7, the third marker MK41 includes a third outer frame image PF3, and each of the two fourth markers MK42 includes a fourth outer frame image PF4, and a first determination unit 355A is provided that determines whether or not the third outer frame image PF3 and each of the two fourth outer frame images PF4 can be detected from the photographed image PB2, and the first determination unit 355A determines that the third outer frame image PF3 and one of the two fourth outer frame images PF4 can be detected, and If the first judgment unit 355A judges that the other of the two fourth outer frame images PF4 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the storage case 24B, and if the first judgment unit 355A judges that the third outer frame image PF3 can be detected but the two fourth outer frame images PF4 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 24A.

[0254] According to this, if the first determination unit 355A determines that the third outer frame image PF3 and one of the two fourth outer frame images PF4 can be detected, but determines that the other of the two fourth outer frame images PF4 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in storage case 24B. If the first determination unit 355A determines that the third outer frame image PF3 can be detected, but determines that the two fourth outer frame images PF4 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in drawer 24A. Thus, it is possible to properly determine whether the captured image PB2 includes an image of food stored in drawer 24A or in storage case 24B. This allows for accurate management of the food stored in refrigerator 2.

[0255] Furthermore, in the photographing system 7, the third marker MK41 is formed inside the third outer frame image PF3 and includes a third identification image MD3 indicating the third identification information, and each of the two fourth markers MK42 is formed inside the fourth outer frame image PF4 and includes a fourth identification image MD4 indicating the fourth identification information, and a second determination unit 355B is provided that determines whether or not the third identification image MD3 and each of the two fourth identification images MD4 can be detected from the photographed image PB2, and When the second judgment unit 355B determines that the third identification image MD3 can be detected and the second judgment unit 355B determines that the other of the two fourth identification images MD4 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the storage case 24B, and when the second judgment unit 355B determines that the third identification image MD3 can be detected and the second judgment unit 355B determines that the two fourth identification images MD4 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 24A.

[0256] According to this, if the second determination unit 355B determines that the third identification image MD3 and one of the two fourth identification images MD4 can be detected, but determines that the other of the two fourth identification images MD4 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the storage case 24B. If the second determination unit 355B determines that the third identification image MD3 can be detected, but determines that the two fourth identification images MD4 cannot be detected, the determination unit 355C determines that the captured image PB2 includes an image of food stored in the drawer 24A. Therefore, it is possible to properly determine whether the captured image PB2 includes an image of food stored in the drawer 24A or in the storage case 24B. This allows for accurate management of the food stored in the refrigerator 2.

[0257] Furthermore, the photographing method in the first embodiment photographs refrigerator 2 having freezer compartment 24 with drawer 24A having an opening formed on the top surface thereof, and uses narrow-angle camera 302 arranged on top surface 20A of refrigerator 2, third marker MK41 arranged on drawer 24A, drawer 24A having storage case 24B configured to be able to be drawn out at the upper stage of drawer 24A, storage case 24B having an opening / closing body that opens and closes the opening of storage case 24B, and the opening / closing body is two plate-shaped opening / closing members 241B, 242 arranged to be able to slide in the left-right direction of storage case 24B. B and two fourth markers MK42 arranged on each of the two plate-shaped opening / closing members 241B, 242B, the photographing method for the photographing system 7 includes a photographing control step of photographing the freezer compartment 24 with the narrow-angle camera 302 and generating a photographed image PB2, and a determination step of determining whether the photographed image PB2 includes an image of food stored in the drawer 24A or an image of food stored in the storage case 24B based on the appearance of the third marker MK41 and the two fourth markers MK42 in the photographed image PB2.

[0258] As a result, the photographing method according to the second embodiment has the same effect as the photographing system 7 according to the second embodiment.

[0259] Furthermore, control program 361 in the first embodiment photographs refrigerator 2 having freezer compartment 24 with drawer 24A having an opening formed on the top surface thereof, and uses narrow-angle camera 302 arranged on the top surface of freezer compartment 24, third marker MK41 arranged on drawer 24A, drawer 24A has storage case 24B configured to be able to be drawn out at the upper stage part of drawer 24A, storage case 24B has an opening / closing body that opens and closes the opening of storage case 24B, and the opening / closing body has two plate-like opening / closing members 241B and 242B that are arranged to be able to slide in the left-right direction of storage case 24B, and A control program 361 for an imaging system 7 comprising two fourth markers MK42 placed in each of drawers 24A and 24B, and a second processor 350 that controls the narrow-angle camera 302, causes the second processor 350 to function as an imaging control unit 354 that images the freezer compartment 24 using the narrow-angle camera 302 and generates a captured image PB2, and a determination unit 255C that determines whether the captured image PB2 includes an image of food stored in drawer 24A or in storage case 24B based on the appearance of the third marker MK41 and the two fourth markers MK42 in the captured image PB2.

[0260] According to this, the control program 361 in the second embodiment achieves the same effects as the imaging system 7 in the second embodiment.

[0261] (Embodiment 3) [3-1. Configuration] The following explanation will mainly focus on the differences from embodiment 1 described with reference to Figures 1 to 19. The configuration of food management system 1 and refrigerator 2 are the same as those in embodiments 1 and 2. The configuration of imaging device 3 is also the same as those in embodiments 1 and 2. Here, in this embodiment, the vegetable drawer shown in Figure 13 will be used as the drawer for explanation.

[0262] Fig. 25 is a block diagram showing a control configuration in the third embodiment. As shown in Fig. 25, this embodiment differs from the first and second embodiments in that it includes a third determination unit 357. In this embodiment, the photography control unit 354 acquires a captured image when the user opens the drawer 25A. The third determination unit 357 generates a best shot of the drawer 25A based on the captured image acquired by the photography control unit 354. When the user opens the drawer 25A and takes food or the like in or out, the user's arm and the food move, which may result in an inappropriate image of the inside of the drawer 25A.

[0263] The third determination unit 357 calculates the amount of motion of the captured image based on the captured image. The amount of motion is calculated, for example, by detecting the difference between each image frame, i.e., by detecting pixel changes in the captured image. If the difference between each image frame is large, the amount of motion is determined to be large, and if the difference is small, the amount of motion is determined to be small. Alternatively, the amount of motion may be calculated using a weighted moving average of approximately 10 captured images. In this case, the amount of motion may be calculated based on the movement of feature points (e.g., markers) in the captured image or changes in edges (linear components of the drawer 25A).

[0264] FIG. 26 is an explanatory diagram showing an example of calculation of the amount of movement in the third embodiment. As shown in FIG. 26 , when drawer 25A is opened, the amount of movement increases. When drawer 25A is fully opened, the upper marker is detected, and drawer 25A is stopped, resulting in a decrease in the amount of movement. Subsequently, when storage case 25B, which is the tray for drawer 25A, is moved toward the back, the amount of movement increases. When storage case 25B has completed its movement and drawer 25A is stationary, the amount of movement decreases. When the user puts or takes out food or the like from drawer 25A, the amount of movement increases. When drawer 25A is taken out and storage case 25B is pulled out, the amount of movement increases. However, when storage case 25B is fully pulled out, the amount of movement decreases. In this way, the amount of movement changes with the drawer 25A and storage case 25B pull-out actions and the user's actions of putting or taking out food.

[0265] In this embodiment, the third determination unit 357 is configured to set a threshold value for the amount of motion in advance, and to select the image frame with the latest amount of motion after the amount of motion falls below the threshold value and then exceeds the threshold value as the best shot. That is, when the amount of motion next exceeds the threshold value after falling below the threshold value, it is determined that there is a high possibility that no motion is captured in the captured image. The threshold value is set in advance depending on the model of the refrigerator. If all the amounts of motion exceed the threshold value, the best shot may be selected based on the time immediately before closing drawer 25A, the time when the amount of motion is minimum, or the inflection point of the amount of motion while the markers are detected. The threshold value may also be set automatically based on data including the latest amount of motion.

[0266] The best shot of the storage case 25B is adopted when both the upper and lower markers are detected, and the best shot of the drawer 25A is adopted when only the lower marker is detected without detecting the upper marker.

[0267] [3-2. Operation] Next, the operation of the third embodiment will be described. FIG. 27 is a flowchart showing the operation of the third embodiment. Note that the flowchart shown in FIG. 27 has the same basic operation as the flowchart shown in FIG. 19. Therefore, only the differences from FIG. 19 will be described. As shown in the flowchart HH in FIG. 27, the generation unit 356 reads out the captured image PB2 included in the moving image indicated by the second recorded data DV2 of the recorded data 363 (step SH1). Next, the first determination unit 355A determines whether or not the first outer frame image MF1 can be detected from the read captured image PB2 (step SH2).

[0268] If the first determination unit 355A determines that the first outer frame image MF1 cannot be detected (step SH2: NO), the process proceeds to step SH8. If the first determination unit 355A determines that the first outer frame image MF1 can be detected (step SH2: YES), the first determination unit 355A determines whether the second outer frame image MF2 can be detected for the read-out captured image PB2 (step SH3).

[0269] If the first determination unit 355A determines that the second outer frame image MF2 cannot be detected (step SH3: NO), the process proceeds to step SH9. If the first determination unit 355A determines that the second outer frame image MF2 can be detected (step SH3: YES), the second determination unit 355B determines whether the first identification image MD1 can be detected for the read-out captured image PB2 (step SH4). Note that the second determination unit 355B determines that the first identification image MD1 can be detected if the second determination unit 355B can detect first identification information corresponding to the first identification image MD1 for the captured image PB2 and if the detected first identification information matches the identification information indicated by the first marker MK51 included in the captured image PA2 stored in the state storage unit 364.

[0270] If the second determination unit 355B determines that the first identification image MD1 cannot be detected (step SH4: NO), the process proceeds to step SH8. If the second determination unit 355B determines that the first identification image MD1 can be detected (step SH4: YES), the second determination unit 355B determines whether the second identification image MD2 can be detected for the read-out photographed image PB2 (step SH5). Note that the second determination unit 355B determines that the second identification image MD2 can be detected if it can detect second identification information corresponding to the second identification image MD2 for the photographed image PB2 and if the detected second identification information matches the identification information indicated by the second marker MK52 included in the photographed image PA2 stored in the state storage unit 364. If the second determination unit 355B determines that the second identification image MD2 cannot be detected (step SH5: NO), the process proceeds to step SH8. If the second determination section 355B determines that the second identification image MD2 can be detected (step SH5: YES), the best shot of the second identification image MD2 is selected.

[0271] In this embodiment, the third determination unit 357 determines whether or not there is motion in the captured image based on the threshold value for the amount of motion of the captured image (step SH6). If it determines that there is no motion in the captured image (step SH6: YES), the image frame with the latest amount of motion after the amount of motion falls below the threshold value and then exceeds the threshold value is selected as the best shot (step SH7). Then, the process proceeds to step SH8.

[0272] If the first determination unit 355A determines that the second outer frame image MF2 cannot be detected (step SH3: NO), the second determination unit 355B determines whether the first identification image MD1 can be detected in the read-out captured image PB2 (step SH9). If the second determination unit 355B determines that the first identification image MD1 cannot be detected (step SH9: NO), the process proceeds to step SH8. If the second determination unit 355B determines that the first identification image MD1 can be detected (step SH9: YES), the process proceeds to step SH10.

[0273] In this embodiment, the third determination unit 357 determines whether or not there is movement in the captured image based on a threshold value for the amount of motion of the captured image (step SH10). If it determines that there is no movement in the captured image (step SH10: YES), the image frame with the most recent amount of motion after the amount of motion falls below the threshold value and then exceeds the threshold value is selected as the best shot (step SH11). Then, the process proceeds to step SH8.

[0274] [3-3. Effects, etc.] As described above, the photography system in this embodiment includes a third determination unit 357 that selects the best shot of the photographed images. The third determination unit 357 calculates the amount of movement of the photographed images and selects the photographed image with the least amount of movement as the best shot. This makes it possible to prevent the selection of photographed images with a large amount of movement caused by the user taking food in and out, and to select photographed images that properly capture the food in the storage compartment.

[0275] In addition, in the photography system of this embodiment, the third determination unit 357 sets a threshold for the amount of motion in advance, and after the amount of motion in the photographed image falls below the threshold, the image frame with the latest amount of motion before exceeding the threshold is selected as the best shot. This makes it possible to prevent the selection of photographed images with large amounts of motion, such as when a user is taking food in or out, and to select photographed images that properly capture the food in the storage compartment.

[0276] (Embodiment 4) [4-1. Configuration] Next, an embodiment of the marker will be described. Fig. 28 is a perspective view from above showing the attached state of the refrigerator compartment marker. Fig. 29 is a perspective view from below showing the attached state of the refrigerator compartment marker. Fig. 30 is a plan view showing the attached state of the refrigerator compartment marker. Fig. 31 is a bottom view showing the attached state of the refrigerator compartment marker.

[0277] 28 to 31, the refrigerator compartment marker 55 is attached to a hinge part 80 that supports the refrigerator compartment door. The hinge part 80 includes a generally flat hinge plate 81 and a support shaft 82 that is provided at one end of the hinge plate 81 and supports the door (not shown). The hinge plate 81 includes a wide part 83 that is large in width in the front-to-rear direction, and a narrow part 84 that is at the other end of the hinge plate 81 and is small in width in the front-to-rear direction.

[0278] The refrigerator compartment marker 55 comprises a marker main body 56 in the shape of a substantially square frame with an open bottom, and a marker support part 57 provided on the bottom side of the marker main body 56 and clamping a hinge part 80 of the refrigerator compartment door. The top surface of the marker main body 56 is formed as a marker attachment part 58 to which a predetermined marker is attached. The refrigerator compartment marker 55 is attached to the narrow part 84 of the hinge plate 81 by the marker main body 56 and the marker support part 57.

[0279] Next, the drawer marker will be described. FIG. 32 is a perspective view showing the drawer marker. FIG. 33 is a side view showing the drawer marker. FIG. 34 is a perspective view showing the attached state of the drawer marker. FIG. 35 is a side view showing the attached state of the drawer marker. As shown in FIGS. 32 to 35 , the drawer marker 60 is attached to the drawer of a drawer-type vegetable compartment 85 of a refrigerator. A box-shaped vegetable compartment case 86 with an open top is installed in the drawer-type vegetable compartment 85. The vegetable compartment case 86 is placed on the upper edges of both side panels 87 of the vegetable compartment 85 and is configured to be movable back and forth within the vegetable compartment 85. A handle 88 for pulling out the vegetable compartment case 86 is formed on the upper front side of the vegetable compartment case 86 so as to extend diagonally forward.

[0280] The drawer marker 60 includes a marker body 61. The marker body 61 is composed of a generally flat marker attachment portion 62, an inner piece 63 that extends downward from the rear edge of the marker body 61 and is located on the inside of the drawer, and an outer piece 64 that extends downward from the front edge of the marker body 61 and is located on the outside of the drawer. A locking portion 65 that protrudes inward is formed at the lower end of the outer piece 64. The drawer marker 60 is attached so as to cover the top of the handle portion 88. When the drawer marker 60 is attached, the locking portion 65 engages with the front lower end of the handle portion 88, thereby enabling the drawer marker 60 to be held in place.

[0281] Next, a case marker to be installed inside a drawer-type freezer will be described. Fig. 36 is a plan view showing the case marker. Fig. 37 is a bottom view showing the case marker. Fig. 38 is a side view showing the case marker. Fig. 39 is a plan view showing the state in which the case marker is attached to a case with one slide plate attached. Fig. 40 is a plan view showing the state in which the case marker is attached to a case with two slide plates attached. Fig. 41 is a side cross-sectional view showing the state in which the case marker is attached.

[0282] 39 and 40 , the case marker 70 is attached to a freezer case 90 that is installed inside a drawer-type freezer compartment 89 of a refrigerator. A box-shaped freezer case 90 with an open top is installed in the drawer-type freezer compartment 89. The freezer case 90, like the vegetable compartment case 86, is placed on the upper edges of both side panels 91 of the freezer compartment 89 and is configured to be movable back and forth inside the freezer compartment 89.

[0283] A sliding plate 92 is attached to the freezer case 90 to cover the top opening of the freezer case 90. A large number of circular pores 93 are formed in the sliding plate 92. In the example shown in Fig. 39, the sliding plate 92 has a width that is approximately half the width of the top opening of the freezer case 90 in the left-right direction. Fig. 39 shows an example in which one sliding plate is provided, while Fig. 40 shows an example in which two sliding plates 92 are provided. In both examples, the sliding plate 92 can be moved left and right to allow food and other items stored inside the freezer case 90 to be inserted or removed.

[0284] 36 to 38, the case marker 70 attached to the slide plate 92 of the freezer case 90 includes a substantially rectangular marker body 71, the upper surface of which serves as a marker attachment portion 72. Support claws 73 protrude from the outer peripheral edge of the lower surface of the marker body 71. Two support claws 73 are provided along each long side of the marker body 71. The support claws 73 are formed so as to be slightly inclined outward relative to the marker body 71, and a locking portion 74 protruding outward is formed at the lower end of each support claw 73.

[0285] Furthermore, support pins 75 are formed to protrude from the underside of the marker body 71, one at each of the front, rear, left and right. The support claws 73 and support pins 75 can each be inserted into the narrow holes 93 of the slide plate 92. By inserting the support claws 73 and support pins 75 into the narrow holes 93 of the slide plate 92, the locking portions 74 of the support claws 73 engage with the lower sides of the narrow holes 93, thereby attaching the case marker 70 to the slide plate 92.

[0286] [4-2. Operation] Next, the operation of this embodiment will be described. When attaching the refrigerator compartment marker 55, it is attached by sandwiching the narrow portion 84 of the hinge plate 81 between the marker main body 56 and the marker support portion 57. When attaching the drawer marker 60, it is attached by fitting the inner piece 63 and the outer piece 64 onto the handle portion 88 of the drawer from above. When the drawer marker 60 is attached, the locking portion 65 engages with the front lower end of the handle portion 88, thereby holding the drawer marker 60 to the handle portion 88.

[0287] When attaching the case marker 70, the support claw 73 and support pin 75 are inserted into the small hole 93 of the slide plate 92, so that the locking portion 74 of the support claw 73 engages with the underside of the small hole 93, thereby attaching the case marker 70 to the slide plate 92.

[0288] [4-3. Effects, etc.] As described above, the marker in this embodiment is a refrigerator compartment marker 55 that is attached to a refrigerator compartment, and includes a marker main body 56 and a marker support part 57 provided on the underside of the marker main body 56. The marker is attached by sandwiching the hinge part 80 of the refrigerator compartment door between the marker main body 56 and the marker support part 57. This allows the refrigerator compartment marker 55 to be easily attached to the refrigerator compartment.

[0289] The marker in this embodiment is a drawer marker 60 to be attached to a drawer-type storage compartment, and includes a marker main body 61, an inner piece 63 that extends downward from the rear edge of the marker main body 61 and is located on the inside of the drawer, and an outer piece 64 that extends downward from the front edge of the marker main body 61 and is located on the outside of the drawer, and is attached to the drawer by sandwiching the handle part 88 of the drawer between the inner piece 63 and the outer piece 64. This allows the drawer marker 60 to be easily attached to a drawer-type storage compartment.

[0290] The marker in this embodiment is a case marker 70 to be attached to a case provided in a drawer-type storage chamber, and includes a marker body 71, support claws 73 provided on the outer peripheral edge of the underside of the marker body 71, and support pins 75 provided on the underside of the marker body 71. The marker is attached to a slide plate 92 by inserting the support claws 73 and support pins 75 into the multiple holes 93 in the slide plate 92 that is provided on the upper surface of the case. This allows the case marker 70 to be easily attached to a case for a drawer-type storage chamber.

[0291] (Other Embodiments) As described above, the above-mentioned first and second embodiments have been described as examples disclosed in the present application. However, the technology in the present disclosure is not limited to these, and can also be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above-mentioned first and second embodiments to create new embodiments. Therefore, other embodiments will be described below as examples.

[0292] In the above-described first and second embodiments, the storage cabinet is a refrigerator 2, but the present disclosure is not limited to this. The storage cabinet may include a storage compartment having a drawer with an opening formed on the top surface. The storage compartment may be, for example, a heat-retaining compartment or a constant temperature compartment.

[0293] In the above-described first embodiment, the case where the first marker MK51 and the second marker MK52 are arranged in the vegetable compartment 25 has been described, but the first marker MK51 and the second marker MK52 may also be arranged in the ice making compartment 22, the fresh freezing compartment 23, or the freezer compartment 24. In the above-described second embodiment, the case where the third marker MK41 and the fourth marker MK42 are arranged in the freezer compartment 24 has been described, but the third marker MK41 and the fourth marker MK42 may also be arranged in the ice making compartment 22, the fresh freezing compartment 23, or the vegetable compartment 25.

[0294] In the above-described first and second embodiments, the identification image MD of the marker MK is composed of an information image MDA and a code image MDB, but the identification image MD may be composed of only the information image MDA. In other words, the identification image MD does not have to include the code image MDB.

[0295] In the above-described first and second embodiments, the first rectangular image PD1 and the second rectangular image PD2 in the marker MK are described as being square, but the first rectangular image PD1 and the second rectangular image PD2 may also be rectangular.

[0296] In the above-described first and second embodiments, the first captured image P1 and the second captured image P2 are used for display purposes on the terminal device 4. However, the first captured image P1 and the second captured image P2 may be used for other purposes, such as detecting food stored in the refrigerator 2.

[0297] In other embodiments, the marker MK may include characters or illustrations that indicate information about the storage compartment or storage case to be installed (for example, the characters "upper vegetable compartment case," an illustration of the upper vegetable compartment case, or a fruit mark). This allows the user P to easily identify the storage compartment or storage case to be installed when installing the marker MK in the refrigerator 2. This can prevent users from making installation mistakes and improve user convenience.

[0298] In another embodiment, the marker affixing portion may include characters, illustrations, or the like that indicate information about the storage compartment or storage case to be installed (for example, a number corresponding to the upper vegetable compartment case). This allows the user P to easily identify the storage compartment or storage case to be installed when installing the marker MK in the refrigerator 2. This can prevent users from making installation errors and improve user convenience.

[0299] In another embodiment, the marker MK may be made of a reflection-reducing material, which can reduce reflection of the marker MK by the second LED 308 and / or the third LED 309, thereby improving the recognition accuracy of the marker MK.

[0300] In another embodiment, the generation unit 356 may compare the image captured by the camera with pre-stored images of the drawer in the pulled-out state and select the image of the drawer in the pulled-out state.

[0301] In other embodiments, in addition to the functional units of second processor 350 in Embodiments 1 and 2, second processor 350 may function as a detection unit that detects food. During the time from when recording control unit 355 starts recording with wide-angle camera 301 and narrow-angle camera 302 until the time it ends, the detection unit may cut out an image of food being put in or taken out of refrigerator compartment 21 from the video. Furthermore, if a marker is attached to the food, the detection unit may detect the food with the marker.

[0302] In another embodiment, the first captured image P1 may be corrected before being uploaded to the food management server 5. An example of image correction is viewpoint conversion correction.

[0303] In other embodiments, the main body 20 of the refrigerator 2 may have fewer or more types of storage compartments than the five types described above. Also, in other embodiments, the main body 20 of the refrigerator 2 may have only one drawer-type storage compartment.

[0304] Each of the first processor 320, the second processor 350, the server processor 50, and the terminal processor 40 may be configured with a single processor or multiple processors. Each of these processors may be hardware programmed to implement the corresponding functional unit. That is, each of these processors may be configured with, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0305] The configurations of the image capture device 3, refrigerator 2, terminal device 4, and food management server 5 shown in Figures 8, 15, and 25 are merely examples, and the specific implementation form is not particularly limited. In other words, it is not necessarily necessary to implement hardware corresponding to each unit individually, and it is also possible to implement a configuration in which a single processor executes a program to realize the functions of each unit. Furthermore, some of the functions realized by software in the above-mentioned embodiments may be implemented by hardware, or some of the functions realized by hardware may be implemented by software.

[0306] The step units of the operations shown in Figures 16 to 19, 24, and 27 are divided according to the main processing content in order to make the operations easier to understand, and the operation is not limited by the way in which the processing units are divided or the names of the processing units. The operations may be divided into more step units depending on the processing content. Furthermore, one step unit may be divided so that it includes more processing. Furthermore, the order of the steps may be changed as appropriate within the scope that does not interfere with the purpose of this disclosure.

[0307] It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents.

[0308] (Additional Notes) The above description of the embodiments discloses the following techniques.

[0309] (Technology 1) A photography system for photographing a storage facility having a storage compartment with a drawer having an opening on its top surface, the photography system comprising: a camera disposed on the top surface of the storage facility; a first marker disposed on the drawer; the drawer having a storage case configured to be removable in its upper portion; a second marker disposed on the storage case; a photography control unit that photographs the storage compartment with the camera and generates a photographed image; and a determination unit that determines whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the first marker and the second marker in the photographed image. With this configuration, it is determined whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the first marker and the second marker in the photographed image. Therefore, it is possible to appropriately determine whether the photographed image includes an image of the drawer or an image of the storage case. This allows for accurate management of items stored in the storage facility.

[0310] (Technology 2) The photography system according to Technology 1, wherein the first marker is placed on the upper surface of the front side of the drawer, and the second marker is placed on the upper surface of the front side of the storage case. With this configuration, the first marker is placed on the upper surface of the front side of the drawer, and the second marker is placed on the upper surface of the front side of the storage case, so the camera can easily capture the first marker and the second marker. Therefore, it is possible to properly determine whether the captured image includes an image of the drawer or an image of the storage case. Therefore, the items stored in the storage facility can be managed with high accuracy.

[0311] (Technology 3) The photographing system described in Technology 1 or Technology 2, wherein the first marker includes a first outer frame image, and the second marker includes a second outer frame image, and further comprising a first determination unit that determines whether or not each of the first outer frame image and the second outer frame image can be detected from the photographed image, and when the first determination unit determines that each of the first outer frame image and the second outer frame image can be detected, the determination unit determines that the photographed image includes an image of the storage case, and when the first determination unit determines that the first outer frame image can be detected and the first determination unit determines that the second outer frame image cannot be detected, the determination unit determines that the photographed image includes an image of the drawer. According to this configuration, when the first determination unit determines that both the first outer frame image and the second outer frame image can be detected, the determination unit determines that the captured image includes an image of a storage case. When the first determination unit determines that the first outer frame image can be detected but the first determination unit determines that the second outer frame image cannot be detected, the determination unit determines that the captured image includes an image of a drawer. Thus, it is possible to properly determine whether the captured image includes an image of a drawer or an image of a storage case. Therefore, it is possible to accurately manage the items stored in the storage facility.

[0312] (Technology 4) The photographing system described in Technology 3, wherein the first marker is formed inside the first outer frame image and includes a first identification image that indicates first identification information, and the second marker is formed inside the second outer frame image and includes a second identification image that indicates second identification information, and further comprising a second determination unit that determines whether or not each of the first identification image and the second identification image can be detected from the photographed image, and when the second determination unit determines that each of the first identification image and the second identification image can be detected, the determination unit determines that the photographed image includes an image of the storage case, and when the second determination unit determines that the first identification image can be detected and the second identification image cannot be detected, the determination unit determines that the photographed image includes an image of the drawer. According to this configuration, when the second determination unit determines that both the first identification image and the second identification image can be detected, the determination unit determines that the captured image includes an image of a storage case. When the second determination unit determines that the first identification image can be detected but determines that the second identification image cannot be detected, the determination unit determines that the captured image includes an image of a drawer. Therefore, it is possible to properly determine whether the captured image includes an image of a drawer or an image of a storage case. Therefore, it is possible to accurately manage the items stored in the storage facility.

[0313] (Technology 5) A photography system for photographing a storage cabinet including a storage compartment with a drawer having an opening on its top surface, the photography system comprising: a camera arranged on the top surface of the storage cabinet; a third marker arranged on the drawer; the drawer having a storage case configured to be removable in its upper section; the storage case having an opening / closing body for opening and closing the opening of the storage case, the opening / closing body having two plate-shaped opening / closing members arranged to slide freely in the left-right direction of the storage case; two fourth markers arranged on each of the two plate-shaped opening / closing members; a photography control unit that photographs the storage compartment with the camera and generates a photographed image; and a determination unit that determines whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the third marker and the two fourth markers in the photographed image. According to this configuration, it is determined whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the third marker and the two fourth markers in the photographed image. Therefore, it is possible to appropriately determine whether the photographed image includes an image of the drawer or an image of the storage case. Therefore, the items stored in the storehouse can be managed with high accuracy.

[0314] (Technology 6) The photography system according to Technology 5, wherein the third marker is arranged on the upper surface of the front side of the drawer, and each of the two fourth markers is arranged on the upper surface of the front side of each of the two plate-shaped opening / closing members. With this configuration, the third marker is arranged on the upper surface of the front side of the drawer, and each of the two fourth markers is arranged on the upper surface of the front side of each of the two plate-shaped opening / closing members. Therefore, the camera can easily capture the third marker and the two fourth markers. Therefore, it is possible to properly determine whether the captured image includes an image of the drawer or an image of the storage case. Therefore, the items stored in the storage facility can be managed with high accuracy.

[0315] (Technology 7) The third marker includes a third outer frame image, and each of the two fourth markers includes a fourth outer frame image, and the photographing system is provided with a first determination unit that determines whether the third outer frame image and each of the two fourth outer frame images can be detected from the photographed image, and when the first determination unit determines that the third outer frame image and one of the two fourth outer frame images can be detected and the first determination unit determines that the other of the two fourth outer frame images cannot be detected, the determination unit determines that the photographed image includes an image of the storage case, and when the first determination unit determines that the third outer frame image can be detected and the two fourth outer frame images cannot be detected, the determination unit determines that the photographed image includes an image of the drawer. According to this configuration, if the first determination unit determines that the third outer frame image and one of the two fourth outer frame images can be detected but the first determination unit determines that the other of the two fourth outer frame images cannot be detected, the determination unit determines that the captured image includes an image of a storage case. If the first determination unit determines that the third outer frame image can be detected but the two fourth outer frame images cannot be detected, the determination unit determines that the captured image includes an image of a drawer. Therefore, it is possible to properly determine whether the captured image includes an image of a drawer or an image of a storage case. This allows for accurate management of the items stored in the storage facility.

[0316] (Technology 8) The photographing system described in Technology 7, wherein the third marker is formed inside the third outer frame image and includes a third identification image that indicates third identification information, and each of the two fourth markers is formed inside the fourth outer frame image and includes a fourth identification image that indicates fourth identification information, and further comprising a second determination unit that determines whether the third identification image and each of the two fourth identification images can be detected from the photographed image, wherein if the second determination unit determines that the third identification image and one of the two fourth identification images can be detected and the other of the two fourth identification images cannot be detected, the determination unit determines that the photographed image includes an image of the storage case, and if the second determination unit determines that the third identification image can be detected and the two fourth identification images cannot be detected, the determination unit determines that the photographed image includes an image of the drawer. According to this configuration, when the second determination unit determines that the third identification image and one of the two fourth identification images can be detected but the second determination unit determines that the other of the two fourth identification images cannot be detected, the determination unit determines that the captured image includes an image of a storage case. When the second determination unit determines that the third identification image can be detected but the second determination unit determines that the two fourth identification images cannot be detected, the determination unit determines that the captured image includes an image of a drawer. Thus, it is possible to properly determine whether the captured image includes an image of a drawer or an image of a storage case. This allows for accurate management of the items stored in the storage facility.

[0317] (Technology 9) The photography system according to Technology 1 further includes a third determination unit that selects the best shot of the photographed images, the third determination unit calculating the amount of movement of the photographed images and selecting the photographed image with the smallest amount of movement as the best shot. This configuration makes it possible to prevent the selection of photographed images with a large amount of movement caused by a user taking food in and out, and to select photographed images that properly capture the food in the storage compartment.

[0318] (Technology 10) The photography system according to Technology 9, wherein the third determination unit sets a threshold value for the amount of motion in advance, and after the amount of motion of the photographed image falls below the threshold value, selects the image frame with the latest amount of motion before exceeding the threshold value as the best shot. This configuration makes it possible to prevent the selection of photographed images with a large amount of motion caused by a user taking food in and out, and to select photographed images that properly capture the food in the storage compartment.

[0319] (Technology 11) An imaging method for an imaging system that photographs a storage cabinet that includes a storage compartment with a drawer having an opening on its top surface, the imaging system including a camera arranged on the top surface of the storage cabinet, a first marker arranged on the drawer, the drawer having a storage case configured to be able to be pulled out in an upper section of the drawer, and a second marker arranged on the storage case, the imaging method including: an imaging control step of photographing the storage compartment with the camera and generating a photographed image; and a determination step of determining whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the first marker and the second marker in the photographed image. This configuration achieves the same effects as the imaging system described in Technology 1.

[0320] (Technology 12) A photography method for a photography system that photographs a storage cabinet including a storage compartment with a drawer having an opening on its top surface, the photography system including a camera arranged on the top surface of the storage cabinet, a third marker arranged on the drawer, the drawer having a drawer-configured storage case in its upper section that can be pulled out, the storage case having an opening / closing body that opens and closes the opening of the storage case, the opening / closing body having two plate-shaped opening / closing members arranged to be slidable in the left-right direction of the storage case, and two fourth markers arranged on each of the two plate-shaped opening / closing members, the photography method including: a photography control step of photographing the storage compartment with the camera and generating a photographed image; and a determination step of determining whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the third marker and the two fourth markers in the photographed image. This configuration achieves the same effect as the photography system described in Technology 5.

[0321] (Technology 13) A photography method for a photography system that photographs a storage cabinet that includes a storage compartment with a drawer having an opening on its top surface and that includes a camera arranged on the top surface of the storage cabinet, the photography method including: a photography control step of photographing the storage compartment with the camera and generating photographed images; and a step of calculating the amount of movement of the photographed images in the photographed images and selecting the photographed image with the least amount of movement as the best shot. This configuration makes it possible to prevent the selection of photographed images in which the amount of movement of the user is large when taking food in and out, and to select photographed images that properly photograph the food in the storage compartment.

[0322] (Technology 14) A control program for an imaging system for imaging a storage cabinet including a storage compartment with a drawer having an opening on its top surface, the system including a camera arranged on the top surface of the storage cabinet, a first marker arranged on the drawer, the drawer having a storage case configured to be able to be pulled out in an upper section of the drawer, a second marker arranged on the storage case, and a processor for controlling the camera, the control program causing the processor to function as an imaging control unit that images the storage compartment with the camera and generates a captured image, and a determination unit that determines whether the captured image includes an image of the drawer or an image of the storage case based on the appearance of the first marker and the second marker in the captured image. This configuration achieves the same effects as the imaging system described in Technology 1.

[0323] (Technology 15) A control program for an imaging system for photographing a storage cabinet including a storage compartment with a drawer having an opening on its top surface, the control program including: a camera arranged on the top surface of the storage cabinet; a third marker arranged on the drawer; the drawer having a storage case configured to be able to be pulled out at an upper section of the drawer; the storage case having an opening / closing body for opening and closing the opening of the storage case, the opening / closing body having two plate-shaped opening / closing members arranged to be able to slide left and right in the storage case; a fourth marker arranged on each of the two plate-shaped opening / closing members; and a processor for controlling the camera, the control program causing the processor to function as an imaging control unit that photographs the storage compartment with the camera and generates a photographed image, and a determination unit that determines whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the third marker and the two fourth markers in the photographed image. This configuration achieves the same effect as the imaging system described in Technology 5.

[0324] (Technology 16) A control program for a photography system that photographs a storage cabinet that includes a storage compartment with a drawer having an opening on its top surface, and that includes a camera placed on the top surface of the storage cabinet and a processor that controls the camera, the control program causing the processor to function as a photography control step of photographing the storage compartment with the camera and generating photographed images, and a determination unit that calculates the amount of movement of the photographed images in the photographed images and selects the photographed image with the least amount of movement as the best shot. With this configuration, it is possible to prevent the selection of photographed images in which the amount of movement of the user is large when taking food in and out, and to select photographed images that properly photograph the food in the storage compartment.

[0325] (Technology 17) A marker for a refrigerator compartment that is attached to the refrigerator compartment, comprising a marker main body and a marker support part provided on the underside of the marker main body, and that is attached by sandwiching the marker main body and the marker support part around the hinge part of the door of the refrigerator compartment. With this configuration, the marker can be easily attached to the refrigerator compartment.

[0326] (Technology 18) A marker for a drawer that is attached to a drawer-type storage chamber, the marker comprising: a marker body; an inner piece that extends downward from the rear edge of the marker body and is positioned inside the drawer; and an outer piece that extends downward from the front edge of the marker body and is positioned outside the drawer, the marker being attached to the drawer by sandwiching a handle of the drawer between the inner piece and the outer piece. With this configuration, the marker can be easily attached to a drawer-type storage chamber.

[0327] (Technology 19) A marker for a case to be attached to a case provided in a drawer-type storage chamber, the marker comprising a marker body, a support claw provided on the outer peripheral edge of the underside of the marker body, and a support pin provided on the underside of the marker body, the marker being attached to the slide plate by inserting the support claw and the support pin into the holes in the slide plate that has a plurality of holes on the upper surface of the case. With this configuration, the marker can be easily attached to the case of the drawer-type storage chamber.

[0328] As described above, the photographing system, photographing method, control program, and marker according to the present invention can be used to determine whether a photographed image includes an image of a drawer or an image of a storage case.

[0329] REFRIGERATOR LIST 1 Food management system 2 Refrigerator 3 Photography device 4 Terminal device 5 Food management server 7 Photography system 20 Main box (box) 20A Top surface (upper surface) 21 Refrigerator compartment (storage compartment) 21A Door 22 Ice making compartment (storage compartment) 23 Fresh freezing compartment (storage compartment) 24 Freezer compartment (storage compartment) 25 Vegetable compartment (storage compartment) 26 Storage compartment 22A, 23A, 24A, 25A, 26A Drawer 24B, 25B Storage case 55 Refrigerator compartment marker 56 Marker body 57 Marker support part 58 Marker attachment part 60 Drawer marker 61 Marker body 62 Marker attachment part 63 Inner piece 64 Outer piece 65 Locking part 70 Case marker 71 Marker body 72 Marker attachment part 73 Support claw 74 Engagement portion 75 Support pin 80 Hinge portion 81 Hinge plate 82 Support shaft 83 Wide portion 84 Narrow portion 85 Vegetable compartment 86 Vegetable compartment case 87 Side plates 88 Handle portion 89 Freezer compartment 90 Freezer compartment case 91 Side plates 92 Sliding plate 93 Slots 241B, 242B Opening and closing member 301 Wide-angle camera 302 Narrow-angle camera (camera) 350 Second processor (processor) 351 First communication control unit 352 Second communication control unit 353 Opening / closing determination unit 354 Shooting control unit 355 Recording control unit 355A First determination unit 355B Second determination unit 355C Determination unit 356 Generation unit 357 Third determination unit 360 Second memory 361 Control program 363 Recorded data DV1 First recorded data DV2 Second recorded data MK Marker MK41 Third marker MF41 Third outer frame image MD41 Third identification image MK421, MK422 MK42 Fourth marker MF42 Fourth outer frame image MD42 Fourth identification image MK51 First marker MF51 First outer frame image MD51 First identification image MK52 Second marker MF52 Second outer frame image MD52 Second identification image PB2 Captured image P1 First captured image P2 Second captured image

Claims

1. An photography system comprising: a camera arranged on the top surface of a storage facility which photographs a storage room having a drawer with an opening formed on the top surface thereof; a first marker arranged on the drawer; the drawer having a storage case configured to be removable in an upper section of the drawer, and a second marker arranged on the storage case; a photography control unit which photographs the storage room with the camera and generates a photographed image; and a determination unit which determines whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the first marker and the second marker in the photographed image.

2. The photography system according to claim 1, wherein the first marker is arranged on the upper surface of the front side of the drawer, and the second marker is arranged on the upper surface of the front side of the storage case.

3. The photographing system according to claim 1 or claim 2, further comprising a first determination unit that determines whether or not each of the first outer frame image and the second outer frame image can be detected from the captured image, wherein when the first determination unit determines that each of the first outer frame image and the second outer frame image can be detected, the determination unit determines that the captured image includes an image of the storage case, and when the first determination unit determines that the first outer frame image can be detected and the first determination unit determines that the second outer frame image cannot be detected, the determination unit determines that the captured image includes an image of the drawer.

4. The photographing system of claim 3, further comprising: a second determination unit that determines whether or not each of the first identification image and the second identification image can be detected from the captured image; when the second determination unit determines that each of the first identification image and the second identification image can be detected, the determination unit determines that the captured image includes an image of the storage case; and when the second determination unit determines that the first identification image can be detected and the second identification image cannot be detected, the determination unit determines that the captured image includes an image of the drawer.

5. An imaging system for photographing a storage facility having a storage compartment with a drawer having an opening formed on its top surface, comprising: a camera arranged on the top surface of the storage facility; a third marker arranged on the drawer; the drawer having a storage case configured to be able to be pulled out at an upper section of the drawer, the storage case having an opening and closing body for opening and closing the opening of the storage case, the opening and closing body having two plate-shaped opening and closing members arranged to be able to slide freely in the left and right direction of the storage case, and two fourth markers arranged on each of the two plate-shaped opening and closing members; an imaging control unit that photographs the storage compartment with the camera and generates a photographed image; and a determination unit that determines whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the third marker and the two fourth markers in the photographed image.

6. The photographing system described in claim 5, wherein the third marker is arranged on the upper surface of the front side of the drawer, and each of the two fourth markers is arranged on the upper surface of the front side of each of the two plate-shaped opening and closing members.

7. The photographing system according to claim 5 or claim 6, further comprising: a first determination unit that determines whether or not the third outer frame image and each of the two fourth outer frame images can be detected from the photographed image; when the first determination unit determines that the third outer frame image and one of the two fourth outer frame images can be detected, and when the first determination unit determines that the other of the two fourth outer frame images cannot be detected, the determination unit determines that the photographed image includes an image of the storage case; and when the first determination unit determines that the third outer frame image can be detected, and when the first determination unit determines that the two fourth outer frame images cannot be detected, the determination unit determines that the photographed image includes an image of the drawer.

8. The photographing system according to claim 7, further comprising: a second determination unit that determines whether or not the third identification image and each of the two fourth identification images can be detected from the captured image; when the second determination unit determines that the third identification image and one of the two fourth identification images can be detected and the other of the two fourth identification images cannot be detected, the determination unit determines that the captured image includes an image of the storage case; and when the second determination unit determines that the third identification image can be detected and the two fourth identification images cannot be detected, the determination unit determines that the captured image includes an image of the drawer.

9. The photographing system according to claim 1, further comprising a third determination unit that selects a best shot from among the photographed images, wherein the third determination unit calculates an amount of motion of the photographed images and selects an image having a small amount of motion as the best shot.

10. The photographing system according to claim 9, wherein the third judgment unit sets a threshold value for the amount of motion in advance, and after the amount of motion of the photographed image falls below the threshold value, adopts as the best shot the image frame with the latest amount of motion before exceeding the threshold value.

11. A method for photographing a storage facility having a storage room with a drawer with an opening formed on the top surface thereof, the method comprising: a camera arranged on the top surface of the storage facility; a first marker arranged on the drawer; the drawer having a storage case configured to be removable in an upper portion of the drawer; and a second marker arranged on the storage case, the method comprising: a photographing control step of photographing the storage room with the camera and generating a photographed image; and a determination step of determining whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the first marker and the second marker in the photographed image.

12. A photography method for a photography system that photographs a storage room having a drawer with an opening formed on the top surface thereof, the photography system comprising: a camera arranged on the top surface of the storage room; a third marker arranged on the drawer; the drawer having a storage case configured to be able to be pulled out at an upper section of the drawer, the storage case having an opening / closing body that opens and closes the opening of the storage case, the opening / closing body having two plate-shaped opening / closing members arranged to slide freely in the left-right direction of the storage case, and two fourth markers arranged on each of the two plate-shaped opening / closing members, the photography method comprising: a photography control step of photographing the storage room with the camera and generating a photographed image; and a determination step of determining whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the third marker and the two fourth markers in the photographed image.

13. A method for photographing a storage facility having a storage compartment with a drawer with an opening formed on the top surface thereof, using an imaging system including a camera disposed on the top surface of the storage facility, the method comprising: an imaging control step of photographing the storage compartment with the camera and generating a photographed image; and a step of calculating an amount of movement of the photographed image in the photographed image and selecting the photographed image with the least amount of movement as a best shot.

14. A control program for an photography system that photographs a storage facility having a storage room with a drawer with an opening formed on the top surface, the control program comprising: a camera placed on the top surface of the storage facility; a first marker placed on the drawer, the drawer having a storage case configured to be pulled out at an upper section of the drawer, a second marker placed on the storage case, and a processor that controls the camera, the control program causing the processor to function as: an photography control unit that photographs the storage room with the camera and generates a photographed image; and a determination unit that determines whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the first marker and the second marker in the photographed image.

15. A control program for an imaging system for photographing a storage facility having a storage room with a drawer with an opening formed on the top surface thereof, the control program comprising: a camera arranged on the top surface of the storage facility; a third marker arranged on the drawer; the drawer having a storage case configured to be removable in an upper section of the drawer, the storage case having an opening / closing body for opening and closing the opening of the storage case, the opening / closing body having two plate-shaped opening / closing members arranged to slide freely in the left-right direction of the storage case, two fourth markers arranged on each of the two plate-shaped opening / closing members; and a processor for controlling the camera, the control program causing the processor to function as: an imaging control unit that photographs the storage room with the camera and generates a photographed image; and a determination unit that determines whether the photographed image includes an image of the drawer or an image of the storage case based on the appearance of the third marker and the two fourth markers in the photographed image.

16. A control program for an photography system for photographing a storage facility having a storage compartment with a drawer with an opening formed on the top surface thereof, the control program comprising a camera disposed on the top surface of the storage facility and a processor for controlling the camera, the control program comprising: a photography control step of photographing the storage compartment with the camera and generating a photographed image; and a control program for causing the processor to function as a determination unit that calculates the amount of movement of the photographed image in the photographed image and selects the photographed image with the least amount of movement as the best shot.

17. A marker for a refrigerator compartment to be attached to a refrigerator compartment, comprising: a marker body; and a marker support part provided on the underside of the marker body, the marker being attached by clamping the marker body and the marker support part to a hinge part of the door of the refrigerator compartment.

18. A drawer marker to be attached to a drawer-type storage compartment, comprising: a marker body, an inner piece extending downward from the rear edge of the marker body and positioned on the inside of the drawer, and an outer piece extending downward from the front edge of the marker body and positioned on the outside of the drawer, the marker being attached to the drawer by sandwiching a handle portion of the drawer between the inner piece and the outer piece.

19. A marker for a case to be attached to a case provided in a drawer-type storage chamber, comprising: a marker body, a support claw provided on the outer peripheral edge of the underside of the marker body, and a support pin provided on the underside of the marker body, the marker being attached to the slide plate by inserting the support claw and support pin into the holes of the slide plate which has a plurality of holes on the upper surface of the case.