Photographing device, photographing method, and control program

Abstract

To make a narrow-angle camera start photographing at an appropriate timing.SOLUTION: A photographing device includes: a narrow-angle camera 302 photographing a refrigerator 2 provided with a vegetable chamber 25 having a drawer 25A forming an opening at its upper surface and disposed at a top surface 20A of a main box body 20 of the refrigerator 2; a thermopile 305 disposed at the top surface 20A of the main box body 20 of the refrigerator 2 and detecting the temperature of an object disposed at the front surface side of the refrigerator 2; and a photographing control part 354 starting photographing by the narrow-angle camera 302 on the basis of the detection result of the thermopile 305.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to an imaging device, an imaging method, and a control program. [Background technology]

[0002] Patent Document 1 discloses a technology in which a camera is installed on the top of a refrigerator, photographs an object when it detects passing through the opening of the refrigerator, and manages the food stored in the refrigerator based on the photographed result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-070476 A Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an imaging device, an imaging method, and a control program for accurately managing food stored in a refrigerator. [Means for solving the problem]

[0005] The photographing device of the present disclosure photographs a refrigerator having a first storage compartment with a drawer with a first opening formed on its top surface, and includes a first camera arranged on the top surface of the refrigerator's box, a temperature sensor arranged on the top surface of the refrigerator's box and detecting the temperature of an object arranged on the front side of the refrigerator, and an photographing control unit that starts photographing by the first camera based on the detection result of the temperature sensor.

[0006] Furthermore, an imaging method disclosed herein is a method for imaging a refrigerator having a first storage compartment with a drawer with a first opening formed on an upper surface thereof, using an imaging device that includes a first camera arranged on an upper surface of a body of the refrigerator and a temperature sensor arranged on the upper surface of the body of the refrigerator and detects the temperature of an object arranged on the front side of the refrigerator, and includes an imaging start step of starting imaging by the first camera based on the detection result of the temperature sensor.

[0007] Further, the control program of the present disclosure is a control program for an imaging device that photographs a refrigerator having a first storage compartment with a drawer with a first opening formed on an upper surface thereof, the control program including a first camera arranged on an upper surface of a body of the refrigerator, a temperature sensor arranged on the upper surface of the body of the refrigerator and detecting the temperature of an object arranged on the front side of the refrigerator, and a processor, and causes the processor to function as an imaging control unit that starts imaging by the first camera based on the detection result of the temperature sensor. Effect of the Invention

[0008] The photographing device, photographing method, and control program disclosed herein are arranged on the top surface of the refrigerator box, and start photographing with the first camera based on the detection result of a temperature sensor that detects the temperature of an object arranged on the front side of the storage compartment. Therefore, it is possible to make the first camera start photographing at an appropriate timing. Therefore, it is possible to accurately manage the food stored in the refrigerator. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 shows a configuration of a food management system according to a first embodiment. [Diagram 2] FIG. 1 is a perspective view of a refrigerator according to a first embodiment. [Diagram 3] FIG. 1 is a side view of the imaging device according to the first embodiment, seen from the right side. [Figure 4] FIG. 11 is a perspective view of a restricting member abutting against a front edge of a main box body according to the first embodiment, as viewed from the left front side; [Diagram 5]FIG. 1 is a front view of a refrigerator and an image capturing device according to a first embodiment; [Figure 6] FIG. 1 is a side view of a refrigerator and an imaging device according to a first embodiment, viewed from the right. [Figure 7] FIG. 1 is a plan view of an image capturing device attached to a refrigerator according to the first embodiment, seen from above; [Figure 8] FIG. 1 is a block diagram showing a configuration of an imaging device according to a first embodiment. [Figure 9] FIG. 1 is a diagram showing an example of a temperature detection state of a thermopile in the first embodiment; [Figure 10] FIG. 13 is a diagram showing another example of a temperature detection state of the thermopile in the first embodiment. [Figure 11] FIG. 11 is a diagram showing yet another example of a temperature detection state of a thermopile in the first embodiment. [Figure 12] FIG. 1 is a diagram showing an example of an image capturing state of a narrow-angle camera in the first embodiment; [Figure 13] FIG. 13 is a diagram showing another example of the image capturing state of the narrow-angle camera in the first embodiment. [Figure 14] FIG. 1 shows an example of a marker placed in a refrigerator compartment in the first embodiment. [Figure 15] FIG. 1 is a diagram showing an example of a hinge cover according to the first embodiment; [Figure 16] FIG. 1 shows an example of a marker configuration according to the first embodiment. [Figure 17] FIG. 1 is a block diagram showing the configuration of a terminal device and a food management server according to a first embodiment. [Figure 18] 1 is a flowchart showing the operation of the photographing device according to the first embodiment. [Figure 19] A flowchart showing the operation of the photographing device and the food management server in the first embodiment. [Figure 20] A flowchart showing the operation of the terminal device and the food management server in the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (The knowledge and other information that formed the basis of this disclosure) At the time the inventors came up with the idea of ​​the present disclosure, there was a technology in which a camera was installed on top of a refrigerator, a photograph was taken when an object passing through the opening of the refrigerator was detected, and the food stored in the refrigerator was managed based on the photographed results. The inventors have discovered that in order to generate a proper captured image, it is necessary to have the camera start capturing images at the proper timing, and have come to form the subject of the present disclosure in order to solve this problem. Therefore, the present disclosure provides an imaging device, an imaging method, and a control program that cause a camera to start imaging at an appropriate timing.

[0011] Hereinafter, the embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed explanations than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations of substantially the same configurations may be omitted. It should be noted that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0012] (Embodiment 1) [1-1. Configuration] [1-1-1. Food control system configuration] FIG. 1 is a diagram showing the configuration of a food management system 1 in the first embodiment. Food management system 1 is a system that manages food stored in refrigerator 2. Food management system 1 of this embodiment manages images of food stored in refrigerator 2 as an example of food management.

[0013] In this embodiment, the food stored in refrigerator 2 includes the food itself, the container in which the food is stored, etc. The food may be marked with a predetermined marker.

[0014] The food management system 1 includes a refrigerator 2 . The refrigerator 2 includes a main box 20 with an opening at the front. A refrigerator compartment 21, an ice-making compartment 22, a fresh-freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25 are formed in the main box 20. These compartments are cooled with cold air. A revolving door 21A is provided in an opening 201 at the front of the refrigerator compartment 21. The door 21A opens and closes the opening 201. The door 21A includes a revolving right door 21B and a revolving left door 21C. The refrigerator compartment 21 corresponds to an example of a "second storage compartment." The opening 201 corresponds to an example of a "second opening." The door 21A corresponds to an example of an "opening / closing body."

[0015] Drawers 22A, 23A, 24A and 25A capable of accommodating food are installed in each of ice making compartment 22, fresh freezing compartment 23, freezing compartment 24 and vegetable compartment 25. Each of drawers 22A to 25A has an open top when installed. The openings formed in the upper surface of each of drawers 22A to 25A correspond to an example of a "first opening." In the following description, when drawers 22A, 23A, 24A, and 25A are referred to collectively without distinction, they will be given the symbol "26A" and described as "drawer 26A." Also, when ice-making compartment 22, fresh-freezing compartment 23, freezer compartment 24, and vegetable compartment 25 are referred to collectively without distinction, they will be given the symbol "26" and described as storage compartment 26. Each of ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25 corresponds to an example of a "first storage compartment" of the present disclosure. In the first embodiment, a case will be described in which the "first storage compartment" of the present disclosure is the vegetable compartment 25. Also, in the first embodiment, a case will be described in which the "second storage compartment" of the present disclosure is the refrigerator compartment 21.

[0016] The food management system 1 includes an imaging device 3. The imaging device 3 is provided on a top surface 20A of a main box 20 of a refrigerator 2. The top surface 20A of the main box 20 is made of metal. The imaging device 3 takes an image of the refrigerator 2. The top surface 20A corresponds to an example of the "upper surface" in the disclosure.

[0017] Food management system 1 includes terminal device 4. Terminal device 4 is a portable computer such as a smartphone. An application program related to the management of food stored in refrigerator 2 is installed on terminal device 4, and communicates with food management server 5 using the functions of this application program. In the following description, this application program will be referred to as the "food management application" and will be given the reference number "411."

[0018] 1, a user P who is at home is indicated by a solid line, and a user P who has left home H is indicated by a dotted line. When used by user P who is at home, the terminal device 4 communicates with the food management server 5 via the communication device 6 or without the communication device 6. When used by user P who has left home H and a communication connection with the communication device 6 cannot be established, the terminal device 4 communicates with the food management server 5 without the communication device 6.

[0019] The communication device 6 is connected to a network NW constituted by a public line network or the like, and communicates with the food management server 5 via the network NW. The communication device 6 functions as an interface device for connecting the refrigerator 2, the photographing device 3, and the terminal device 4 to the network NW. The communication device 6 establishes a local network in home H.

[0020] Food management system 1 includes food management server 5. Food management server 5 is a device that manages the food stored in refrigerator 2. Food management server 5 stores images of food as part of its food management. Food management server 5 is connected to network NW. Note that in FIG. 1, food management server 5 is represented by a single block, but this does not necessarily mean that food management server 5 is composed of a single device.

[0021] [1-1-2. Refrigerator configuration] FIG. 2 is a perspective view of the refrigerator 2 in an installed state. FIG. 2 illustrates an X-axis, a Y-axis, and a Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to each other. The Z-axis indicates the up-down direction, which corresponds to the height direction of the refrigerator 2. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction. The left-right direction corresponds to the width direction of the refrigerator 2. The Y-axis indicates the front-rear direction. The positive direction of the X-axis indicates the rightward direction. The positive direction of the Y-axis indicates the forward direction. The positive direction of the Z-axis indicates the upward direction. Each of Figures 3-7 and 9-14 also illustrates an X-axis, a Y-axis, and a Z-axis.

[0022] The refrigerator 2 comprises a main box 20, a door 21A, and a drawer 26A. The main box body 20 is a box-shaped member forming the main body of the refrigerator 2. The main box body 20 includes an outer box 202 that opens to the front, an inner box 203 that is housed inside the outer box 202 and opens to the front, and a heat insulating material filled in the space between the outer box 202 and the inner box 203. The outer box 202 is formed of, for example, a metal such as iron or a hard resin such as ABS. The upper surface of the outer box 202, i.e., the top surface 20A of the main box 20, is formed of a metal. The inner box 203 is formed of, for example, a hard resin such as ABS. For example, a hard urethane foam is used as the heat insulating material. Note that the heat insulating material is not limited to this, and may be any material that has heat insulating properties and can be filled in the space between the outer box 202 and the inner box 203. The main box 20 corresponds to an example of a "box" of the present disclosure.

[0023] The internal space of the inner box 203 is divided into a plurality of spaces by partition members. In this embodiment, the internal space of the inner box 203 is divided into five spaces, and the five spaces function as a refrigerator compartment 21, an ice-making compartment 22, a fresh-freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25, respectively.

[0024] The right door 21B is rotatably supported by a right door hinge member 204. The right door hinge member 204 is a hinge member installed on the top surface 20A of the main box body 20. The left door 21C is rotatably supported by a left door hinge member 205. The left door hinge member 205 is a hinge member installed on the top surface 20A of the main box body 20.

[0025] A second wireless communication module 206 is installed in the right door hinge member 204. The second wireless communication module 206 is a module that receives control data for controlling the refrigerator 2 from an external device and transmits predetermined data to the external device.

[0026] [1-1-3. Configuration of the imaging device] The photographing device 3 will be described with reference to FIGS. 3 to 7 show the same X-axis, Y-axis, and Z-axis as those shown in Fig. 2. The X-axis direction corresponds to the width direction and the left-right direction of the imaging device 3. The Y-axis direction corresponds to the front-rear direction of the imaging device 3. The Z-axis direction corresponds to the up-down direction of the imaging device 3.

[0027] Fig. 3 is a side view of the imaging device 3 attached to the refrigerator 2, seen from the right. Fig. 4 is a perspective view of the restricting member 34 abutting against the front edge 20B of the main box body 20, seen from the left front.

[0028] 3 and 4, the photographing device 3 includes a main body 30 installed on the top surface 20A of the main box 20, and a photographing member 31 extending forward from the main body 30. The photographing member 31 is provided above the front of the main body 30.

[0029] The main body 30 has, on the front surface, a power button 32 for turning on and off the power of the photographing device 3. The power button 32 is provided with a first LED (Light emitting diode) 33 for notifying predetermined information.

[0030] Furthermore, main body 30 has, on the front surface, a shooting start button 32A for starting shooting with imaging device 3. When shooting start button 32A is pressed by user P, shooting control unit 354 shown in Fig. 8 causes wide-angle camera 301 and narrow-angle camera 302 to start shooting. When shooting start button 32A is pressed, recording control unit 355 shown in Fig. 8 starts recording the results of shooting with wide-angle camera 301 and narrow-angle camera 302.

[0031] Note that user P presses shooting start button 32A when wide-angle camera 301 and narrow-angle camera 302 have not started shooting even though door 21A or drawer 26A is in the open state. When making wide-angle camera 301 and narrow-angle camera 302 start shooting, shooting control unit 354 turns on second LED 308 and third LED 309. Thus, user P can recognize that wide-angle camera 301 and narrow-angle camera 302 have not started shooting because second LED 308 and third LED 309 are not turned on.

[0032] The main body 30 includes a regulating member 34. The regulating member 34 abuts against the front end edge 20B of the main box body 20 to regulate the position of the image capturing member 31 in the front-rear direction of the refrigerator 2. The regulating member 34 is a plate-like member extending downward from the bottom surface 30A of the main body 30. The regulating member 34 extends in the width direction of the main body 30 (the width direction of the image capturing device 3) on the front side of the main body 30, and has a mark 35 at the center of the width direction of the main body 30. The mark 35 indicates the center of the image capturing device 3 in the width direction. The length of the regulating member 34 extending from the bottom surface 30A is shorter than the length from the top surface 20A of the main box body 20 to a door packing provided approximately parallel to the top surface 20A at the upper inner surface of the door 21A.

[0033] Fig. 5 is a front view of refrigerator 2 and the camera device 3 attached to refrigerator 2, as viewed from the front. Door 21A of refrigerator 2 shown in Fig. 5 is in an open state. Fig. 6 is a side view of refrigerator 2 and the camera device 3 attached to refrigerator 2, as viewed from the right. The refrigerator 2 shown in Fig. 6 has drawer 25A in an open state, i.e., pulled out. Fig. 7 is a plan view of camera device 3 attached to refrigerator 2, as viewed from above. The configuration of the photographing device 3 will be described with reference to FIGS.

[0034] The photographing device 3 includes a wide-angle camera 301 and a narrow-angle camera 302. The wide-angle camera 301 and the narrow-angle camera 302 are installed at the front end of the bottom surface 31A of the photographing member 31. When the photographing device 3 is attached to the refrigerator 2 and the door 21A is in a closed state, the wide-angle camera 301 and the narrow-angle camera 302 are located in front of the door 21A.

[0035] Wide-angle camera 301 is a camera with a wider angle than narrow-angle camera 302. With imaging device 3 attached to refrigerator 2, wide-angle camera 301 captures an image of refrigerating compartment 21 from above and in front of refrigerator 2. For example, the shooting range of wide-angle camera 301 includes range A1 shown in Fig. 5 when viewed from the front of refrigerator 2. Range A1 is a range that includes door pockets 21E of right door 21B and left door 21C and opening 201 of refrigerator compartment 21 when right door 21B and left door 21C are in the maximum open state. Moreover, the shooting range of wide-angle camera 301 includes range A3 shown in Fig. 6 when viewed from the right of refrigerator 2. Range A3 is a range that includes, in the front-to-back direction, from the front ends of right door 21B and left door 21C in the open state to the front end of shelf 21F, and also includes opening 201 of refrigerator compartment 21 in the up-down direction. The wide-angle camera 301 corresponds to an example of the "second camera" of the present disclosure.

[0036] Narrow-angle camera 302 is a camera with a narrower angle than wide-angle camera 301. In other words, the shooting angle of narrow-angle camera 302 in the left-right direction, i.e., the X-axis direction, is smaller than the shooting angle of wide-angle camera 301 in the left-right direction, i.e., the X-axis direction. Also, the shooting angle of narrow-angle camera 302 in the front-back direction, i.e., the Y-axis direction, is smaller than the shooting angle of wide-angle camera 301 in the front-back direction, i.e., the Y-axis direction. With the imaging device 3 attached to the refrigerator 2, the narrow-angle camera 302 captures an image of the drawer 26A from above and in the front of the refrigerator 2. For example, the imaging range of narrow-angle camera 302 includes range A2 shown in FIG. 5 when viewed from the front of refrigerator 2. Range A2 is a range that includes drawer 26A when viewed from the front of refrigerator 2. Also, for example, the imaging range of narrow-angle camera 302 includes range A4 shown in FIG. 6 when viewed from the right of refrigerator 2. Range A4 is a range that includes drawer 26A when it is pulled out to the maximum in the front-to-rear direction. The imaging range of narrow-angle camera 302 may be a range that includes drawers provided in refrigeration compartment 21. The narrow-angle camera 302 corresponds to an example of a "first camera" in the present disclosure.

[0037] 5-7, narrow-angle camera 302 constitutes a part of image capturing device 3 in embodiment 1. Image capturing device 3 in embodiment 1 includes narrow-angle camera 302, thermopile 305, and image capturing control unit 354. In other words, image capturing device 3 includes narrow-angle camera 302, thermopile 305, and image capturing control unit 354. The imaging control unit 354 will be described with reference to FIG.

[0038] The photographing device 3 includes a first distance measuring sensor 303 and a second distance measuring sensor 304. The first distance measuring sensor 303 and the second distance measuring sensor 304 are installed in the main body 30. The first distance measuring sensor 303 is a sensor that detects the opening and closing of the right door 21B by detecting the distance between itself and the right door 21B. For example, the first distance measuring sensor 303 is an infrared sensor. The first distance measuring sensor 303 detects the distance between itself and the right door 21B, and outputs a detection value indicating the detected distance. The first distance measuring sensor 303 detects the distance by, for example, a change in the optical axis. The second distance measuring sensor 304 is a sensor that detects the opening and closing of the left door 21C by detecting the distance between itself and the left door 21C. For example, the second distance measuring sensor 304 is an infrared sensor. The second distance measuring sensor 304 detects the distance between itself and the left door 21C and outputs a detection value indicating the detected distance. The second distance measuring sensor 304 detects the distance by, for example, a change in the optical axis. Each of the first distance measurement sensor 303 and the second distance measurement sensor 304 corresponds to an example of a "second detection sensor."

[0039] In this embodiment, the second detection sensor is a distance sensor, but the present disclosure is not limited to this. The second detection sensor may be any of a photo sensor, a hall sensor, an acceleration sensor, and a temperature sensor.

[0040] The imaging device 3 includes a thermopile 305. The thermopile 305 is a sensor that detects whether the drawer 26A is open or closed. The thermopile 305 is installed at the front end of the bottom surface 31A of the imaging member 31. The thermopile 305 outputs a detection value indicating the detected temperature. The thermopile 305 corresponds to an example of a "temperature sensor" in the present disclosure. Thermopile 305 is further described with reference to Figures 9-11.

[0041] The photographing device 3 includes a buzzer 306. The buzzer 306 notifies the user of a change in the state of the refrigerator 2. For example, when at least one of the right door 21B and the left door 21C is set to a predetermined opening / closing angle θ, the buzzer 306 outputs a buzzer sound. The predetermined opening / closing angle θ is, for example, 80 degrees to 100 degrees. Whether the opening / closing angle θ is the predetermined opening / closing angle θ is determined based on a photographed image PA1 captured by the narrow-angle camera 302. The photographed image PA1 includes an image of the first marker MK1. The photographed image PA1 is stored in the state storage unit 364. For example, when the right door 21B is opened from a closed state to a predetermined opening / closing angle θ, the buzzer 306 outputs a buzzer sound. Also, for example, when the right door 21B is opened from a predetermined opening / closing angle θ to a closed state, the buzzer 306 outputs a buzzer sound. The first marker MK1 will be further described with reference to FIGS.

[0042] Furthermore, for example, when the state of the second marker MK2 arranged in the drawer 26A changes from not being at a predetermined position to being at a predetermined position, the buzzer 306 outputs a buzzer sound. Whether or not the second marker MK2 is at the predetermined position is determined based on the captured image PA2 captured by the narrow-angle camera 302. The captured image PA2 includes an image of the second marker MK2. The captured image PA2 is stored in the state storage unit 364. Also, for example, when the state where the second marker MK2 arranged in the drawer 26A is in a predetermined position changes to a state where the second marker MK2 is not in the predetermined position, the buzzer 306 outputs a buzzer sound. The second marker MK2 will be further described with reference to Figures 12-13. Moreover, when there is no need to distinguish between the first marker MK1 and the second marker MK2, they will be referred to as marker MK. The marker MK includes the first marker MK1 and the second marker MK2.

[0043] The photographing device 3 includes an illuminance sensor 307. The illuminance sensor 307 is a sensor for adjusting the exposure of the wide-angle camera 301 and the narrow-angle camera 302. The illuminance sensor 307 outputs a detection value. The illuminance sensor 307 is installed at the front end of the photographing member 31.

[0044] The photographing device 3 is equipped with a second LED 308 and a third LED 309. The second LED 308 is installed at the right front end of the bottom surface 31A of the photographing member 31. The third LED 309 is installed at the left front end of the bottom surface 31A of the photographing member 31. The second LED 308 and the third LED 309 illuminate the refrigerator 2 to which the photographing device 3 is attached from above.

[0045] As shown in FIG. 7, the photographing device 3 includes a first control board 311 and a second control board 312.

[0046] The first control board 311 is a board that controls various sensors included in the photographing device 3 and the power supply of the photographing device 3. The first control board 311 is installed inside the photographing device 3. The power button 32, the shooting start button 32A, the first LED 33, the second LED 308, the third LED 309, the first distance measuring sensor 303, the second distance measuring sensor 304, the thermopile 305, the buzzer 306, and the illuminance sensor 307 are connected to the first control board 311. The first control board 311 controls these devices. The first control board 311 is installed behind the second control board 312 and the first wireless communication module 313. The first control board 311 may be configured as a board that controls various sensors included in the photographing device 3 and a board that controls the power supply of the photographing device 3.

[0047] Second control board 312 is a board that controls wide-angle camera 301 and narrow-angle camera 302, and also controls first wireless communication module 313. Second control board 312 is installed inside imaging device 3. Wide-angle camera 301, narrow-angle camera 302, and first wireless communication module 313 are connected to second control board 312. Second control board 312 controls these devices. Second control board 312 is installed in front of first control board 311, alongside first control board 311.

[0048] The first wireless communication module 313 is a device equipped with hardware such as an antenna and a wireless communication circuit. The first wireless communication module 313 is installed to the left of the second control board 312 inside the photographing member 31. The first wireless communication module 313 is installed in the photographing device 3 so as to be located forward of the opening 201 of the main box 20 when the photographing device 3 is attached to the refrigerator 2. More specifically, the first wireless communication module 313 is installed forward of the restricting member 34 in the photographing device 3.

[0049] As shown in FIG. 7, when the image capturing device 3 is attached to the refrigerator 2, a metal plate 314 is placed between the first wireless communication module 313 and the second wireless communication module 206 in the left-right direction.

[0050] FIG. 8 is a block diagram showing the configuration of the photographing device 3. As shown in FIG. 8, the first control board 311 includes a first processor 320 such as a CPU (Central Processing Unit), a first memory 330, a first communication module 340, and an interface circuit to which other devices and sensors are connected.

[0051] The first memory 330 is a storage device that stores programs and data executed by the first processor 320. The first memory 330 is configured by a non-volatile storage device such as a ROM (Read Only Memory). The first memory 330 may include a volatile storage device that configures a work area of ​​the first processor 320, such as a RAM (Random Access Memory). The first memory 330 stores data processed by the first processor 320 and a control program 331 executed by the first processor 320.

[0052] The first communication module 340 includes hardware such as a communication circuit and a connector that comply with a predetermined wired communication standard, and communicates with the second control board 312 under the control of the first processor 320 .

[0053] The first processor 320 reads and executes the control program 331 to turn the power of the image capturing device 3 on and off in response to the operation of the power button 32 . In addition, the first processor 320 reads and executes a control program 331 to turn on and off the first LED 33, the second LED 308, and the third LED 309. In addition, the first processor 320 reads and executes the control program 331 to transmit detection values ​​output by each sensor connected to the first control board 311 to the second control board 312 via the first communication module 340.

[0054] The second control board 312 includes a second processor 350 such as a CPU, a second memory 360, a second communication module 370, and an interface circuit to which other devices and sensors are connected.

[0055] The second memory 360 is a storage device that stores programs and data executed by the second processor 350. The second memory 360 is configured by a non-volatile storage device such as a ROM. The second memory 360 may also include a volatile storage device such as a RAM that configures a work area for the second processor 350. The second memory 360 stores data processed by the second processor 350, a control program 361 executed by the second processor 350, and an imaging device ID (Identification) 362. The camera device ID 362 is information for uniquely identifying the camera device 3 . The second processor 350 corresponds to an example of a "processor."

[0056] The second communication module 370 includes hardware such as a communication circuit and a connector that comply with a predetermined wired communication standard, and communicates with the first control board 311 under the control of the second processor 350.

[0057] The second processor 350 reads out and executes the control program 361 stored in the second memory 360, thereby functioning as a first communication control unit 351, a second communication control unit 352, an open / close determination unit 353, an image capture control unit 354, a recording control unit 355, and a generation unit 356. The second processor 350 also reads out and executes the control program 361 stored in the second memory 360, thereby causing the second memory 360 to function as a state storage unit 364. The control program 361 corresponds to an example of a "control program" of the present disclosure.

[0058] The first communication control unit 351 communicates with the food management server 5 via the first wireless communication module 313.

[0059] The second communication control unit 352 communicates with the first control board 311 via the second communication module 370. The second communication control unit 352 receives detection values ​​of various sensors connected to the first control board 311 from the first control board 311.

[0060] The open / close determination unit 353 determines whether the right door 21B is in an open state or a closed state based on the detection value of the first distance measurement sensor 303 received by the second communication control unit 352. When the second communication control unit 352 receives a detection value different from the detection value corresponding to the closed state, the open / close determination unit 353 determines that the right door 21B is in an open state. For example, when the detection value of the first distance measurement sensor 303 is equal to or greater than a predetermined value, the open / close determination unit 353 determines that the right door 21B is in an open state. On the other hand, when the second communication control unit 352 receives a detection value corresponding to the closed state, the open / close determination unit 353 determines that the right door 21B is in a closed state. For example, when the detection value of the second distance measurement sensor 304 is less than a predetermined value, the open / close determination unit 353 determines that the right door 21B is in a closed state.

[0061] The open / close determination unit 353 determines whether the left door 21C is open or closed based on the detection value of the second distance measurement sensor 304 by the second communication control unit 352, in the same manner as the determination for the right door 21B.

[0062] Open / close determination section 353 determines whether drawer 26A is open or closed, based on the detection value of thermopile 305 received by second communication control section 352. For example, when the second communication control unit 352 receives a detection value of the thermopile 305 indicating a temperature rise equal to or greater than the first threshold value TH1, the open / close determination unit 353 determines that the drawer 26A has changed from the closed state to the open state. The first threshold value TH1 is, for example, 5°C. Furthermore, when the second communication control unit 352 receives a detection value of the thermopile 305 indicating a temperature drop equal to or greater than the second threshold value TH2, the open / close determination unit 353 determines that the drawer 26A has changed from the closed state to the open state. The second threshold value TH2 is, for example, 5°C. The processing of the open / close determination unit 353 will be further described with reference to FIGS.

[0063] Furthermore, for example, when the second communication control unit 352 receives a detection value of the thermopile 305 indicating that a temperature within a predetermined range continues for a predetermined period of time or longer, the open / close determination unit 353 determines that the drawer 26A is closed. The predetermined temperature range is, for example, 15° C. or more and 30° C. or less. The predetermined period is, for example, 5 seconds. The predetermined temperature range corresponds to the temperature of a region FLA of the floor surface FL shown in FIG.

[0064] When the opening / closing determination unit 353 determines that the door 21A or the drawer 26A is in the open state, the photographing control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start photographing. When the photographing control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start photographing, the second communication control unit 352 transmits lighting request information to the first control board 311. The lighting request information is information requesting lighting of the second LED 308 and the third LED 309. When the first processor 320 receives the lighting request information, it turns on the second LED 308 and the third LED 309, and notifies the user P that photographing has started. Furthermore, when the opening / closing determination unit 353 determines that the door 21A or the drawer 26A is in the open state, the first processor 320 may turn on the second LED 308 and the third LED 309.

[0065] Furthermore, when the opening / closing determination unit 353 determines that the door 21A and the drawer 26A are in the closed state, the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end photography. When the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end photography, the second communication control unit 352 transmits turn-off request information to the first control board 311. The turn-off request information is information requesting turning off the second LED 308 and the third LED 309. When the first processor 320 receives the turn-off request information, it turns off the second LED 308 and the third LED 309 and notifies the user P that photography has ended. Furthermore, when the opening / closing determination unit 353 determines that the door 21A or the drawer 26A is in the closed state, the first processor 320 may turn off the second LED 308 and the third LED 309.

[0066] In addition, when the open / close determination unit 353 determines that the door 21A or the drawer 26A has been opened, the photography control unit 354 may cause the wide-angle camera 301 and the narrow-angle camera 302 to start photographing, and then, when a predetermined time (e.g., one minute) has elapsed, the photography control unit 354 may cause the wide-angle camera 301 and the narrow-angle camera 302 to stop photographing. Furthermore, when the opening / closing determination unit 353 determines that the door 21A or the drawer 26A has been opened, the photography control unit 354 may extend the predetermined time after causing the wide-angle camera 301 and the narrow-angle camera 302 to start photographing and when the opening / closing determination unit 353 determines that the door 21A or the drawer 26A has been opened. For example, when the opening / closing determination unit 353 determines that the door 21A has been opened, the photography control unit 354 may extend the predetermined time after causing the wide-angle camera 301 and the narrow-angle camera 302 to start photographing and when the opening / closing determination unit 353 determines that the drawer 26A has been opened. In this case, a captured image of the drawer 26A can be reliably generated.

[0067] After the imaging control unit 354 causes the narrow-angle camera 302 to start imaging, the recording control unit 355 determines whether or not the door 21A is at a predetermined opening / closing angle θ based on the captured image PA1 by the narrow-angle camera 302. The predetermined opening / closing angle θ is, for example, 80 degrees to 100 degrees. When the recording control unit 355 determines that the door 21A is at a predetermined opening / closing angle θ, the recording control unit 355 starts recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302. That is, the recording control unit 355 starts recording the recorded data 363 in the second memory 360. The recorded data 363 includes the first recorded data DV1 indicating the images captured by the wide-angle camera 301 and the second recorded data DV2 indicating the images captured by the narrow-angle camera 302. In the following description, door 21A being at a predetermined opening / closing angle θ may be described as “satisfying the recording start condition.” The “recording start condition” refers to a condition under which recording control unit 355 starts recording the results of shooting by wide-angle camera 301 and narrow-angle camera 302. After the shooting control unit 354 has caused the narrow-angle camera 302 to start shooting, the recording control unit 355 may start recording the shooting results of the wide-angle camera 301 and the narrow-angle camera 302 when the narrow-angle camera 302 has captured an image of the first marker MK1.

[0068] Furthermore, when the determination of the recording control unit 355 changes from a state in which the door 21A is not at the predetermined opening / closing angle θ to a state in which the door 21A is at the predetermined opening / closing angle θ, the second communication control unit 352 transmits output request information for a buzzer sound to the first control board 311. Furthermore, when the determination of the recording control unit 355 changes from a state in which the door 21A is at the predetermined opening / closing angle θ to a state in which the door 21A is not at the predetermined opening / closing angle θ, the second communication control unit 352 transmits output request information for a buzzer sound to the first control board 311. The buzzer sound output request information is information that requests the buzzer 306 to output a buzzer sound. When the first processor 320 receives the buzzer sound output request information, it causes the buzzer 306 to output a buzzer sound, and notifies the user P that the door 21A is at a predetermined opening / closing angle θ.

[0069] After the shooting control unit 354 causes the narrow-angle camera 302 to start shooting, the recording control unit 355 determines whether or not the drawer 26A is pulled out to a predetermined extent based on the image PA2 captured by the narrow-angle camera 302. The predetermined amount of pull-out is, for example, 80% to 100%. When recording control unit 355 determines that drawer 26A is pulled out to a predetermined extent, recording control unit 355 starts recording the results of shooting by wide-angle camera 301 and narrow-angle camera 302. That is, recording control unit 355 starts recording recording data 363 in second memory 360. Recording data 363 includes first recording data DV1 indicating the results of shooting by wide-angle camera 301 and second recording data DV2 indicating the results of shooting by narrow-angle camera 302. In the following description, when drawer 26A is pulled out to a predetermined extent, this may be referred to as "satisfying the recording start condition." The recording start condition is that the door 21A is at a predetermined opening / closing angle θ, or that the drawer 26A is pulled out to a predetermined extent. After the shooting control unit 354 has caused the narrow-angle camera 302 to start shooting, the recording control unit 355 may start recording the shooting results of the wide-angle camera 301 and the narrow-angle camera 302 when the narrow-angle camera 302 is able to capture an image of the second marker MK2.

[0070] Furthermore, when the determination of the recording control unit 355 changes from a state in which drawer 26A is not pulled out to the specified amount to a state in which drawer 26A is pulled out to the specified amount, the second communication control unit 352 transmits information requesting the first control board 311 to output a buzzer sound. Furthermore, when the determination of the recording control unit 355 changes from a state in which drawer 26A is pulled out to the specified amount to a state in which drawer 26A is not pulled out to the specified amount, the second communication control unit 352 transmits information requesting the first control board 311 to output a buzzer sound. The buzzer sound output request information is information that requests the buzzer 306 to output a buzzer sound. Upon receiving the buzzer sound output request information, the first processor 320 causes the buzzer 306 to output a buzzer sound, and notifies the user P that the marker MK is at a predetermined position.

[0071] When open / close determination unit 353 determines that door 21A and drawer 26A are closed, recording control unit 355 ends recording of the images captured by wide-angle camera 301 and narrow-angle camera 302. Recording control unit 355 may end recording of the images captured by wide-angle camera 301 and narrow-angle camera 302 a predetermined time (e.g., one minute) after narrow-angle camera 302 is no longer able to capture an image of second marker MK2. Furthermore, after the image capture control unit 354 has ended recording of the images captured by the wide-angle camera 301 and the narrow-angle camera 302 via the recording control unit 355 , the image capture control unit 354 ends the image capture by the wide-angle camera 301 and the narrow-angle camera 302 .

[0072] In the first embodiment, when the recording control unit 355 determines that the recording start condition is satisfied, the recording control unit 355 starts recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302, but the present disclosure is not limited to this. For example, when open / close determination unit 353 determines that door 21A or drawer 26A is open, recording control unit 355 may start recording the images captured by wide-angle camera 301 and narrow-angle camera 302. In this case, recording of the images captured by narrow-angle camera 302 can be started at an appropriate timing with simple processing.

[0073] The state storage unit 364 stores a captured image PA1 captured by the narrow-angle camera 302 when the opening / closing angle of the door 21A is a predetermined opening / closing angle θ. The captured image PA1 includes an image of the first marker MK1. The predetermined opening / closing angle θ is, for example, 80 degrees to 100 degrees. The first marker MK1 will be described with reference to FIGS.

[0074] The state storage unit 364 stores the captured image PA2 captured by the narrow-angle camera 302 when the drawer 26A is pulled out to a predetermined extent. When the drawn-out amount is zero, it indicates that drawer 26A is in a closed state. The larger the predetermined drawn-out amount, the more open drawer 26A is. When the drawn-out amount is 100%, it indicates that drawer 26A is in a fully open state. The predetermined drawn-out amount is, for example, 80% to 100%.

[0075] Second marker MK2 is placed at the top end of a front panel that is located at the front end of drawer 26A of storage chamber 26, and the state of shooting by narrow-angle camera 302 changes depending on the amount that drawer 26A is pulled out. Note that the amount that drawer 26A is pulled out refers to the amount that drawer 26A is pulled out relative to storage chamber 26. In the first embodiment, a case will be described in which storage chamber 26 is vegetable chamber 25 and drawer 26A is drawer 25A. The second marker MK2 will be further described with reference to Figures 12-13.

[0076] The generating unit 356 selects the first captured image P1 based on the captured image PA1 from the captured image PB1 included in the moving image indicated by the first recorded data DV1 of the recorded data 363. The first captured image P1 is an image captured when the door 21A is at a predetermined opening / closing angle θ. Furthermore, the generating unit 356 selects a second captured image P2 based on the captured image PA2 from the captured image PB2 included in the moving image indicated by the second recorded image data DV2 of the recorded image data 363. The second captured image P2 is an image captured when the drawer 26A is pulled out to a predetermined extent. Then, the generation unit 356 generates management data D1 from the first captured image P1 and the second captured image P2. The management data D1 is data to be uploaded to the food management server 5 and is data for managing the foods stored in the refrigerator 2. The first communication control unit 351 transmits the management data D1 generated by the generation unit 356 to the food management server 5. In this embodiment, a case where the image capturing device 3 includes the generating unit 356 will be described, but the present invention is not limited to this. For example, the food management server 5 may include the generating unit 356. In this case, the management data D1 includes the first recorded data DV1 and the second recorded data DV2.

[0077] In the first embodiment, the generation unit 356 selects the first captured image P1 based on the captured image PA1 from the captured image PB1 included in the moving image indicated by the first recorded data DV1 of the recorded data 363, but the present disclosure is not limited to this. The generation unit 356 may select the captured image PB1 that has the least amount of user P reflected therein as the first captured image P1 from the captured images PB1 included in the moving image indicated by the first recorded data DV1 of the recorded data 363 based on the captured image PA1. In this case, the generation unit 356 further selects the first captured image P1 from the first captured image P1 selected based on the captured image PA1 as follows: That is, the generation unit 356 recognizes an image of the user P by a method such as pattern matching, and selects the captured image PB1 having the smallest area of ​​the recognized image of the user P as the first captured image P1.

[0078] Similarly, in the first embodiment, the generation unit 356 selects the second captured image P2 based on the captured image PA2 from the captured image PB2 included in the moving image indicated by the second recorded data DV2 of the recorded data 363, but is not limited to this. The generation unit 356 may select the captured image PB2 that has the least amount of user P reflected therein as the second captured image P2 from the captured images PB2 included in the moving image indicated by the second recorded data DV2 of the recorded data 363 based on the captured image PA2.

[0079] [1-1-4. Thermopile temperature detection state] 9 to 11, a description will now be given of the temperature detection state ST of the thermopile 305. The temperature detection state ST indicates the state of the detection range of the thermopile 305 when the thermopile 305 detects temperature. 9 to 11 show the same X-axis, Y-axis, and Z-axis as those shown in FIG. 2. The X-axis direction corresponds to the width direction and the left-right direction of the refrigerator 2. The Y-axis direction corresponds to the front-rear direction of the refrigerator 2. The Z-axis direction corresponds to the up-down direction of the refrigerator 2.

[0080] Fig. 9 is a diagram showing an example of a temperature detection state ST1 of thermopile 305 in embodiment 1. As shown in Fig. 9, thermopile 305 is placed on the upper surface of main box 20 of refrigerator 2. In other words, as described with reference to Figs. 5 to 7, thermopile 305 is installed at the front end of bottom surface 31A of imaging member 31 constituting imaging device 3. Also, as shown in Fig. 9, thermopile 305 detects the temperature of an object placed on the front side of refrigerator 2.

[0081] As shown in Fig. 9, thermopile 305 detects the temperature of an object placed within a detection range of detection angle θ1. The detection range is a range that extends in the vertical direction along the front surface of refrigerator 2. The front surface of refrigerator 2 includes, for example, the front surface of door 21A in a closed state and the front surface of drawer 26A in a closed state. The detection range extends approximately parallel to the Z axis. Detection angle θ1 is, for example, 10 degrees. Thermopile 305 is installed at the front end of bottom surface 31A of imaging member 31 so that the detection range of thermopile 305 is inclined forward, i.e., in the positive direction of the Y-axis, as shown in Fig. 9. Thermopile 305 is also installed at the front end of bottom surface 31A of imaging member 31 so that the detection range of thermopile 305 is inclined in the negative direction of the X-axis, so that the detection range of thermopile 305 includes the central position of refrigerator 2 in the left-right direction. In other words, thermopile 305 is preferably installed to include the central position of refrigerator 2 in the left-right direction. In FIG. 9, since no object is placed between the thermopile 305 and the floor surface FL within the detection range of the thermopile 305, in the temperature detection state ST1, the thermopile 305 detects the temperature of a partial area FLA of the floor surface FL. The temperature of the area FLA is, for example, 20°C.

[0082] Fig. 10 is a diagram showing another example of the temperature detection state ST of the thermopile 305 in the first embodiment. As shown in Fig. 10, in the temperature detection state ST2, the hand HD of the user P is placed between the thermopile 305 and the floor surface FL within the detection range of the thermopile 305. For example, in order to pull out the drawer 25A of the vegetable compartment 25, the user P hooks the fingers of the hand HD on the upper end of the front panel of the drawer 25A. In this case, the thermopile 305 detects the temperature of a region HA that is a part of the upper surface of the hand HD (here, the back of the hand HD). The temperature of the region HA is, for example, 36°C.

[0083] For example, when a transition occurs from temperature detection state ST1 shown in Fig. 9 to temperature detection state ST2 shown in Fig. 10, the detection value of thermopile 305 indicates a temperature rise of, for example, 16°C (=36°C-20°C). Since the detection value of thermopile 305 indicates a temperature rise equal to or greater than first threshold value TH1, open / close determination unit 353 determines that drawer 26A has changed from a closed state to an open state. Furthermore, since open / close determination unit 353 determines that drawer 26A has become open, shooting control unit 354 causes wide-angle camera 301 and narrow-angle camera 302 to start shooting.

[0084] 10, the temperature detection state ST2 is described as a case where the user P pulls out the drawer 25A of the vegetable compartment 25, but the present disclosure is not limited to this. The temperature detection state ST2 may be a case where the user P pulls out the drawer 26A. In other words, the temperature detection state ST2 may be a case where the user P pulls out any one of the drawers 22A, 23A, 24A, and 25A.

[0085] Fig. 11 is a diagram showing yet another example of the temperature detection state ST of the thermopile 305 in the first embodiment. As shown in Fig. 11, the temperature detection state ST3 corresponds to a state in which the user P has pulled out the drawer 25A of the vegetable compartment 25 so that the pulled-out amount is equal to or greater than a predetermined amount. The predetermined amount is, for example, 30%. In the temperature detection state ST3, the bottom surface of the drawer 25A of the vegetable compartment 25 is located between the thermopile 305 and the floor surface FL within the detection range of the thermopile 305. In this case, thermopile 305 detects the temperature of area BA, which is a part of the bottom surface of drawer 25A. The temperature of area BA is, for example, 5°C.

[0086] For example, when the temperature detection state ST1 shown in Fig. 9 transitions to the temperature detection state ST3 shown in Fig. 11, the detection value of thermopile 305 indicates a temperature drop of, for example, 15°C (=20°C-5°C). Since the detection value of thermopile 305 indicates a temperature drop of equal to or greater than second threshold value TH2, open / close determination unit 353 determines that drawer 26A has changed from the closed state to the open state. Note that second threshold value TH2 is, for example, 10°C. Furthermore, since open / close determination unit 353 determines that drawer 26A has entered the open state, shooting control unit 354 causes wide-angle camera 301 and narrow-angle camera 302 to start shooting.

[0087] 11 illustrates a case where user P pulls out drawer 25A of vegetable compartment 25, but the present disclosure is not limited to this. Temperature detection state ST3 may be a case where user P pulls out drawer 26A. In other words, temperature detection state ST3 may be a case where user P pulls out any one of drawers 22A, 23A, 24A, and 25A.

[0088] 11, a case where no food such as vegetables is stored in drawer 25A of vegetable compartment 25 is described, but the present disclosure is not limited to this. Food such as vegetables may be stored in drawer 25A of vegetable compartment 25. In this case, thermopile 305 detects the temperature of a part of an area on the top surface of food such as vegetables stored in drawer 25A. The temperature of the area is, for example, 5°C.

[0089] [1-1-5. Narrow-angle camera shooting conditions] Next, the shooting state of narrow-angle camera 302 will be described with reference to FIGS. 12 and 13 show the same X-axis, Y-axis, and Z-axis as those shown in FIG. 2. The X-axis direction corresponds to the width direction and the left-right direction of the refrigerator 2. The Y-axis direction corresponds to the front-rear direction of the refrigerator 2. The Z-axis direction corresponds to the up-down direction of the refrigerator 2.

[0090] Fig. 12 is a diagram showing an example of the image capturing state of narrow-angle camera 302 in embodiment 1. As shown in Fig. 12, narrow-angle camera 302 is disposed on the upper surface of main box 20 of refrigerator 2. In other words, as described with reference to Figs. 5 to 7, narrow-angle camera 302 is installed at the front end portion of bottom surface 31A of imaging member 31 constituting imaging device 3. Furthermore, as shown in Fig. 12, narrow-angle camera 302 captures an image of drawer 26A from above and in the front of refrigerator 2.

[0091] As shown in FIG. 12, narrow-angle camera 302 captures an image of an object placed within a capture range of capture angle θ2. The capture range is a range along the front of refrigerator 2. The capture range extends roughly parallel to the Z axis. Capture angle θ2 is, for example, 25 degrees. Narrow-angle camera 302 captures an image of food stored inside drawer 26A. Drawer 26A refers to drawers 22A, 23A, 24A, and 25A. Drawers 22A, 23A, 24A, and 25A capable of storing food are installed in ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively. 12 and 13, a case where drawer 26A is drawer 25A arranged in crisper 25 will be described.

[0092] 12, when drawer 25A is closed, narrow-angle camera 302 is blocked by drawer 24A and cannot capture an image of second marker MK2 arranged in drawer 25A. Note that second marker MK2 is arranged at the upper end of the front panel of drawer 25A. As shown by the solid line in FIG. 12, when drawer 25A is pulled out to a position where second marker MK2 is not blocked by drawer 24A, narrow-angle camera 302 can capture an image of second marker MK2.

[0093] Figure 13 is a diagram showing another example of the shooting state of narrow-angle camera 302 in embodiment 1. In Figure 13, drawer 25A is at its maximum open position. In other words, in Figure 13, drawer 25A is at its maximum open position. In this case, second marker MK2 is not blocked by drawer 24A, etc., so narrow-angle camera 302 can shoot the image of second marker MK2. Also, as shown in FIG. 13, when the drawer 25A is at the maximum open position, the second marker MK2 is located at a predetermined position. Therefore, for example, when drawer 25A transitions from the closed state indicated by the dashed line in Fig. 12, via the state indicated by the solid line in Fig. 12, to the maximum open position indicated in Fig. 13, recording control unit 355 starts recording the images captured by wide-angle camera 301 and narrow-angle camera 302, and second communication control unit 352 transmits information requesting output of a buzzer sound to first control board 311. Upon receiving the information requesting output of a buzzer sound, first processor 320 causes buzzer 306 to output a buzzer sound, thereby notifying user P that second marker MK2 is in a predetermined position.

[0094] 12 and 13, a case has been described in which drawer 26A is drawer 25A arranged in vegetable compartment 25, but the present disclosure is not limited to this. Drawer 26A may be any of drawers 22A, 23A, 24A, and 25A arranged in ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively.

[0095] [1-1-6. Markers to be placed in the refrigerator] The first marker MK1 arranged in the refrigerating compartment 21 will be described with reference to Figs. 14 and 15 show the same X-axis, Y-axis, and Z-axis as those shown in FIG. 2. The X-axis direction corresponds to the width direction and the left-right direction of the refrigerator 2. The Y-axis direction corresponds to the front-rear direction of the refrigerator 2. The Z-axis direction corresponds to the up-down direction of the refrigerator 2.

[0096] FIG. 14 is a diagram showing an example of a first marker MK1 arranged in refrigeration compartment 21 in the first embodiment. As shown in Fig. 14, the first marker MK1 is placed on the hinge cover AS. The left diagram of Fig. 14 shows a state in which the hinge cover AS is not attached, and the right diagram of Fig. 14 shows a state in which the hinge cover AS is attached. In the left diagram of Fig. 14, the hinge cover AS is not attached, so the right door hinge member 204 is exposed. The right door hinge member 204 axially supports the right door 21B so as to be rotatable.

[0097] The hinge cover AS is formed in an L-shape in a plan view, and an imaging surface PF2 is formed on the upper surface. A first marker MK1 including identification information indicating the refrigeration chamber 21 is formed on the imaging surface PF2. For example, a sticker on which the first marker MK1 is formed is affixed to the imaging surface PF2. The first marker MK1 will be further described with reference to FIG.

[0098] Fig. 15 is a perspective view of the hinge cover AS. As shown in Fig. 15, a notch AS1 is formed in the hinge cover AS. The notch AS1 fits into the upper part of the frame of the drawer 23A (e.g., the right door hinge member 204). The hinge cover AS is made of, for example, resin. The hinge cover AS may be the same color as the main box body 20 and the hinge, or may be a different color. The hinge cover AS is made of, for example, white color.

[0099] The state storage unit 364 stores the first captured image PA1 including the first marker MK1. Based on the first captured image PA1 stored in the state memory unit 364, the generation unit 356 selects the first captured image P1 when the right door 21B is at a predetermined opening / closing angle θ from the captured image PB1 included in the moving image indicated by the first recording data DV1 of the recording data 363.

[0100] 14 and 15, the case where the first marker MK1 is placed on the right door 21B of the refrigerator compartment 21 has been described, but the present disclosure is not limited thereto. The first marker MK1 may also be placed on the left door 21C of the refrigerator compartment 21, for example.

[0101] [1-1-7. Marker configuration] Next, the configuration of the marker MK will be described with reference to Fig. 16. Fig. 16 is a diagram showing an example of the configuration of the marker MK in embodiment 1. In Fig. 16, a case will be described in which the marker MK is placed on any of door 21A, drawer 22A, drawer 23A, drawer 24A, and drawer 25A. 16, the marker MK includes a rectangular outer frame image MF and an identification image MD formed inside the outer frame image MF. The identification image MD corresponds to identification information indicating each of the refrigerator compartment 21, the ice making compartment 22, the fresh freezing compartment 23, the freezer compartment 24, and the vegetable compartment 25 in the refrigerator 2. The outer frame image MF is formed in black, for example. In the following description, a case where the marker MK is placed in the drawer 25A of the vegetable compartment 25 will be described.

[0102] The identification image MD is composed of a first rectangular image PD1 in a first color and a second rectangular image PD2 in a second color different from the first color, and the arrangement pattern of the first rectangular image PD1 and the second rectangular image PD2 indicates the identification information. The first color is, for example, white, and the second color is, for example, black. The size of the first rectangular image PD1 is the same as the size of the second rectangular image PD2, and each of the first rectangular image PD1 and the second rectangular image PD2 is, for example, a square. In other words, the width WP1 of the first rectangular image PD1 and the width WP2 of the second rectangular image PD2 are the same as the height HP of the first rectangular image PD1 and the second rectangular image PD2.

[0103] The thickness HF of the outer frame image MF in the short side direction (up and down direction) and the thickness WF of the outer frame image MF in the long side direction (left and right direction) are each the same as, for example, the width WP1 of the first rectangular image PD1, the width WP2 of the second rectangular image PD2, the height HP of the first rectangular image PD1, and the height HP of the second rectangular image PD2. Each of the width HF in the short side direction (up-down direction) of the outer frame image MF and the width WF in the long side direction (left-right direction) of the outer frame image MF is determined based on the resolution of the narrow-angle camera 302 and the distance between the narrow-angle camera 302 and the marker MK when the drawer 25A is pulled out to a predetermined amount (e.g., 100%). Also, each of the width HF in the short side direction (up-down direction) of the outer frame image MF and the width WF in the long side direction (left-right direction) of the outer frame image MF is determined based on the shooting range or angle of view of the narrow-angle camera 302. For example, the higher the resolution of narrow-angle camera 302, the smaller the width HF in the long side direction of outer frame image MF and the width WF in the short side direction of outer frame image MF can be. Also, the longer the distance between narrow-angle camera 302 and marker MK at a predetermined drawn-out amount of drawer 25A (e.g., 100%), the larger the width HF in the long side direction of outer frame image MF and the width WF in the short side direction of outer frame image MF must be. Also, the larger the imaging range or angle of view of narrow-angle camera 302, the larger the width HF in the long side direction of outer frame image MF and the width WF in the short side direction of outer frame image MF must be. In the marker MK shown in FIG. 16, the width HF of the outer frame image MF in the long side direction and the width WF of the outer frame image MF in the short side direction are each, for example, 6 mm.

[0104] The first rectangular images PD1 and the second rectangular images PD2 are arranged along the longitudinal direction (here, the left-right direction) of the outer frame image MF. In Fig. 16, five first rectangular images PD1 and four second rectangular images PD2 are arranged as the identification images MD. The first rectangular images PD1 represent, for example, "0," and the second rectangular images PD2 represent, for example, "1." That is, the identification images MD represent a nine-digit binary number. The identification image MD is composed of an information image MDA and a code image MDB. The information image MDA is composed of five rectangular images PD, and the code image MDB is composed of four rectangular images PD. The rectangular images PD are the first rectangular images PD1 or the second rectangular images PD2. The information image MDA corresponds to identification information indicating the drawer 25A in the refrigerator 2. The code image MDB indicates a code for correcting an error in the identification information indicated by the information image MDA. The information image MDA is composed of five rectangular images PD and therefore indicates a five-digit binary number. That is, the information image MDA indicates 2 to the power of 5, that is, 32 types of identification information. For example, the information image MDA shown in FIG. 16 indicates a five-digit binary number, "01010."

[0105] The size of the first rectangular image PD1 and the size of the second rectangular image PD2 are determined based on the resolution of the narrow-angle camera 302 and the distance between the narrow-angle camera 302 and the marker MK at a predetermined pull-out amount (e.g., 80% to 100%) of the drawer 25A. The size of the first rectangular image PD1 and the size of the second rectangular image PD2 are determined based on the shooting range or angle of view of the narrow-angle camera 302. For example, the higher the resolution of the narrow-angle camera 302, the smaller the size of the first rectangular image PD1 and the size of the second rectangular image PD2 can be. Also, the longer the distance between the narrow-angle camera 302 and the marker MK at a predetermined pull-out amount of the drawer 25A (e.g., 80% to 100%), the larger the size of the first rectangular image PD1 and the size of the second rectangular image PD2 must be. Also, the larger the shooting range or angle of view of the narrow-angle camera 302, the larger the size of the first rectangular image PD1 and the size of the second rectangular image PD2 must be. In the marker MK shown in FIG. 16, the size of the first rectangular image PD1 and the size of the second rectangular image PD2 are squares with one side measuring 6 mm.

[0106] 16, a case where the outer frame image MF is formed in black will be described, but is not limited to this. For example, the outer frame image MF may be formed in blue. Also, a case where the first color is white and the second color is black will be described, but is not limited to this. For example, the first color may be yellow and the second color may be blue.

[0107] As described with reference to Figures 12 and 13, a marker MK is placed on the upper end of the front panel of drawer 25A. Also, as shown in Figure 16, the marker MK includes a rectangular outer frame image MF and an identification image MD formed inside the outer frame image MF. Therefore, generation unit 356 executes the following process.

[0108] That is, the generation unit 356 first determines whether or not an outer frame image MF can be detected from each captured image PB2 included in the moving image indicated by the second recorded data DV2 of the recorded data 363. Then, when the outer frame image MF can be detected from the captured image PB2, the generation unit 356 determines whether or not the identification information indicated by the identification image MD inside the detected outer frame image MF can be read. Next, when the identification information indicated by the identification image MD can be read and the read identification information matches the identification information included in the second marker MK2 stored in the state storage unit 364, the generation unit 356 selects the captured image PB2 as the second captured image P2.

[0109] Furthermore, in order for the generating unit 356 to easily detect the outer frame image MF from each captured image PB2 included in the moving image indicated by the second video recording data DV2, it is preferable that the color of the outer frame image MF is different from the color of the inner surface of the vegetable compartment 25 and the color of the food stored in the vegetable compartment 25. Therefore, the color of the outer frame image MF is preferably, for example, black, blue, or the like.

[0110] [1-1-8. Food management server configuration] FIG. 17 is a block diagram showing the configuration of the terminal device 4 and the food management server 5. The food management server 5 includes a server processor 50, which is a processor such as a CPU, a server memory 51, and a server communication module 52. The server processor 50 is connected to the server memory 51 and the server communication module 52.

[0111] The server memory 51 is a storage device that stores programs and data executed by the server processor 50. The server memory 51 is composed of a non-volatile storage device. The server memory 51 may also include a non-volatile storage device that constitutes a work area for the server processor 50. The server memory 51 stores data processed by the server processor 50, a control program 511 executed by the server processor 50, and a food management DB (database) 512. The food management DB 512 has one record for each photographing device 3. This one record includes the photographing device ID 362, image data of the first photographed image P1, and image data of the second photographed image P2.

[0112] The server communication module 52 includes a communication circuit, connector, and the like that comply with a predetermined communication standard, and communicates with the refrigerator 2, the image capturing device 3, and the terminal device 4 via the network NW.

[0113] The server processor 50 reads out and executes a control program 511 stored in the server memory 51 , thereby functioning as a server communication control unit 501 and a server processing unit 502 . The server communication control unit 501 communicates with the refrigerator 2, the image capturing device 3, and the terminal device 4 via the server communication module 52. The server processing unit 502 performs various information processes using the image capturing device 3 and the terminal device 4 as clients. The server processing unit 502 may also use the refrigerator 2 as a client.

[0114] [1-1-9. Terminal device configuration] Next, the configuration of the terminal device 4 will be described. The terminal device 4 includes a terminal processor 40, which is a processor such as a CPU, a terminal memory 41, a terminal communication module 42, and a touch panel 43.

[0115] The terminal memory 41 is a storage device that non-volatilely stores programs and data executed by the terminal processor 40. The terminal memory 41 is configured by a non-volatile storage device. The terminal memory 41 may also include a RAM that configures a work area for the terminal processor 40. The terminal memory 41 stores data processed by the terminal processor 40, control programs executed by the terminal processor 40, and a food management application 411.

[0116] The terminal communication module 42 includes a communication circuit, a connector, and the like that comply with a predetermined communication standard, and communicates with the food management server 5 under the control of the terminal processor 40 . The touch panel 43 includes a display panel and a touch sensor that is provided over the display panel or is integral with the display panel.

[0117] The terminal processor 40 functions as an application execution unit 400 . The application execution unit 400 communicates with the food management server 5 via the terminal communication module 42. The application execution unit 400 displays a user interface related to management of food stored in the refrigerator 2 on the touch panel 43. In the following description, this user interface is referred to as the "application UI." The display control unit 402 of this embodiment displays images of food stored in the refrigerator 2 by displaying the application UI. The application execution unit 400 receives various inputs from the user P via the application UI.

[0118] [1-2. Operation] Next, the operation of each part of food management system 1 will be described.

[0119] [1-2-1. Recording-related operations] First, the operation of the image capturing device 3 relating to recording will be described. FIG. 18 is a flowchart FA showing the operation of the photographing device 3.

[0120] The open / close determination unit 353 determines whether the door 21A or the drawer 26A is in an open state (step SA1). The door 21A includes the right door 21B and the left door 21C. The open / close determination unit 353 determines whether the right door 21B is in an open state or a closed state based on the detection value of the first distance measurement sensor 303. The open / close determination unit 353 also determines whether the left door 21C is in an open state or a closed state based on the detection value of the second distance measurement sensor 304. The open / close determination unit 353 also determines whether the drawer 26A is in an open state or a closed state based on the detection value of the thermopile 305. The drawer 26A refers to the drawers 22A, 23A, 24A, and 25A. Drawers 22A, 23A, 24A, and 25A are arranged in ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively.

[0121] If the open / close determination unit 353 determines that the door 21A or the drawer 26A is not open (step SA1: NO), the second processor 350 returns the process to step SA1.

[0122] If open / close determining unit 353 determines that door 21A or drawer 26A is open (step SA1: YES), image capturing control unit 354 causes wide-angle camera 301 and narrow-angle camera 302 to start capturing images (step SA2). Step SA1 and step SA2 correspond to an example of a "photography start step."

[0123] Next, the recording control unit 355 determines whether or not a recording start condition is satisfied (step SA3). As described with reference to FIG. 8, the recording start condition is that the door 21A is at a predetermined opening / closing angle θ or the drawer 26A is pulled out to a predetermined extent. If the recording control unit 355 determines that the recording start condition is not satisfied (step SA3: NO), the recording control unit 355 performs the determination in step SA3 again.

[0124] When the recording control unit 355 determines that the recording start condition is met (step SA3: YES), the recording control unit 355 starts recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302 (step SA4). Note that the recording control unit 355 may start recording only the images captured by the wide-angle camera 301 when the first marker MK1 is captured. Also, the recording control unit 355 may start recording only the images captured by the narrow-angle camera 302 when the second marker MK2 is captured.

[0125] Next, the open / close determination unit 353 determines whether the door 21A and the drawer 26A are closed (step SA5). If the open / close determination unit 353 determines that the door 21A and the drawer 26A are not closed (step SA5: NO), the open / close determination unit 353 performs the determination of step SA5 again.

[0126] On the other hand, if it is determined that the door 21A and the drawer 26A are closed (step SA5: YES), the recording control unit 355 ends recording of the captured image (step SA6). As a result, the recorded image data 363 is generated in the second memory 360.

[0127] Next, the shooting control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end shooting (step SA7).

[0128] [1-2-2. Operations related to sending management data] Next, the operation relating to the transmission of the management data D1 will be described. FIG. 19 is a flowchart showing the operations of the photographing device 3 and the food management server 5. In FIG. 19, a flowchart FB shows the operation of the photographing device 3, and a flowchart FC shows the operation of the food management server 5.

[0129] The generating unit 356 determines whether or not the recorded data 363 is stored in the second memory 360 (step SB1). When the generating unit 356 determines that the recorded data 363 is not stored (step SB1: NO), the generating unit 356 performs the determination of step SB1 again.

[0130] On the other hand, if the generation unit 356 determines that the recording data 363 has been stored (step SB1: YES), it selects the first captured image P1 from the captured image PB1 included in the moving image indicated by the first recording data DV1 of the recording data 363 stored in the second memory 360, based on the first captured image PA1 stored in the state memory unit 364 (step SB2).

[0131] Step SB2 will now be described in detail. The first captured image P1 is an image captured when the right door 21B or the left door 21C is open at a predetermined opening / closing angle θ (for example, 80 degrees to 100 degrees).

[0132] The generating unit 356 first judges whether or not an outer frame image MF can be detected from each captured image PB1 included in the moving image indicated by the first recorded data DV1 of the recorded data 363. Then, when the outer frame image MF can be detected from the captured image PB1, the generating unit 356 judges whether or not the identification information indicated by the identification image MD inside the detected outer frame image MF can be read. Next, when the identification information indicated by the identification image MD can be read and the read identification information matches the identification information indicated by the first marker MK1 included in the captured image PA1 stored in the state storage unit 364, the generating unit 356 selects the captured image PB1 as the first captured image P1.

[0133] Returning to the explanation of the flowchart FB, the generation unit 356 selects a second captured image P2 from each of the captured images PB2 included in the moving image indicated by the second recording data DV2 of the stored recording data 363, based on the second captured image PA2 stored in the state memory unit 364 (step SB3).

[0134] Step SB3 will now be described in detail. The second captured image P2 is an image captured when the drawer 26A is pulled out to a predetermined extent.

[0135] The generating unit 356 first judges whether or not an outer frame image MF can be detected from each captured image PB2 included in the moving image indicated by the second recorded data DV2 of the recorded data 363. Then, when the outer frame image MF can be detected from the captured image PB2, the generating unit 356 judges whether or not the identification information indicated by the identification image MD inside the detected outer frame image MF can be read. Next, when the identification information indicated by the identification image MD can be read and the read identification information matches the identification information included in the second marker MK2 of the captured image PA2 stored in the state storage unit 364, the generating unit 356 selects the captured image PB2 as the second captured image P2.

[0136] The generation unit 356 determines whether or not at least one of the first captured image P1 and the second captured image P2 has been selected (step SB4).

[0137] When it is determined that the first captured image P1 and the second captured image P2 have not been selected (step SB4: NO), the generating unit 356 performs the process of step SB7.

[0138] On the other hand, when it is determined that at least one of the first captured image P1 and the second captured image P2 has been selected (step SB4: YES), the generating unit 356 generates management data D1 (step SB5).

[0139] When the first captured image P1 and the second captured image P2 are selected, the generation unit 356 generates management data D1 including image data of the first captured image P1, image data of the second captured image P2, and the photographing device ID 362. If the first captured image P1 can be selected but the second captured image P2 cannot be selected, the generation unit 356 generates management data D1 including image data of the first captured image P1 and the photographing device ID 362. If the second captured image P2 can be selected but the first captured image P1 cannot be selected, the generation unit 356 generates management data D1 including image data of the second captured image P2 and the imaging device ID 362.

[0140] Next, the first communication control unit 351 transmits the management data D1 generated by the generation unit 356 to the food management server 5 (step SB6).

[0141] Next, the generating unit 356 erases the video recording data 363 that has been processed in steps SB2 and after from the second memory 360 (step SB7).

[0142] As shown in the flowchart FC, the server communication control unit 501 determines whether or not it has received management data D1 from the photographing device 3 (step SC1). If the server communication control unit 501 determines that it has received management data D1 (step SC1: YES), the server processing unit 502 updates the food management DB 512 based on the management data D1 received by the server communication control unit 501 (step SC2).

[0143] Step SC2 will now be described in detail. The server processing unit 502 identifies a record from the food management DB 512 that includes the photographing device ID 362 included in the management data D1, and updates the image data of the first photographed image P1 in the identified record to the image data of the first photographed image P1 included in the management data D1. The server processing unit 502 identifies a record from the food management DB 512 that includes the photographing device ID 362 included in the management data D1, and updates the image data of the second photographed image P2 in the identified record to the image data of the second photographed image P2 included in the management data D1.

[0144] [1-2-3. Actions related to app UI] Next, the operation of the food management system 1 related to the application UI will be described. 20 is a flowchart showing the operations of the terminal device 4 and the food management server 5. In FIG.

[0145] As shown in the flowchart FD, the application execution unit 400 determines whether or not to request image data stored in the food management DB 512 from the food management server 5 (step SD1). For example, when the food management application 411 is launched, the terminal communication control unit 401 makes a positive determination in step SD1.

[0146] When the application execution unit 400 determines to request image data stored in the food management DB 512 (step SD1: YES), it transmits request data requesting the image data to the food management server 5 (step SD2). The request data includes the photographing device ID 362 stored in the terminal memory 41. The terminal device 4 acquires the photographing device ID 362 at a predetermined timing.

[0147] As shown in the flowchart FE, the server communication control unit 501 receives the request data (step SE1). Next, the server processing unit 502 acquires image data corresponding to the photographing device ID 362 included in the request data received by the server communication control unit 501 from the food management DB 512 (step SE2). Next, the server communication control unit 501 transmits the image data acquired by the server processing unit 502 to the terminal device 4 as a response to the request data (step SE3).

[0148] Referring to the flowchart FD, the application execution unit 400 receives image data from the food management server 5 (step SD3). Next, the application execution unit 400 causes the touch panel 43 to display an application UI including an image represented by the image data received in step SD3 (step SD4).

[0149] In the first embodiment, the case where the photographing device 3 is configured as a separate entity from the refrigerator 2 has been described, but the present disclosure is not limited to this. The photographing device 3 may be configured as an integral part of the refrigerator 2. In other words, the photographing device 3 may be configured as a part of the refrigerator 2.

[0150] [1-3. Effects, etc.] As described above with reference to Figures 1 to 20, the photographing device 3 in embodiment 1 photographs refrigerator 2 equipped with a vegetable compartment 25 having a drawer 25A with an opening formed on the top surface thereof and comprises: narrow-angle camera 302 arranged on top surface 20A of main box 20 of refrigerator 2; thermopile 305 arranged on top surface 20A of main box 20 of refrigerator 2 and detecting the temperature of an object arranged on the front side of refrigerator 2; and photographing control unit 354 that starts photographing vegetable compartment 25 using narrow-angle camera 302 based on the detection result of thermopile 305.

[0151] According to this, narrow-angle camera 302 starts photographing vegetable compartment 25 based on the detection result of thermopile 305. Therefore, narrow-angle camera 302 can be made to start photographing at an appropriate timing. Therefore, food stored in refrigerator 2 can be managed with high accuracy.

[0152] The image capturing device 3 also includes a recording control unit 355 that starts recording an image captured by the narrow-angle camera 302 based on the detection result of the thermopile 305.

[0153] According to this, recording of images captured by narrow-angle camera 302 starts based on the detection result of thermopile 305. Therefore, recording of images captured by narrow-angle camera 302 can be started at an appropriate timing. Therefore, food stored in refrigerator 2 can be managed with high accuracy.

[0154] Furthermore, in the photographing device 3, when the thermopile 305 detects a temperature rise equal to or greater than a first threshold value TH1 set in advance, the photographing control section 354 starts photographing the vegetable compartment 25 by the narrow-angle camera 302.

[0155] 9 and 10, when thermopile 305 detects a temperature rise equal to or greater than a preset first threshold value TH1, narrow-angle camera 302 starts photographing vegetable compartment 25. By setting first threshold value TH1 to an appropriate value, recording of images captured by narrow-angle camera 302 can be started at an appropriate timing. Therefore, food stored in refrigerator 2 can be managed with high accuracy.

[0156] Furthermore, in the photographing device 3, when the thermopile 305 detects a drop in temperature equal to or greater than a second threshold value TH2 set in advance, the photographing control unit 354 starts photographing by the narrow-angle camera 302.

[0157] 9 and 11, when thermopile 305 detects a temperature drop equal to or greater than the preset second threshold value TH2, narrow-angle camera 302 starts capturing images. Therefore, by setting second threshold value TH2 to an appropriate value, recording of images captured by narrow-angle camera 302 can be started at an appropriate timing. Therefore, food stored in refrigerator 2 can be managed with high accuracy.

[0158] In addition, in the photographing device 3, the refrigerator 2 is arranged above the drawer 25A, has an opening 201 formed on the front, and is equipped with a refrigeration compartment 21 having a door 21A for opening and closing the opening 201, and is arranged on the top surface 20A of the main box body 20 of the refrigerator 2, and is equipped with a wide-angle camera 301 different from the narrow-angle camera 302, a first distance measuring sensor 303 and a second distance measuring sensor 304 which detect the opening and closing angle of the door 21A, and the photographing control unit 354 starts photographing the refrigeration compartment 21 using the wide-angle camera 301 based on the detection results of the first distance measuring sensor 303 and the second distance measuring sensor 304.

[0159] According to this, wide-angle camera 301 starts photographing refrigeration compartment 21 based on the detection results of first distance measurement sensor 303 and second distance measurement sensor 304. Therefore, photographing of refrigeration compartment 21 by wide-angle camera 301 can be started at an appropriate timing. Therefore, food stored in refrigerator 2 can be managed with high accuracy.

[0160] Furthermore, the photographing method in embodiment 1 is a photographing method using photographing device 3 which photographs refrigerator 2 equipped with vegetable compartment 25 having drawer 25A with an opening formed on the top surface thereof, and which is equipped with narrow-angle camera 302 arranged on top surface 20A of main box 20 of refrigerator 2, and thermopile 305 arranged on top surface 20A of main box 20 of refrigerator 2 and detects the temperature of an object arranged on the front side of refrigerator 2, and includes a photographing start step of starting to photograph vegetable compartment 25 using narrow-angle camera 302 based on the detection result of thermopile 305.

[0161] As a result, the photographing method in the first embodiment achieves the same effects as those of the photographing device 3 in the first embodiment.

[0162] Furthermore, the control program 361 in the first embodiment is a control program 361 for an imaging device 3 that photographs a refrigerator 2 equipped with a vegetable compartment 25 having a drawer 25A with an opening formed on the top surface thereof, the control program 361 including a narrow-angle camera 302 arranged on a top surface 20A of a main box 20 of the refrigerator 2, a thermopile 305 arranged on a top surface 20A of the main box 20 of the refrigerator 2 and detecting the temperature of an object arranged on the front side of the refrigerator 2, and a second processor 350, and causes the second processor 350 to function as an imaging control unit 354 that starts imaging the vegetable compartment 25 using the narrow-angle camera 302 based on the detection result of the thermopile 305.

[0163] According to this, the control program 361 in the first embodiment achieves the same effects as the image capturing device 3 in the first embodiment.

[0164] (Other embodiments) As described above, the above-mentioned first embodiment has been described as an example disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above-mentioned first and second embodiments to form a new embodiment. Therefore, other embodiments will be exemplified below.

[0165] In the above-described first embodiment, the sensor for detecting whether drawer 26A is open or closed is configured by thermopile 305. However, the sensor for detecting whether drawer 26A is open or closed may be a temperature sensor. That is, open / close determination unit 353 determines whether drawer 26A is open or closed based on the detection value of the temperature sensor. The temperature sensor may be a radiation temperature type.

[0166] In the above-mentioned embodiment 1, the second marker MK2 is described as being placed in the vegetable compartment 25, but the second marker MK2 may also be placed in the ice making compartment 22, the fresh freezing compartment 23, or the freezer compartment 24.

[0167] In the above-described first embodiment, the identification image MD in the marker MK is composed of an information image MDA and a code image MDB, but the identification image MD may be composed of only the information image MDA. In other words, the identification image MD does not have to include the code image MDB.

[0168] In the above-described first embodiment, the first rectangular image PD1 and the second rectangular image PD2 in the marker MK are square in shape. However, the first rectangular image PD1 and the second rectangular image PD2 may be rectangular.

[0169] In the above-described first embodiment, the first captured image P1 and the second captured image P2 are used for displaying the terminal device 4. However, the first captured image P1 and the second captured image P2 may be used for other purposes, such as detecting food stored in the refrigerator 2.

[0170] In other embodiments, the marker MK may include characters, illustrations, or the like that indicate information about the storage compartment or storage case to be installed (for example, the characters "upper vegetable compartment case," an illustration of the upper vegetable compartment case, or a fruit mark). This allows the user P to easily identify the storage compartment or storage case to be installed when installing the marker MK in the refrigerator 2. This can therefore prevent the user from making an installation error, improving user convenience.

[0171] In another embodiment, the generation unit 356 may compare the image captured by the camera with pre-stored images of the drawer in the pulled-out state and select the image of the drawer in the pulled-out state.

[0172] In another embodiment, in addition to the functional units of second processor 350 in embodiment 1, second processor 350 may function as a detection unit that detects food. During the time from when recording control unit 355 starts recording with wide-angle camera 301 and narrow-angle camera 302 to when it ends the recording, the detection unit may cut out an image of food being taken in or out of refrigerator compartment 21 from the video. Furthermore, if a marker is attached to the food, the detection unit may detect the food with the marker.

[0173] In another embodiment, the first captured image P1 may be corrected before being uploaded to the food management server 5. An example of image correction is viewpoint conversion correction.

[0174] In other embodiments, the main box 20 of the refrigerator 2 may have a number of rooms other than the five types described above, or may have fewer rooms. Also, in other embodiments, the main box 20 of the refrigerator 2 may have only one drawer-type storage room.

[0175] Each of the first processor 320, the second processor 350, the server processor 50, and the terminal processor 40 may be configured by a single processor or multiple processors. Each of these processors may be hardware programmed to realize the corresponding functional unit. That is, each of these processors may be configured by, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0176] The configurations of the image capture device 3, refrigerator 2, terminal device 4, and food management server 5 shown in Figures 8 and 17 are merely examples, and the specific implementation form is not particularly limited. In other words, it is not necessarily necessary to implement hardware corresponding to each unit individually, and it is also possible to implement a configuration in which one processor executes a program to realize the functions of each unit. Also, some of the functions realized by software in the above-mentioned embodiment may be implemented as hardware, or some of the functions realized by hardware may be realized by software.

[0177] The step units of the operations shown in Figures 18 to 20 are divided according to the main processing contents in order to make the operations easier to understand, and the manner in which the processing units are divided or the names of the processing units do not limit the operations. The operations may be divided into more step units according to the processing contents. Furthermore, one step unit may be divided so as to include more processing. Furthermore, the order of the steps may be changed as appropriate within the scope of the present disclosure.

[0178] It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, substitutions, additions, omissions, and the like can be made within the scope of the claims or their equivalents.

[0179] (Additional Note) The above description of the embodiments discloses the following techniques.

[0180] (Technology 1) An imaging device for photographing a refrigerator having a first storage compartment with a drawer with a first opening formed on an upper surface thereof, the imaging device comprising: a first camera arranged on an upper surface of a box of the refrigerator; a temperature sensor arranged on the upper surface of the box of the refrigerator and detecting the temperature of an object arranged on the front side of the refrigerator; and an imaging control unit that starts imaging by the first camera based on a detection result of the temperature sensor. According to this configuration, the first camera starts taking pictures based on the detection result of the temperature sensor. Therefore, the first camera can start taking pictures at an appropriate timing. Therefore, the food stored in the refrigerator can be managed with high accuracy.

[0181] (Technology 2) The imaging device according to Technology 1, further comprising a recording control unit that starts recording an image captured by the first camera based on a detection result of the temperature sensor. According to this configuration, the recording of the image captured by the first camera is started based on the detection result of the temperature sensor. Therefore, the recording of the image captured by the first camera can be started at an appropriate timing. Therefore, the food stored in the refrigerator can be managed with high accuracy.

[0182] (Technology 3) The imaging device according to Technology 1 or Technology 2, wherein when the temperature sensor detects a temperature rise equal to or greater than a first threshold set in advance, the imaging control unit starts imaging using the first camera. According to this configuration, when the temperature sensor detects a temperature rise equal to or greater than a first threshold value set in advance, the first camera starts taking pictures. Therefore, by setting the first threshold value to an appropriate value, the first camera can start taking pictures at an appropriate timing. Therefore, the food stored in the refrigerator can be managed with high accuracy.

[0183] (Technology 4) An imaging device described in any one of Technology 1 to Technology 3, wherein when the temperature sensor detects a drop in temperature equal to or greater than a predetermined second threshold, the imaging control unit starts imaging using the first camera. According to this configuration, when the temperature sensor detects a temperature drop equal to or greater than a preset second threshold, the first camera starts taking pictures. Therefore, by setting the second threshold to an appropriate value, the first camera can start taking pictures at an appropriate time. Therefore, the food stored in the refrigerator can be managed with high accuracy.

[0184] (Technology 5) The photographing device according to any one of Technology 1 to Technology 4, wherein the photographing control unit ends photographing when a predetermined time has elapsed after photographing by the first camera is started. According to this configuration, the first camera stops taking pictures when a predetermined time has elapsed after the first camera starts taking pictures. Therefore, by setting the predetermined time to an appropriate time, the picture taking can be stopped at an appropriate timing. Therefore, the food stored in the refrigerator can be managed with high accuracy.

[0185] (Technology 6) The photographing device described in Technology 5, further comprising an opening / closing determination unit that detects the opening of a door of the first storage chamber, and the photographing control unit extends the specified time when the opening / closing determination unit detects the opening of the door of the first storage chamber after photographing by the first camera has started. According to this configuration, when the opening / closing determination unit detects the door of the first storage compartment being open after the first camera starts taking pictures, the predetermined time is extended. Therefore, by extending the predetermined time by an appropriate amount, it is possible to end the taking of pictures at an appropriate timing. Therefore, it is possible to reliably generate a captured image of the first storage compartment. As a result, it is possible to accurately manage the food stored in the refrigerator.

[0186] (Technology 7) The imaging device according to any one of Technology 1 to Technology 4, wherein the refrigerator is provided with a second storage compartment arranged above the drawer, having a second opening formed on a front surface and having an opening / closing body for opening and closing the second opening, and is provided with a second camera arranged on an upper surface of a box body of the refrigerator, the second camera being different from the first camera, and a second detection sensor for detecting opening of a door of the second storage compartment, and the imaging control unit starts imaging by the second camera based on a detection result of the second detection sensor. According to this configuration, the second camera starts taking pictures based on the detection result of the second detection sensor. Therefore, the second camera can start taking pictures at an appropriate timing. Therefore, the food stored in the refrigerator can be managed with high accuracy.

[0187] (Technology 8) A photographing method for an imaging device that photographs a refrigerator having a first storage compartment with a drawer with a first opening formed on an upper surface thereof, the imaging method including a first camera arranged on an upper surface of a body of the refrigerator, and a temperature sensor arranged on the upper surface of the body of the refrigerator and detecting the temperature of an object arranged on the front side of the refrigerator, the method including a photographing start step of starting photographing by the first camera based on a detection result of the temperature sensor. According to this configuration, the same effects as those of the imaging device described in the first technique can be achieved.

[0188] (Technology 9) A control program for an imaging device that photographs a refrigerator having a first storage compartment with a drawer with a first opening formed on its top surface, the control program comprising: a first camera arranged on an upper surface of a body of the refrigerator; a temperature sensor arranged on the upper surface of the body of the refrigerator and detecting the temperature of an object arranged on the front side of the refrigerator; and a processor, the control program causing the processor to function as an imaging control unit that starts imaging by the first camera based on the detection result of the temperature sensor. According to this configuration, the same effects as those of the imaging device described in the first technique can be achieved. [Industrial Applicability]

[0189] INDUSTRIAL APPLICABILITY As described above, the photographing device, photographing method, and control program according to the present invention can be used to cause the first camera to start photographing at the appropriate timing. [Explanation of symbols]

[0190] 1. Food Control System 2. Refrigerator 3. Imaging Equipment 4 Terminal Equipment 5. Food Management Server 20 Main box (box) 20A Top surface (upper surface) 21 Refrigerator room (second storage room) 21A Door (opening and closing body) 22 Ice room (first storage room) 23 Fresh freezing room (first storage room) 24 Freezer (first storage room) 25 Vegetable room (first storage room) 26 Storage Room 22A, 23A, 24A, 25A, 26A drawer 201 Opening (2nd Opening) 301 Wide-angle camera (second camera) 302 Narrow-angle camera (first camera) 303 First distance measuring sensor (second detection sensor) 304 Second distance measuring sensor (second detection sensor) 305 Thermopile (temperature sensor) 350 Second Processor (Processor) 351 First communication control unit 352 Second communication control section 353 Open / Close Judgment Unit 354 Shooting control section 355 Recording Control Unit 356 Generation part 360 Second Memory 361 Control Program 363 Recording Data DV1 1st recording data DV2 2nd recording data MK Marker MK1 First Marker MK2 2nd Marker P1 First captured image P2 2nd captured image

Claims

1. A first camera is disposed on a top surface of a box body of the refrigerator and photographs a refrigerator including a first storage compartment having a drawer with a first opening formed on an upper surface of the refrigerator; a temperature sensor that is disposed on an upper surface of a box body of the refrigerator and detects a temperature of an object disposed on a front side of the refrigerator; an imaging control unit that starts imaging by the first camera based on a detection result of the temperature sensor; An imaging device comprising:

2. a recording control unit that starts recording an image captured by the first camera based on a detection result of the temperature sensor; The imaging device according to claim 1 .

3. When the temperature sensor detects a temperature increase equal to or greater than a first threshold value, The photographing control unit starts photographing by the first camera. The imaging device according to claim 1 .

4. When the temperature sensor detects a temperature drop equal to or greater than a second threshold value set in advance, The photographing control unit starts photographing by the first camera. The imaging device according to claim 1 .

5. the photographing control unit ends photographing when a predetermined time has elapsed after photographing by the first camera has started; The imaging device according to any one of claims 1 to 4.

6. an opening / closing determination unit that detects whether a door of the first storage chamber is open; The photographing control unit extends the predetermined time when the opening / closing determination unit detects that the door of the first storage chamber is open after the first camera starts photographing. The imaging device according to claim 5 .

7. the refrigerator includes a second storage compartment that is disposed above the drawer, has a second opening formed in a front surface thereof, and has an opening / closing body that opens and closes the second opening; A second camera that is disposed on an upper surface of a box body of the refrigerator and is different from the first camera; a second detection sensor that detects whether the door of the second storage chamber is open; The photographing control unit starts photographing by the second camera based on a detection result of the second detection sensor. The imaging device according to claim 1 .

8. 1. An imaging method for an imaging device that images a refrigerator having a first storage compartment with a drawer having a first opening formed on an upper surface thereof, the imaging method comprising: a first camera arranged on an upper surface of a box body of the refrigerator; and a temperature sensor arranged on the upper surface of the box body of the refrigerator and detecting a temperature of an object arranged on a front side of the refrigerator, an image capture start step of starting image capture by the first camera based on a detection result of the temperature sensor; Including, how to shoot.

9. A control program for an imaging device that captures an image of a refrigerator having a first storage compartment with a drawer having a first opening formed on an upper surface thereof, the control program including: a first camera arranged on an upper surface of a box body of the refrigerator; a temperature sensor arranged on the upper surface of the box body of the refrigerator and detecting a temperature of an object arranged on a front side of the refrigerator; and a processor, the control program including: an imaging control unit that starts imaging by the first camera based on a detection result of the temperature sensor; A control program that acts as a

Images