Imaging apparatus, imaging method, and storage medium

Abstract

A narrow-angle camera is enabled to start shooting at a proper time. Comprises: a narrow-angle camera (302) that captures an image of a refrigerator (2) including a vegetable chamber (25) having a drawer (25A) having an opening formed in the upper surface thereof, the narrow-angle camera (302) being disposed on the top surface (20A) of a main case (20) of the refrigerator (2); a thermoelectric pile (305) that is disposed on the top surface (20A) of the main case (20) of the refrigerator (2) and that detects the temperature of an object disposed on the front surface side of the refrigerator (2); and an imaging control unit (354) that starts imaging by the narrow-angle camera (302) on the basis of the detection result of the thermopile (305).

Description

Technical Field

[0001] The present disclosure relates to a photographing device, a photographing method and a storage medium. Background Art

[0002] Patent Document 1 discloses a technology in which a camera is installed on the upper part of a refrigerator, an image is taken when an object passing through an opening of the refrigerator is detected, and food stored in the refrigerator is managed based on the imaged result.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-070476 Summary of the invention

[0006] Problems to be solved by the invention

[0007] The present disclosure provides an imaging device, an imaging method, and a storage medium for managing food stored in a cold storage with high accuracy.

[0008] Technical solutions to solve problems

[0009] The shooting device in the present disclosure includes: a first camera, which shoots a cold storage including a first storage room having a drawer with a first opening formed on the upper surface, and is arranged on the upper surface of the box body of the cold storage; a temperature sensor, which is arranged on the upper surface of the box body of the cold storage and detects the temperature of an object arranged on the front surface side of the cold storage; and a shooting control unit, which starts shooting by the first camera based on the detection result of the temperature sensor.

[0010] In addition, the shooting method in the present disclosure is a shooting method of a shooting device, including a shooting start step of starting shooting by the first camera based on the detection result of the temperature sensor, wherein the shooting device includes: a first camera, which shoots a cold storage including a first storage room having a drawer with a first opening formed on the upper surface, and is configured on the upper surface of the box body of the cold storage; and a temperature sensor, which is configured on the upper surface of the box body of the cold storage and detects the temperature of an object configured on the front surface side of the cold storage.

[0011] In addition, the storage medium in the present disclosure is a storage medium storing a control program of a shooting device, which enables the processor to function as a shooting control unit that starts shooting by the first camera based on the detection result of the temperature sensor, wherein the shooting device includes: a first camera, which shoots a cold storage including a first storage room having a drawer with a first opening formed on the upper surface, and is arranged on the upper surface of the box body of the cold storage; a temperature sensor, which is arranged on the upper surface of the box body of the cold storage and detects the temperature of an object arranged on the front surface side of the cold storage; and a processor.

[0012] Effects of the Invention

[0013] The photographing device, photographing method and storage medium disclosed in the present invention start photographing by the first camera based on the detection result of the temperature sensor disposed on the upper surface of the box body of the cold storage and detecting the temperature of the object disposed on the front surface side of the storage. Thus, for the first camera, photographing can be started at an appropriate time. Thus, the food stored in the cold storage can be managed with high precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a diagram showing the structure of the food management system according to the first embodiment.

[0015] Figure 2 This is a perspective view of the refrigerator according to the first embodiment.

[0016] Figure 3 This is a side view of the imaging device according to Embodiment 1 as viewed from the right side.

[0017] Figure 4 A perspective view of a restricting member in contact with the front end edge of the main housing according to Embodiment 1 as viewed from the left front.

[0018] Figure 5 This is a front view of the refrigerator and the imaging device according to Embodiment 1 as seen from the front.

[0019] Figure 6 This is a side view of the refrigerator and the imaging device according to Embodiment 1 as seen from the right side.

[0020] Figure 7 This is a plan view showing the imaging device installed in the refrigerator according to Embodiment 1 when viewed from above.

[0021] Figure 8 This is a block diagram showing the structure of the imaging device according to the first embodiment.

[0022] Fig. 9 This is a diagram showing an example of a temperature detection state of the thermopile according to the first embodiment.

[0023] Fig.10This is a diagram showing another example of the temperature detection state of the thermopile according to the first embodiment.

[0024] Fig.11 This is a diagram showing another example of the temperature detection state of the thermopile according to the first embodiment.

[0025] Fig.12 This is a diagram showing an example of a photographing state of the narrow-angle camera according to the first embodiment.

[0026] Fig.13 This is a diagram showing another example of the imaging state of the narrow-angle camera according to the first embodiment.

[0027] Fig.14 This is a diagram showing an example of a mark arranged in the refrigerating room according to the first embodiment.

[0028] Fig.15 This is a diagram showing an example of the hinge cover according to the first embodiment.

[0029] Fig.16 This is a diagram showing an example of the structure of a mark according to the first embodiment.

[0030] Fig.17 This is a block diagram showing the configuration of a terminal device and a food management server according to the first embodiment.

[0031] Fig.18 This is a flowchart showing the operation of the imaging device according to the first embodiment.

[0032] Fig.19 This is a flowchart showing the operations of the imaging device and the food management server according to the first embodiment.

[0033] Fig. 20 This is a flowchart showing the operations of the terminal device and the food management server in the first embodiment.

[0034] Description of Reference Numerals

[0035] 1 Food Management System

[0036] 2 Cold Storage

[0037] 3 Photographic Installation

[0038] 4 Terminal Devices

[0039] 5 Food Management Server

[0040] 20 Main cabinet (cabinet)

[0041] 20A top surface (upper surface)

[0042] 21 Refrigerator (Storage Room 2)

[0043] 21A door (opening and closing body)

[0044] 22Ice making room (1st storage room)

[0045] 23Fresh freezing room (1st storage room)

[0046] 24 Freezer (Storage Room 1)

[0047] 25 Vegetable room (1st storage room)

[0048] 26Storage Room

[0049] 22A, 23A, 24A, 25A, 26A drawers

[0050] 201 opening (2nd opening)

[0051] 301 wide-angle camera (second camera)

[0052] 302 narrow angle camera (first camera)

[0053] 303 1st distance measuring sensor (2nd detection sensor)

[0054] 304 2nd distance measuring sensor (2nd detection sensor)

[0055] 305 Thermocouple (Temperature Sensor)

[0056] 350 2nd processor (processor)

[0057] 351 1st Communication Control Unit

[0058] 352 2nd Communication Control Unit

[0059] 353 Open / Close Judgment Unit

[0060] 354 Shooting Control Unit

[0061] 355 Video Control Department

[0062] 356 Generation Department

[0063] 360 2nd Memory

[0064] 361 Control Program

[0065] 363 video data

[0066] DV1 1st video data

[0067] DV2 2nd video data

[0068] MK Mark

[0069] MK1 Mark 1

[0070] MK2 Mark 2

[0071] P1 1st captured image

[0072] P2 2nd captured image DETAILED DESCRIPTION

[0073] (Insights, etc. that form the basis of this disclosure)

[0074] At the time when the inventors conceived of the present disclosure, there was a technology in which a camera was installed on the top of a refrigerator, an object passing through the opening of the refrigerator was photographed when an object was detected, and food stored in the refrigerator was managed based on the photographed result.

[0075] The present inventors have discovered the problem of causing a camera to start shooting at an appropriate time in order to generate an appropriate captured image, and have developed the subject matter of the present disclosure in order to solve this problem.

[0076] Therefore, the present disclosure provides a photographing device, a photographing method, and a storage medium that enable a camera to start photographing at an appropriate time.

[0077] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. However, sometimes, more detailed descriptions than necessary will be omitted. For example, sometimes, detailed descriptions of already well-known matters or repeated descriptions of substantially the same structures will be omitted.

[0078] In addition, the drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0079] (Implementation Method 1)

[0080] [1-1. Structure]

[0081] [1-1-1. Structure of food management system]

[0082] Figure 1 This is a diagram showing the structure of the food management system 1 in the first embodiment.

[0083] The food management system 1 is a system for managing the food stored in the refrigerator 2. As an example of the management of the food, the food management system 1 of the present embodiment manages an image in which the food stored in the refrigerator 2 is captured.

[0084] In the present embodiment, the food stored in the refrigerator 2 includes the food itself, a container storing the food, etc. The food may be labeled with a predetermined mark.

[0085] The food management system 1 includes a cold storage 2 .

[0086] The cold storage 2 includes a main body 20 with an opening at the front. A refrigerating chamber 21, an ice making chamber 22, a fresh freezing chamber 23, a freezing chamber 24, and a vegetable chamber 25 are formed on the main body 20. These storage chambers are cooled by cold air. A revolving door 21A is provided at an opening 201 on the front surface of the refrigerating chamber 21. The door 21A opens and closes the opening 201. The door 21A includes a revolving right door 21B and a revolving left door 21C.

[0087] Refrigerating room 21 corresponds to an example of a "second storage room".

[0088] The opening 201 corresponds to an example of the “second opening”.

[0089] The door 21A corresponds to an example of an “opening and closing body”.

[0090] Drawers 22A, 23A, 24A and 25A capable of storing food are provided in ice making room 22, fresh freezing room 23, freezing room 24 and vegetable room 25. In the installed state, the upper surfaces of drawers 22A to 25A are open.

[0091] The opening formed in the upper surface of each of the drawers 22A to 25A corresponds to an example of the “first opening”.

[0092] In the following description, when the drawer 22A, the drawer 23A, the drawer 24A, and the drawer 25A are collectively referred to without distinguishing them, the symbol "26A" is marked and recorded as "drawer 26A". In addition, when the ice making chamber 22, the fresh freezing chamber 23, the freezing chamber 24, and the vegetable chamber 25 are collectively referred to without distinguishing them individually, the symbol "26" is marked and recorded as storage chamber 26.

[0093] The ice making room 22, the fresh freezing room 23, the freezing room 24, and the vegetable room 25 respectively correspond to an example of the "first storage room" of the present disclosure.

[0094] In Embodiment 1, the case where the "first storage room" of the present disclosure is vegetable room 25 is described. Moreover, in Embodiment 1, the case where the "second storage room" of the present disclosure is refrigeration room 21 is described.

[0095] The food management system 1 includes an imaging device 3. The imaging device 3 is provided on a top surface 20A of a main body 20 of the refrigerator 2. The top surface 20A of the main body 20 is made of metal. The imaging device 3 captures an image of the refrigerator 2.

[0096] The top surface 20A corresponds to an example of the “upper surface” in the present disclosure.

[0097] The food management system 1 includes a terminal device 4. The terminal device 4 is a portable computer such as a smartphone. The terminal device 4 has an application program related to the management of the food stored in the refrigerator 2 installed therein, and communicates with the food management server 5 according to the function of the application program.

[0098] In the following description, this application is described as "food management application" and is denoted by the symbol "411".

[0099] exist Figure 1 In FIG. 1 , a user P at home is indicated by a solid line, and a user P outside the home H is indicated by a dotted line. When the user P at home uses the terminal device 4, the terminal device 4 communicates with the food management server 5 via the communication device 6 or not via the communication device 6. In addition, when the user P outside the home H uses the terminal device 4 and cannot establish a communication connection with the communication device 6, the terminal device 4 communicates with the food management server 5 not via the communication device 6.

[0100] The communication device 6 is connected to a network NW such as a public network, and communicates with the food management server 5 via the network NW. The communication device 6 functions as an interface device for connecting the refrigerator 2, the imaging device 3, and the terminal device 4 to the network NW. The communication device 6 constructs a local area network for the home H.

[0101] The food management system 1 includes a food management server 5. The food management server 5 is a device for managing the food stored in the cold storage 2. The food management server 5 stores images of the food as food management. The food management server 5 is connected to the network NW. Figure 1 In the figure, the food management server 5 is represented by one frame, but this does not mean that the food management server 5 is composed of a single device.

[0102] [1-1-2. Structure of cold storage]

[0103] Figure 2 It is a perspective view of the refrigerator 2 in the installed state.

[0104] Figure 2 The X-axis, Y-axis, and Z-axis are shown in the figure. The X-axis, Y-axis, and Z-axis are orthogonal to each other. The Z-axis represents the up-down direction, which is equivalent to the height direction of the cold storage 2. The X-axis and the Y-axis are parallel to the horizontal direction. The X-axis represents the left-right direction. The left-right direction is equivalent to the width direction of the cold storage 2. The Y-axis represents the front-back direction. The positive direction of the X-axis represents the right direction. The positive direction of the Y-axis represents the front direction. The positive direction of the Z-axis represents the upper direction.

[0105] Figure 3 to Figure 7 and Figures 9 to 14 The X-axis, Y-axis and Z-axis are also shown respectively.

[0106] Refrigerator 2 includes a main body 20, a door 21A, and a drawer 26A.

[0107] The main box body 20 is a box-shaped component that forms the main body of the cold storage 2. The main box body 20 includes: an outer box 202 that is open on the front surface; an inner box 203 that is accommodated inside the outer box 202 and is open on the front surface; and a heat-insulating material that fills the space between the outer box 202 and the inner box 203. The outer box 202 is formed of, for example, metal such as iron, hard resin such as ABS, etc. The upper surface of the outer box 202, that is, the top surface 20A of the main box 20, is formed of metal. The inner box 203 is formed of, for example, a hard resin such as ABS. The heat-insulating material is, for example, hard polyurethane foam. In addition, the heat-insulating material is not limited to this, and it can be a material that has heat-insulating properties and can be filled in the space between the outer box 202 and the inner box 203.

[0108] The main housing 20 corresponds to an example of the “housing” in the present disclosure.

[0109] The inner space of the inner box 203 is divided into a plurality of spaces by a partition member. In the present embodiment, the inner space of the inner box 203 is divided into five spaces, which function as a refrigerating chamber 21, an ice making chamber 22, a fresh freezing chamber 23, a freezing chamber 24, and a vegetable chamber 25, respectively.

[0110] The right door 21B is rotatably supported by a right door hinge member 204. The right door hinge member 204 is a hinge member provided on the top surface 20A of the main housing 20. The left door 21C is rotatably supported by a left door hinge member 205. The left door hinge member 205 is a hinge member provided on the top surface 20A of the main housing 20.

[0111] Second wireless communication module 206 is provided on right door hinge member 204. Second wireless communication module 206 is a module that receives control data for controlling refrigerator 2 from an external device or transmits predetermined data to the external device.

[0112] [1-1-3. Structure of the imaging device]

[0113] Reference Figure 3 to Figure 7 The imaging device 3 will be described.

[0114] exist Figure 3 to Figure 7 In the figure, it is shown that Figure 2 The X-axis, Y-axis, and Z-axis shown are the same as the X-axis, Y-axis, and Z-axis. The X-axis direction corresponds to the width direction of the camera 3 and the left-right direction of the camera 3. The Y-axis direction corresponds to the front-back direction of the camera 3. The Z-axis direction corresponds to the up-down direction of the camera 3.

[0115] Figure 3 This is a side view of imaging device 3 attached to refrigerator 2 as seen from the right side. Figure 4This is a perspective view of the restriction member 34 in contact with the front end edge 20B of the main housing 20 as viewed from the left front.

[0116] like Figure 3 and Figure 4 As shown in FIG. 1 , the photographing device 3 includes a main body 30 disposed on the top surface 20A of the main housing 20 and a photographing component 31 extending forward from the main body 30 . The photographing component 31 is disposed above the front portion of the main body 30 .

[0117] The main body 30 includes a power button 32 on the front surface for turning on and off the power of the imaging device 3. The power button 32 is provided with a first LED (Light emitting diode) 33 for notifying predetermined information.

[0118] In addition, the main body 30 includes a shooting start button 32A on the front surface for starting shooting of the shooting device 3. When the shooting start button 32A is pressed by the user P, Figure 8 The shooting control unit 354 shown in the figure starts shooting with the wide-angle camera 301 and the narrow-angle camera 302. In addition, when the shooting start button 32A is pressed, Figure 8 The video recording control unit 355 shown starts recording the image capturing results of the wide-angle camera 301 and the narrow-angle camera 302 .

[0119] Furthermore, even if the door 21A or the drawer 26A is already in the open state, the user P presses the shooting start button 32A when the wide-angle camera 301 and the narrow-angle camera 302 have not started shooting. When the wide-angle camera 301 and the narrow-angle camera 302 start shooting, the shooting control unit 354 turns on the second LED 308 and the third LED 309. Thus, the user P can recognize that the wide-angle camera 301 and the narrow-angle camera 302 have not started shooting because the second LED 308 and the third LED 309 are not turned on.

[0120] The main body 30 includes a limiting component 34. The limiting component 34 limits the position of the photographing component 31 in the front-to-back direction of the cold storage 2 by abutting against the front end edge 20B of the main body 20. The limiting component 34 is a plate-shaped component extending downward from the bottom surface 30A of the main body 30. The limiting component 34 extends along the width direction of the main body 30 (the width direction of the photographing device 3) on the front surface of the main body 30, and has a mark 35 in the center of the width direction of the main body 30. The mark 35 indicates the center in the width direction of the photographing device 3. The length of the limiting component 34 extending from the bottom surface 30A is formed to be shorter than the length from the top surface 20A of the main body 20 to the door seal provided on the upper inner surface of the door 21A and substantially parallel to the top surface 20A.

[0121] Figure 5This is a front view of the refrigerator 2 and the imaging device 3 attached to the refrigerator 2 as seen from the front. Figure 5 Door 21A of refrigerator 2 is shown in an open state. Figure 6 It is a side view which looked at the refrigerator 2 and the imaging device 3 attached to the refrigerator 2 from the right side. Figure 6 Drawer 25A of refrigerator 2 is shown in an open state, that is, in a pulled-out state. Figure 7 It is a top view of imaging device 3 installed in refrigerator 2 as seen from above.

[0122] Reference Figure 5 to Figure 7 The structure of the imaging device 3 will be described.

[0123] The imaging device 3 includes a wide-angle camera 301 and a narrow-angle camera 302. The wide-angle camera 301 and the narrow-angle camera 302 are disposed at the front end portion of the bottom surface 31A of the imaging component 31. When the imaging device 3 is installed in the cold storage 2 and the door 21A is closed, the wide-angle camera 301 and the narrow-angle camera 302 are located in front of the door 21A.

[0124] Wide-angle camera 301 is a camera having a wider angle than narrow-angle camera 302. In a state where imaging device 3 is attached to refrigerator 2, wide-angle camera 301 captures refrigerating room 21 from the upper front of refrigerator 2.

[0125] For example, when viewed from the front of the cold storage 2, the shooting range of the wide-angle camera 301 includes Figure 5 When the right door 21B and the left door 21C are opened to the maximum, the range A1 includes the door shelves 21E of the right door 21B and the left door 21C and the opening 201 of the refrigerator 21. In addition, when viewed from the right side of the refrigerator 2, the shooting range of the wide-angle camera 301 includes Figure 6 Range A3 is a range from the front ends of the right door 21B and the left door 21C in the open state to the front end of the shelf 21F in the front-rear direction, and is a range including the opening 201 of the refrigerator compartment 21 in the up-down direction.

[0126] The wide-angle camera 301 corresponds to an example of the “second camera” in the present disclosure.

[0127] The narrow-angle camera 302 is a camera with a narrower angle than the wide-angle camera 301. In other words, the shooting angle of the narrow-angle camera 302 in the left-right direction, i.e., the X-axis direction, is smaller than the shooting angle of the wide-angle camera 301 in the left-right direction, i.e., the X-axis direction. In addition, the shooting angle of the narrow-angle camera 302 in the front-back direction, i.e., the Y-axis direction, is smaller than the shooting angle of the wide-angle camera 301 in the front-back direction, i.e., the Y-axis direction.

[0128] When imaging device 3 is attached to refrigerator 2 , narrow-angle camera 302 captures an image of drawer 26A from the upper front of refrigerator 2 .

[0129] For example, when viewed from the front of the cold storage 2, the shooting range of the narrow-angle camera 302 includes Figure 5 Range A2 is a range including drawer 26A when viewed from the front of refrigerator 2. For example, when viewed from the right of refrigerator 2, the shooting range of narrow-angle camera 302 includes Figure 6 The range A4 is a range including the drawer 26A in the maximum pulled-out state in the front-rear direction. In addition, the imaging range of the narrow-angle camera 302 may also be a range including the drawers provided in the refrigerator compartment 21.

[0130] The narrow-angle camera 302 corresponds to an example of the “first camera” in the present disclosure.

[0131] like Figure 5 to Figure 7 As shown, the narrow-angle camera 302 constitutes a part of the imaging device 3 of Embodiment 1. The imaging device 3 of Embodiment 1 includes the narrow-angle camera 302, the thermopile 305, and the imaging control unit 354. In other words, the imaging device 3 includes the narrow-angle camera 302, the thermopile 305, and the imaging control unit 354.

[0132] Reference Figure 8 The imaging control unit 354 will be described.

[0133] The imaging device 3 includes a first distance measuring sensor 303 and a second distance measuring sensor 304 . The first distance measuring sensor 303 and the second distance measuring sensor 304 are provided in the main body 30 .

[0134] The first distance measuring sensor 303 is a sensor that detects the opening and closing of the right door 21B by detecting the distance between the first distance measuring sensor 303 and the right door 21B. For example, the first distance measuring sensor 303 is an infrared sensor. The first distance measuring sensor 303 detects the distance between the right door 21B and itself, and outputs a detection value indicating the detected distance. The first distance measuring sensor 303 detects the distance, for example, by the change of the optical axis.

[0135] The second distance measuring sensor 304 is a sensor that detects the opening and closing of the left door 21C by detecting the distance between the second distance measuring sensor 304 and the left door 21C. For example, the second distance measuring sensor 304 is an infrared sensor. The second distance measuring sensor 304 detects the distance between the left door 21C and itself, and outputs a detection value indicating the detected distance. The second distance measuring sensor 304 detects the distance, for example, by the change of the optical axis.

[0136] The first distance measuring sensor 303 and the second distance measuring sensor 304 each correspond to an example of a “second detection sensor”.

[0137] In this embodiment, the case where the "second detection sensor" is a distance measuring sensor is described, but the present disclosure is not limited thereto. The "second detection sensor" may be any one of a photoelectric sensor, a Hall sensor, an acceleration sensor, and a temperature sensor.

[0138] The imaging device 3 includes a thermopile 305. The thermopile 305 is a sensor that detects the opening and closing of the drawer 26A. The thermopile 305 is provided at the front end of the bottom surface 31A of the imaging member 31. The thermopile 305 outputs a detection value indicating the detected temperature.

[0139] The thermopile 305 corresponds to an example of a “temperature sensor” in the present disclosure.

[0140] Reference Figures 9 to 11 The thermopile 305 is further described.

[0141] The camera 3 includes a buzzer 306. The buzzer 306 informs the user of the change in the state of the cold storage 2. For example, when at least one of the right door 21B and the left door 21C reaches a specified opening and closing angle θ, the buzzer 306 outputs a buzzer sound. The specified opening and closing angle θ is, for example, 80 degrees to 100 degrees. Whether it is the specified opening and closing angle θ is determined based on the captured image PA1 captured by the narrow-angle camera 302. The captured image PA1 includes an image of the first marker MK1. The captured image PA1 is stored in the state storage unit 364.

[0142] For example, when the right door 21B changes from the closed state to a predetermined opening / closing angle θ, the buzzer 306 outputs a buzzer sound. Also, for example, when the right door 21B changes from the predetermined opening / closing angle θ to the closed state, the buzzer 306 outputs a buzzer sound.

[0143] Reference Figure 14-15 The first marker MK1 will be further described.

[0144] In addition, for example, when the state where the second mark MK2 arranged in the drawer 26A is not in the prescribed position changes to the state where the second mark MK2 is in the prescribed position, the buzzer 306 outputs a buzzer sound. Whether the second mark MK2 is in the prescribed position is determined based on the captured image PA2 captured by the narrow-angle camera 302. The captured image PA2 includes an image of the second mark MK2. The captured image PA2 is stored in the state storage unit 364.

[0145] In addition, for example, when the state where the second mark MK2 arranged in the drawer 26A is located at the prescribed position changes to the state where the second mark MK2 is not located at the prescribed position, the buzzer 306 outputs a buzzer sound.

[0146] Reference Figure 12-13 The second mark MK2 is further described.

[0147] In addition, when the first mark MK1 and the second mark MK2 are not distinguished, they are described as marks MK. The marks MK include the first mark MK1 and the second mark MK2.

[0148] The imaging device 3 includes an illuminance sensor 307. The illuminance sensor 307 is a sensor for adjusting the exposure of the wide-angle camera 301 and the narrow-angle camera 302. The illuminance sensor 307 outputs a detection value. The illuminance sensor 307 is provided at the front end of the imaging unit 31.

[0149] The imaging device 3 includes a second LED 308 and a third LED 309. The second LED 308 is provided at the right front end of the bottom surface 31A of the imaging member 31. The third LED 309 is provided at the left front end of the bottom surface 31A of the imaging member 31. The second LED 308 and the third LED 309 illuminate the refrigerator 2 on which the imaging device 3 is mounted from above.

[0150] like Figure 7 As shown, the imaging device 3 includes a first control substrate 311 and a second control substrate 312 .

[0151] The first control substrate 311 is a substrate for controlling various sensors provided in the imaging device 3 and a power supply of the imaging device 3. The first control substrate 311 is provided inside the imaging device 3. The power button 32, the shooting start button 32A, the first LED 33, the second LED 308, the third LED 309, the first distance measuring sensor 303, the second distance measuring sensor 304, the thermopile 305, the buzzer 306, and the illuminance sensor 307 are connected to the first control substrate 311. And the first control substrate 311 controls these devices. The first control substrate 311 is provided behind the second control substrate 312 and the first wireless communication module 313. In addition, the first control substrate 311 can also be divided into a substrate for controlling various sensors provided in the imaging device 3 and a substrate for controlling the power supply of the imaging device 3.

[0152] The second control substrate 312 is a substrate for controlling the wide-angle camera 301 and the narrow-angle camera 302, and also controlling the first wireless communication module 313. The second control substrate 312 is provided inside the imaging device 3. The wide-angle camera 301, the narrow-angle camera 302, and the first wireless communication module 313 are connected to the second control substrate 312. And the second control substrate 312 controls these devices. The second control substrate 312 is arranged in front of the first control substrate 311 and aligned with the first control substrate 311.

[0153] The first wireless communication module 313 is a device including hardware such as an antenna and a wireless communication circuit. The first wireless communication module 313 is provided inside the imaging component 31 to the left of the second control substrate 312. The first wireless communication module 313 is provided in the imaging device 3 so as to be located in front of the opening 201 of the main body 20 when the imaging device 3 is installed in the cold storage 2. More specifically, the first wireless communication module 313 is provided in the imaging device 3 at a position in front of the limiting component 34.

[0154] like Figure 7 As shown, in a state where the imaging device 3 is mounted in the refrigerator 2 , a metal plate 314 is provided between the first wireless communication module 313 and the second wireless communication module 206 in the left-right direction.

[0155] Figure 8 It is a block diagram showing the structure of the imaging device 3.

[0156] like Figure 8 As shown, the first control substrate 311 includes a first processor 320 such as a CPU (Central Processing Unit), a first memory 330, a first communication module 340, and an interface circuit to which other devices and sensors are connected.

[0157] The first memory 330 is a storage device (storage medium) that stores programs and data executed by the first processor 320. The first memory 330 is composed of a non-volatile storage device such as a ROM (Read Only Memory). The first memory 330 may also include a volatile storage device such as a RAM (Random Access Memory) that constitutes a working area of ​​the first processor 320. The first memory 330 stores data processed by the first processor 320 and a control program 331 executed by the first processor 320.

[0158] The first communication module 340 includes hardware such as a communication circuit and a connector in compliance with a predetermined wired communication standard, and communicates with the second control board 312 under the control of the first processor 320 .

[0159] The first processor 320 reads and executes the control program 331 to turn the power of the imaging device 3 on and off according to the operation of the power button 32 .

[0160] Furthermore, the first processor 320 reads out and executes the control program 331 to turn on and off the first LED 33 , the second LED 308 , and the third LED 309 .

[0161] Furthermore, the first processor 320 reads out and executes the control program 331 to transmit the detection values ​​output by each sensor connected to the first control board 311 to the second control board 312 via the first communication module 340 .

[0162] The second control substrate 312 includes a second processor 350 such as a CPU, a second memory 360 , a second communication module 370 , and an interface circuit to which other devices and sensors are connected.

[0163] The second memory 360 is a storage device (storage medium) that stores programs and data executed by the second processor 350. The second memory 360 is composed of a non-volatile storage device such as a ROM. In addition, the second memory 360 may also include a volatile storage device such as a RAM that constitutes a working area of ​​the second processor 350. The second memory 360 stores data processed by the second processor 350, a control program 361 executed by the second processor 350, and an image capture device ID (Identification) 362.

[0164] The imaging device ID 362 is information that unambiguously identifies the imaging device 3 .

[0165] The second processor 350 corresponds to an example of a “processor”.

[0166] The second communication module 370 includes hardware such as a communication circuit and a connector in accordance with a predetermined wired communication standard, and communicates with the first control substrate 311 under the control of the second processor 350 .

[0167] The second processor 350 reads out and executes the control program 361 stored in the second memory 360, thereby functioning as the first communication control unit 351, the second communication control unit 352, the opening / closing determination unit 353, the imaging control unit 354, the recording control unit 355, and the generating unit 356. In addition, the second processor 350 reads out and executes the control program 361 stored in the second memory 360, thereby causing the second memory 360 to function as a state storage unit 364.

[0168] The control program 361 corresponds to an example of the “control program” of the present disclosure.

[0169] The first communication control unit 351 communicates with the food management server 5 via the first wireless communication module 313 .

[0170] The second communication control unit 352 communicates with the first control board 311 via the second communication module 370. The second communication control unit 352 receives detection values ​​of various sensors connected to the first control board 311 from the first control board 311.

[0171] The opening and closing judgment unit 353 judges whether the right door 21B is in the open state or the closed state based on the detection value of the first distance measuring sensor 303 received by the second communication control unit 352. When the second communication control unit 352 receives a detection value different from the detection value corresponding to the closed state, the opening and closing judgment unit 353 judges that the right door 21B is in the open state. For example, when the detection value of the first distance measuring sensor 303 is greater than a prescribed value, the opening and closing judgment unit 353 judges that the right door 21B is in the open state. On the other hand, when the second communication control unit 352 receives a detection value corresponding to the closed state, the opening and closing judgment unit 353 judges that the right door 21B is in the closed state. For example, when the detection value of the second distance measuring sensor 304 is smaller than a prescribed value, the opening and closing judgment unit 353 judges that the right door 21B is in the closed state.

[0172] The opening / closing determination unit 353 determines whether the left door 21C is in the open state or the closed state based on the detection value of the second distance measuring sensor 304 by the second communication control unit 352 in the same manner as the determination for the right door 21B.

[0173] The opening / closing determination unit 353 determines whether the drawer 26A is in the open state or the closed state based on the detection value of the thermopile 305 received by the second communication control unit 352 .

[0174] For example, when the second communication control unit 352 receives the detection value of the thermopile 305 indicating a temperature rise equal to or higher than the first threshold TH1, the opening / closing determination unit 353 determines that the drawer 26A has changed from the closed state to the open state. The first threshold TH1 is 5°C, for example.

[0175] When the second communication control unit 352 receives the detection value of the thermopile 305 indicating a temperature drop above the second threshold TH2, for example, the opening / closing determination unit 353 determines that the drawer 26A has changed from the closed state to the open state. The second threshold TH2 is 5°C, for example.

[0176] Reference Figures 9 to 11 The processing of the opening / closing determination unit 353 will be further described.

[0177] Furthermore, for example, when the second communication control unit 352 receives a detection value from the thermopile 305 indicating that a temperature within a predetermined range has continued for more than a predetermined period, the opening / closing determination unit 353 determines that the drawer 26A is in the closed state.

[0178] The temperature in the predetermined range is, for example, in the range of 15°C or more and 30°C or less. The predetermined period is, for example, 5 seconds. Fig. 9 The temperature of the area FLA shown corresponds to a portion of the ground FL.

[0179] When the opening / closing judgment unit 353 judges that the door 21A or the drawer 26A is in the open state, the shooting control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start shooting. In addition, when the shooting control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start shooting, the second communication control unit 352 sends lighting request information to the first control substrate 311. The lighting request information is information requesting the second LED 308 and the third LED 309 to light up. When receiving the lighting request information, the first processor 320 lights up the second LED 308 and the third LED 309 and notifies the user P that shooting has started. In addition, the first processor 320 may also light up the second LED 308 and the third LED 309 when the opening / closing judgment unit 353 judges that the door 21A or the drawer 26A is in the open state.

[0180] Furthermore, when the opening / closing determination unit 353 determines that the door 21A and the drawer 26A are in a closed state, the shooting control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end shooting. Furthermore, when the shooting control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end shooting, the second communication control unit 352 sends a extinguishing request message to the first control substrate 311. The extinguishing request message is information requesting the second LED 308 and the third LED 309 to be extinguished. Upon receiving the extinguishing request message, the first processor 320 extinguishes the second LED 308 and the third LED 309 and notifies the user P that shooting is ended. Furthermore, the first processor 320 may also extinguish the second LED 308 and the third LED 309 when the opening / closing determination unit 353 determines that the door 21A or the drawer 26A is in a closed state.

[0181] In addition, when the opening and closing judgment unit 353 determines that the door 21A or the drawer 26A is in an open state, the shooting control unit 354 may also cause the wide-angle camera 301 and the narrow-angle camera 302 to stop shooting when a specified time (for example, 1 minute) has passed after the wide-angle camera 301 and the narrow-angle camera 302 started shooting.

[0182] In addition, the shooting control unit 354 may also extend the prescribed time when the opening and closing judgment unit 353 determines that the door 21A or the drawer 26A is in the open state after the wide-angle camera 301 and the narrow-angle camera 302 start shooting when the opening and closing judgment unit 353 determines that the door 21A or the drawer 26A is in the open state. For example, the shooting control unit 354 may also extend the prescribed time when the opening and closing judgment unit 353 determines that the drawer 26A is in the open state after the wide-angle camera 301 and the narrow-angle camera 302 start shooting when the opening and closing judgment unit 353 determines that the door 21A is in the open state. In this case, the shooting image of the drawer 26A can be reliably generated.

[0183] After the imaging control unit 354 causes the narrow-angle camera 302 to start imaging, the recording control unit 355 determines whether the door 21A is at a predetermined opening / closing angle θ based on the image PA1 captured by the narrow-angle camera 302. The predetermined opening / closing angle θ is, for example, 80 to 100 degrees.

[0184] When the recording control unit 355 determines that the door 21A is at the predetermined opening / closing angle θ, the recording control unit 355 starts recording the image capturing results of the wide-angle camera 301 and the narrow-angle camera 302. That is, the recording control unit 355 starts recording the image capturing data 363 in the second memory 360. The image capturing data 363 includes the first image capturing data DV1 representing the image capturing results of the wide-angle camera 301 and the second image capturing data DV2 representing the image capturing results of the narrow-angle camera 302.

[0185] In the following description, door 21A being at a predetermined opening / closing angle θ may be described as “satisfying the recording start condition.” The “recording start condition” refers to a condition for recording control unit 355 to start recording the image capturing results of wide-angle camera 301 and narrow-angle camera 302 .

[0186] After the imaging control unit 354 causes the narrow-angle camera 302 to start imaging, the recording control unit 355 may start recording the imaging results of the wide-angle camera 301 and the narrow-angle camera 302 when the narrow-angle camera 302 can capture an image of the first mark MK1.

[0187] Furthermore, when the determination of the video recording control unit 355 changes from the state that the door 21A is not at the prescribed opening and closing angle θ to the state that the door 21A is at the prescribed opening and closing angle θ, the second communication control unit 352 transmits the output request information of the buzzer sound to the first control substrate 311. Furthermore, when the determination of the video recording control unit 355 changes from the state that the door 21A is at the prescribed opening and closing angle θ to the state that the door 21A is not at the prescribed opening and closing angle θ, the second communication control unit 352 transmits the output request information of the buzzer sound to the first control substrate 311.

[0188] The buzzer sound output request information is information for requesting the output of a buzzer sound to the buzzer 306. Upon receiving the buzzer sound output request information, the first processor 320 outputs a buzzer sound to the buzzer 306 and notifies the user that the P door 21A is at a predetermined opening / closing angle θ.

[0189] After the imaging control unit 354 starts imaging with the narrow-angle camera 302, the recording control unit 355 determines whether the drawer 26A is pulled out by a predetermined amount based on the image PA2 captured by the narrow-angle camera 302. The predetermined amount of pulling out is, for example, 80% to 100%.

[0190] When the video recording control unit 355 determines that the drawer 26A is pulled out to a predetermined amount, the video recording control unit 355 starts recording the shooting results of the wide-angle camera 301 and the narrow-angle camera 302. That is, the video recording control unit 355 starts recording the video recording data 363 in the second memory 360. The video recording data 363 includes the first video recording data DV1 indicating the shooting results of the wide-angle camera 301 and the second video recording data DV2 indicating the shooting results of the narrow-angle camera 302.

[0191] In the following description, the fact that the drawer 26A is pulled out to a predetermined amount may be described as “satisfying the recording start condition”.

[0192] The recording start condition is that the door 21A is opened or closed at a predetermined angle θ or the drawer 26A is pulled out by a predetermined amount.

[0193] The recording control unit 355 may start recording the shooting results of the wide-angle camera 301 and the narrow-angle camera 302 when the narrow-angle camera 302 is able to capture the image of the second marker MK2 after the shooting control unit 354 causes the narrow-angle camera 302 to start shooting.

[0194] Furthermore, when the judgment of the video recording control unit 355 changes from the state that the drawer 26A is not pulled out by the prescribed amount to the state that the drawer 26A is pulled out by the prescribed amount, the second communication control unit 352 sends a buzzer sound output request information to the first control substrate 311. Furthermore, when the judgment of the video recording control unit 355 changes from the state that the drawer 26A is pulled out by the prescribed amount to the state that the drawer 26A is not pulled out by the prescribed amount, the second communication control unit 352 sends a buzzer sound output request information to the first control substrate 311.

[0195] The buzzer sound output request information is information for requesting the output of a buzzer sound to the buzzer 306. Upon receiving the buzzer sound output request information, the first processor 320 outputs a buzzer sound to the buzzer 306 to notify the user that the P mark MK is at a predetermined position.

[0196] When the opening / closing determination unit 353 determines that the door 21A and the drawer 26A are in the closed state, the recording control unit 355 ends the recording of the shooting results of the wide-angle camera 301 and the narrow-angle camera 302. Alternatively, the recording control unit 355 may end the recording of the shooting results of the wide-angle camera 301 and the narrow-angle camera 302 after a predetermined time (e.g., 1 minute) has passed since the narrow-angle camera 302 could no longer capture the image of the second marker MK2.

[0197] Furthermore, after the recording control unit 355 completes recording of the recording results of the wide-angle camera 301 and the narrow-angle camera 302 , the imaging control unit 354 completes imaging by the wide-angle camera 301 and the narrow-angle camera 302 .

[0198] In the first embodiment, when the recording control unit 355 determines that the recording start condition is satisfied, the recording control unit 355 starts recording the image capturing results of the wide-angle camera 301 and the narrow-angle camera 302 , but the present disclosure is not limited thereto.

[0199] For example, when the opening / closing determination unit 353 determines that the door 21A or the drawer 26A is in the open state, the video recording control unit 355 may start recording the shooting results of the wide-angle camera 301 and the narrow-angle camera 302. In this case, the recording of the captured image by the narrow-angle camera 302 can be started at an appropriate time through simple processing.

[0200] The state storage unit 364 stores the captured image PA1 captured by the narrow-angle camera 302 when the opening and closing angle of the door 21A is a predetermined opening and closing angle θ. The captured image PA1 includes the image of the first marker MK1. The predetermined opening and closing angle θ is, for example, 80 to 100 degrees.

[0201] Reference Figure 14-15 The first marker MK1 is described.

[0202] When the drawer 26A is pulled out to a predetermined amount, the state storage unit 364 stores the captured image PA2 captured by the narrow-angle camera 302 .

[0203] When the pull-out amount is zero, the drawer 26A is in a closed state. The larger the specified pull-out amount is, the more the drawer 26A is opened. A pull-out amount of 100% indicates that the drawer 26A is fully opened. The specified pull-out amount is, for example, 80% to 100%.

[0204] The second mark MK2 is arranged at the upper end of the front panel arranged at the front end of the drawer 26A of the storage room 26, and the shooting state of the narrow-angle camera 302 changes according to the pull-out amount of the drawer 26A. In addition, the pull-out amount of the drawer 26A is the pull-out amount of the drawer 26A relative to the storage room 26. In the first embodiment, the storage room 26 is the vegetable room 25, and the drawer 26A is the drawer 25A.

[0205] Reference Figure 12-13 The second mark MK2 is further described.

[0206] Based on the photographed image PA1, the generator 356 selects the first photographed image P1 from the photographed images PB1 included in the moving image shown in the first recorded data DV1 of the recorded data 363. The first photographed image P1 is a photographed image when the door 21A is at a predetermined opening / closing angle θ.

[0207] Furthermore, based on the photographed image PA2, the generator 356 selects the second photographed image P2 from the photographed image PB2 included in the moving image shown in the second recorded data DV2 of the recorded data 363. The second photographed image P2 is a photographed image when the drawer 26A is pulled out to a predetermined amount.

[0208] Furthermore, the generator 356 generates management data D1 based on the first captured image P1 and the second captured image P2. The management data D1 is data uploaded to the food management server 5 and is data for managing the food stored in the cold storage 2. The first communication control unit 351 transmits the management data D1 generated by the generator 356 to the food management server 5.

[0209] In this embodiment, the case where the imaging device 3 includes the generating unit 356 is described, but the present invention is not limited thereto. For example, the food management server 5 may include the generating unit 356. In this case, the management data D1 includes the first video data DV1 and the second video data DV2.

[0210] In the first embodiment, the generator 356 selects the first captured image P1 from the captured images PB1 included in the moving image shown in the first recorded data DV1 of the recorded data 363 based on the captured image PA1, but the present disclosure is not limited thereto. The generator 356 may select the captured image PB1 in which the least amount of the user P is captured as the first captured image P1 from the captured images PB1 included in the moving image shown in the first recorded data DV1 of the recorded data 363 based on the captured image PA1.

[0211] At this time, the generation unit 356 further selects the first captured image P1 from the first captured image P1 selected based on the captured image PA1 in the following manner. That is, the generation unit 356 recognizes the image of the user P by a method such as pattern matching, and selects the captured image PB1 having the smallest area of ​​the recognized image of the user P as the first captured image P1.

[0212] Similarly, in the first embodiment, the generator 356 selects the second captured image P2 from the captured images PB2 included in the moving image shown in the second recorded data DV2 of the recorded data 363 based on the captured image PA2, but the present invention is not limited thereto. The generator 356 may select the captured image PB2 in which the least amount of the user P is captured as the second captured image P2 from the captured images PB2 included in the moving image shown in the second recorded data DV2 of the recorded data 363 based on the captured image PA2.

[0213] [1-1-4. Temperature detection status of the thermopile]

[0214] Next, refer to Figures 9 to 11The temperature detection state ST of the thermopile 305 will be described. The temperature detection state ST indicates the state of the detection range of the thermopile 305 when the thermopile 305 detects temperature.

[0215] exist Figures 9 to 11 Shown in Figure 2 The X-axis, Y-axis, and Z-axis shown are the same. The X-axis direction corresponds to the width direction of the cold storage 2 and the left-right direction of the cold storage 2. The Y-axis direction corresponds to the front-back direction of the cold storage 2. The Z-axis direction corresponds to the up-down direction of the cold storage 2.

[0216] Fig. 9 1 is a diagram showing an example of the temperature detection state ST1 of the thermopile 305 according to the first embodiment. Fig. 9 As shown, the thermopile 305 is arranged on the upper surface of the main box 20 of the cold storage 2. In other words, as shown in FIG. Figure 5 to Figure 7 As described above, the thermopile 305 is provided at the front end of the bottom surface 31A of the imaging member 31 constituting the imaging device 3. Fig. 9 As shown, thermopile 305 detects the temperature of an object disposed on the front surface side of refrigerator 2 .

[0217] like Fig. 9 As shown, the thermopile 305 detects the temperature of an object disposed in a detection range of the detection angle θ1. The detection range is a range extending in the up-down direction along the front surface of the cold storage 2. In addition, the front surface of the cold storage 2 includes, for example, the front surface of the door 21A in a closed state and the front surface of the drawer 26A in a closed state. The detection range extends approximately parallel to the Z axis. The detection angle θ1 is, for example, 10 degrees.

[0218] like Fig. 9 As shown, the thermopile 305 is arranged at the front end of the bottom surface 31A of the photographing component 31 in a manner that the detection range of the thermopile 305 is tilted forward, that is, in the positive direction of the Y axis. In addition, the thermopile 305 is arranged at the front end of the bottom surface 31A of the photographing component 31 in a manner that is tilted in the negative direction of the X axis in a manner that the detection range of the thermopile 305 includes the central position in the left-right direction of the cold storage 2. That is, it is preferred that the thermopile 305 is arranged in a manner that includes the central position in the left-right direction of the cold storage 2.

[0219] exist Fig. 9 In the detection range of the thermopile 305 , no object is arranged between the thermopile 305 and the ground FL. Therefore, in the temperature detection state ST1 , the thermopile 305 detects the temperature of a partial area FLA of the ground FL.

[0220] The temperature of the area FLA is, for example, 20°C.

[0221] Fig.10 FIG. 2 is a diagram showing another example of the temperature detection state ST of the thermopile 305 in the first embodiment. Fig.10 As shown, in the temperature detection state ST2, the hand HD of the user P is arranged between the thermopile 305 and the floor FL in the detection range of the thermopile 305. For example, the user P hooks the finger of the hand HD on the upper end of the front panel of the drawer 25A in order to pull out the drawer 25A of the vegetable compartment 25.

[0222] At this time, the thermopile 305 detects the temperature of a partial region HA of the upper surface of the hand HD (here, the nail of the hand HD). The temperature of the region HA is, for example, 36°C.

[0223] For example, in Fig. 9 The temperature detection state ST1 shown in FIG. Fig.10 In the temperature detection state ST2 shown, the detection value of the thermopile 305 indicates, for example, a temperature rise of 16°C (=36°C to 20°C). Therefore, since the detection value of the thermopile 305 indicates a temperature rise above the first threshold TH1, the opening and closing determination unit 353 determines that the drawer 26A changes from the closed state to the open state. In addition, since the opening and closing determination unit 353 determines that the drawer 26A is in the open state, the shooting control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start shooting.

[0224] exist Fig.10 In the description, the temperature detection state ST2 is described as the case where the user P pulls out the drawer 25A of the vegetable chamber 25, but the present disclosure is not limited thereto. The temperature detection state ST2 may be the case where the user P pulls out the drawer 26A. That is, the temperature detection state ST2 may be the case where the user P pulls out any one of the drawers 22A, 23A, 24A, and 25A.

[0225] Fig.11 FIG. 2 is a diagram showing another example of the temperature detection state ST of the thermopile 305 in the first embodiment. Fig.11 As shown, the temperature detection state ST3 corresponds to the state in which the user P pulls out the drawer 25A of the vegetable compartment 25 to a pull-out amount greater than a prescribed amount. The prescribed amount is, for example, 30%. In the temperature detection state ST3, the bottom surface of the drawer 25A of the vegetable compartment 25 is arranged between the thermopile 305 and the floor FL within the detection range of the thermopile 305.

[0226] At this time, the thermopile 305 detects the temperature of a portion of the bottom surface of the drawer 25A, the area BA. The temperature of the area BA is 5°C, for example.

[0227] For example, in Fig. 9 The temperature detection state ST1 shown in FIG. Fig.11In the temperature detection state ST3 shown, the detection value of the thermopile 305 indicates, for example, a temperature drop of 15°C (=20°C to 5°C). Therefore, since the detection value of the thermopile 305 indicates a temperature drop above the second threshold TH2, the opening and closing judgment unit 353 judges that the drawer 26A changes from a closed state to an open state. In addition, the second threshold TH2 is, for example, 10°C. In addition, since the opening and closing judgment unit 353 judges that the drawer 26A is in an open state, the shooting control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start shooting.

[0228] exist Fig.11 In the example, the case where the user P pulls out the drawer 25A of the vegetable chamber 25 is described, but the present disclosure is not limited thereto. The temperature detection state ST3 may be the case where the user P pulls out the drawer 26A. That is, the temperature detection state ST3 may be the case where the user P pulls out any one of the drawers 22A, 23A, 24A, and 25A.

[0229] exist Fig.11 In the description, the case where no food such as vegetables is stored in the drawer 25A of the vegetable compartment 25 is described, but the present disclosure is not limited thereto. It may also be the case where food such as vegetables is stored in the drawer 25A of the vegetable compartment 25. At this time, the thermopile 305 detects the temperature of a part of the upper surface of the food such as vegetables stored in the drawer 25A. The temperature of the area is, for example, 5°C.

[0230] [1-1-5. Shooting status of narrow-angle camera]

[0231] Next, refer to Figure 12-13 The imaging state of the narrow-angle camera 302 will be described.

[0232] exist Figure 12-13 In the figure, it is shown that Figure 2 The X-axis, Y-axis, and Z-axis shown are the same. The X-axis direction corresponds to the width direction of the cold storage 2 and the left-right direction of the cold storage 2. The Y-axis direction corresponds to the front-back direction of the cold storage 2. The Z-axis direction corresponds to the up-down direction of the cold storage 2.

[0233] Fig.12 FIG. 2 is a diagram showing an example of the shooting state of the narrow-angle camera 302 in Embodiment 1. Fig.12 As shown, the narrow-angle camera 302 is disposed on the upper surface of the main body 20 of the cold storage 2. In other words, as shown in FIG. Figure 5 to Figure 7 As described above, the narrow-angle camera 302 is provided at the front end portion of the bottom surface 31A of the imaging member 31 constituting the imaging device 3. Fig.12 As shown, narrow-angle camera 302 captures drawer 26A from the upper front of refrigerator 2 .

[0234] like Fig.12 As shown, the narrow-angle camera 302 captures an object within a capture range of a capture angle θ2. The capture range is a range along the front surface of the cold storage 2. The capture range extends approximately parallel to the Z axis. The capture angle θ2 is, for example, 25 degrees. The narrow-angle camera 302 captures food stored inside the drawer 26A. Drawer 26A represents drawer 22A, drawer 23A, drawer 24A, and drawer 25A. The ice making chamber 22, the fresh freezing chamber 23, the freezer chamber 24, and the vegetable chamber 25 are each provided with a drawer 22A, a drawer 23A, a drawer 24A, and a drawer 25A capable of storing food.

[0235] exist Figure 12-13 In the description, a case where drawer 26A is drawer 25A disposed in vegetable chamber 25 will be described.

[0236] like Fig.12 As shown in the dotted line, when the drawer 25A is in the closed state, the narrow-angle camera 302 is blocked by the drawer 24A, so the second mark MK2 configured in the drawer 25A cannot be photographed. In addition, the second mark MK2 is configured at the upper end of the front panel of the drawer 25A.

[0237] like Fig.12 As shown by the solid line, when the drawer 25A is pulled out to the position where the second mark MK2 is not blocked by the drawer 24A, the narrow-angle camera 302 can capture the second mark MK2.

[0238] Fig.13 FIG. 2 is a diagram showing another example of the shooting state of the narrow-angle camera 302 in the first embodiment. Fig.13 In other words, in Fig.13 In the embodiment, the drawer 25A is in the state where the opening degree of the drawer 25A is the largest. At this time, the second mark MK2 is not blocked by the drawer 24A or the like, so the narrow-angle camera 302 can shoot the second mark MK2.

[0239] In addition, if Fig.13 As shown, when the drawer 25A is at its maximum opening, the second mark MK2 is located at a specified position.

[0240] Thus, for example, drawer 25A is Fig.12 The closed state shown by the dashed line is Fig.12 The state shown by the solid line changes to Fig.13 When the aperture is at the maximum opening shown in , the video recording control unit 355 starts recording the shooting results of the wide-angle camera 301 and the narrow-angle camera 302, and the second communication control unit 352 sends a buzzer output request information to the first control substrate 311. Upon receiving the buzzer output request information, the first processor 320 causes the buzzer 306 to output a buzzer sound and informs the user P that the second marker MK2 is at a specified position.

[0241] exist Figure 12-13 In the description, drawer 26A is described as drawer 25A disposed in vegetable compartment 25, but the present disclosure is not limited thereto. Drawer 26A may be any one of drawers 22A, 23A, 24A, and 25A disposed in ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively.

[0242] [1-1-6. Markings placed in the refrigerator]

[0243] Reference Figure 14-15 The first marker MK1 arranged in the refrigerating room 21 will be described.

[0244] exist Figure 14-15 In the figure, it is shown that Figure 2 The X-axis, Y-axis, and Z-axis shown are the same. The X-axis direction corresponds to the width direction of the cold storage 2 and the left-right direction of the cold storage 2. The Y-axis direction corresponds to the front-back direction of the cold storage 2. The Z-axis direction corresponds to the up-down direction of the cold storage 2.

[0245] Fig.14 This is a diagram showing an example of the first marker MK1 arranged in the refrigerating room 21 in the first embodiment.

[0246] like Fig.14 As shown, the first mark MK1 is arranged on the hinge cover AS. Fig.14 The left figure shows the state where the hinge cover AS is not installed. Fig.14 The right figure shows the state where the hinge cover AS is installed. Fig.14 In the left figure, because the hinge cover AS is not installed, the right door hinge member 204 is exposed. The right door hinge member 204 rotatably supports the right door 21B.

[0247] The hinge cover AS is formed in an L-shape in a plan view, and has an imaging surface PF2 formed on the upper surface. A first mark MK1 including identification information indicating the refrigerating chamber 21 is formed on the imaging surface PF2. For example, a sticker having the first mark MK1 is attached to the imaging surface PF2.

[0248] Reference Fig.16 The first marker MK1 will be further described.

[0249] Fig.15 is a three-dimensional diagram of the hinged cover AS. Fig.15As shown, a notch AS1 is formed in the hinge cover AS. The notch AS1 is engaged with the upper part of the frame of the drawer 23A (for example, the right door hinge member 204). The hinge cover AS is formed of, for example, resin. In addition, the hinge cover AS may be the same color as the main box body 20 and the hinge, or may be a different color. The hinge cover AS is formed in, for example, white.

[0250] The state storage unit 364 stores the first captured image PA1 including the first marker MK1.

[0251] Based on the first captured image PA1 stored in the state storage unit 364 , the generator 356 selects the first captured image P1 when the right door 21B is at a predetermined opening / closing angle θ from the captured image PB1 included in the moving image shown in the first recorded data DV1 of the recorded data 363 .

[0252] Reference Figure 14-15 Although the case where the first marker MK1 is arranged on the right door 21B of the refrigerating room 21 has been described, the present disclosure is not limited thereto. The first marker MK1 is also arranged on the left door 21C of the refrigerating room 21, for example.

[0253] [1-1-7. Marking structure]

[0254] Next, refer to Fig.16 The structure of the marker MK will be described. Fig.16 FIG. 1 is a diagram showing an example of the structure of the marker MK in the first embodiment. Fig.16 In the description, a case where the mark MK is arranged on any one of the door 21A, the drawer 22A, the drawer 23A, the drawer 24A, and the drawer 25A is described.

[0255] like Fig.16 As shown, the mark MK includes a rectangular outer frame image MF and an identification image MD formed inside the outer frame image MF. The identification image MD corresponds to identification information respectively indicating the refrigerating chamber 21, the ice making chamber 22, the fresh freezing chamber 23, the freezing chamber 24, and the vegetable chamber 25 of the cold storage 2. The outer frame image MF is formed in black, for example.

[0256] In the following description, a case where the mark MK is arranged in the drawer 25A of the vegetable compartment 25 is described.

[0257] The identification image MD is composed of a first rectangular image PD1 of a first color and a second rectangular image PD2 of a second color different from the first color, and the arrangement pattern of the first rectangular image PD1 and the second rectangular image PD2 represents identification information. The first color is, for example, white, and the second color is, for example, black.

[0258] The size of the first rectangular image PD1 is the same as that of the second rectangular image PD2, and the first rectangular image PD1 and the second rectangular image PD2 are, for example, squares. In other words, the width WP1 of the first rectangular image PD1 and the width WP2 of the second rectangular image PD2 are the same as the height HP of the first rectangular image PD1 and the second rectangular image PD2.

[0259] The thickness HF in the short side direction (up and down direction) of the outer frame image MF and the thickness WF in the long side direction (left and right direction) of the outer frame image MF are respectively the same as, for example, the width WP1 of the first rectangular image PD1, the width WP2 of the second rectangular image PD2, the height HP of the first rectangular image PD1 and the height HP of the second rectangular image PD2.

[0260] The thickness HF in the short-side direction (up-down direction) of the outer frame image MF and the thickness WF in the long-side direction (left-right direction) of the outer frame image MF are determined based on the resolution of the narrow-angle camera 302 and the distance between the narrow-angle camera 302 and the mark MK at a predetermined pulling amount (e.g., 100%) of the drawer 25A. In addition, the thickness HF in the short-side direction (up-down direction) of the outer frame image MF and the thickness WF in the long-side direction (left-right direction) of the outer frame image MF are determined based on the shooting range or viewing angle of the narrow-angle camera 302.

[0261] For example, the greater the resolution of the narrow-angle camera 302, the smaller the thickness HF in the long-side direction of the outer frame image MF and the thickness WF in the short-side direction of the outer frame image MF can be. In addition, the longer the distance between the narrow-angle camera 302 and the mark MK at a predetermined pulling amount (e.g., 100%) of the drawer 25A, the greater the thickness HF in the long-side direction of the outer frame image MF and the thickness WF in the short-side direction of the outer frame image MF need to be. In addition, the greater the shooting range or viewing angle of the narrow-angle camera 302, the greater the thickness HF in the long-side direction of the outer frame image MF and the thickness WF in the short-side direction of the outer frame image MF need to be.

[0262] exist Fig.16 In the illustrated mark MK, the thickness HF in the long-side direction of the outer-frame image MF and the thickness WF in the short-side direction of the outer-frame image MF are each, for example, 6 mm.

[0263] The first rectangular image PD1 and the second rectangular image PD2 are arranged along the longitudinal direction (here, the left-right direction) of the outer-frame image MF. Fig.16 In FIG. 1 , five first rectangular images PD1 and four second rectangular images PD2 are arranged as identification image MD. The first rectangular image PD1 represents, for example, "0", and the second rectangular image PD2 represents, for example, "1". That is, identification image MD represents a 9-bit binary number.

[0264] The identification image MD is composed of an information image MDA and a symbol image MDB. The information image MDA is composed of five rectangular images PD, and the symbol image MDB is composed of four rectangular images PD. The rectangular image PD is the first rectangular image PD1 or the second rectangular image PD2. The information image MDA corresponds to the identification information representing the drawer 25A of the cold storage 2. The symbol image MDB represents a symbol for correcting errors in the identification information shown in the information image MDA.

[0265] The information image MDA is composed of five rectangular images PD, and thus represents a 5-bit binary number. That is, the information image MDA represents 2 to the fifth power, that is, 32 types of identification information. For example, Fig.16 The information image MDA shown indicates a 5-bit binary number "01010".

[0266] The size of the first rectangular image PD1 and the size of the second rectangular image PD2 are determined based on the resolution of the narrow-angle camera 302 and the distance between the narrow-angle camera 302 and the mark MK at a predetermined pull-out amount (e.g., 80% to 100%) of the drawer 25A. In addition, the size of the first rectangular image PD1 and the size of the second rectangular image PD2 are determined based on the shooting range or viewing angle of the narrow-angle camera 302.

[0267] For example, the larger the resolution of the narrow-angle camera 302, the smaller the size of the first rectangular image PD1 and the size of the second rectangular image PD2 can be. In addition, the longer the distance between the narrow-angle camera 302 and the mark MK at a predetermined pull-out amount (e.g., 80% to 100%) of the drawer 25A, the larger the size of the first rectangular image PD1 and the size of the second rectangular image PD2 need to be. In addition, the larger the shooting range or viewing angle of the narrow-angle camera 302, the larger the size of the first rectangular image PD1 and the size of the second rectangular image PD2 need to be.

[0268] exist Fig.16 In the illustrated mark MK, the size of the first rectangular image PD1 and the size of the second rectangular image PD2 are squares each having a side of 6 mm.

[0269] exist Fig.16 In the description, the case where the outer frame image MF is formed in black is described, but the present invention is not limited thereto. For example, the outer frame image MF may be formed in blue. In addition, the case where the first color is white and the second color is black is described, but the present invention is not limited thereto. For example, the first color may be yellow and the second color may be blue.

[0270] As reference Figure 12-13 As described above, a mark MK is disposed on the upper end of the front panel of the drawer 25A. Fig.16As shown, the mark MK includes a rectangular outer-frame image MF and an identification image MD formed inside the outer-frame image MF. Thus, the generation unit 356 executes the following processing.

[0271] That is, the generation unit 356 first determines whether the outer frame image MF can be detected from the captured image PB2 contained in the dynamic image represented by the second video data DV2 of the video data 363. Then, when the outer frame image MF can be detected from the captured image PB2, the generation unit 356 determines whether the identification information indicated by the identification image MD inside the detected outer frame image MF can be read. Next, when the identification information indicated by the identification image MD can be read and the read identification information is consistent with the identification information contained in the second mark MK2 stored in the state storage unit 364, the generation unit 356 selects the captured image PB2 as the second captured image P2.

[0272] Furthermore, in order to allow the generation unit 356 to easily detect the outer frame image MF from each of the captured images PB2 included in the moving image represented by the second video data DV2, the color of the outer frame image MF is preferably different from the color of the inner surface of the vegetable compartment 25 and the color of the food stored in the vegetable compartment 25. Therefore, the color of the outer frame image MF is preferably black, blue, or the like, for example.

[0273] [1-1-8. Structure of food management server]

[0274] Fig.17 It is a block diagram showing the configuration of the terminal device 4 and the food management server 5.

[0275] The food management server 5 includes a server processor 50 such as a CPU as a processor, a server memory 51, and a server communication module 52. The server processor 50 is connected to the server memory 51 and the server communication module 52.

[0276] The server memory 51 is a storage device (storage medium) that stores programs and data executed by the server processor 50. The server memory 51 is composed of a non-volatile storage device. In addition, the server memory 51 may also include a non-volatile storage device that constitutes a working area of ​​the server processor 50. The server memory 51 stores data processed by the server processor 50, a control program 511 executed by the server processor 50, and a food management DB (database) 512.

[0277] The food management DB 512 has one record for each imaging device 3. This one record includes the imaging device ID 362, the image data of the first image P1, and the image data of the second image P2.

[0278] Server communication module 52 includes a communication circuit, a connector, etc. in accordance with a predetermined communication standard, and communicates with refrigerator 2, imaging device 3, and terminal device 4 via network NW.

[0279] The server processor 50 functions as a server communication control unit 501 and a server processing unit 502 by reading and executing a control program 511 stored in the server memory 51 .

[0280] The server communication control unit 501 communicates with the refrigerator 2 , the imaging device 3 , and the terminal device 4 via the server communication module 52 .

[0281] The server processing unit 502 performs various information processing with the image capturing device 3 and the terminal device 4 as clients. The server processing unit 502 may also use the refrigerator 2 as a client.

[0282] [1-1-9. Configuration of terminal device]

[0283] Next, the configuration of the terminal device 4 will be described.

[0284] The terminal device 4 includes a terminal processor 40 such as a CPU as a processor, a terminal memory 41 , a terminal communication module 42 , and a touch panel 43 .

[0285] The terminal memory 41 is a storage device (storage medium) that non-volatilely stores programs and data executed by the terminal processor 40. The terminal memory 41 is composed of a non-volatile storage device. In addition, the terminal memory 41 may also include a RAM that constitutes a work area of ​​the terminal processor 40. The terminal memory 41 stores data processed by the terminal processor 40, a control program executed by the terminal processor 40, and a food management application 411.

[0286] The terminal communication module 42 includes a communication circuit, a connector, etc. in accordance with a prescribed communication standard, and communicates with the food management server 5 under the control of the terminal processor 40 .

[0287] The touch panel 43 includes a display panel and a touch sensor overlapped with or provided integrally with the display panel.

[0288] The terminal processor 40 functions as an application execution unit 400 .

[0289] The application execution unit 400 communicates with the food management server 5 via the terminal communication module 42 .

[0290] The application execution unit 400 causes the touch panel 43 to display a user interface related to the management of the food stored in the refrigerator 2. In the following description, this user interface is referred to as "application UI." The display control unit 402 of this embodiment displays images of the food stored in the refrigerator 2 and the like by causing the application UI to be displayed.

[0291] The application execution unit 400 receives various inputs from the user P via the application UI.

[0292] [1-2. Action]

[0293] Next, the operation of each part of the food management system 1 will be described.

[0294] [1-2-1. Actions related to recording]

[0295] First, the operation of the imaging device 3 related to video recording will be described.

[0296] Fig.18 It is a flowchart FA showing the operation of the imaging device 3 .

[0297] The opening and closing judgment unit 353 judges whether the door 21A or the drawer 26A is in an open state (step SA1). The door 21A includes the right door 21B and the left door 21C. The opening and closing judgment unit 353 judges whether the right door 21B is in an open state or a closed state based on the detection value of the first distance measuring sensor 303. In addition, the opening and closing judgment unit 353 judges whether the left door 21C is in an open state or a closed state based on the detection value of the second distance measuring sensor 304. In addition, the opening and closing judgment unit 353 judges whether the drawer 26A is in an open state or a closed state based on the detection value of the thermopile 305. Drawer 26A represents drawer 22A, drawer 23A, drawer 24A and drawer 25A. Drawer 22A, drawer 23A, drawer 24A and drawer 25A are respectively arranged in the ice making chamber 22, the fresh freezing chamber 23, the freezer chamber 24 and the vegetable chamber 25.

[0298] When the opening / closing determination unit 353 determines that the door 21A or the drawer 26A is not in the open state (step SA1 : NO), the second processor 350 returns the process to step SA1 .

[0299] When the opening / closing determination unit 353 determines that the door 21A or the drawer 26A is in the open state (step SA1 : Yes), the imaging control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start imaging (step SA2 ).

[0300] Step SA1 and step SA2 correspond to an example of a “photographing start step”.

[0301] Next, the video recording control unit 355 determines whether the video recording start condition is satisfied (step SA3). Figure 8 As described above, the recording start condition is that the door 21A is opened or closed at a predetermined angle θ or the drawer 26A is pulled out by a predetermined amount.

[0302] When the recording control unit 355 determines that the recording start condition is not satisfied (step SA3 : No), the determination of step SA3 is performed again.

[0303] When the recording control unit 355 determines that the recording start condition is satisfied (step SA3: Yes), the recording control unit 355 starts recording the shooting results of the wide-angle camera 301 and the narrow-angle camera 302 (step SA4). In addition, the recording control unit 355 may start recording only the shooting results of the wide-angle camera 301 when the first mark MK1 cannot be shot. In addition, the recording control unit 355 may start recording only the shooting results of the narrow-angle camera 302 when the second mark MK2 cannot be shot.

[0304] Next, the opening / closing determination unit 353 determines whether the door 21A and the drawer 26A are closed (step SA5 ). When the opening / closing determination unit 353 determines that the door 21A and the drawer 26A are not closed (step SA5 : No), the determination in step SA5 is repeated.

[0305] On the other hand, when it is determined that the door 21A and the drawer 26A are in the closed state (step SA5 : Yes), the recording control unit 355 ends the recording of the photographing result (step SA6 ). As a result, the recording data 363 is generated in the second memory 360 .

[0306] Next, the imaging control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end imaging (step SA7 ).

[0307] [1-2-2. Operations related to sending management data]

[0308] Next, the operation related to the transmission of the management data D1 will be described.

[0309] Fig.19 It is a flowchart showing the operation of the imaging device 3 and the food management server 5.

[0310] exist Fig.19 , the flowchart FB represents the operation of the imaging device 3 , and the flowchart FC represents the operation of the food management server 5 .

[0311] The generation unit 356 determines whether the recorded data 363 is stored in the second memory 360 (step SB1). When the generation unit 356 determines that the recorded data 363 is not stored (step SB1: No), the determination of step SB1 is performed again.

[0312] On the other hand, when the generation unit 356 determines that the video data 363 is stored (step SB1: yes), based on the first captured image PA1 stored in the status storage unit 364, the first captured image P1 is selected from the captured image PB1 contained in the dynamic image represented by the first video data DV1 of the video data 363 stored in the second memory 360 (step SB2).

[0313] Step SB2 will be described in detail.

[0314] The first captured image P1 is a captured image when the right door 21B or the left door 21C is opened at a predetermined opening / closing angle θ (for example, 80 degrees to 100 degrees).

[0315] The generation unit 356 first determines whether the outer frame image MF can be detected from the captured image PB1 included in the dynamic image represented by the first recorded data DV1 of the recorded data 363. Then, when the outer frame image MF cannot be detected from the captured image PB1, the generation unit 356 determines whether the identification information represented by the identification image MD inside the detected outer frame image MF can be read. Next, when the identification information represented by the identification image MD can be read, and the read identification information is consistent with the identification information represented by the first mark MK1 included in the captured image PA1 stored in the state storage unit 364, the generation unit 356 selects the captured image PB1 as the first captured image P1.

[0316] Returning to the description of flowchart FB, the generation unit 356 selects the second captured image P2 from the captured images PB2 included in the moving image represented by the second recorded data DV2 of the stored recorded data 363 based on the second captured image PA2 stored in the state storage unit 364 (step SB3).

[0317] Step SB3 will be described in detail.

[0318] The second captured image P2 is a captured image when the drawer 26A is pulled out to a predetermined amount.

[0319] The generation unit 356 first determines whether the outer frame image MF can be detected from the captured image PB2 included in the dynamic image represented by the second video data DV2 of the video data 363. Then, when the outer frame image MF can be detected from the captured image PB2, the generation unit 356 determines whether the identification information indicated by the identification image MD inside the detected outer frame image MF can be read. Next, when the identification information indicated by the identification image MD can be read and the read identification information is consistent with the identification information included in the second mark MK2 of the captured image PA2 stored in the state storage unit 364, the generation unit 356 selects the captured image PB2 as the second captured image P2.

[0320] The generation unit 356 determines whether or not at least one of the first captured image P1 and the second captured image P2 can be selected (step SB4 ).

[0321] When the generation unit 356 determines that the first captured image P1 and the second captured image P2 cannot be selected (step SB4 : No), it performs the process of step SB7 .

[0322] On the other hand, when the generating unit 356 determines that at least one of the first captured image P1 and the second captured image P2 can be selected (step SB4 : Yes), it generates the management data D1 (step SB5 ).

[0323] When the first captured image P1 and the second captured image P2 are selectable, the generating unit 356 generates the management data D1 including the image data of the first captured image P1 , the image data of the second captured image P2 , and the imaging device ID 362 .

[0324] When the first captured image P1 can be selected but the second captured image P2 cannot be selected, the generating unit 356 generates the management data D1 including the image data of the first captured image P1 and the imaging device ID 362 .

[0325] When the second captured image P2 can be selected but the first captured image P1 cannot be selected, the generating unit 356 generates the management data D1 including the image data of the second captured image P2 and the imaging device ID 362 .

[0326] Next, the first communication control unit 351 transmits the management data D1 generated by the generating unit 356 to the food management server 5 (step SB6).

[0327] Next, the generating unit 356 deletes the recorded data 363 to be processed from the second memory 360 after step SB2 (step SB7 ).

[0328] As shown in flowchart FC, the server communication control unit 501 determines whether the management data D1 is received from the camera 3 (step SC1). When the server communication control unit 501 determines that the management data D1 is received (step SC1: Yes), the server processing unit 502 updates the food management DB 512 based on the management data D1 received by the server communication control unit 501 (step SC2).

[0329] Step SC2 is described in detail.

[0330] The server processing unit 502 identifies the record including the imaging device ID 362 included in the management data D1 from the food management DB 512, and updates the image data of the first captured image P1 of the identified record to the image data of the first captured image P1 included in the management data D1.

[0331] The server processing unit 502 identifies the record including the imaging device ID 362 included in the management data D1 from the food management DB 512, and updates the image data of the second captured image P2 of the identified record to the image data of the second captured image P2 included in the management data D1.

[0332] [1-2-3. Actions related to application UI]

[0333] Next, the operation of the food management system 1 related to the application UI will be described.

[0334] Fig. 20 4 is a flowchart showing the operation of the terminal device 4 and the food management server 5. Fig. 20 Flowchart FD represents the operation of the terminal device 4, and flowchart FE represents the operation of the food management server 5.

[0335] As shown in flowchart FD, the application execution unit 400 determines whether to request image data stored in the food management DB 512 from the food management server 5 (step SD1). For example, when the food management application 411 is activated, the terminal communication control unit 401 makes an affirmative determination in step SD1.

[0336] When the application execution unit 400 determines that the image data stored in the food management DB 512 is requested (step SD1: Yes), it transmits request data for requesting the image data to the food management server 5 (step SD2). The request data includes the camera ID 362 stored in the terminal memory 41. In addition, the terminal device 4 obtains the camera ID 362 at a predetermined time.

[0337] As shown in flowchart FE, the server communication control unit 501 receives the request data (step SE1). Then, the server processing unit 502 obtains the image data corresponding to the camera ID 362 included in the request data received by the server communication control unit 501 from the food management DB 512 (step SE2). Next, the server communication control unit 501 sends the image data obtained by the server processing unit 502 to the terminal device 4 as a response to the request data (step SE3).

[0338] Referring to flowchart FD, the application execution unit 400 receives image data from the food management server 5 (step SD3). Next, the application execution unit 400 causes the touch panel 43 to display an application UI including an image indicated by the image data received in step SD3 (step SD4).

[0339] In the first embodiment, the case where the imaging device 3 and the refrigerator 2 are configured as separate components is described, but the present disclosure is not limited thereto. The imaging device 3 may be configured integrally with the refrigerator 2. In other words, the imaging device 3 may be configured as a part of the refrigerator 2.

[0340] [1-3. Effects, etc.]

[0341] Above, as reference Figures 1 to 20 As described, the shooting device 3 in embodiment 1 includes: a narrow-angle camera 302, which shoots the cold storage 2 including the vegetable chamber 25 having a drawer 25A with an opening formed on the upper surface, and is arranged on the top surface 20A of the main body 20 of the cold storage 2; a thermopile 305, which is arranged on the top surface 20A of the main body 20 of the cold storage 2 and detects the temperature of an object arranged on the front surface side of the cold storage 2; and a shooting control unit 354, which starts shooting of the vegetable chamber 25 by the narrow-angle camera 302 based on the detection result of the thermopile 305.

[0342] Thus, the narrow-angle camera 302 starts photographing the vegetable compartment 25 based on the detection result of the thermopile 305. Thus, the narrow-angle camera 302 can start photographing at an appropriate time. Therefore, the food stored in the refrigerator 2 can be managed with high accuracy.

[0343] Furthermore, the imaging device 3 includes a recording control unit 355 that starts recording of the captured image by the narrow-angle camera 302 based on the detection result of the thermopile 305 .

[0344] Thus, the recording of the captured image by the narrow-angle camera 302 is started based on the detection result of the thermopile 305. Thus, the recording of the captured image by the narrow-angle camera 302 can be started at an appropriate time. Therefore, the food stored in the refrigerator 2 can be managed with high accuracy.

[0345] Furthermore, in the imaging device 3 , when the thermopile 305 detects a temperature rise that is equal to or higher than the first threshold value TH1 set in advance, the imaging control unit 354 starts imaging of the vegetable compartment 25 by the narrow-angle camera 302 .

[0346] Therefore, as reference Fig. 9 and Fig.10 As described above, when the thermopile 305 detects a temperature rise above the preset first threshold TH1, the narrow-angle camera 302 starts photographing the vegetable compartment 25. Thus, by setting the first threshold TH1 to an appropriate value, the recording of the photographed image by the narrow-angle camera 302 can be started at an appropriate time. Therefore, the food stored in the cold storage 2 can be managed with high accuracy.

[0347] Furthermore, in the imaging device 3 , when the thermopile 305 detects a drop in temperature equal to or higher than the second threshold value TH2 set in advance, the imaging control unit 354 starts imaging by the narrow-angle camera 302 .

[0348] Therefore, as reference Fig. 9 and Fig.11 As described above, when the thermopile 305 detects a drop in temperature above the preset second threshold value TH2, the narrow-angle camera 302 starts shooting. Thus, by setting the second threshold value TH2 to an appropriate value, the recording of the captured image by the narrow-angle camera 302 can be started at an appropriate time. Therefore, the food stored in the cold storage 2 can be managed with high accuracy.

[0349] In addition, in the shooting device 3, the cold storage 2 includes a refrigerating chamber 21 which is arranged above the drawer 25A, has an opening 201 formed on the front surface and has a door 21A that opens and closes the opening 201, and includes a wide-angle camera 301 which is arranged on the top surface 20A of the main box 20 of the cold storage 2 and is different from the narrow-angle camera 302, and a first ranging sensor 303 and a second ranging sensor 304 for detecting the opening and closing angles of the door 21A. The shooting control unit 354 starts shooting of the refrigerating chamber 21 by the wide-angle camera 301 based on the detection results of the first ranging sensor 303 and the second ranging sensor 304.

[0350] Thus, the wide-angle camera 301 starts photographing the refrigerating chamber 21 based on the detection results of the first distance measuring sensor 303 and the second distance measuring sensor 304. Thus, the wide-angle camera 301 can start photographing the refrigerating chamber 21 at an appropriate time. Therefore, the food stored in the refrigerating chamber 2 can be managed with high accuracy.

[0351] In addition, the shooting method in embodiment 1 is a shooting method of a shooting device 3 including a narrow-angle camera 302 and a thermopile 305, which includes a shooting start step of starting shooting of the vegetable chamber 25 by the narrow-angle camera 302 based on the detection result of the thermopile 305, wherein the shooting narrow-angle camera 302 shoots the cold storage 2 including the vegetable chamber 25 having a drawer 25A with an opening formed on the upper surface, and is arranged on the top surface 20A of the main body 20 of the cold storage 2, and the thermopile 305 is arranged on the top surface 20A of the main body 20 of the cold storage 2, and detects the temperature of the object arranged on the front surface side of the cold storage 2.

[0352] Therefore, the imaging method of the first embodiment has the same effect as the imaging device 3 of the first embodiment.

[0353] In addition, the control program 361 of embodiment 1 is a control program 361 of the shooting device 3 including a narrow-angle camera 302, a thermopile 305 and a second processor 350, which enables the second processor 350 to function as a shooting control unit 354 that starts shooting the vegetable room 25 by the narrow-angle camera 302 based on the detection result of the thermopile 305, wherein the narrow-angle camera 302 shoots the cold storage 2 including the vegetable room 25 having a drawer 25A with an opening formed on the upper surface, and is arranged on the top surface 20A of the main body 20 of the cold storage 2, and the thermopile 305 is arranged on the top surface 20A of the main body 20 of the cold storage 2, and detects the temperature of the object arranged on the front surface side of the cold storage 2.

[0354] Thus, the control program 361 in the first embodiment achieves the same effect as that of the imaging device 3 in the first embodiment.

[0355] (Other Embodiments)

[0356] As described above, the above-mentioned embodiment 1 is described as an example disclosed in this application. However, the technology in the present invention is not limited to this, and can also be applied to embodiments that have been changed, replaced, added, omitted, etc. In addition, the various components described in the above-mentioned embodiments 1 and 2 can be combined as a new embodiment. Therefore, other embodiments are exemplified below.

[0357] In the above-mentioned embodiment 1, the case where the sensor for detecting the opening and closing of the drawer 26A is composed of the thermopile 305 is exemplified. However, the sensor for detecting the opening and closing of the drawer 26A may be a temperature sensor. That is, the opening and closing judgment unit 353 judges whether the drawer 26A is in the open state or the closed state based on the detection value of the temperature sensor. In addition, the temperature sensor may also be a radiation temperature type.

[0358] In the above-mentioned first embodiment, the case where the second marker MK2 is arranged in the vegetable compartment 25 is described, but the second marker MK2 may be arranged in the ice making compartment 22 , the fresh freezing compartment 23 , or the freezing compartment 24 .

[0359] In the first embodiment, the identification image MD in the marker MK is composed of the information image MDA and the symbol image MDB, but the identification image MD may be composed of only the information image MDA. In other words, the identification image MD may not include the symbol image MDB.

[0360] In the above-mentioned first embodiment, the case where the first rectangular image PD1 and the second rectangular image PD2 in the mark MK are squares has been described, but the first rectangular image PD1 and the second rectangular image PD2 may be rectangular.

[0361] In the first embodiment, the first image P1 and the second image P2 are used for display on the terminal device 4. However, the first image P1 and the second image P2 may be used for other purposes such as detecting food stored in the cold storage 2.

[0362] In other embodiments, the mark MK may also include text, illustrations, etc. that can identify the information of the storage room or storage housing to be set (for example, text such as vegetable compartment upper box, illustrations of vegetable compartment upper box, and fruit marks). Thus, when the mark MK is set in the cold storage 2, the user P can easily recognize the storage room or storage housing to be set. Thus, the user's setting errors can be suppressed, and the user's convenience can be improved.

[0363] In other embodiments, the generator 356 may compare the image captured by the camera with a pre-stored image of the drawer being pulled out, and select the image of the drawer being pulled out.

[0364] In other embodiments, in addition to the functional units of the second processor 350 in Embodiment 1, the second processor 350 may also function as a detection unit for detecting food. During the period from the start to the end of the recording of the wide-angle camera 301 and the narrow-angle camera 302 by the video recording control unit 355, the detection unit may also capture the image of the food entering and leaving the refrigerating chamber 21 from the dynamic image. In addition, when the food is marked, the detection unit may also detect the marked food.

[0365] In other embodiments, the first captured image P1 may be subjected to image correction and uploaded to the food management server 5. As an example of image correction, correction by viewpoint conversion can be cited.

[0366] In other embodiments, the compartment formed in the main box 20 of the cold storage 2 may also have storage rooms other than the above 5 types, or may be less. In addition, in other embodiments, the drawer-type storage room formed in the main box 20 of the cold storage 2 may also be one.

[0367] The first processor 320, the second processor 350, the server processor 50, and the terminal processor 40 may each be composed of a single processor or a plurality of processors. These processors may be hardware programmed to implement corresponding functional units. That is, these processors may be composed of, for example, ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0368] Figure 8 and Fig.17 The structures of the camera 3, the cold storage 2, the terminal device 4 and the food management server 5 shown are examples, and the specific implementation is not particularly limited. In other words, it is not necessary to install corresponding hardware in each part, and it is also possible to adopt a structure in which the functions of each part are realized by executing a program by a processor. In addition, in the above-mentioned implementation, hardware can be used for part of the functions realized by software, or software can be used to realize part of the functions realized by hardware.

[0369] Figures 18 to 20 The step units of the actions shown are divided according to the main processing content for easy understanding of the actions, and the actions are not limited by the division method and name of the processing units. It can also be further divided into more step units according to the processing content. In addition, it can also be divided in a way that one step unit contains more processing. In addition, the order of the steps can be appropriately replaced within the scope of not hindering the purpose of the present disclosure.

[0370] In addition, since the above-mentioned embodiment is intended to illustrate the technology of the present invention, various changes, substitutions, additions, omissions, etc. can be made within the scope of the claims or their equivalents.

[0371] (Note)

[0372] The following technology is disclosed through the description of the above embodiments.

[0373] (Technology 1) A photographing device, comprising: a first camera, which photographs a cold storage including a first storage room having a drawer with a first opening formed on an upper surface, and is arranged on the upper surface of a body of the cold storage; a temperature sensor, which is arranged on the upper surface of the body of the cold storage and detects the temperature of an object arranged on the front surface side of the cold storage; and a photographing control unit, which starts photographing by the first camera based on the detection result of the temperature sensor.

[0374] According to this configuration, the first camera starts photographing based on the detection result of the temperature sensor. This allows the first camera to start photographing at an appropriate time. Therefore, the food stored in the refrigerator can be managed with high accuracy.

[0375] (Technique 2) The imaging device according to Technique 1 further includes a video recording control unit that starts recording of the captured image by the first camera based on the detection result of the temperature sensor.

[0376] According to this structure, the recording of the captured image by the first camera is started based on the detection result of the temperature sensor. Therefore, the recording of the captured image by the first camera can be started at an appropriate time. Therefore, the food stored in the cold storage can be managed with high accuracy.

[0377] (Technique 3) In the imaging device according to Technique 1 or Technique 2, when the temperature sensor detects a temperature rise exceeding a preset first threshold value, the imaging control unit starts imaging by the first camera.

[0378] According to this structure, when the temperature sensor detects a rise in temperature above a preset first threshold, the first camera starts shooting. Thus, by setting the first threshold to an appropriate value, the first camera can start shooting at an appropriate time. Therefore, the food stored in the cold storage can be managed with high precision.

[0379] (Technique 4) In the imaging device according to any one of Techniques 1 to 3, when the temperature sensor detects a drop in temperature above a preset second threshold value, the imaging control unit starts imaging by the first camera.

[0380] According to this structure, when the temperature sensor detects a drop in temperature above a preset second threshold value, the first camera starts shooting. Thus, by setting the second threshold value to an appropriate value, the first camera can start shooting at an appropriate time. Therefore, the food stored in the cold storage can be managed with high precision.

[0381] (Technique 5) The imaging device according to any one of Techniques 1 to 4, wherein the imaging control unit ends imaging when a predetermined time has elapsed after imaging by the first camera starts.

[0382] According to this structure, the shooting by the first camera ends when a predetermined time has passed after the shooting starts. Thus, by setting the predetermined time to an appropriate time, the shooting can be ended at an appropriate time. Therefore, the food stored in the cold storage can be managed with high accuracy.

[0383] (Technique 6) A photographing device as described in Technique 5, wherein the photographing control unit includes an opening and closing judgment unit for detecting the opening of the door of the first storage room, and when the opening and closing judgment unit detects that the door of the first storage room is opened after the photographing by the first camera starts, the photographing control unit extends the specified time.

[0384] According to this structure, after the shooting by the first camera starts, when the opening and closing judgment unit detects that the door of the first storage room is opened, the prescribed time is extended. Thus, by extending the prescribed time by an appropriate time, the shooting can be ended at an appropriate time. Therefore, the shot image of the first storage room can be reliably generated. As a result, the food stored in the cold storage can be managed with high precision.

[0385] (Technique 7) A photographing device as described in any one of Techniques 1 to 4, wherein the cold storage includes a second storage room, which is arranged above the drawer, has a second opening formed on the front surface, and has an opening and closing body for opening and closing the second opening, the cold storage includes: a second camera different from the first camera, which is arranged on the upper surface of the box body of the cold storage; and a second detection sensor for detecting opening of the door of the second storage room, and the photographing control unit starts photographing by the second camera based on the detection result of the second detection sensor.

[0386] According to this structure, the second camera starts photographing based on the detection result of the second detection sensor. Thus, the second camera can start photographing at an appropriate time. Therefore, the food stored in the cold storage can be managed with high accuracy.

[0387] (Technique 8) A shooting method, which is a shooting method of a shooting device including a first camera and a temperature sensor, comprising a shooting start step of starting shooting by the first camera based on the detection result of the temperature sensor, wherein the first camera shoots a cold storage including a first storage room having a drawer with a first opening formed on the upper surface, and is arranged on the upper surface of the box body of the cold storage, and the temperature sensor is arranged on the upper surface of the box body of the cold storage to detect the temperature of an object arranged on the front surface side of the cold storage.

[0388] According to this structure, the same effect as the imaging device described in technique 1 can be achieved.

[0389] (Technology 9) A storage medium storing a control program for a shooting device including a first camera, a temperature sensor and a processor, causing the processor to function as a shooting control unit that starts shooting by the first camera based on a detection result of the temperature sensor, wherein the first camera shoots a cold storage including a first storage room having a drawer with a first opening formed on an upper surface, and is arranged on an upper surface of a body of the cold storage, and the temperature sensor is arranged on the upper surface of the body of the cold storage to detect the temperature of an object arranged on the front surface side of the cold storage.

[0390] According to this structure, the same effect as the imaging device described in technique 1 can be achieved.

[0391] Industrial Application Possibility

[0392] As described above, the imaging device, imaging method, and storage medium according to the present invention can be used to enable the first camera to start imaging at an appropriate time.

Claims

1. A photographing device, characterized in that: include: a first camera for photographing a refrigerator including a first storage room having a drawer having a first opening formed on an upper surface thereof, and arranged on an upper surface of a housing of the refrigerator; a temperature sensor disposed on an upper surface of a housing of the refrigerator and detecting a temperature of an object disposed on a front surface side of the refrigerator; and An imaging control unit starts imaging by the first camera based on a detection result of the temperature sensor.

2. The photographing device according to claim 1, wherein: A video recording control unit is included that starts recording the image captured by the first camera based on the detection result of the temperature sensor.

3. The photographing device according to claim 1, wherein: The imaging control unit starts imaging by the first camera when the temperature sensor detects a temperature rise that is equal to or higher than a preset first threshold value.

4. The photographing device according to claim 1, wherein: The imaging control unit starts imaging by the first camera when the temperature sensor detects a drop in temperature that is equal to or higher than a preset second threshold value.

5. The photographing device according to any one of claims 1 to 4, characterized in that: The imaging control unit ends imaging when a predetermined time has elapsed after imaging by the first camera starts.

6. The photographing device according to claim 5, characterized in that: comprising an opening / closing determination unit for detecting that the door of the first storage chamber is opened, The imaging control unit extends the predetermined time when the opening / closing determination unit detects that the door of the first storage room is opened after the imaging of the first camera starts.

7. The photographing device according to claim 1, wherein: The refrigerator includes a second storage room, which is arranged above the drawer, has a second opening formed on the front surface, and has an opening and closing body for opening and closing the second opening. The imaging device includes: a second camera, which is arranged on the upper surface of the box body of the cold storage and is different from the first camera; and a second detection sensor for detecting that the door of the second storage room is opened, The imaging control unit starts imaging by the second camera based on the detection result of the second detection sensor.

8. A shooting method of a shooting device, the shooting device comprising: a first camera for photographing a refrigerator including a first storage room having a drawer having a first opening formed on an upper surface thereof, and arranged on an upper surface of a housing of the refrigerator; and a temperature sensor disposed on the upper surface of the box body of the cold storage to detect the temperature of an object disposed on the front surface side of the cold storage, wherein the photographing method is characterized in that: The method includes a photographing start step of starting photographing by the first camera based on the detection result of the temperature sensor.

9. A storage medium storing a control program of a photographing device, the photographing device comprising: a first camera for photographing a refrigerator including a first storage room having a drawer having a first opening formed on an upper surface thereof, and arranged on an upper surface of a housing of the refrigerator; a temperature sensor disposed on the upper surface of the box body of the cold storage and detecting the temperature of an object disposed on the front surface side of the cold storage; and a processor, wherein the storage medium is characterized in that: The processor is caused to function as an imaging control unit that starts imaging by the first camera based on a detection result of the temperature sensor.

Images