Notification system, control method for notification system, and program

Abstract

The present disclosure provides a notification system capable of notifying a user of a refrigerator of information that cannot be acquired from an imaging result of an outside imaging device.　A notification system according to the present disclosure comprises: an outside imaging device that is provided outside a refrigerator and captures an image of a storage chamber of the refrigerator; a sensor that is provided inside the refrigerator and performs sensing of the storage chamber; a first acquisition unit that acquires input / output information related to input / output of food from an imaging result of the outside imaging device; a second acquisition unit that acquires first food information related to the food from a sensing result of the sensor; a food information detection unit that detects second food information related to the food on the basis of the input / output information acquired by the first acquisition unit and the first food information acquired by the second acquisition unit; and a notification unit that notifies a user of the refrigerator of the second food information detected by the food information detection unit.

Description

Notification system, notification system control method, and program

[0001] The present disclosure relates to a notification system, a control method for a notification system, and a program.

[0002] Patent Literature 1 discloses a food management system that includes a food detection device that is installed outside a refrigerator and takes images of the refrigerator's storage compartment. The system disclosed in Patent Literature 1 notifies a user of the type of food detected by the food detection device, images of the food captured by the food detection device, and the destinations of the food detected by the food detection device.

[0003] Japanese Patent Application Laid-Open No. 2022-042854

[0004] The present disclosure provides a notification system, a control method for the notification system, and a program that can notify a refrigerator user of information that cannot be obtained from the image capture results of an outside-fridge image capture device.

[0005] The notification system of the present disclosure includes an outside-fridge photography device that is provided outside the refrigerator and photographs the storage compartment of the refrigerator; a sensor that is provided inside the refrigerator and senses the storage compartment; a first acquisition unit that acquires in- and out-of-stock information related to the in- and out-of-stock of food from the photography results of the outside-fridge photography device; a second acquisition unit that acquires first food information related to the food from the sensing results of the sensor; a food information detection unit that detects second food information related to the food based on the in- and out-of-stock information acquired by the first acquisition unit and the first food information acquired by the second acquisition unit; and a notification unit that notifies a user of the refrigerator of the second food information detected by the food information detection unit.

[0006] Further, a control method for a notification system in the present disclosure is a control method for a notification system that includes an outside-fridge photography device that is provided outside a refrigerator and photographs the storage compartment of the refrigerator, and a sensor that is provided inside the refrigerator and senses the storage compartment, the control method obtaining in- and out-of-fridge information related to the in- and out-of-fridge of food from the photography results of the outside-fridge photography device, obtaining first food information related to the food from the sensing results of the sensor, detecting second food information related to the food based on the obtained in- and out-of-fridge information and the first food information, and notifying the user of the refrigerator of the detected second food information.

[0007] The program of the present disclosure also causes a processor to function as a first acquisition unit that acquires information about the entry and exit of food from the results of photography by an outside photography device installed outside the refrigerator that photographs the storage compartment of the refrigerator, a second acquisition unit that acquires first food information about the food from the results of sensing by a sensor installed inside the refrigerator that senses the storage compartment, a food information detection unit that detects second food information about the food based on the entry and exit information acquired by the first acquisition unit and the first food information acquired by the second acquisition unit, and a notification unit that notifies the user of the refrigerator of the second food information detected by the food information detection unit. This specification is intended to include all of the contents of Japanese Patent Application No. 2024-027882, filed on February 27, 2024.

[0008] The notification system, notification system control method, and program disclosed herein notify the refrigerator user of second food information that takes into account first food information including changes in the food over time in addition to the inventory information, thereby making it possible to notify the refrigerator user of information that cannot be obtained from the photographing results of the outside-fridge photography device.

[0009] FIG. 1 is a diagram showing the configuration of a notification system in embodiment 1. FIG. 2 is a plan view of a refrigerator in embodiment 1 and an outside-fridge photography device attached to the refrigerator, as viewed from the front. FIG. 3 is a plan view of a refrigerator in embodiment 1 and an outside-fridge photography device attached to the refrigerator, as viewed from the right. FIG. 4 is a plan view of the outside-fridge photography device attached to the refrigerator, as viewed from above. FIG. 5 is a diagram showing the configuration of a control system for the outside-fridge photography device in embodiment 1. FIG. 6 is a diagram showing the configuration of a control system for the refrigerator in embodiment 1. FIG. 7 is a diagram showing the configuration of a control system for a notification management server and a terminal device in embodiment 1. FIG. 8 is a diagram showing the configuration of a control system for receiving information in embodiment 1. FIG. 12 is a flowchart showing the operation of the notification management server in the first embodiment. FIG. 13 is a flowchart showing the operation of the notification management server and the terminal device in the first embodiment. FIG. 14 is a flowchart showing the operation of the terminal device in the first embodiment. FIG. 15 is a diagram showing an example of an application UI in the first embodiment.

[0010] (Knowledge Forming the Basis of the Present Disclosure) At the time the inventors conceived the present disclosure, food management systems existed that included a food detection device installed outside a refrigerator and photographing the refrigerator's storage compartment. In this type of system, information obtained from the image capture results of the image capture device outside the refrigerator has traditionally been notified to the user. This information includes, for example, the type of food stored in or removed from the refrigerator, images of the food stored in or removed from the refrigerator, and the destination of the food captured by the image capture device. However, the inventors discovered a problem with conventional technology in that the information that can be notified to the user may be limited to information obtainable from the image capture results of the image capture device outside the refrigerator. To solve this problem, the present disclosure has constituted the subject matter of the present disclosure. Therefore, the present disclosure provides a notification system, a control method for the notification system, and a program that can notify a refrigerator user of information that cannot be obtained from the image capture results of the image capture device outside the refrigerator.

[0011] Hereinafter, embodiments will be described in detail with reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed description of already well-known matters or redundant description of substantially the same configuration may be omitted. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0012] (First Embodiment) [1-1. Configuration] [1-1-1. Configuration of the notification system] Fig. 1 is a diagram showing the configuration of a notification system 1 in the first embodiment. The notification system 1 is a system that notifies information to a user P of a refrigerator 2. The information notified by the notification system 1 will be described later. The notification system 1 also functions as a management system that manages food stored in the refrigerator 2. As an example of food management, the notification system 1 of this embodiment manages images of food stored in the refrigerator 2. Note that the management of food performed by the notification system 1 is not limited to these images, and other targets such as food expiration dates, food arrival and departure dates, and food types may also be managed.

[0013] In this embodiment, the food stored in refrigerator 2 may be marked with a predetermined marker.

[0014] The notification system 1 includes a refrigerator 2. The refrigerator 2 includes a main box 20 with an open front. The main box 20 is formed with a refrigerator compartment 21, an ice making compartment 22, a fresh freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25. These storage compartments are cooled with cold air. A revolving door 21A is installed at the opening in the front of the refrigerator compartment 21. The door 21A includes a revolving right door 21B and a revolving left door 21C. The ice making compartment 22, the fresh freezing compartment 23, the freezer compartment 24, and the vegetable compartment 25 are each equipped with drawers 22A, 23A, 24A, and 25A, respectively, capable of storing food. Each of the drawers 22A to 25A has an open top when installed. Hereinafter, when drawers 22A, 23A, 24A, and 25A are collectively referred to without distinction, they will be designated by the symbol "26A" and referred to as "drawer 26A." The refrigerator compartment 21 is an example of a "storage compartment."

[0015] The refrigerator 2 is equipped with an interior camera 27. The interior camera 27 is located lower than the exterior camera 3, which will be described later, in the vertical direction of the installed refrigerator 2. In this embodiment, the interior camera 27 is located on the door 21A. The interior camera 27 captures an image of the refrigeration compartment 21 when the door 21A is closed. This embodiment illustrates a case where the interior camera 27 is located on the door 21B. The image capturing range of the interior camera 27 in this embodiment includes the opening at the front of the refrigeration compartment 21. In this embodiment, the image captured by the interior camera 27 is a still image. The image capturing result of the interior camera 27, i.e., the image captured by the interior camera 27, is associated with the date and time of capture as information. The interior camera 27 is an example of a "sensor." The image capturing result of the interior camera 27 is an example of a "sensing result."

[0016] The notification system 1 includes an outside-fridge camera 3. The outside-fridge camera 3 is a camera installed outside the refrigerator 2. The outside-fridge camera 3 is detachably installed on the top surface 20A of the main box 20 of the refrigerator 2. The outside-fridge camera 3 photographs the refrigerator 2. The photographing results of the outside-fridge camera 3, i.e., the images obtained by photographing with the outside-fridge camera 3, are linked to the date and time of the photograph as information.

[0017] The notification system 1 includes a terminal device 4. The terminal device 4 is a portable computer such as a smartphone. An application program related to the management of food stored in the refrigerator 2 is installed on the terminal device 4, and the terminal device 4 communicates with the notification management server 5 using the functions of this application program. In the following description, this application program is referred to as a "notification management app" and is designated by the reference number "421."

[0018] 1, a user P who is at home is indicated by a solid line, and a user P who has left home H is indicated by a dotted line. When the terminal device 4 is used by the user P who is at home, it communicates with the notification management server 5 via the communication device 6 or without the communication device 6. When the terminal device 4 is used by the user P who has left home H and a communication connection with the communication device 6 cannot be established, it communicates with the notification management server 5 without the communication device 6.

[0019] The communication device 6 is connected to a network NW configured by a public line network or the like, and communicates with the notification management server 5 via the network NW. The communication device 6 functions as an interface device for connecting the refrigerator 2, the exterior photography device 3, and the terminal device 4 to the network NW. The communication device 6 establishes a local network in the home H.

[0020] The notification system 1 includes a notification management server 5. The notification management server 5 is a server device that notifies information to the user P. The notification management server 5 is also a server device that manages the food stored in the refrigerator 2. The notification management server 5 is connected to a network NW. Note that in each diagram, the notification management server 5 is represented by a single block, but this does not necessarily mean that the notification management server 5 is composed of a single device.

[0021] [1-1-2. Configuration of the outside-warehouse photography device] The configuration of the outside-warehouse photography device 3 will be described with reference to Figures 2 to 4. Figures 2 to 4 illustrate the X-axis, Y-axis, and Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to one another. The Z-axis indicates the up-down direction and corresponds to the height direction of the outside-warehouse photography device 3 when installed. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction. The left-right direction corresponds to the left-right direction of the outside-warehouse photography device 3 when installed, in other words, the width direction of the outside-warehouse photography device 3. The Y-axis indicates the front-to-back direction. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the forward direction. The positive direction of the Z-axis indicates the upward direction.

[0022] Fig. 2 is a plan view of refrigerator 2 and the exterior photography device 3 attached to refrigerator 2, viewed from the front. In Fig. 2, door 21A of refrigerator 2 is in an open state. Fig. 3 is a plan view of refrigerator 2 and the exterior photography device 3 attached to refrigerator 2, viewed from the right. In Fig. 3, drawer 25A of refrigerator 2 is in an open state, i.e., pulled out. Fig. 4 is a plan view of exterior photography device 3 attached to refrigerator 2, viewed from above.

[0023] The outside-fridge photography device 3 includes a wide-angle camera 301 and a narrow-angle camera 302. The wide-angle camera 301 and the narrow-angle camera 302 are installed at the front end of the underside 31A of the photography member 31. When the outside-fridge photography device 3 is attached to the refrigerator 2 and the door 21A is closed, the wide-angle camera 301 and the narrow-angle camera 302 are located in front of the door 21A.

[0024] Wide-angle camera 301 has a wider angle than narrow-angle camera 302. When exterior photography device 3 is attached to refrigerator 2, wide-angle camera 301 photographs refrigeration compartment 21 from above and in front of refrigerator 2. For example, the photography range of wide-angle camera 301 includes range A1 shown in FIG. 2 when viewed from the front of refrigerator 2. Range A1 is a range that includes door pockets 21D of right door 21B and left door 21C and the opening of refrigeration compartment 21 when right door 21B and left door 21C are fully open. Furthermore, the photography range of wide-angle camera 301 includes range A3 shown in FIG. 3 when viewed from the right of refrigerator 2. Range A3 is a range that includes, in the front-to-back direction, the front ends of right door 21B and left door 21C in the open state to the front end of shelf 21E, and also includes the opening of refrigeration compartment 21 in the up-down direction.

[0025] Narrow-angle camera 302 has a narrower angle than wide-angle camera 301. When exterior photography device 3 is attached to refrigerator 2, narrow-angle camera 302 photographs drawer 26A from above and in the front of refrigerator 2. For example, the photography range of narrow-angle camera 302 includes range A2 shown in FIG. 2 when viewed from the front of refrigerator 2. Range A2 is a range that includes drawer 26A when viewed from the front of refrigerator 2. Also, for example, the photography range of narrow-angle camera 302 includes range A4 shown in FIG. 3 when viewed from the right of refrigerator 2. Range A4 is a range that includes drawer 26A when it is pulled out to the furthest position in the front-to-back direction.

[0026] The outside-compartment photography device 3 includes a first distance measurement sensor 303 and a second distance measurement sensor 304. The first distance measurement sensor 303 and the second distance measurement sensor 304 are installed on the main body 30. The first distance measurement sensor 303 is a distance measurement sensor for detecting the opening and closing of the right door 21B. The first distance measurement sensor 303 is, for example, an optical sensor that emits infrared rays. The first distance measurement sensor 303 detects the distance between the right door 21B and itself and outputs a detection value indicating the detected distance. The sensing target of the first distance measurement sensor 303 is a portion of the right door 21B above the upper door gasket. The second distance measurement sensor 304 is a sensor for detecting the opening and closing of the left door 21C. The second distance measurement sensor 304 is, for example, an optical sensor that emits infrared rays. The second distance measurement sensor 304 detects the distance between the left door 21C and itself and outputs a detection value indicating the detected distance. The sensing target of the second distance measurement sensor 304 is the portion of the left door 21C above the upper door packing.

[0027] The outside-compartment photography device 3 includes a thermopile 305. The thermopile 305 is a sensor for detecting whether the drawer 26A is open or closed. The thermopile 305 is installed at the front end of the underside 31A of the photography member 31. The thermopile 305 outputs a detection value indicating the detected temperature.

[0028] The outside-fridge photography device 3 may be equipped with one or more LEDs (Light Emitting Diodes) that illuminate the inside of the fridge. The outside-fridge photography device 3 may also be equipped with an LED that notifies the user of the status of the outside-fridge photography device 3.

[0029] 5 is a diagram showing the configuration of a control system for the outside-compartment photography device 3. The outside-compartment photography device 3 is equipped with an outside-compartment control device 32. The outside-compartment control device 32 is a device that controls each part of the outside-compartment photography device 3, and may be formed from a circuit board or a housing.

[0030] The outside-compartment control device 32 includes an outside-compartment processor 310 such as a CPU (Central Processing Unit), an outside-compartment memory 320, and an interface circuit to which other devices and sensors are connected. Figure 5 illustrates an example in which a wide-angle camera 301, a narrow-angle camera 302, a first distance measurement sensor 303, a second distance measurement sensor 304, a thermopile 305, and an outside-compartment wireless communication module 306 are connected to the outside-compartment control device 32. Note that devices connected to the outside-compartment control device 32 are not limited to these devices, and other devices such as LEDs may also be connected.

[0031] The outside-compartment memory 320 is a storage device that stores programs and data executed by the outside-compartment processor 310. The outside-compartment memory 320 is configured by a non-volatile storage device such as a read-only memory (ROM). The outside-compartment memory 320 includes a volatile storage device that configures a work area for the outside-compartment processor 310, such as a random access memory (RAM). The outside-compartment memory 320 stores data processed by the outside-compartment processor 310, a control program 321 executed by the outside-compartment processor 310, an image capture device ID (Identification) 322, and the like. The image capture device ID 322 is identification information that uniquely identifies the outside-compartment image capture device 3.

[0032] The outside-fridge wireless communication module 306 includes hardware such as a communication circuit and a connector that conform to a predetermined wired communication standard, and communicates with the notification management server 5 under the control of the inside-fridge control device 32 .

[0033] The outside-compartment processor 310 reads and executes the control program 311 to function as an outside-compartment communication unit 311, an outside-compartment open / close determination unit 312, an outside-compartment photography control unit 313, a recording control unit 314, and an outside-compartment generation unit 315.

[0034] The outside-fridge communication unit 311 communicates with the notification management server 5 via the inside-fridge wireless communication module 306.

[0035] The outside-compartment opening / closing determination unit 312 determines whether the right door 21B is open or closed based on the detection value output by the first distance measurement sensor 303. If the outside-compartment opening / closing determination unit 312 receives a detection value that differs from the detection value corresponding to the closed state, it determines that the right door 21B is open. On the other hand, if the outside-compartment opening / closing determination unit 312 receives a detection value corresponding to the closed state, it determines that the right door 21B is closed.

[0036] The outside-compartment open / close determination unit 312 determines whether the left door 21C is open or closed based on the detection value output by the second distance measurement sensor 304, in the same manner as the determination for the right door 21B.

[0037] The outside-compartment open / close determination unit 312 determines whether the drawer 26A is open or closed based on the detection value output by the thermopile 305. When the outside-compartment open / close determination unit 312 receives a detection value from the thermopile 305 indicating a temperature below a predetermined temperature, it determines that the drawer 26A is open. On the other hand, when the outside-compartment open / close determination unit 312 receives a detection value from the thermopile 305 indicating a temperature above the predetermined temperature, it determines that the drawer 26A is closed.

[0038] When the outside-fridge opening / closing determination unit 312 determines that the door 21A or the drawer 26A is in the open state, the outside-fridge photography control unit 313 causes the wide-angle camera 301 and the narrow-angle camera 302 to start photographing. Furthermore, when the outside-fridge opening / closing determination unit 312 determines that the door 21A and the drawer 26A are in the closed state, the outside-fridge photography control unit 313 causes the wide-angle camera 301 and the narrow-angle camera 302 to stop photographing.

[0039] When the exterior opening / closing determination unit 312 determines that the door 21A or the drawer 26A is open, the recording control unit 314 starts recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302. In other words, the recording control unit 314 starts recording the image data 323 in the in-fridge memory 320. The image data 323 includes first image data indicating the image captured by the wide-angle camera 301 and second image data indicating the image captured by the narrow-angle camera 302.

[0040] When the outside-compartment open / close determination unit 312 determines that the door 21A and the drawer 26A are closed, the recording control unit 314 ends recording of the images captured by the wide-angle camera 301 and the narrow-angle camera 302.

[0041] The outside-compartment generating unit 315 generates first upload data D1. The first upload data D1 is data to be uploaded to the notification management server 5 and is data for managing the food stored in the refrigerator 2. In this embodiment, the first upload data D1 is data for managing the food stored in the refrigerator compartment 21.

[0042] The outside-fridge generation unit 315 generates the first upload data D1 as follows: The outside-fridge generation unit 315 acquires the first recorded data from the recorded data 323 stored in the on-fridge memory 320. Next, the outside-fridge generation unit 315 selects a first captured image G1 for display and a group of captured images for acquisition from the moving images indicated by the first recorded data.

[0043] The first display image G1 is a photographed image displayed by the terminal device 4, and is a photographed image obtained by the outside-fridge photography device 3.

[0044] The outside-fridge generation unit 315 selects the first captured image G1 for display as follows. For example, the outside-fridge generation unit 315 selects a captured image that shows a predetermined marker attached to door 21A from the video images indicated by the first recorded data. This predetermined marker is attached in advance to a predetermined position on the refrigerator 2 so that it appears in the captured image when the angles of right door 21B and left door 21C are α°≦θ≦β°. Next, the outside-fridge generation unit 315 selects from the selected captured images the image that shows the least amount of user P as the first captured image G1 for display. The outside-fridge generation unit 315 recognizes user P appearing in the captured images using a method such as pattern matching, and sets the captured image with the smallest area of ​​the image of the recognized user P as the image with the least amount of user P.

[0045] The acquisition-purpose photographed image group is a group of acquisition-purpose photographed images, which are photographed images for acquiring the inventory information NJ described later.

[0046] The outside-compartment generation unit 315 selects a plurality of captured images for acquisition. The outside-compartment generation unit 315 selects the captured images for acquisition as follows. For example, the outside-compartment generation unit 315 selects a plurality of captured images showing the above-mentioned predetermined marker attached to door 21A from the video images indicated by the first video recording data. Next, the outside-compartment generation unit 315 selects a plurality of captured images showing food that has not been placed in refrigerator compartment 21 from the selected plurality of captured images. The selection of captured images showing food that has not been placed in refrigerator compartment 21 is performed based on a determination of whether or not food that is not superimposed on refrigerator 2 is shown in the captured images. Note that refrigerator 2 is identified in the captured images based on, for example, pattern matching. Furthermore, food is identified in the captured images based on feature quantities such as shape and color. Next, the outside-fridge generation unit 315 selects a photographed image showing food that has not been stored in the refrigerator compartment 21, and then selects from the selected photographed images a plurality of photographed images that show food that has not been stored in the refrigerator compartment 21 and in which the food is shown in different positions in the photographed images, and selects these selected photographed images as a group of photographed images for acquisition.

[0047] When the outside-fridge generation unit 315 selects the first display-use captured image G1 and the group of capture-use captured images, it generates first upload data D1 including image data of the selected captured images.

[0048] When the first upload data D1 is sent to the notification management server 5, the image capturing device ID 322 is added to the first upload data D1.

[0049] [1-1-3. Configuration of the Refrigerator] Next, a description will be given of the configuration of the refrigerator 2. Fig. 6 is a diagram showing the configuration of the control system of the refrigerator 2.

[0050] The refrigerator 2 includes a refrigerator control device 28, an interior photography device 27, a refrigerator wireless communication module 29, a cooling unit 201, and an opening / closing sensor group 202.

[0051] Before describing the refrigerator control device 28, the refrigerator wireless communication module 29, the cooling unit 201, and the open / close sensor group 202 will be described.

[0052] The refrigerator wireless communication module 29 has hardware such as a communication circuit and connector that conforms to a predetermined wired communication standard, and communicates with the notification management server 5 under the control of the refrigerator control device 28.

[0053] Cooling unit 201 includes mechanisms for cooling each storage compartment of refrigerator 2, such as a compressor, a condenser, a capillary tube, a cooler, a cooling fan that sends the cool air generated by the cooler to each storage compartment, and a damper that divides the cool air sent by the cooling fan. Cooling unit 201 cools each storage compartment of refrigerator 2 under the control of refrigerator control device 28.

[0054] The open / close sensor group 202 includes a first open / close sensor 202A, a second open / close sensor 202B, a third open / close sensor 202C, a fourth open / close sensor 202D, a fifth open / close sensor 202E, and a sixth open / close sensor 202F. These open / close sensors are formed of, for example, Hall ICs.

[0055] First opening / closing sensor 202A is a sensor for detecting the opening / closing state of right door 21B. Second opening / closing sensor 202B is a sensor for detecting the opening / closing state of left door 21C. Third opening / closing sensor 202C is a sensor for detecting the opening / closing state of drawer 22A. Fourth opening / closing sensor 202D is a sensor for detecting the opening / closing state of drawer 23A. Fifth opening / closing sensor 202E is a sensor for detecting the opening / closing state of drawer 24A. Sixth opening / closing sensor 202F is a sensor for detecting the opening / closing state of drawer 25A. Each opening / closing sensor outputs a detection value to refrigerator control device 28.

[0056] Refrigerator control device 28 is a device that controls each part of refrigerator 2, and may be configured on a board or a housing. Refrigerator control device 28 includes refrigerator processor 210 such as a CPU, refrigerator memory 220, and an interface circuit to which other devices and sensors are connected.

[0057] Refrigerator memory 220 is a storage device that stores programs and data executed by refrigerator processor 210. Refrigerator memory 220 is configured with a non-volatile storage device such as ROM. Refrigerator memory 220 includes a volatile storage device that configures a work area for refrigerator processor 210, such as RAM. Refrigerator memory 220 stores data processed by refrigerator processor 210, control program 221 executed by refrigerator processor 210, refrigerator ID (Identification) 222, and the like. Refrigerator ID 222 is identification information that uniquely identifies refrigerator 2.

[0058] The refrigerator processor 210 reads and executes the control program 221 to function as a refrigerator communication unit 211, a refrigerator open / close determination unit 212, a refrigerator photography control unit 213, and a refrigerator generation unit 214.

[0059] The refrigerator communication unit 211 communicates with the notification management server 5 via the refrigerator wireless communication module 29.

[0060] Refrigerator open / close determination unit 212 determines whether right door 21B is open or closed based on the detection value output by first open / close sensor 202A. When refrigerator open / close determination unit 212 receives a detection value that differs from the detection value corresponding to the closed state, it determines that right door 21B is open. On the other hand, when refrigerator open / close determination unit 212 receives a detection value corresponding to the closed state, it determines that right door 21B is closed.

[0061] Similar to the determination for the right door 21B, the refrigerator open / close determination unit 212 determines whether the left door 21C is open or closed based on the detection value output by the second open / close sensor 202B.

[0062] Refrigerator open / close determination unit 212 determines whether each of drawers 22A to 25A is open or closed based on the detection values ​​output by each of third open / close sensor 202C to sixth open / close sensor 202F.

[0063] After refrigerator open / close determination unit 212 determines that door 21A has entered the open state, refrigerator photography control unit 213 causes interior photography device 27 to take an image if refrigerator open / close determination unit 212 determines that door 21A has entered the closed state. Interior photography device 27 may not be operated simultaneously with exterior photography device 3, but may be activated after a predetermined time has passed after it has confirmed that exterior photography device 3 has finished photographing. Furthermore, interior photography device 27 may be activated at regular intervals or at specified times if it is determined that door 21A has entered the closed state, and data may be periodically acquired and saved.

[0064] Refrigerator generation unit 214 generates second upload data D2. When refrigerator photography control unit 213 captures an image, refrigerator generation unit 214 generates second upload data D2. Second upload data D2 includes second captured image for display G2. Second captured image for display G2 is a captured image displayed on terminal device 4 and is an image captured by fridge interior photography device 27. Refrigerator generation unit 214 includes the image captured by fridge interior photography device 27 as second captured image for display G2 in second upload data D2.

[0065] In addition, when the second upload data D2 is sent to the notification management server 5, the refrigerator ID 222 is added.

[0066] 7 is a diagram showing the configuration of the notification management server 5 and the control system of the terminal device 4. The notification management server 5 includes a server processor 51, which is a processor such as a CPU, a server memory 52, and a server communication module 53. The server processor 51 is connected to the server memory 52 and the server communication module 53. The server processor 51 is an example of a "processor."

[0067] The server memory 52 is a storage device that stores programs and data executed by the server processor 51. The server memory 52 is configured with a non-volatile storage device. The server memory 52 also includes a non-volatile storage device that forms a work area for the server processor 51. The server memory 52 stores data processed by the server processor 51, a control program 521 executed by the server processor 51, a first database (DB) 522, a second DB 523, and a third DB 524. The control program 521 is an example of a "program."

[0068] The first DB 522 is a database having a first record R1. Each first record R1 has an image capturing device ID 322, a refrigerator ID 222, a first captured image for display G1, a second captured image for display G2, a group of captured images for acquisition, and a second group of captured images for display. The second group of captured images for display is a group of the second captured images for display G2.

[0069] The second DB 523 is a database that has second records R2. Each second record R2 has a user ID 422, a refrigerator ID 222, and communication information. The user ID 422 is identification information that uniquely identifies a user P. The communication information is information for communicating with the terminal device 4 of the user P identified by the user ID, such as address information.

[0070] The third DB 524 is a database that includes third records R3. Each third record R3 includes a refrigerator ID 222, inventory information NJ, and first food information FJ1. Details of the first food information FJ1 will be described later. The third record R3 may be configured to be deleted from the third DB 524 after a predetermined period of time has elapsed since it was stored in the third DB 524, or may be configured to be deleted from the third DB 524 when it becomes the target of a process for acquiring second food information, which will be described later.

[0071] The server communication module 53 includes a communication circuit and connectors that comply with a predetermined communication standard, and communicates with the refrigerator 2, the exterior photography device 3, and the terminal device 4 via the network NW.

[0072] The server processor 51 reads and executes a control program 521 stored in the server memory 52, thereby functioning as a server communication unit 511, an update unit 512, a first acquisition unit 513, a second acquisition unit 514, a management unit 515, and a third acquisition unit 516. The server communication unit 511 is an example of a "notification control unit."

[0073] The server communication unit 511 communicates with the refrigerator 2, the exterior photography device 3, and the terminal device 4 via the server communication module 53.

[0074] The update unit 512 updates the contents of the first record R1 stored in the first DB 522. When the server communication unit 511 receives the first upload data D1 from the outside-compartment photography device 3, the update unit 512 identifies the first record R1 of the photography device ID 322 added to the first upload data D1 from the first DB 522. Next, the update unit 512 divides the first display captured image G1 included in the received first upload data D1 into a left door image LDG showing an image of the left door 21C, a refrigerator compartment image RFG showing an image of the refrigerator compartment 21, and a right door image RDG showing an image of the right door 21B. This division is performed based on pattern matching, color difference, or the like. Next, the update unit 512 updates the first display captured image G1 of the identified first record R1 to a first display captured image G1 divided into three images. Next, the update unit 512 updates the acquisition-purpose captured image group included in the identified first record R1 to the acquisition-purpose captured image group included in the received first upload data D1.

[0075] When the server communication unit 511 receives the second upload data D2 from the refrigerator 2, the update unit 512 identifies the second record R2 of the refrigerator ID 222 added to the second upload data D2 in the first DB 522. Next, the update unit 512 updates the second captured image for display G2 of the identified second record R2 to the second captured image for display G2 included in the received second upload data D2. Next, the update unit 512 includes the second captured image for display G2 before the update and the second captured image for display G2 after the update in a group of multiple second captured images for display included in the identified second record R2.

[0076] The first acquisition unit 513 acquires inventory information NJ. The first acquisition unit 513 acquires inventory information NJ based on the group of captured images for acquisition. The inventory information NJ is information related to the inventory of food. The inventory information NJ has a type item, a destination item, and an arrival date item, and information is recorded in each item. The type item is an item in which the type of food is recorded. The destination item is an item in which the destination of the food is recorded. The arrival date item is an item in which the arrival date is recorded. In other words, the inventory information NJ in this embodiment is information indicating the type of food, the destination of the food, and the arrival date.

[0077] The first acquisition unit 513 will be described in detail. The first acquisition unit 513 identifies a first record R1 from the first DB 522 and reads out a group of capture images from the identified first record R1. Next, the first acquisition unit 513 detects food items being stored in or removed from the refrigerator compartment 21 from the read out group of capture images. The first acquisition unit 513 detects the food items based on feature quantities such as shape and color. Next, the first acquisition unit 513 determines whether the detected food items are food items that have been stored in or removed from the refrigerator compartment 21. This determination is made based on the movement direction of the detected food items in the read out plurality of capture images. That is, if the movement direction of the detected food items is a direction toward the refrigerator 2, the first acquisition unit 513 determines that the detected food items are food items that have been stored in the refrigerator compartment 21. Furthermore, if the detected direction of movement of the food is a direction away from the refrigerator 2 , the first acquisition unit 513 determines that the detected food is food that has been removed from the refrigeration compartment 21 .

[0078] If the first acquisition unit 513 determines that the detected food is a released food, it terminates the process of acquiring the inventory information NJ. On the other hand, if the first acquisition unit 513 determines that the detected food is a received food, it identifies the type of the detected food. The first acquisition unit 513 identifies the type of the detected food based on features such as shape and color. Note that the first acquisition unit 513 may also identify the type of the detected food using AI (artificial intelligence).

[0079] Next, the first acquisition unit 513 identifies the storage destination of the detected food. The first acquisition unit 513 performs image analysis on the retrieved plurality of capture images to identify the storage destination of the detected food. For example, the storage destination is identified based on the type of food, the size of the food shown in the capture images, the type of shelf 21E shown in the capture images on which the food is superimposed when being stored, the movement direction of the detected food in the capture images, etc.

[0080] Next, the first acquisition unit 513 identifies the photographing date and time associated with the most recent capture-purpose photographed image among the plurality of capture-purpose photographed images as the storage date.

[0081] In this way, the first acquisition unit 513 acquires the inventory information NJ by identifying the type of detected food, the storage destination of the detected food, and the storage date from the acquisition captured image.

[0082] The second acquisition unit 514 acquires the first food information FJ1. The second acquisition unit 514 acquires the first food information FJ1 based on the second group of captured images for display. The first food information FJ1 is information related to food stored in the refrigerator 2. The first food information has a type item, a storage destination item, and a change-over-time item, and information is recorded in each item. The change-over-time item is an item in which the change in the food over time is recorded, and in this embodiment, is an item in which the change in the color of the food over time is recorded. In other words, the first food information FJ1 in this embodiment is information indicating the type of food, the storage destination of the food, and the change in the color of the food over time.

[0083] The second acquisition unit 514 will be described in detail. The second acquisition unit 514 reads out a group of second display images from the first record R1 identified by the first acquisition unit 513 when acquiring the inventory information NJ. Next, the second acquisition unit 514 detects food from the read group of second display images. The second acquisition unit 514 detects food in the same manner as the first acquisition unit 513. Note that the second acquisition unit 514 detects at least food that appears in both the second display image with the most recent shooting date and time and the second display image with the second most recent shooting date and time from the group of second display images.

[0084] Next, for each food item detected from the second group of captured images for display, the second acquisition unit 514 identifies the type of food item, the storage location of the food item, and the change in color of the food item over time.

[0085] The second acquisition unit 514 identifies the type of food in the same manner as the first acquisition unit 513 .

[0086] The second acquisition unit 514 identifies the storage destination of the food based on the position of the food in the second display image G2 with the latest image capture date and time.

[0087] The second acquisition unit 514 determines the change in color of the food over time based on the RGB values ​​of the food in the second display captured image G2. For example, if the detected food changes from "green to yellow" in the second display captured image group, the second acquisition unit 514 determines the change in color of the food over time as "green to yellow."

[0088] After performing the above three specifications for each detected food item, the second acquisition unit 514 acquires first food information FJ1 for each detected food item from the second group of captured images for display.

[0089] The management unit 515 associates and manages the inventory information NJ acquired by the first acquisition unit 513 and the first food information FJ1 acquired by the second acquisition unit 514. As part of its management, the management unit 515 generates a third record R3 including the associated inventory information NJ and first food information FJ1, and stores the generated third record R3 in the third DB 524.

[0090] The management unit 515 associates the inventory information NJ and the first food information FJ1, assuming that the inventory information NJ and the first food information FJ1 are in a corresponding relationship when the degree of similarity between the contents of the inventory information NJ and the contents of the first food information FJ1 is greater than or equal to a predetermined value.

[0091] 8 is a diagram showing an example of the correspondence between inventory information NJ and first food information FJ1. FIG. 8 illustrates an example in which inventory information NJ1 is acquired by the first acquisition unit 513. In inventory information NJ1, information "AAA" is recorded in the type field, information "top left" is recorded in the destination field, and information "yellow" is recorded in the color field.

[0092] FIG. 8 illustrates an example in which the second acquisition unit 514 acquires first food information FJ11, FJ12, and FJ13. The first food information FJ11 has information "AAA" recorded in the type field, information "top right" recorded in the destination field, and information "yellow → red" recorded in the time-varying change field. The first food information FJ12 has information "AAA" recorded in the type field, information "top left" recorded in the destination field, and information "yellow → red" recorded in the time-varying change field. The first food information FJ13 has information "BBB" recorded in the type field, information "top right" recorded in the destination field, and information "yellow → red" recorded in the time-varying change field.

[0093] The inventory information NJ and the first food information FJ1 share several common fields. That is, the inventory information NJ and the first food information FJ1 share a type field and a destination field. The management unit 515 compares the information recorded for each common field and determines the corresponding item based on the number of items for which the recorded information matches. In this embodiment, if there are two common fields for which the recorded information matches, the management unit 515 associates the inventory information NJ with the first food information FJ1. That is, since the inventory information NJ and the first food information FJ1 share three fields, the management unit 515 associates the inventory information NJ with the first food information FJ1 if the matching rate between the contents of the inventory information NJ and the contents of the first food information FJ1 is 66.6% or higher.

[0094] The inventory information NJ1 and the first food information FJ11 have the same information recorded in the type field, but the information recorded in the destination field is different. Therefore, the management unit 515 does not associate these two.

[0095] The information recorded in the type field and the information recorded in the destination field are the same between the inventory information NJ1 and the first food information FJ12. Therefore, the management unit 515 associates these two.

[0096] The inventory information NJ1 and the first food information FJ13 have the same information recorded in the inventory destination field, but different information recorded in the type field. Therefore, the management unit 515 does not associate these two.

[0097] After associating the inventory information NJ with the first food information FJ1, the management unit 515 generates a third record R3 that includes the refrigerator ID 222 of the second record R2 referenced by the first acquisition unit 513 and the second acquisition unit 514 when acquiring the inventory information NJ and the first food information FJ1. The management unit 515 then stores the generated third record R3 in the third DB 524.

[0098] The third acquisition unit 516 acquires second food information. The second food information is information related to food and is different from the first food information FJ11. In this embodiment, the second food information indicates at least one of the food's best-before date, the food's expiry date, the best time to eat the food, and the food's capacity.

[0099] For example, the third acquisition unit 516 acquires second food information indicating the best-before date or expiration date of a food product as follows: The third acquisition unit 516 reads the third record R3 from the third DB 524, and then reads the inventory information NJ and the first food information FJ1 from the read third record R3. The third acquisition unit 516 then identifies the change in color of the food product over time recorded in the first food information FJ1. Based on the identified change in color of the food product over time, the third acquisition unit 516 then shortens or maintains the best-before date or expiration date determined from the date of entry in the inventory information NJ, as appropriate, to determine the current best-before date or expiration date of the food product. In this way, the third acquisition unit 516 acquires second food information indicating the best-before date or expiration date of the food product.

[0100] The third acquisition unit 516 may acquire the best-before date or expiration date determined from the date of receipt of the inventory information NJ by referencing a table that defines the type of food, the date of receipt of the food, and the expiration date of the food. In this case, the table may be stored in the server memory 52 or may be stored in an external server for reference. The third acquisition unit 516 may also acquire the best-before date or expiration date determined from the date of receipt of the inventory information NJ using AI that outputs the best-before date or expiration date of the food in response to the input of the type of food and the date of receipt of the food. The third acquisition unit 516 may also acquire the shortened period of the best-before date or expiration date corresponding to the change in color of the food over time by referencing a table that defines the type of food, the color change of the food, and the shortened period. In this case, the table may be stored in the server memory 52 or may be stored in an external server for reference. In addition, the third acquisition unit 516 may use AI that outputs the shortened period of the expiration date or best-before date of a food product based on the change in color of the food product over time, and acquire the shortened period of the expiration date or best-before date of the food product according to the change in color of the food product over time.

[0101] For example, the third acquisition unit 516 acquires second food information indicating the ripeness of a food item as follows: The third acquisition unit 516 reads the third record R3 from the third DB 524, and then reads the inventory information NJ and the first food information FJ1 from the read third record R3. The third acquisition unit 516 then identifies the change in color of the food over time recorded in the first food information FJ1. Based on the identified change in color of the food over time, the third acquisition unit 516 then appropriately corrects the best-to-eat date of the food, which is determined from the date of arrival in the inventory information NJ, by extending, shortening, or leaving it unchanged, to determine the best-to-eat date of the food item at the current time. In this way, the third acquisition unit 516 acquires second food information indicating the ripeness of the food item.

[0102] The third acquisition unit 516 may acquire the best-to-eat date determined from the arrival date of the inventory information NJ by referencing a table that defines the type of food, the arrival date of the food, and the best-to-eat date of the food. In this case, the table may be stored in the server memory 52 or may be stored in an external server that can be referenced. The third acquisition unit 516 may acquire the best-to-eat date determined from the arrival date and time of the inventory information NJ using AI that outputs the best-to-eat date of the food based on the input of the type of food and the arrival date of the food. The third acquisition unit 516 may acquire the best-to-eat extension or shortening period corresponding to the change in color of the food over time by referencing a table that defines the type of food, the color change of the food, and the extension or shortening period. In this case, the table may be stored in the server memory 52 or may be stored in an external server that can be referenced. In addition, the third acquisition unit 516 may use AI that outputs an extension or shortening period for the best time to eat food based on the change in food color over time, and acquire the extension or shortening period for the best time to eat food based on the change in food color over time.

[0103] For example, the third acquisition unit 516 acquires second food information indicating the food storage capacity as follows: The third acquisition unit 516 reads third record R3 from the third DB 524, and then reads inventory information NJ and first food information FJ1 from the read third record R3. Next, the third acquisition unit 516 identifies the type of food recorded in the read inventory information NJ. Next, the third acquisition unit 516 identifies third record R3 from the third DB 524 that includes the refrigerator ID 222 of the read third record R3 and the first food information FJ1 in which the identified food type is recorded. Next, the third acquisition unit 516 determines the number of identified third records R3 as the food storage capacity. In this way, the third acquisition unit 516 acquires second food information indicating the food storage capacity.

[0104] [1-1-5. Configuration of Terminal Device] Next, we will explain the configuration of the terminal device 4. The terminal device 4 includes a terminal processor 41, which is a processor such as a CPU, a terminal memory 42, a terminal communication module 43, and a touch panel 44.

[0105] The terminal memory 42 is a storage device that non-volatilely stores programs and data executed by the terminal processor 41. The terminal memory 42 is configured from a non-volatile storage device. The terminal memory 42 also includes RAM that forms the work area of ​​the terminal processor 41. The terminal memory 42 stores data processed by the terminal processor 41, control programs executed by the terminal processor 41, a notification management application 421, and a user ID 422.

[0106] The terminal communication module 43 includes a communication circuit and a connector that comply with a predetermined communication standard, and communicates with the notification management server 5 under the control of the terminal processor 41. The touch panel 44 includes a display panel and a touch sensor that is superimposed on or integrated with the display panel.

[0107] The terminal processor 41 functions as an application execution unit 410. The application execution unit 410 communicates with the notification management server 5 via the terminal communication module 43. The application execution unit 410 displays a user interface related to the management of food stored in the refrigerator 2 on the touch panel 44. In the following description, this user interface is referred to as an "application UI" and is denoted by the reference numeral "441." In this embodiment, the application execution unit 410 displays images of food stored in the refrigerator 2 by displaying the application UI 441. The application execution unit 410 receives various inputs from the user P via the application UI 441. The application execution unit 410 is an example of a "notification unit" and a "display unit."

[0108] [1-2. Operation] Next, the operation of each part of the notification system 1 according to this embodiment will be described.

[0109] 9 is a flowchart FA showing the operation of the outside-fridge photography device 3.

[0110] The outside-compartment open / close determination unit 312 determines whether the door 21A or the drawer 26A is open (step SA1).

[0111] If the outside-compartment open / close determination unit 312 determines that the door 21A or the drawer 26A is not open (step SA1: NO), the determination in step SA1 is performed again.

[0112] If the outside-fridge open / close determination unit 312 determines that the door 21A or the drawer 26A is open (step SA1: YES), the outside-fridge photography control unit 313 causes the wide-angle camera 301 and the narrow-angle camera 302 to start photographing (step SA2).

[0113] Next, the recording control unit 314 starts recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302 (step SA3).

[0114] Next, the outside-compartment opening / closing determination unit 312 determines whether the door 21A and the drawer 26A are closed (step SA4). If the outside-compartment opening / closing determination unit 312 determines that the door 21A and the drawer 26A are not closed (step SA4: NO), it performs the determination in step SA4 again.

[0115] On the other hand, if it is determined that the door 21A and the drawer 26A are closed (step SA4: YES), the recording control unit 314 ends recording of the photographed image (step SA5). As a result, new recording data 323 is generated in the external memory 320.

[0116] Next, the outside-fridge photography control unit 313 causes the wide-angle camera 301 and the narrow-angle camera 302 to end photography (step SA6).

[0117] [1-2-2. Operations Related to First Upload Data] Next, operations related to the first upload data D1 will be described. Figure 10 is a flowchart showing the operations of the outside-warehouse photography device 3 and the notification management server 5. In Figure 10, flowchart FB shows the operations of the outside-warehouse photography device 3, and flowchart FC shows the operations of the notification management server 5.

[0118] The outside-compartment generation unit 315 determines whether or not new video recording data 323 has been stored in the outside-compartment memory 320 (step SB1). If the outside-compartment generation unit 315 determines that new video recording data 323 has not been stored (step SB1: NO), it performs the determination in step SB1 again.

[0119] On the other hand, if it is determined that the video recording data 323 has been stored (step SB1: YES), the outside-fridge generating unit 315 generates the first upload data D1 (step SB2).

[0120] Next, the outside-fridge communication unit 311 transmits the first upload data D1 generated by the outside-fridge generation unit 315 to the notification management server 5 (step SB3).

[0121] Next, the recording control unit 314 erases the recording data 323 stored in the external memory 320 from the external memory 320 (step SB4).

[0122] As shown in flowchart FC, server communication unit 511 determines whether or not first upload data D1 has been received from outside-warehouse photography device 3 (step SC1). If server communication unit 511 determines that first upload data D1 has been received (step SC1: YES), update unit 512 updates first DB 522 based on the first upload data D1 received by server communication unit 511 (step SC2).

[0123] [1-2-3. Operations Related to Second Upload Data] Next, operations related to the second upload data D2 will be described. Figure 11 is a flowchart showing the operations of the refrigerator 2 and the notification management server 5. In Figure 11, flowchart FD shows the operations of the refrigerator 2, and flowchart FE shows the operations of the notification management server 5.

[0124] Refrigerator generation unit 214 determines whether fridge interior photography device 27 has taken an image (step SD1). If refrigerator generation unit 214 determines that fridge interior photography device 27 has not taken an image (step SD1: NO), refrigerator generation unit 214 performs the determination in step SD1 again.

[0125] On the other hand, if refrigerator generation unit 214 determines that interior photography device 27 has taken an image (step SD1: YES), refrigerator generation unit 214 generates second upload data D2 (step SD2).

[0126] Next, the refrigerator communication unit 211 transmits the second upload data D2 generated by the refrigerator generation unit 214 to the notification management server 5 (step SD3).

[0127] As shown in flowchart FE, server communication unit 511 determines whether second upload data D2 has been received from refrigerator 2 (step SE1). If server communication unit 511 determines that second upload data D2 has been received (step SE1: YES), update unit 512 updates first DB 522 based on the second upload data D2 received by server communication unit 511 (step SE2).

[0128] [1-2-4. Operations Related to Corresponding Inventory Information and First Food Product Information] Next, operations related to correlating inventory information NJ and first food product information FJ1 will be described. Figure 12 is a flowchart FF showing the operations of the notification management server 5. The operations shown in Figure 12 are performed for each first record R1 stored in the first DB 522.

[0129] The management unit 515 determines whether a trigger related to the association between the inventory information NJ and the first food information FJ1 has occurred (step SF1). An example of such a trigger is when the contents of the first record R1 to be processed are updated.

[0130] If the management unit 515 determines that a trigger related to the association of the inventory information NJ and the first food information FJ1 has not occurred (step SF1: NO), it performs the determination of step SF1 again.

[0131] On the other hand, if it is determined that a trigger related to the correspondence between the inventory information NJ and the first food information FJ1 has occurred (step SF1: YES), the first acquisition unit 513 acquires the inventory information NJ based on the group of acquisition-purpose photographed images recorded in the first record R1 to be processed (step SF2).

[0132] Next, the second acquisition unit 514 acquires the first food information FJ1 based on the second group of captured images for display recorded in the first record R1 to be processed (step SF3). In step SF3, the first food information FJ1 is acquired for each food detected from the second group of captured images for display.

[0133] Next, the management unit 515 associates the inventory information NJ acquired in step SF2 with the first food information FJ1 acquired in step SF3 (step SF4).

[0134] Next, the management unit 515 generates a third record R3 having the inventory information NJ and the first food information FJ1 associated in step SF4, and stores the generated third record R3 in the third DB 524 (step SF5).

[0135] [1-2-5. Operations Related to Notification of Second Food Information] Next, operations related to notification of second food information will be described. Fig. 13 is a flowchart showing operations related to notification of second food information. In Fig. 13, flowchart FG shows operations of the terminal device 4, and flowchart FH shows operations of the notification management server 5.

[0136] When the notification management application 421 installed on the terminal device 4 starts up, the application execution unit 410 transmits start-up information indicating that the notification management application 421 has started up to the notification management server 5 (step SG1). The start-up information includes the user ID 422 stored in the terminal memory 42.

[0137] As shown in flowchart FH, the server communication unit 511 receives start-up information from the terminal device 4 (step SH1).

[0138] Next, third acquisition unit 516 identifies refrigerator ID 222 corresponding to user ID 422 added to the start-up information (step SH2). More specifically, third acquisition unit 516 identifies second record R2 including user ID 422 added to the start-up information from second DB 523. Third acquisition unit 516 then identifies refrigerator ID 222 in the identified second record R2 as refrigerator ID 222 corresponding to user ID 422 added to the start-up information.

[0139] Next, the third obtaining unit 516 obtains second food information by processing the third record R3 including the refrigerator ID 222 identified in step SH2 (step SH3).

[0140] Next, the server communication unit 511 transmits the second food information acquired in step SH3 to the terminal device 4 (step SH4). To describe step SH4 in more detail, the server communication unit 511 identifies, from the second DB 523, communication information corresponding to the user ID 422 added to the startup information received in step SH1, and transmits the second food information based on the identified communication information.

[0141] As shown in flowchart FG, the application execution unit 410 receives the second food information from the notification management server 5 (step SG2), and displays the received second food information on the application UI 441 (step SG3).

[0142] 13 is a notification that is sent when the notification management app 421 is launched. However, the timing of the notification of the second food information is not limited to when the notification management app 421 is launched, and may be a predetermined timing when the third acquisition unit 516 acquires the second food information. The predetermined timing may be, for example, when a new third record R3 is stored in the third DB 524 or when a predetermined period has elapsed since the previous notification.

[0143] [1-2-6. Operations Related to Display of Captured Images] Next, operations of the notification system 1 related to display of captured images will be described. Fig. 14 is a flowchart showing the operations of the terminal device 4 and the notification management server 5. In Fig. 14, flowchart FI shows the operations of the terminal device 4, and flowchart FJ shows the operations of the notification management server 5.

[0144] As shown in flowchart FI, the application execution unit 410 determines whether or not to request a captured image (step SI1). For example, the application execution unit 410 makes a positive determination in step SI1 when the notification management application 421 is launched or when an instruction to start displaying a captured image is issued in the application UI 441.

[0145] If the application execution unit 410 determines that the captured image is to be requested (step SI1: YES), the application execution unit 410 transmits request information requesting the captured image to the notification management server 5 (step SI2). The request information includes the user ID 422 stored in the device memory 42.

[0146] As shown in flowchart FJ, server communication unit 511 receives request information (step SJ1). Next, server communication unit 511 identifies refrigerator ID 222 corresponding to user ID 422 included in the received request information (step SJ2).

[0147] Next, the server communication unit 511 acquires the first captured image for display and the second captured image for display corresponding to the identified refrigerator ID 222 from the first DB 522 (step SJ3).

[0148] Next, the server communication unit 511 transmits the image data of the acquired first captured image for display and second captured image for display to the terminal device 4 (step SJ4).

[0149] As shown in flowchart FI, the application execution unit 410 receives image data from the notification management server 5 (step SI3). Next, the application execution unit 410 displays the image indicated by the image data received in step SI3 on ​​the application UI 441 (step SI4).

[0150] 15 is a diagram showing an example of an application UI 441 that displays a captured image. In FIG. 15, the upward direction in the figure is indicated by the symbol UP, the downward direction in the figure is indicated by the symbol DW, the left direction in the figure is indicated by the symbol L, and the right direction in the figure is indicated by the symbol R.

[0151] The application UI 441 that displays the captured image has a display area HA. ​​The display area HA has a first display area HA1, a second display area HA2, and a third display area HA3.

[0152] The first display area HA1 is an area for displaying the left door image LDG, the second display area HA2 is an area for displaying the refrigerator compartment image RFG or the second display captured image G2, and the third display area HA3 is an area for displaying the right door image RDG.

[0153] The app UI 441 identifies image data of the left door image LDG from the image data received in step SI3 and displays the left door image LDG in the first display area HA1. The app UI 441 also identifies image data of the refrigerator compartment image RFG or the second captured image for display G2 from the image data received in step SI3 and displays the refrigerator compartment image RFG or the second captured image for display G2 in the second display area HA2. The app UI 441 identifies image data of the right door image RDG from the image data received in step SI3 and displays the right door image RDG in the third display area HA3.

[0154] The application executing unit 410 accepts a swipe operation in the up or down direction in the second display area HA2. A swipe operation is an operation of moving an operator, such as a finger, while in contact with the touch panel 44. When a downward swipe operation is accepted in the second display area HA1 while the second display area HA1 is displaying the refrigerator compartment image RFG, the application executing unit 410 switches the image displayed in the second display area HA1 from the refrigerator compartment image RFG to the second display image. Furthermore, when an upward swipe operation is accepted in the second display area HA1 while the second display area HA1 is displaying the refrigerator compartment image RFG, the application executing unit 410 switches the image displayed in the second display area HA1 from the second display captured image to the refrigerator compartment image RFG.

[0155] [1-3. Effects, etc.] As described above, notification system 1 includes exterior photography device 3 that is installed outside refrigerator 2 and photographs the storage compartment of refrigerator 2. Notification system 1 also includes interior photography device 27 that is installed inside refrigerator 2 and photographs the storage compartment of refrigerator 2. Notification system 1 also includes first acquisition unit 513 that acquires inventory information NJ related to the storage of food from the image capture results of exterior photography device 3. Notification system 1 also includes second acquisition unit 514 that acquires first food information FJ1 related to food and including changes in the food over time from the image capture results of interior photography device 27. Notification system 1 also includes third acquisition unit 516 that acquires second food information related to food based on the inventory information NJ acquired by first acquisition unit 513 and the first food information FJ1 acquired by second acquisition unit 514. The notification system 1 also includes an application execution unit 410 that notifies the user P of the refrigerator 2 of the second food information acquired by the third acquisition unit 516.

[0156] According to this, the inventory information NJ includes second food information that takes into account the first food information FJ1, which includes changes in the food over time, so that the user P of the refrigerator 2 can be notified of information that cannot be obtained from the photographing results of the outside-fridge photography device 3.

[0157] The second food information indicates at least one of the expiration date of the food, the expiration date of the food, the best time to eat the food, and the amount of food that can be stored in the refrigerator 2.

[0158] This allows the user P to be notified of at least one of the best-before date of the food, the expiration date of the food, the best time to eat the food, and the amount of food that can be stored in the refrigerator 2. In particular, since the second food information takes into account changes in the food over time, the accuracy of the notification content can be increased, and the user P can be notified of at least one of the best-before date of the food, the expiration date of the food, the best time to eat the food, and the amount of food that can be stored in the refrigerator 2.

[0159] The notification system 1 includes a management unit 515 that manages the inventory information NJ and the first food information FJ1 in association with each other. The management unit 515 manages the inventory information NJ and the first food information FJ1 in association with each other based on the contents of the inventory information NJ and the contents of the first food information FJ1. The third acquisition unit 516 acquires second food information based on the pair of the inventory information NJ and the first food information FJ1 managed by the management unit 515.

[0160] This allows matching based on the content of the information, so that the inventory information NJ and the first food information FJ1 can be accurately matched for each food item stored in the refrigerator 2. This allows the third acquisition unit 516 to accurately acquire the second food information for each food item. This allows the user P of the refrigerator 2 to be notified of information for each food item that cannot be acquired from the photographing results of the outside-fridge photography device 3.

[0161] The management unit 515 selects inventory information NJ and first food information FJ1 whose degree of similarity is greater than or equal to a predetermined value based on multiple pieces of information contained in the inventory information NJ and multiple pieces of information contained in the first food information FJ1, and matches the selected inventory information NJ and first food information FJ1.

[0162] This allows the management unit 515 to accurately associate the receiving information NJ and the first food information FJ1, which have similar information content, with each food item stored in the refrigerator 2. This allows the user P of the refrigerator 2 to be accurately notified of information for each food item that cannot be obtained from the photographing results of the outside-fridge photography device 3.

[0163] The inventory information NJ indicates the type of food and the destination of the food. The first food information FJ1 indicates the type of food, the destination of the food, and changes in the food over time. If the type of food indicated by the inventory information NJ matches the type of food indicated by the first food information FJ1, and the destination of the food indicated by the inventory information NJ matches the destination of the food indicated by the first food information FJ1, the management unit 515 associates the inventory information NJ with the first food information FJ1.

[0164] This allows the management unit 515 to more accurately associate the inventory information NJ and the first food information FJ1 for each food item stored in the refrigerator 2. This allows the user P of the refrigerator 2 to be more accurately notified of information for each food item that cannot be obtained from the photographing results of the outside-fridge photography device 3.

[0165] The application execution unit 410 displays the photographed results of the outside-fridge photography device 3 and the photographed results of the inside-fridge photography device 27 in a switchable manner.

[0166] This allows the user P of the refrigerator 2 to understand the inside of the refrigeration compartment 21 from different viewpoints. Therefore, the user P of the refrigerator 2 can understand the inside of the refrigeration compartment 21 more accurately.

[0167] The control method of the notification system 1 is to obtain inventory information NJ related to the storage of food from the photographing results of the outside photographing device 3, obtain first food information FJ1 related to the food and including changes in the food over time from the photographing results of the inside photographing device 27, obtain second food information related to the food based on the inventory information NJ obtained by the first acquisition unit 513 and the first food information FJ1 obtained by the second acquisition unit 514, and notify the user P of the refrigerator 2 of the second food information obtained by the third acquisition unit 516.

[0168] This achieves the effects of the notification system 1 described above.

[0169] The control program 521 causes the server processor 51 to function as a first acquisition unit 513 that acquires inventory information NJ related to the inventory of food from the image capture results of the outside-fridge image capture device 3, a second acquisition unit 514 that acquires first food information FJ1 that is information related to food and includes changes in the food over time from the image capture results of the inside-fridge image capture device 27, a third acquisition unit 516 that acquires second food information related to the food based on the inventory information NJ acquired by the first acquisition unit 513 and the first food information FJ1 acquired by the second acquisition unit 514, and a server communication unit 511 that notifies the user P of the refrigerator 2 of the second food information acquired by the third acquisition unit 516.

[0170] This achieves the effects of the notification system 1 described above.

[0171] (Other Embodiments) As described above, the first embodiment has been described as an example disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. It is also possible to combine the components described in the first embodiment to create new embodiments. Therefore, other embodiments will be described below as examples.

[0172] In another embodiment, the degree of agreement may be the ratio of the number of items whose recorded information matches to the number of items shared by the inventory information NJ and the first food information FJ1. Also, in another embodiment, the degree of agreement may be the ratio of the number of items whose recorded information matches to the number of items in the inventory information NJ. Also, in another embodiment, the degree of agreement may be the ratio of the number of items whose recorded information matches to the number of items in the first food information FJ1.

[0173] In another embodiment, the degree of match may be calculated by combining the probability of recorded information for items shared by the inventory information NJ and the first food information FJ1. Also, in another embodiment, the degree of match may be calculated by combining the probability of recorded information for items included in the inventory information NJ or the first food information FJ1.

[0174] In the first embodiment described above, interior photography device 27 is provided on door 21B, and is configured to photograph the interior of refrigerator compartment 21. In other embodiments, interior photography device 27 may be provided on the ceiling of the storage compartment that houses drawer 26A, together with or instead of door 21B. In this configuration, notification system 1 can notify user P of second food information regarding the food stored in drawer 26A, based on storage information NJ of the food stored in drawer 26A and first food information FJ1 acquired from photographed images of the inside of drawer 26A.

[0175] In the first embodiment described above, the interior photography device 27 is provided on the door 21B, and is configured to photograph the interior of the refrigerator compartment 21. In other embodiments, the interior photography device 27 may be provided on the door 21C, and the interior photography device 27 may be configured to photograph the interior of the refrigerator compartment 21.

[0176] In the first embodiment described above, the app UI 441 is configured to switch between displaying the image capture results from the outside-fridge image capture device 3 and the image capture results from the inside-fridge image capture device 27. In other embodiments, the app UI 441 may fixedly display the second display captured image G2 instead of the refrigerator compartment image RFG.

[0177] In the first embodiment described above, the outside-compartment photography device 3 and the refrigerator 2 are configured to each send upload data to the notification management server 5 without cooperating with each other. In other embodiments, it is assumed that the outside-compartment photography device 3 and the refrigerator 2 are able to communicate with each other, and the outside-compartment photography device 3 or the refrigerator 2 may send the first upload data D1 and the second upload data D2. In this case, when the first upload data D1 is generated, the outside-compartment photography device 3 sends the first upload data D1 to the refrigerator 2, and the refrigerator 2 sends the second upload data D2 together with the first upload data D1 received at the time the second upload data D2 was generated. Alternatively, when the second upload data D2 is generated, the refrigerator 2 sends the second upload data D2 to the outside-compartment photography device 3, and the outside-compartment photography device 3 sends the first upload data D1 together with the second upload data D2 received at the time the first upload data D1 was generated.

[0178] In another embodiment, the operation of refrigerator 2 may be controlled based on the second food information acquired by third acquisition unit 516. For example, in another embodiment, if the second food information indicates when a food is ripe to eat, and the ripe food is ripe for eating and the current date and time are within a predetermined number of days, the internal temperature of refrigerator 2 is reduced.

[0179] In the first embodiment described above, the second food information is notified to user P by display. In other embodiments, the second food information may be notified by printing, audio, or the like. If the notification is by printing, terminal device 4 transmits print data to a printing device to print the second food information on a print medium. If the notification is by audio, terminal device 4 outputs the second food information through its speaker. For example, terminal device 4 outputs audio, saying, "The AAA in the refrigerator is best eaten on June 4, 2025."

[0180] In other embodiments, information other than the second food information may be provided. Examples of information other than the inventory information NJ and the first food information FJ1 include the availability of the refrigerator 2, the food to be removed from the refrigerator 2, and the user P's food preferences (such as frequently purchased foods and the origins of frequently purchased foods). In this other embodiment, the availability of the refrigerator 2 is determined based on the area relationship between the area in which no food is captured in the image captured by the interior photography device 27 and the area in which food is captured in the image captured by the interior photography device 27. In another embodiment, the food to be removed from the refrigerator 2 indicated by the inventory information NJ is determined from the color change over time of the food recorded in the first food information FJ1. In another embodiment, the user P's food preferences are determined based on past image capture results captured by the exterior photography device 3.

[0181] In another embodiment, the determination data used for determining the type or state of food may be used as learning data. In this case, it is preferable for the app UI 441 to reflect feedback information from the user P. This improves the accuracy of the determination results. Furthermore, the refrigerator 2 may learn from the determination data and, based on the timing of use of the refrigerator 2 and the aging of food, such as deterioration, change the temperature, humidity, or gas of a specified space or food to change its freshness or maturity. Furthermore, when it is confirmed that food has been removed from the refrigerator, information provided to the user P, such as an image, may be deleted after a specified time has elapsed.

[0182] In other embodiments, the dedicated space within the refrigerator may be part of a separate storage compartment or may be in the form of a box that can be installed afterwards.

[0183] In another embodiment, the installation and fixing portion and power connection portion of the outside-compartment photography device 3 are provided on the refrigerator 2, and the outside-compartment photography device 3 can be connected to the refrigerator 2 to communicate sensing data and the like between the refrigerator 2 and the outside-compartment photography device 3. The outside-compartment photography device 3 may also be provided on the ceiling of the refrigerator 2, so that the outside-compartment photography device 3 is integrated with the refrigerator 2. The center of the door of the refrigerator compartment 21 may be recessed, and the outside-compartment photography device 3 may be positioned opposite the recessed shape, so that an image of the drawer 26A can be acquired without protruding from the door when the refrigerator compartment 21 is closed.

[0184] In the above-described embodiment, the interior image capturing device 27 is used as an example of the "sensor." In other embodiments, the "sensor" may be a sensor other than the interior image capturing device 27, such as a weight sensor or an odor sensor. When the "sensor" is a weight sensor, the weight sensor transmits the type of food whose weight is being detected and the weight value as the sensing result to the refrigerator control device 28. The refrigerator 2 then transmits second upload data D2, which records the sensing result of the weight sensor, to the notification management server 5. The second acquisition unit 514 then acquires first food information FJ11, which includes changes in the weight of the food over time. The third acquisition unit 516 then acquires second food information taking this first food information FJ11 into consideration. When the "sensor" is an odor sensor, the odor sensor transmits the type of food whose odor is being detected and the detection value as the sensing result to the refrigerator control device 28. The refrigerator 2 then transmits second upload data D2, which records the sensing result of the odor sensor, to the notification management server 5. The second acquisition unit 514 acquires first food information FJ11 including changes in the smell of the food over time, and the third acquisition unit 516 acquires second food information taking into account the first food information FJ11.

[0185] In another embodiment, the accuracy of determining the type of food may be improved by combining the image captured by the outside-chamber photography device 3 and the sensing results of the "sensor" with the color of the food, the shape of the food, the texture of the food's surface, the shipping date and region of the food, the amount of food purchased by user P in the past, the gas components emitted by the food, the weight of the food, the number of food items, etc. In another embodiment, the notification content may be acquired by taking this information into account when notifying the second food information. This allows the content of the second food information to be expanded.

[0186] In the first embodiment described above, the inventory information NJ is information indicating the type of food, the destination of the food, and the date of arrival. In other embodiments, the inventory information NJ may indicate even more information. For example, in addition to the above three pieces of information, the inventory information NJ may also indicate the color of the food, the type of marker if the food is marked with a marker, the action of user P when putting the food in, etc.

[0187] In the first embodiment described above, the second food information is information indicating at least one of the expiration date, the expiration date, the best time to eat the food, and the amount of the food. In other embodiments, the content indicated by the second food information is not limited to the above-described content, as long as it is information based on the inventory information NJ and the first food information FJ1.

[0188] In the first embodiment described above, the server processor 51 functions as the first acquisition unit 513, but the outside-fridge processor 310 may also function as the first acquisition unit 513. In this configuration, the first upload data D1 sent from the outside-fridge photography device 3 includes the storage information NJ instead of or in addition to the group of captured images for acquisition.

[0189] In the first embodiment described above, server processor 51 is configured to function as second acquisition unit 514, but refrigerator processor 210 may also function as second acquisition unit 514. In this configuration, second upload data D2 transmitted from outside-fridge photography device 3 further includes first food information FJ1.

[0190] In the first embodiment described above, the wide-angle camera 301, the narrow-angle camera 302, and the thermopile 305 are arranged in this order at the front end of the imaging member 31. However, this arrangement order is merely an example, and the arrangement order of these devices is not limited to that of the first embodiment.

[0191] In the first embodiment described above, the right door 21B and the left door 21C have different widths. In other embodiments, the right door 21B and the left door 21C may have the same width, or the left door 21C may have a width greater than that of the right door 21B.

[0192] In the first embodiment described above, the sensor for detecting the open / closed state of drawer 26A is configured with thermopile 305. However, the sensor for detecting the open / closed state of drawer 26A may be a distance sensor. In this case, the outside-compartment open / close determination unit 312 determines whether drawer 26A is open or closed based on the detection value of the distance sensor. When drawer 26A is pulled out, it detects that there has been movement. The distance sensor may be an infrared type or an ultrasonic type.

[0193] In the above-described second embodiment, the first open / close sensor 202A to the sixth open / close sensor 202F are configured with Hall ICs. However, the first open / close sensor 202A and the second open / close sensor 202B may be other sensors such as a distance sensor, an angle sensor, a camera, or a temperature sensor. Furthermore, the third open / close sensor 202C to the sixth open / close sensor 202F may be other sensors such as an acceleration sensor, a mechanical switch, an angle sensor, a camera, or a temperature sensor.

[0194] In other embodiments, the outside-fridge image capture device 3 may be configured to end recording or capturing images a predetermined time after door 21A or drawer 26A is opened. For example, the predetermined time is one minute. The predetermined time may be extended if it is detected that door 21A or drawer 26A is open, that a person is present near refrigerator 2, or that user P is taking or putting items in or out. Furthermore, if it is detected that door 21A or drawer 26A is open, that a person is present near refrigerator 2, or that user P is taking or putting items in or out, the outside-fridge image capture device 3 may end recording or capturing images a predetermined time after the detection.

[0195] In another embodiment, the outside-compartment photography device 3 may divide the first display image G1 into three images (left door image LDG, refrigerator compartment image RFG, and right door image RDG), and the outside-compartment photography device 3 may perform image correction on the three divided images and upload them to the notification management server 5. An example of this image correction is viewpoint conversion correction.

[0196] In other embodiments, the number of compartments formed in the main box 20 of the refrigerator 2 may be less than or equal to the five types of compartments described above. Also, in other embodiments, the number of doors provided at the opening of the refrigeration compartment 21 may be one.

[0197] In the first embodiment described above, the terminal device 4 is a smartphone or a tablet PC. In other embodiments, the terminal device 4 may be a desktop or laptop PC.

[0198] In the first embodiment described above, an application program that can be installed on the terminal device 4 is exemplified as the "program," but the "program" may also be a software program that already exists on the terminal device 4, such as a browser.

[0199] In the first embodiment described above, the terminal device 4 generates the various screens displayed by the application UI 441, and the terminal device 4 displays the various screens generated. In other embodiments, the notification management server 5 may generate the various screens displayed by the application UI 441, and the terminal device 4 may display the various screens generated by the notification management server 5.

[0200] Outside processor 310, refrigerator processor 210, server processor 51, and terminal processor 41 may be configured by a single processor or multiple processors. These processors may be hardware programmed to implement corresponding functional units. That is, these processors may be configured by, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0201] The configurations of the outside-fridge photography device 3, refrigerator 2, terminal device 4, and notification management server 5 shown in Figures 5, 6, and 7 are merely examples, and the specific implementation form is not particularly limited. In other words, it is not necessary to implement hardware corresponding to each unit individually, and it is also possible to implement a configuration in which a single processor executes a program to realize the functions of each unit. Furthermore, some of the functions realized by software in the above-described embodiments may be implemented by hardware, or some of the functions realized by hardware may be implemented by software.

[0202] The step units of the operations shown in Figures 9 to 14 are divided according to the main processing content in order to make the operations easier to understand, and the operation is not limited by the way in which the processing units are divided or the names of the processing units. The operations may be divided into more step units depending on the processing content. Furthermore, one step unit may be divided so that it includes more processing. Furthermore, the order of the steps may be changed as appropriate within the scope that does not interfere with the purpose of this disclosure.

[0203] It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents.

[0204] (Additional Notes) The above description of the embodiments discloses the following techniques.

[0205] (Technology 1) A notification system including: an outside-fridge photography device installed outside a refrigerator for photographing a storage compartment of the refrigerator; a sensor installed inside the refrigerator for sensing the storage compartment; a first acquisition unit for acquiring entry information related to the entry of food from the image capture results of the outside-fridge photography device; a second acquisition unit for acquiring first food information related to the food from the sensor's sensing results, the first food information including changes in the food over time; a third acquisition unit for acquiring second food information related to the food based on the entry information acquired by the first acquisition unit and the first food information acquired by the second acquisition unit; and a notification unit for notifying a user of the refrigerator of the second food information acquired by the third acquisition unit. This system notifies the user of the refrigerator of the second food information that takes into account the first food information including changes in the food over time in addition to the entry information, thereby making it possible to notify the user of the refrigerator of information that cannot be acquired from the image capture results of the outside-fridge photography device.

[0206] (Technology 2) The notification system according to Technology 1, wherein the second food information notified by the notification unit indicates at least one of the best-before date of the food, the expiration date of the food, the best time to eat the food, and the amount of the food that can be stored in the refrigerator. This allows user P to be notified of at least one of the best-before date of the food, the expiration date of the food, the best time to eat the food, and the amount of food that can be stored in the refrigerator. In particular, because the second food information takes into account changes in the food over time, the accuracy of the notification content can be increased, and user P can be notified of at least one of the best-before date of the food, the expiration date of the food, the best time to eat the food, and the amount of food that can be stored in the refrigerator.

[0207] (Technology 3) The notification system according to Technology 1 or Technology 2 includes a management unit that associates and manages the inventory information and the first food information, the management unit associates and manages the inventory information and the first food information based on the contents of the inventory information and the first food information, and the third acquisition unit acquires the second food information based on the pair of the inventory information and the first food information managed by the management unit. Since the association is based on the content of the information, the inventory information and the first food information can be accurately associated for each food item stored in the refrigerator. Therefore, the third acquisition unit can accurately acquire the second food information for each food item. Therefore, information that cannot be acquired from the photographing results of the exterior photography device can be notified to the refrigerator user for each food item.

[0208] (Technology 4) In the notification system described in Technology 3, the management unit selects the inventory information and the first food information that have a degree of similarity equal to or greater than a predetermined value based on the plurality of pieces of information included in the inventory information and the plurality of pieces of information included in the first food information, and associates the selected inventory information with the first food information. This allows the management unit to accurately associate the inventory information and the first food information, which have similar information content, for each food item stored in the refrigerator. This allows the refrigerator user to be accurately notified of information that cannot be obtained from the photographing results of the outside-fridge photography device for each food item.

[0209] (Technology 5) In the notification system described in Technology 4, the inventory information indicates the type of food and its destination, the first food information indicates the type of food, its destination, and changes in the food over time, and the management unit associates the inventory information with the first food information when the type of food indicated by the inventory information matches the type of food indicated by the first food information and when the destination of the food indicated by the inventory information matches the destination of the food indicated by the first food information. This allows the management unit to more accurately associate the inventory information with the first food information for each food item stored in the refrigerator. This allows the refrigerator user to more accurately be notified of information for each food item that cannot be obtained from the image capture results of the exterior photography device.

[0210] (Technology 6) The notification system according to any one of Technology 1 to Technology 5, wherein the storage compartment is a refrigerator compartment, the sensor is an interior photography device provided inside the refrigerator and photographing the refrigerator compartment, the sensing result is the photography result of the interior photography device, the notification unit is a display unit that displays the image, and the display unit is capable of switching between the photography result of the exterior photography device and the photography result of the interior photography device. This allows the user of the refrigerator to understand the interior of the refrigerator compartment from different perspectives. This allows the user of the refrigerator to more accurately understand the interior of the refrigerator compartment.

[0211] (Technology 7) A control method for a notification system including an outside-fridge photography device installed outside a refrigerator for photographing a storage compartment of the refrigerator, and a sensor installed inside the refrigerator for sensing the storage compartment, the control method for the notification system comprising: acquiring inventory information related to the storage of food from the photography results of the outside-fridge photography device; acquiring first food information related to the food and including changes in the food over time from the sensing results of the sensor; acquiring second food information related to the food based on the acquired inventory information and the first food information; and notifying a user of the refrigerator of the acquired second food information. This achieves effects similar to those of the notification system described in Technology 1.

[0212] (Technology 8) A program that causes a processor to function as a first acquisition unit that acquires storage information related to the storage of food from image capture results from an outside photography device that is installed outside the refrigerator and captures images of the storage compartment of the refrigerator, a second acquisition unit that acquires first food information that is information related to the food and includes changes in the food over time from sensing results from a sensor that is installed inside the refrigerator and senses the storage compartment, a third acquisition unit that acquires second food information related to the food based on the storage information acquired by the first acquisition unit and the first food information acquired by the second acquisition unit, and a notification control unit that notifies a user of the refrigerator of the second food information acquired by the third acquisition unit. This achieves effects similar to those of the notification system described in Technology 1.

[0213] As described above, the notification system, the control method for the notification system, and the program according to the present invention can be used to notify information about food stored in a refrigerator.

[0214] REFRIGERATOR REFRIGERATOR LIST 1 Notification system 2 Refrigerator 3 Outside-fridge image capture device 4 Terminal device 5 Notification management server 27 Inside-fridge image capture device (sensor) 29 Refrigerator wireless communication module 51 Server processor (processor) 201 Cooling unit 202 Open / close sensor group 301 Wide-angle camera 302 Narrow-angle camera 303 First distance measurement sensor 304 Second distance measurement sensor 305 Thermopile 306 Outside-fridge wireless communication module 310 Outside-fridge processor 311 Control program 311 Outside-fridge communication unit 312 Outside-fridge open / close determination unit 313 Outside-fridge image capture control unit 314 Recording control unit 315 Outside-fridge generation unit 320 Outside-fridge memory 321 Control program 322 Image capture device ID 323 Recording data 410 App execution unit (notification unit, display unit) 421 Notification management app 422 User ID 511 Server communication unit (notification control unit) 512 Update unit 513 First acquisition unit 514 Second acquisition unit 515 Management unit 516 Third acquisition unit 521 Control program (program) 522 Refrigerator ID D1 First uploaded data D2 Second uploaded data FJ1, FJ11, FJ12, FJ13 First food information G1 First captured image for display G2 Second captured image for display (sensing result) NJ, NJ1 Stock information

Claims

1. A notification system comprising: an exterior photography device installed outside a refrigerator for photographing the refrigerator's storage compartment; a sensor installed inside the refrigerator for sensing the storage compartment; a first acquisition unit for acquiring inventory information related to the storage of food from the photography results of the exterior photography device; a second acquisition unit for acquiring first food information related to the food, including changes in the food over time, from the sensing results of the sensor; a third acquisition unit for acquiring second food information related to the food based on the inventory information acquired by the first acquisition unit and the first food information acquired by the second acquisition unit; and a notification unit for notifying a user of the refrigerator of the second food information acquired by the third acquisition unit.

2. The notification system of claim 1, wherein the second food information notified by the notification unit indicates at least one of the expiration date of the food, the expiration date of the food, the best time to eat the food, and the amount of the food that can be stored in the refrigerator.

3. A notification system as described in claim 1 or 2, comprising a management unit that associates and manages the inventory information and the first food information, wherein the management unit associates and manages the inventory information and the first food information based on the contents of the inventory information and the contents of the first food information, and the third acquisition unit acquires the second food information based on the pair of inventory information and the first food information managed by the management unit.

4. The notification system described in claim 3, wherein the management unit selects the inventory information and the first food information that have a degree of similarity greater than or equal to a predetermined value based on the multiple pieces of information included in the inventory information and the multiple pieces of information included in the first food information, and associates the selected inventory information with the first food information.

5. The notification system described in claim 4, wherein the inventory information indicates the type of food and the destination of the food, the first food information indicates the type of food, the destination of the food, and changes in the food over time, and the management unit matches the inventory information and the first food information when the type of food indicated by the inventory information matches the type of food indicated by the first food information and the destination of the food indicated by the inventory information matches the destination of the food indicated by the first food information.

6. The notification system of claim 1, wherein the storage compartment is a refrigerator compartment, the sensor is an interior photography device provided inside the refrigerator and taking pictures of the refrigerator compartment, the sensing result is the photography result of the interior photography device, the notification unit is a display unit that displays a display, and the display unit is capable of switching between displaying the photography result of the exterior photography device and the photography result of the interior photography device.

7. A control method for a notification system comprising an exterior photography device installed outside a refrigerator for photographing the storage compartment of the refrigerator, and a sensor installed inside the refrigerator for sensing the storage compartment, the control method comprising: obtaining inventory information related to the storage of food from the photography results of the exterior photography device; obtaining first food information related to the food, including changes in the food over time, from the sensing results of the sensor; obtaining second food information related to the food based on the obtained inventory information and the first food information; and notifying the obtained second food information to a user of the refrigerator.

8. A program that causes a processor to function as: a first acquisition unit that acquires inventory information related to the entry of food from the image capture results of an outside photography device that is installed outside the refrigerator and captures images of the refrigerator's storage compartment; a second acquisition unit that acquires first food information that is information related to the food and includes changes in the food over time from the sensing results of a sensor that is installed inside the refrigerator and senses the storage compartment; a third acquisition unit that acquires second food information related to the food based on the inventory information acquired by the first acquisition unit and the first food information acquired by the second acquisition unit; and a notification control unit that notifies the user of the refrigerator of the second food information acquired by the third acquisition unit.

Images