Detection system, program, and inventory management method

Abstract

To further certainly obtain information on articles stored in a refrigerator or delivered from the refrigerator.SOLUTION: A detection system is equipped with a marker attached to an article, and a detection device that detects the marker. The detection device is equipped with a shooting portion that shoots a range including at least a part of the inside of a refrigerator from an upper side of the inside of the refrigerator, while a door of the refrigerator is opened, a detecting portion that detects an image of the marker from the shooting image shot by the shooting portion, and a determining portion that determines the storing of the article attached with the marker to the inside of the refrigerator or the delivery of the article from the inside of the refrigerator, on the basis of the image of the marker detected by the detecting portion.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a detection system, a program, and a method for managing inventory. [Background technology]

[0002] Patent Document 1 discloses a refrigerator in which a passing object detection unit detects an object passing through an opening provided in a storage compartment, and when the passing object detection unit detects the object, a camera takes an image of the object, and obtains information about the item being stored in the storage compartment from the image obtained by the image capture. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-070476 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides a detection system, a program, and an incoming / outgoing management method that can more reliably acquire information about items entering or leaving a refrigerator. [Means for solving the problem]

[0005] The detection system of the present disclosure is housed within a refrigerator compartment. At least one of a food product and a container for containing the food productA detection system including a marker image attached to an item and a detection device that detects the marker image, wherein the detection device includes: an imaging unit that captures an image of an area including at least a part of the interior of the refrigerator from above the interior when a door of the refrigerator is in an open state; a detection unit that performs image analysis on an image of a range that corresponds to a detection area that is on the bottom of a refrigerating compartment of the refrigerator and is provided in front of a chilled compartment, among the images captured by the imaging unit, and detects the marker image attached to the item; and a determination unit that determines whether the item attached with the marker image has been placed into the refrigerator or whether the item has been removed from the refrigerator, based on a trajectory of movement of the marker image detected from the image of the range that corresponds to the detection area. The marker image includes a rectangular background image and a pattern image that is superimposed on the background image and includes identification information that identifies the marker image, the detection unit determines a range to be cut out from the photographed image by adding a preset setting value to four vertices of the background image detected from the photographed image, and cuts out an image of the determined range from the photographed image, and the determination unit determines whether the item has been put into the warehouse or taken out from the warehouse based on a trajectory of the marker image included in the image of the range cut out by the detection unit. A detection system.

[0006] The program in the present disclosure is stored in the refrigerator compartment. At least one of a food product and a container for containing the food product The computer mounted on the detection device that detects the marker image attached to the item executes the following processes: a process of causing a computer mounted on the detection device to acquire an image captured by an image capturing unit that captures an image of an area including at least a part of the interior of the refrigerator from above the interior when the door of the refrigerator is in an open state; a process of detecting the marker image attached to the item by performing image analysis on an area of ​​the acquired captured image that corresponds to a detection area that is on the bottom of the refrigerating compartment of the refrigerator and is provided in front of a chilled compartment, as a detection area; and a process of determining whether the item to which the marker image is attached has been put into the refrigerator or whether the item has been taken out of the refrigerator, based on a trajectory of movement of the marker image detected from an image of the area that corresponds to the detection area. The marker image includes a rectangular background image and a pattern image that is superimposed on the background image and includes identification information that identifies the marker image, Detection process a process of determining a range to be cut out from the photographed image by adding a preset value to four vertices of the background image detected from the photographed image, and cutting out an image of the determined range from the photographed image; and a process of determining whether the item has been put into the warehouse or whether the item has been taken out from the warehouse based on a trajectory of the marker image included in the image of the range cut out by the detection process. It is a program.

[0007] The method for managing incoming and outgoing goods in the present disclosure is At least one of a food product and a container for containing the food productThe detection device executes the following steps: causing a processor mounted on the detection device to acquire an image captured by an image capturing unit that captures an image of an area including at least a part of the interior of the refrigerator from above the interior when the door of the refrigerator is open; detecting the marker image attached to the item by performing image analysis on an area of ​​the acquired captured image that corresponds to a detection area that is on the bottom of the refrigerating compartment of the refrigerator and is provided in front of a chilled compartment, as a detection area; and determining whether the item to which the marker image is attached has been put into the refrigerator or whether the item has been taken out of the refrigerator, based on a trajectory of movement of the marker image detected from an image of the area that corresponds to the detection area. The marker image includes a rectangular background image and a pattern image that is superimposed on the background image and includes identification information that identifies the marker image; the detecting step determines a range to be cut out from the photographed image by adding a preset value to four vertices of the background image detected from the photographed image, and cuts out an image of the determined range from the photographed image; and the determining step determines whether the item has been put into the warehouse or taken out from the warehouse based on a trajectory of the marker image included in the image of the range cut out by the detecting process. This is a method of managing incoming and outgoing goods. [Effects of the Invention]

[0008] The detection system, program, and inventory management method disclosed herein can more reliably obtain information on items entering or leaving a refrigerator. [Brief explanation of the drawings]

[0009] [Figure 1] A diagram showing the configuration of an item management system [Figure 2] Front view of the detection device [Figure 3] Right side view of the detector [Figure 4] Block diagram showing the configuration of the detection system [Figure 5] FIG. 10 is a diagram showing an example of a marker attached to an item. [Figure 6] Block diagram showing the configuration of a terminal device and an item management server [Figure 7] FIG. 10 is a diagram showing an example of a record stored in an item management DB. [Figure 8] Flowchart showing the operation of the detection device [Figure 9] An explanatory diagram illustrating the detection area [Figure 10] Enlarged view of the detection area captured in the recorded image [Figure 11] An explanatory diagram explaining the cutout range [Figure 12] An explanatory diagram explaining the entry and exit judgment [Figure 13] An explanatory diagram explaining the entry and exit judgment [Figure 14] Flowchart showing the operation of the article management system [Figure 15] Flowchart showing the operation of the article management system [Figure 16] A diagram showing an example of an input screen DETAILED DESCRIPTION OF THE INVENTION

[0010] (Findings that formed the basis of this disclosure) At the time when the inventors came up with the present disclosure, there was a technology for detecting items entering and leaving a refrigerator. However, the inventors discovered that the conventional technology required more reliable acquisition of information on items entering and leaving a refrigerator, and in order to solve this problem, they came up with the subject matter of the present disclosure. Therefore, the present disclosure provides a detection system, a program, and an incoming / outgoing management method that can more reliably acquire information about items entering or leaving a refrigerator.

[0011] Hereinafter, embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or redundant explanation of substantially the same configuration may be omitted. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0012] (Embodiment 1) The first embodiment will be described. [1. Configuration] [1-1. Composition of the inventory management system] FIG. 1 is a diagram showing the configuration of an article management system 1000. The product management system 1000 is a system that manages the names of the products BP stored in the refrigerator compartment 11, the storage start date and time of the products BP stored in the refrigerator compartment 11, and the like.

[0013] In this embodiment, the item BP includes food, containers for storing food, etc. Also, in this embodiment, the item BP managed by the item management system 1000 is an item to which a marker MK is attached. The marker MK may be a marker printed on the item BP, or a marker attached to the item BP. The marker MK functions as a mark for making the attached item BP a management target in the item management system 1000. The marker MK may be a two-dimensional code such as a QR code (registered trademark), a barcode, or a character string, and the specific form is not limited.

[0014] The product management system 1000 includes a refrigerator 1. The refrigerator 1 includes a main box 10 with an open front. The main box 10 is formed with a refrigerator compartment 11, an ice-making compartment 12, a fresh-freezing compartment 13, a freezer compartment 14, and a vegetable compartment 15. A chilled compartment 11D is provided within the refrigerator compartment 11. A rotating left door 11A and a rotating right door 11B are provided at the opening on the front of the refrigerator compartment 11. The ice-making compartment 12, the fresh-freezing compartment 13, the freezer compartment 14, and the vegetable compartment 15 are each provided with drawers 12A, 13A, 14A, and 15A for storing products BP. In the following description, when there is no need to distinguish between the left door 11A and the right door 11B, they will be referred to as "doors" and given the symbol "11C." Also, in the following description, when there is no need to distinguish between the drawers 12A, 13A, 14A, and 15A, they will be referred to as "drawers" and given the symbol "16A."

[0015] The item management system 1000 includes a detection device 2. The detection device 2 detects items BP being stored in the refrigerator 1. The detection device 2 also detects items BP being removed from the refrigerator 1. The detection device 2 is provided on the top surface 10A, which is the upper part of the main box body 10 of the refrigerator 1.

[0016] The product management system 1000 includes a terminal device 3. The terminal device 3 is, for example, a smartphone or a tablet terminal. An application program for managing the products BP stored in the refrigerator compartment 11 is installed on the terminal device 3, and the terminal device 3 communicates with the product management server 4 using the functions of this application program. In the following description, this application program will be referred to as the "item management application" and will be given the reference number "311."

[0017] In FIG. 1, a user P who is at home is indicated by a solid line, and a user P who has left home H is indicated by a dotted line. When the terminal device 3 is used by a user P who is at home, the terminal device 3 communicates with the item management server 4 via the communication device 5 or without the communication device 5 using the function of the item management application 311. When the terminal device 3 is used by a user P who has left home H and a communication connection with the communication device 5 cannot be established, the terminal device 3 communicates with the item management server 4 without the communication device 5 using the function of the item management application 311.

[0018] The communication device 5 is connected to the communication network NW and communicates with the item management server 4 via the communication network NW. The communication network NW is a network consisting of a public line network, a dedicated line, other communication lines, and various communication facilities, and the specific form is not limited. For example, the communication network NW may be a wide area network. The communication network NW may be configured to include at least one of a wireless communication circuit and a wired communication circuit. The communication device 5 functions as an interface device for connecting the detection device 2 and the terminal device 3 to the communication network NW.

[0019] The item management system 1000 includes an item management server 4. The item management server 4 is a server device that manages the items BP stored in the refrigerator compartment 11. The item management server 4 is connected to a communication network NW. Note that in each figure, the item management server 4 is represented by a single block, but this does not necessarily mean that the item management server 4 is composed of a single server device. For example, the item management server 4 may be composed of multiple server devices with different processing contents.

[0020] [1-2. Configuration of the detection device] The detection device 2 will be described with reference to FIGS. 2 and 3 illustrate the X-axis, Y-axis, and Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to each other. The Z-axis indicates the up-down direction. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction. The Y-axis indicates the front-back direction. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the forward direction. The positive direction of the Z-axis indicates the upward direction.

[0021] Fig. 2 is a front view of refrigerator 1. In Fig. 2, left door 11A and right door 11B are open. Fig. 3 is a right side view of refrigerator 1 and detection device 2. In Fig. 3, refrigerator 1 is in an open state, i.e., a state in which drawer 15A is pulled out.

[0022] The detection device 2 includes a refrigerator compartment camera 27 and a drawer camera 28. A camera with a wider angle of view than drawer camera 28 is used for refrigerator compartment camera 27. Refrigerator compartment camera 27 photographs the area from above and in front of refrigerator 1 when detection device 2 is installed on top of refrigerator 1. Refrigerator compartment camera 27 photographs an area including at least a portion of the interior of refrigerator 1 from above when door 11C of refrigerator 1 is open. Refrigerator compartment camera 27 corresponds to the "photographing unit" in this disclosure.

[0023] For example, the shooting range of refrigerator compartment camera 27 includes range A1 shown in Fig. 2 when viewed from the front side of refrigerator 1. Range A1 is a range that includes door pocket 111 of door 11C and the opening of refrigerator compartment 11 when left door 11A and right door 11B are open. Furthermore, the shooting range of refrigerator compartment camera 27 includes range A3 shown in Fig. 3 when viewed from the right side of refrigerator 1. Range A3 is a range that includes, in the front-to-back direction, from door 11C in the open state to the front end of shelf 112, and also includes the opening of refrigerator compartment 11 in the up-down direction.

[0024] Drawer camera 28 photographs the area from above and in front of refrigerator 1 downward. For example, the imaging range of drawer camera 28 includes range A2 shown in Fig. 2 when viewed from the front of refrigerator 1. Range A2 is a range that includes drawer 16A when viewed from the front of refrigerator 1. Also, for example, the imaging range of drawer camera 28 includes range A4 shown in Fig. 3 when viewed from the right side of refrigerator 1. Range A4 is a range that includes drawer 16A when it is pulled out to the furthest position in the front-to-rear direction.

[0025] The detection device 2 includes a distance measuring sensor 23 for the left door, a distance measuring sensor 24 for the right door, and a distance measuring sensor 25 for the drawer. The left door distance measuring sensor 23 is a distance measuring sensor. The left door distance measuring sensor 23 detects the distance between the left door 11A and the left door distance measuring sensor 23 at a predetermined cycle and outputs a detection value indicating the detected distance to the detection device processor 20. The right door distance measuring sensor 24 is a distance measuring sensor that detects the distance between the right door 11B and the right door distance measuring sensor 24 at a predetermined cycle and outputs a detection value indicating the detected distance to the detection device processor 20. The drawer distance measuring sensor 25 is a distance measuring sensor that detects the distance between the drawer 16A and the drawer distance measuring sensor 25 at a predetermined cycle and outputs a detection value indicating the detected distance to the detection device processor 20.

[0026] The left door distance measuring sensor 23, the right door distance measuring sensor 24, and the drawer distance measuring sensor 25 may be optical distance measuring sensors or ultrasonic distance measuring sensors.

[0027] The detection device 2 includes a human presence sensor 26. The human presence sensor 26 outputs a detection value to the detection device processor 20 at a predetermined cycle. The human presence sensor 26 may be an infrared sensor, an ultrasonic sensor, or a visible light sensor.

[0028] Note that the arrangement order and installation positions of each camera and each sensor in Figures 2 and 3 are illustrated for the convenience of explaining the detection device 2, and the arrangement order and installation positions of each camera and each sensor are not limited to the arrangement order and installation positions shown in Figures 2 and 3.

[0029] FIG. 4 is a block diagram showing the configuration of the detection system 100. The detection system 100 includes a marker MK attached to an item BP and a detection device 2 that detects the marker MK. The detection device 2 is a computer device including a detection device processor 20, which is a processor such as a CPU (Central Processing Unit) or an MPC (Micro-processing unit), and a detection device memory 21, which is a memory for storing programs and data. The detection device memory 21 corresponds to the storage unit of the present disclosure.

[0030] The detection device memory 21 has a nonvolatile storage area. The detection device memory 21 may also be configured to have a volatile storage area in addition to the nonvolatile storage area. The detection device memory 21 stores programs executed by the detection device processor 20 and data processed by the detection device processor 20. The detection device memory 21 stores, as programs and data, a control program 211, a detection device ID (Identification) 212, audio data 213, communication information 214, etc.

[0031] The detector ID 212 is identification information of the detector 2. Examples of the detector ID 212 include a serial number and a manufacturing number. The audio data 213 is data of the audio output by the notification unit 209 through the speaker 29 . The communication information 214 may be, for example, address information such as an IP address used for communication with the goods management server 4, or an email address of the user P who owns the terminal device 3.

[0032] Connected to the detection device processor 20 are a detection device communication unit 22, a left door distance measurement sensor 23, a right door distance measurement sensor 24, a drawer distance measurement sensor 25, a human presence sensor 26, a refrigerator compartment camera 27, a drawer camera 28, and a speaker 29. Devices other than these devices may also be connected to the detection device processor 20.

[0033] The detection device communication unit 22 is a communication device having a communication circuit, connector, etc. that conforms to a predetermined communication standard, and communicates with the item management server 4, which is the communication destination of the communication information 214 stored in the detection device memory 21, under the control of the detection device processor 20. In this embodiment, the communication standard used by the detection device communication unit 22 is exemplified as a wireless communication standard, but a wired communication standard may also be used.

[0034] The detection device processor 20 reads and executes a control program 211 stored in the detection device memory 21, thereby functioning as a detection device communication control unit 201, a person presence determination unit 202, an opening / closing determination unit 203, a photography control unit 204, a recording control unit 205, a detection unit 206, an entry / exit determination unit 207, an acquisition unit 208, a notification unit 209, and a generation unit 210. The entry / exit determination unit 207 corresponds to the determination unit of the present disclosure.

[0035] The detection device communication control unit 201 communicates with the item management server 4 via the detection device communication unit 22 .

[0036] The person presence determination unit 202 determines whether or not a person is present around the refrigerator 1 based on the detection value output by the human presence sensor 26. For example, if the human presence determination unit 202 outputs a detection value equal to or greater than a predetermined value, the person presence determination unit 202 determines that a person is present around the refrigerator 1, and if the human presence determination unit 202 outputs a detection value below the predetermined value, the person presence determination unit 202 determines that a person is not present around the refrigerator 1. If the person presence determination unit 202 determines that a person is present around the refrigerator 1, it outputs first presence information indicating that a person is present to the photography control unit 204. If the person presence determination unit 202 determines that no person is present around the refrigerator 1, it outputs second presence information indicating that a person is not present to the photography control unit 204.

[0037] The open / close determination unit 203 determines whether the left door 11A is in the open state or the closed state based on the detection value of the left door distance measuring sensor 23. When the open / close determination unit 203 receives a detection value from the left door distance measuring sensor 23 that indicates a distance greater than a detection value corresponding to the left door 11A being in the closed state, the open / close determination unit 203 determines that the left door 11A is in the open state. The open / close determination unit 203 determines whether the right door 11B is open or closed based on the detection value of the right door distance measuring sensor 24. When the open / close determination unit 203 receives a detection value from the right door distance measuring sensor 24 that indicates a distance greater than a detection value corresponding to the right door 11B being closed, the open / close determination unit 203 determines that the right door 11B is open.

[0038] Furthermore, the open / close determining unit 203 determines whether the drawer 16A is open or closed, depending on the distance indicated by the detection value of the drawer distance measuring sensor 25.

[0039] When the open / close determination unit 203 determines that at least one of the left door 11A and the right door 11B is in the open state, it outputs first door open / close information indicating that the door 11C is in the open state to the recording control unit 205 and the notification unit 209. When the open / close determination unit 203 determines that the left door 11A and the right door 11B are in the closed state, it outputs second door open / close information indicating that the left door 11A and the right door 11B are in the closed state to the recording control unit 205.

[0040] When it is determined that at least one of the drawers 16A is open, the open / close determination unit 203 outputs first drawer open / close information to the recording control unit 205. The first drawer open / close information is information that notifies that at least one of the drawers 16A is open. When it is determined that all of the drawers 16A are closed, the open / close determination unit 203 outputs second drawer open / close information to the recording control unit 205. The second drawer open / close information is information that notifies that all of the drawers 16A are closed.

[0041] When the first presence information is input from the person presence determination unit 202, the photography control unit 204 causes the refrigerator compartment camera 27 and the drawer camera 28 to start photographing. When the second presence information is input from the person presence determination unit 202 after the refrigerator compartment camera 27 and the drawer camera 28 have started photographing, the photography control unit 204 causes the refrigerator compartment camera 27 and the drawer camera 28 to stop photographing.

[0042] When the first door open / close information is input from the open / close determination unit 203, the recording control unit 205 records the image captured by the refrigerator compartment camera 27 in the detection device memory 21. That is, the recording control unit 205 starts recording. The captured image recorded in the detection device memory 21 is hereinafter referred to as a recorded image 215. When the recording control unit 205 starts recording, it outputs recording start information indicating that recording has started to the notification unit 209.

[0043] When the second door open / close information is input from the open / close determination unit 203, the recording control unit 205 ends recording of the captured image in the detection device memory 21. That is, the recording control unit 205 ends recording. Furthermore, when first drawer open / close information is input from open / close determination unit 203, recording control unit 205 records the image captured by drawer camera 28 in detection device memory 21. Furthermore, when second drawer open / close information is input from open / close determination unit 203, recording control unit 205 ends recording of the image captured by drawer camera 28.

[0044] The detection unit 206 detects an image of a marker MK attached to an item BP being stored in the refrigeration compartment 11, or an image of a marker MK attached to an item BP being removed from the refrigeration compartment 11, based on the recorded image 215 recorded in the detection device memory 21. The detection method of the detection unit 206 will be described in detail later. When the detection unit 206 detects an image of a marker MK, it notifies the notification unit 209 that the image of the marker MK has been detected.

[0045] The inbound / outbound determination unit 207 performs inbound / outbound determination processing. The inbound / outbound determination processing is processing for determining whether an item BP with an image of a marker MK attached has been stored in the refrigerator compartment 11 or has been removed from the refrigerator compartment 11. The item BP is an item BP with a marker MK attached whose image has been detected by the detection unit 206. Details of the determination method of the inbound / outbound determination unit 207 will be described later. When the inbound / outbound determination processing is completed, the inbound / outbound determination unit 207 outputs to the notification unit 209 that the inbound / outbound determination processing has been completed.

[0046] The acquisition unit 208 analyzes the image of the marker MK detected by the detection unit 206 and acquires the marker ID 4123. The marker ID 4123 is identification information of the marker MK assigned to the marker MK. For example, if the image of the marker MK is a two-dimensional code such as a QR code, the acquisition unit 208 acquires the marker ID 4123 included in the QR code. The acquisition unit 208 temporarily stores the acquired marker ID 4123 in the detection device memory 21.

[0047] FIG. 5 is a diagram showing an example of a marker MK attached to an item BP. Here, the marker MK will be explained with reference to Fig. 5. Fig. 5 shows a case where the item BP is a food storage container, but the food ingredient itself may be the item BP and the marker MK may be attached to the food ingredient. For example, cooked food prepared by user P is stored in the storage container.

[0048] The marker MK includes a background area MKA and a pattern image MKB formed within the background area MKA. The background area MKA is a rectangular area formed, for example, in a single color. The color of the background area MKA is preferably a color that does not interfere with the detection of the pattern image MKB when detecting the pattern image MKB from the recorded image 215.

[0049] The pattern image MKB may be a two-dimensional code such as a QR code or a one-dimensional code such as a barcode. Also, the manufacturing number attached to the food packaging may be used as the marker MK. Furthermore, a sticker printed with a serial number, which is the pattern image MKB, may be attached to the item BP and used as the marker MK.

[0050] The notification unit 209 performs various notifications by outputting the sound indicated by the sound data 213 stored in the detection device memory 21 to the speaker 29. When recording start information is input from the recording control unit 205 and an image of the marker MK is detected by the detection unit 206, the notification unit 209 notifies the user P. In addition, when recording start information is input from the recording control unit 205 and the entry / exit determination process of the entry / exit determination unit 207 is completed, the notification unit 209 notifies the user P.

[0051] The generation unit 210 generates upload data. The upload data is data to be uploaded to the item management server 4. The generation unit 210 causes the detection device communication control unit 201 to transmit the generated upload data to the item management server 4.

[0052] [1-3. Configuration of the product management server] FIG. 6 is a block diagram showing the configuration of the terminal device 3 and the item management server 4. As shown in FIG. First, the configuration of the item management server 4 will be described. The article management server 4 includes a server processor 40, which is a processor such as a CPU or MPC, a server memory 41, which is a memory for storing programs and data, and a server communication unit .

[0053] The server processor 40 functions as a server communication control unit 401 and a server processing unit 402 by reading and executing a control program 411 stored in the server memory 41 .

[0054] The server memory 41 stores programs executed by the server processor 40 and data processed by the server processor 40. The server memory 41 stores a control program 411 executed by the server processor 40, an item management DB (database) 412, and various other data. The server memory 41 has a non-volatile storage area. The server memory 41 may also have a volatile storage area and constitute a work area for the server processor 40.

[0055] FIG. 7 is a diagram showing an example of a record R stored in the item management DB 412. As shown in FIG. One record R stored in the item management DB 412 has an account ID 4121, communication information 4122, a detection device ID 212, and one or more sub-records SR. A sub-record SR is a record stored within a record R. One sub-record SR has a marker ID 4123, item name information 4124, storage area information 4125, storage start date and time information 4126, and item image data 4127.

[0056] The account ID 4121 is identification information for identifying an account assigned to a user P who uses the goods management application 311 .

[0057] The communication information 4122 is address information such as an IP address used for communication with the terminal device 3 used by the user P.

[0058] The item name information 4124 is information indicating the item name, which is the name of the item BP.

[0059] The storage area information 4125 is information indicating a storage area. The storage area indicated by the storage area information 4125 is an area in the refrigerator compartment 11. The storage area information 4125 of this embodiment indicates a number assigned to the storage area as the storage area.

[0060] The storage start date and time information 4126 is information indicating the date and time when the item was stored in the refrigerator compartment 11.

[0061] The article image data 4127 is image data of an article image BPG, which is an image of the article BP.

[0062] 6, the server communication unit 42 is a communication interface including a communication circuit conforming to a predetermined communication standard, and communicates with devices connected to the communication network NW according to the predetermined communication standard. In this embodiment, the server communication unit 42 communicates with the detection device 2 and the terminal device 3.

[0063] As described above, the server processor 40 functions as the server communication control unit 401 and the server processing unit 402 .

[0064] The server communication control unit 401 communicates with the detection device 2 and the terminal device 3 via the server communication unit 42 .

[0065] The server processing unit 402 performs various information processes using the detection device 2 and the terminal device 3 as clients.

[0066] [1-4. Terminal Device Configuration] Next, the functional configuration of the terminal device 3 will be described. The terminal device 3 includes a terminal processor 30 which is a processor such as a CPU or MPC, a terminal memory 31 which is a memory for storing programs and data, a terminal communication unit 32, and a touch panel 33.

[0067] The terminal processor 30 functions as an application execution unit 300 by reading and executing the goods management application 311 stored in the terminal memory 31 .

[0068] The terminal memory 31 stores programs executed by the terminal processor 30 and data processed by the terminal processor 30. The terminal memory 31 stores an item management application 311 executed by the terminal processor 30 and various other data. The terminal memory 31 has a non-volatile storage area. The terminal memory 31 may also have a volatile storage area and constitute a work area for the terminal processor 30.

[0069] The terminal communication unit 32 is a communication interface having a communication circuit, connector, etc. that conforms to a predetermined communication standard, and communicates with the item management server 4 under the control of the terminal processor 30. In this embodiment, the communication standard used by the terminal communication unit 32 is exemplified as a wireless communication standard, but it may also be a wired communication standard.

[0070] The touch panel 33 includes a display panel such as a liquid crystal display panel, and a touch sensor that is provided over the display panel or is integral with the display panel.

[0071] As described above, the terminal processor 30 functions as the application execution unit 300. The application execution unit 300 communicates with the item management server 4 via the terminal communication unit 32. The application execution unit 300 also displays a user interface related to the management of items stored in the refrigerator 1 on the touch panel 33. In the following description, this user interface is referred to as the "application UI" and is denoted by the reference numeral "330." By displaying the application UI 330 on the touch panel 33, the application execution unit 300 provides the user P with various information related to the items BP stored in the refrigerator 1, and also receives various inputs from the user P related to the items BP stored in the refrigerator 1.

[0072] [2. Operation] Next, the operation of each part of the item management system 1000 will be described. [2-1. Operations related to item detection] First, the operation relating to the detection of an article BP entering or leaving the refrigerating compartment 11 will be described. FIG. 8 is a flowchart showing the operation of the detection device 2.

[0073] The person presence determination unit 202 determines whether or not a person is present around the refrigerator 1 (step SA1).

[0074] If the person presence determination unit 202 determines that a person is present around the refrigerator 1 (step SA1: YES), the photography control unit 204 causes the refrigerator compartment camera 27 and the drawer camera 28 to start photographing (step SA2).

[0075] Next, the open / close determining unit 203 determines whether the door 11C is in the open state (step SA3).

[0076] If the open / close determining unit 203 determines that the door 11C is not in the open state (step SA3: NO), the detection device processor 20 performs the process of step SA14.

[0077] If the open / close determining unit 203 determines that the door 11C is in the open state (step SA3: YES), the recording control unit 205 starts recording the images captured by the refrigerator compartment camera 27 (step SA4).

[0078] Next, the notification unit 209 notifies the user P through the speaker 29 that detection of the item BP has started (step SA5).

[0079] Next, the detection unit 206 determines whether or not a marker MK has been detected within the detection area TA based on the recorded image 215 whose recording has started by the processing of step SA4 (step SA6).

[0080] The detection area TA will be described with reference to FIG. FIG. 9 is an explanatory diagram illustrating the detection area TA.

[0081] The detection area TA is an area within the front area MA formed in front of the chilled compartment 11D on the bottom surface of the refrigerator compartment 11. In this embodiment, the detection area TA is illustrated as having a smaller area than the front area MA, but may have the same area.

[0082] The detection unit 206 analyzes the recorded image 215 for which recording has started, and determines whether or not a marker MK has been detected within the detection area TA. The detection unit 206 recognizes the marker MK appearing in the recorded image 215 from features such as shape and color. Note that Fig. 9 shows a case in which the marker MK is not located within the detection area TA. Therefore, in the case shown in Fig. 9, the detection unit 206 determines that the marker MK has not been detected within the detection area TA.

[0083] Returning to the explanation of the flowchart in FIG. 8, if the detection unit 206 determines that the marker MK has not been detected within the detection area TA (step SA6: NO), the process proceeds to step SA12.

[0084] On the other hand, if the detection unit 206 determines that a marker MK has been detected within the detection area TA (step SA6: YES), it reads out the marker ID 4123 from the detected marker MK (step SA7).

[0085] Next, the detection unit 206 cuts out the cut-out range KH from the recorded image 215 (step SA8) and generates the article image BPG. That is, the image captured in the cut-out range KH of the recorded image 215 becomes the article image BPG. The cut-out range KH will be described with reference to FIG.

[0086] FIG. 10 is an explanatory diagram for explaining the cutout range KH. The detection unit 206 sets the cutout range KH based on, for example, the coordinates of the four vertices C1, C2, C3, and C4 that have already been calculated. For example, the detection unit 206 sets the cutout range KH to a coordinate range obtained by adding or subtracting a preset value to or from the coordinates of the four vertices C1, C2, C3, and C4. At this time, the cutout range KH may be larger or smaller than the background area MKA. Furthermore, the preset value may be different in the vertical and horizontal directions. Furthermore, the detection unit 206 may generate a difference image between before and after the item BP is detected, and set the range where the difference occurs in the image as the cutout range KH. Furthermore, although FIG. 11 illustrates an example in which the cutout range KH is narrower than the detection area TA, the cutout range KH may be the same area as the detection area TA.

[0087] In addition, in this embodiment, a marker MK is detected within the detection area TA and a cut-out range KH of the recorded image 215 is set, but detection of the marker MK may also be started when the detection area TA is cut out from the recorded image 215 and an image of the item BP is included in the recorded image 215 of the cut-out detection area TA.

[0088] Returning to the explanation of the flowchart in FIG. 8, when the detection unit 206 cuts out the item image BPG, the notification unit 209 notifies the user P by the speaker 29 that the item BP has been detected (step SA9).

[0089] Next, the entry / exit determination unit 207 performs entry / exit determination based on the recorded image 215 recorded in the detection device memory 21 (step SA10).

[0090] The incoming / outgoing judgment in step SA10 is a process of judging whether the item BP of the item image BPG cut out in step SA8 is an item BP that has been entered into the refrigerator 1 or an item BP that has been taken out of the refrigerator 1.

[0091] The entry / exit determination unit 207 performs image analysis on the recorded image 215 to set a first area R1, a second area R2, a third area R3, and a fourth area R4. The first area R1 corresponds to the refrigerator compartment 11. The second area R2 corresponds to the left door 11A. The third area R3 corresponds to the right door area 11B. The fourth area R4 corresponds to the outside of the refrigerator 1.

[0092] FIG. 12 is an explanatory diagram for explaining the entry / exit determination, showing the boundary lines between the first area R1, the second area R2, the third area R3, and the fourth area R4. The range of the first region R1 corresponding to the refrigerator compartment 11 is always constant in the recorded image 215, even if the opening degree of the left door 11A or the right door 11B opened or closed by the user varies. For this reason, the entry / exit determination unit 207 sets a preset range of the recorded image 215 as the first region R1. Boundary line 501 shown in Fig. 12 corresponds to the boundaries between the first region R1 and the second region R2, third region R3, and fourth region R4.

[0093] The second region R2 corresponding to the left door 11A and the third region R3 corresponding to the right door 11B change depending on the opening degree of the left door 11A or the right door 11B when the user opens or closes them. Therefore, the entry / exit determination unit 207 may detect, for example, the edges of the door pockets 111 provided on the left door 11A or the right door 11B, and set the extents of the second region R2 and the third region R3 based on the detected edges of the door pockets 111. Alternatively, a mark or the like identifying the door pockets 111 may be placed on the edges of the door pockets 111 on the left door 11A or the right door 11B, and the entry / exit determination unit 207 may set the extents of the second region R2 and the third region R3 based on the position of the mark detected from the recorded image 215. The entry / exit determination unit 207 may set the areas other than the set first region R1, second region R2, and third region R3 as a fourth region R4 outside the refrigerator compartment 11. A boundary line 502 shown in FIG. 12 indicates the boundary of the second region R2, and a boundary line 503 indicates the boundary of the third region R3.

[0094] The in / out determination unit 207 detects the trajectory of the moving marker MK by analyzing the continuously captured recorded images 215. The in / out determination unit 207 determines in which area, the first area R1, the second area R2, or the third area R3, the trajectory of the detected marker MK ends, and determines into which area the item BP has been stored. Furthermore, if the trajectory of the detected marker MK ends in the fourth area R4, the in / out determination unit 207 determines that the item BP has been removed from the refrigerator compartment 11.

[0095] FIG. 13 is a diagram for explaining the warehouse entry / exit determination image HG. Furthermore, the inbound / outbound determination unit 207 may superimpose an inbound / outbound determination image HG on the recorded image 215 to determine whether the item BP is inbound or outbound. FIG. 13 shows a case where an entry / exit determination image HG is superimposed on a recorded image 215 capturing a scene in which an item BP marked with a marker MK is being placed into the refrigerating compartment 11.

[0096] The entry / exit determination image HG has a boundary line L1. Boundary line L1 is a line extending from the left edge to the right edge of the entry / exit determination image HG at approximately the center of the image. Boundary line L1 is located in front of refrigerator 1 from the opening of refrigerator compartment 11 captured by refrigerator compartment camera 27.

[0097] The image area of ​​the entry / exit determination image HG is divided into a first area HA1 and a second area HA2 by a boundary line L1. The first area HA1 corresponds to the front side of the refrigerator 1 from the opening of the refrigerator compartment 11 that appears in the image captured by the refrigerator compartment camera 27 when the entry / exit determination image HG is superimposed on the recorded image 215. The second area HA2 corresponds to the rear side of the refrigerator 1 from the opening of the refrigerator compartment 11 that appears in the image captured by the refrigerator compartment camera 27 when the entry / exit determination image HG is superimposed on the recorded image 215.

[0098] The entry / exit determination unit 207 analyzes the recorded image 215 on which the entry / exit determination image HG is superimposed to determine whether the marker MK detected in the detection area TA has moved to the first area HA1 or the second area HA2. When the entry / exit determination unit 207 determines that the marker MK detected in the detection area TA has moved to the first area HA1, it determines that the item BP detected by the detection unit 206 is an item BP that has been taken out of the refrigerator compartment 11. On the other hand, when the marker MK detected in the detection area TA has moved to the second area HA2, it determines that the item BP detected by the detection unit 206 is an item BP that has been put into the refrigerator compartment 11.

[0099] When it is determined that the item BP has been stored in the warehouse, the storage / retrieval determination unit 207 determines the storage area into which the item BP has been stored. The storage / retrieval determination unit 207 analyzes the recorded image 215 to identify the storage area of ​​the refrigerator compartment 11 into which the item has been stored. The storage / retrieval determination unit 207 then determines the storage area into which the item BP has been stored based on the movement direction of the marker MK and the arrangement within the warehouse of the multiple storage areas provided therein. For example, marks identifying the storage area are attached to the end of the door 11C side of the shelf 112 of the refrigerator 1 and to each of the door pockets 111. The in-stock / out-stock determining unit 207 detects the marks identifying the storage area from the recorded image 215, and specifies the storage area based on the positional relationship between the detected marks and the markers MK attached to the items BP.

[0100] The in / out determination unit 207 may also determine whether the item BP is an item that has been re-stored in the refrigerator 1. For example, it is conceivable that the user P takes out ingredients from the refrigerator 1 for use, uses some of the taken-out ingredients, and re-stores the remaining ingredients in the refrigerator 1, or that the user P uses ingredients stored in a storage container that is the item BP, refills the storage container with new ingredients, and re-stores the container in the refrigerator 1.

[0101] When the acquisition unit 208 acquires again, within a preset time period, a marker ID identical to the marker ID 4123 already acquired by the acquisition unit 208, the in-stock / out-stock determination unit 207 determines that the item BP has been re-entered into the refrigerator 1. Furthermore, when the acquisition unit 208 acquires an ID identical to the marker ID 4123 attached to the item BP determined to have been released, and the determination result of the in-stock / out-stock determination process is that the item BP has been re-entered into the refrigerator 1, the in-stock / out-stock determination unit 207 determines that the item BP has been re-entered into the refrigerator 1.

[0102] At this time, the inventory-entry / outbound determination unit 207 may cause the terminal device 3 to display a message inquiring whether the food is new or not via the item management server 4. Alternatively, the inventory-entry / outbound determination unit 207 may output a voice message inquiring whether the food is new or not from the speaker 29, and have the user P operate the terminal device 3 to input a setting as to whether the item BP is new or not. Alternatively, the detection device 2 may be provided with a display panel and a remote control for operating the detection device 2, and the display panel may display a message inquiring whether the food is new or not, and the remote control may accept an operation by the user P in response to this message.

[0103] When the inbound / outbound determination unit 207 determines that the item BP has been stored in the refrigerator 1, it stores the determination result of the inbound determination in the detection device memory 21. In addition to this determination result, the inbound / outbound determination unit 207 stores the marker ID 4123 of the item BP, the item image BPG cut out from the cut-out range KH from the recorded image 215, and storage area information 4125 indicating the storage area in which the item BP is stored in the detection device memory 21. Furthermore, when the inbound / outbound determination unit 207 determines that the item BP has been re-stored in the refrigerator 1, it stores the determination result of the re-store in the detection device memory 21.

[0104] Furthermore, when the inbound / outbound determination unit 207 determines that the item BP has been taken out from the warehouse, it stores the determination result of the outbound determination in the detection device memory 21. In addition to this determination result, the inbound / outbound determination unit 207 stores the marker ID 4123 of the item BP and the item image BPG cut out from the cut-out range KH of the recorded image 215 in the detection device memory 21.

[0105] Returning to the explanation of the flowchart in FIG. 8, when the inbound / outbound determination unit 207 performs inbound / outbound determination, the notification unit 209 notifies the user P via the speaker 29 that the inbound / outbound of the detected item BP has been determined (step SA11).

[0106] Next, the open / close determining unit 203 determines whether the open door 11C has been closed (step SA12).

[0107] If the open / close determining unit 203 determines that the door 11C that was open is not in the closed state (step SA12: NO), the detection device processor 20 moves the process to step SA6.

[0108] On the other hand, if the open / close determination unit 203 determines that the open door 11C has been closed (step SA12: YES), the recording control unit 205 ends recording of the images captured by the refrigerator compartment camera 27 (step SA13).

[0109] Next, the person presence determination unit 202 determines whether or not a person is present around the refrigerator 1 (step SA14).

[0110] If the person presence determination unit 202 determines that a person is present around the refrigerator 1 (step SA14: YES), the detection device processor 20 moves the process to step SA3.

[0111] On the other hand, if the person presence determination unit 202 determines that no person is present around the refrigerator 1 (step SA14: NO), the photography control unit 204 causes the refrigerator compartment camera 27 and the drawer camera 28 to end photography (step SA15).

[0112] After performing the process of step SA15, the detection device processor 20 shifts the process to step SA1 and performs the processes from step SA1 onwards again.

[0113] [2-2. Operations related to sending uploaded data] Next, the operation of the item management system 1000 related to the transmission of upload data will be described. FIG. 14 is a flowchart showing the operation of the item management system 1000. In FIG. 14, a flowchart FB shows the operation of the detection device 2, and a flowchart FC shows the operation of the article management server 4.

[0114] As shown in the flowchart FB, the generation unit 210 of the detection device 2 determines whether a trigger for transmitting upload data to the article management server 4 has occurred (step SB1).

[0115] The trigger for transmitting the upload data may be when the inbound / outbound determination unit 207 stores the determination result of the inbound / outbound determination process in the detection device memory 21. Other triggers may include the completion of recording of images captured by the refrigerator compartment camera 27, or the current time reaching a predetermined time.

[0116] When a trigger occurs (step SB1: YES), the generation unit 210 determines whether the determination result of the inventory entry / exit determination process is inventory entry or inventory exit (step SB2). When the determination result of the inventory entry / exit determination process is inventory entry (step SB: inventory entry), the generation unit 210 generates first upload data (step SB3). The first upload data includes the marker ID 4123, an item image, storage area information 4125, and detection device ID 212. Furthermore, when the inventory entry / exit determination process determines that the item is re-entered, the generation unit 210 includes determination information indicating re-entry in the first upload data.

[0117] The generation unit 210 outputs the generated first upload data to the detection device communication control unit 201. The detection device communication control unit 201 transmits the input first upload data to the item management server 4 (step SB4). Thereafter, the generation unit 210 proceeds to the determination in step SB7.

[0118] Furthermore, if it is determined in step SB2 that the information indicates leaving the warehouse (step SB2: leaving the warehouse), the generation unit 210 generates second upload data (step SB5). The second upload data includes the marker ID 4123, the article image, and the detection device ID 212.

[0119] The generation unit 210 outputs the generated second upload data to the detection device communication control unit 201. The detection device communication control unit 201 transmits the input second upload data to the item management server 4 (step SB6).

[0120] Next, the generation unit 210 determines whether or not other determination results of the inbound / outbound determination process are recorded in the detection device memory 21 (step SB7). If other determination results of the inbound / outbound determination process are recorded in the detection device memory 21 (step SB7: YES), the generation unit 210 returns the process to step SB2 and performs the processes from step SB2 onwards again. On the other hand, if other determination results of the inbound / outbound determination process are not recorded in the detection device memory 21 (step SB7: NO), the generation unit 210 returns the process to step SB1 and performs the processes from step SB1 onwards again.

[0121] Next, the operation of the item management server 4 will be described with reference to a flowchart FC. The server communication control unit 401 of the item management server 4 determines whether or not either the first upload data or the second upload data has been received (step SC1).

[0122] If server communication control section 401 determines that upload data has not been received (step SC1: NO), it performs the determination of step SC1 again.

[0123] On the other hand, if the server communication control unit 401 determines that the upload data has been received (step SC1: YES), the server processing unit 402 updates the item management DB 412 based on the upload data received by the server communication control unit 401 (step SC2).

[0124] Step SC2 will now be described in detail. If the upload data received by the server communication control unit 401 is the first upload data, the server processing unit 402 identifies a record R having the detection device ID 212 included in the first upload data from the item management DB 412. Next, the server processing unit 402 stores the next sub-record SR for the identified record R. The stored sub-record SR stores the marker ID 4123, storage area information 4125, and item image data 4127 included in the first upload data, in association with blank item name information 4124, and storage start date and time information 4126 indicating the storage start date and time.

[0125] Furthermore, if the first upload data contains information indicating restocking, the server processing unit 402 does not generate a new record R or sub-record SR. In this case, the server processing unit 402 acquires a marker ID 4123 from the first upload data and determines whether there is information to update the sub-record SR corresponding to the acquired marker ID 4123. For example, if the storage area is changed, the server processing unit 402 updates the storage area information 4125 of the sub-record SR.

[0126] If the upload data received by the server communication control unit 401 is the second upload data, the server processing unit 402 identifies a record R having the detection device ID 212 and the marker ID 4123 included in the second upload data from the item management DB 412. Next, the server processing unit 402 deletes the sub-record SR having the marker ID 4123 included in the second upload data from the identified record R.

[0127] [2-3. Actions related to product name registration] Next, the operation of the item management system 1000 related to the registration of item names will be described. FIG. 15 is a flowchart showing the operation of the article management system 1000. In FIG. 15, a flowchart FD shows the operation of the article management server 4, and a flowchart FE shows the operation of the terminal device 3.

[0128] As shown in the flowchart FD, the server communication control unit 401 of the item management server 4 determines whether a new sub-record SR has been stored in the item management DB 412 (step SD1).

[0129] If the server communication control unit 401 of the article management server 4 determines that the sub-record SR has been stored in the article management DB 412 (step SD1: YES), it transmits input screen data to the terminal device 3 (step SD2). The input screen data is data for displaying the input screen NG in the application UI. The input screen NG is a screen for accepting input of the item name, which is the name of the item BP.

[0130] Step SD2 will now be described. The server communication control unit 401 generates display data including the marker ID 4123 and the item image data 4127 included in the newly stored sub-record SR. Next, the server communication control unit 401 transmits the generated input screen data to the terminal device 3 based on the communication information 4122 included in the record R to which the sub-record SR has been added.

[0131] As shown in a flowchart FE, the application execution unit 300 of the terminal device 3 receives input screen data from the article management server 4 (step SE1).

[0132] Next, the application executing unit 300 causes the touch panel 33 to display the application UI 330 displaying the input screen NG (step SE2).

[0133] FIG. 16 is a diagram showing an example of the input screen NG. The input screen NG displays an item image BPG. The item image BPG displayed on the input screen NG is the item image BPG indicated by the item image data 4127 included in the new item notification information.

[0134] The input screen NG displays an input field NL for inputting an item name. The item name is input into the input field NL displayed on the input screen NG by the user P of the terminal device 3. The input screen NG in Fig. 16 illustrates an example in which the item name "boiled egg" is input into the input field NL.

[0135] The input screen NG displays a Confirm button B1, which is a software button for confirming the content entered in the input field NL.

[0136] Returning to the explanation of the flowchart in Fig. 15, when the user P inputs an item name (step SX1) and operates the confirm button B1 (step SC3), the application execution unit 300 transmits update information to the item management server 4 (step SE3). The update information transmitted in step SE3 includes item name information indicating the item name input on the input screen NG and the marker ID 4123 included in the input screen data.

[0137] As shown in the flowchart FD, the server communication control unit 401 of the article management server 4 receives update information from the terminal device 3 (step SD3).

[0138] Next, the server processing unit 402 updates the product management DB 412 based on the update information received in step SD3 (step SD4).

[0139] Step SD4 will now be described in detail. The server processing unit 402 identifies the sub-record SR having the marker ID included in the update information from the item management DB 412. Next, the server processing unit 402 updates the item name information 4124 of the identified sub-record SR to the item name information 4124 included in the update information.

[0140] [3. Effects, etc.] As described above, the detection system 100 of this embodiment includes the marker MK attached to the item BP and the detection device 2 that detects the marker MK. The detection device 2 includes a refrigerator camera 27, a detection unit 206, and an entry / exit determination unit 207.

[0141] The refrigerator compartment camera 27 photographs an area including the interior of the refrigerator 1 from above the interior when the door 11C of the refrigerator 1 is in an open state. The detection unit 206 detects an image of the marker MK from the image captured by the refrigerator compartment camera 27. Based on the image of the marker MK detected by the detection unit 206, the inbound / outbound determination unit 207 determines whether the item BP marked with the marker MK has entered the warehouse or whether the item BP has been taken out of the warehouse.

[0142] Therefore, by detecting the image of the marker MK attached to the item BP, it is possible to determine whether the item BP has entered or left the warehouse. For example, by attaching a different marker MK to each item BP, it is possible to improve the accuracy of determining the item BP and more reliably obtain information about the item BP entering or leaving the refrigerator.

[0143] The detection unit 206 detects the markers MK within a preset detection area TA of the captured image.

[0144] Therefore, the detection time of the marker MK can be shortened, and the processing load of the detection device 2 can be reduced.

[0145] After detecting the marker MK within the detection area TA, the in-stock / out-stock determining unit 207 determines whether the item BP has entered the warehouse or been taken out from the warehouse based on the movement direction of the marker MK.

[0146] Therefore, it is possible to determine whether the item BP with the marker MK has been stored in the refrigerator 1 or has been removed from the refrigerator.

[0147] After detecting the marker MK within the detection area TA, the inventory entry / exit determination unit 207 determines that the item BP marked with the marker MK has entered the inventory if it detects that the marker MK has moved into the area of ​​the captured image corresponding to the door pocket 111 of the door 11C.

[0148] Therefore, it is possible to accurately detect whether an article BP with a marker MK is placed in the door pocket 111.

[0149] After detecting the marker MK in the detection area TA, the inventory entry / exit determination unit 207 determines the storage area in which the item BP is stored based on the movement direction of the marker MK and the arrangement of the multiple storage areas provided within the warehouse.

[0150] Therefore, it is possible to determine in which of the storage areas provided inside the refrigerator 1 the item BP marked with the marker MK has been stored.

[0151] The detector 2 comprises a detector memory 21 for storing communication information 214 . The marker MK includes a rectangular background area MKA and a pattern image MKB formed within the background area MKA and corresponding to the identification information assigned to the marker. The entry / exit determination unit 207 calculates the area of ​​the background area MKA captured in the captured image in the order in which the captured images were taken, and when no change is detected in the area of ​​the calculated background area MKA, it cuts out the range of the captured image determined by the image of the marker and transmits the cut-out image to the destination indicated by the communication information 214.

[0152] Therefore, an image of the stationary item BP can be extracted and sent to the destination of the communication information 214. For this reason, for example, by registering the IP address of the item management server 4 or the email address of the user P who owns the terminal device 3 as the communication information 214, it is possible to check the image of the item BP stored in the refrigerator 1.

[0153] The detection device 2 includes an acquisition unit 208 that acquires identification information corresponding to the pattern image MKB detected by the detection unit 206. If the acquisition unit 208 acquires the same identification information within a preset time, the inventory / delivery determination unit 207 determines that the item BP with the detected marker MK has entered the warehouse again.

[0154] Therefore, it is possible to determine whether an item BP taken out of the refrigerator 1 is re-stored in the refrigerator.

[0155] In addition, if the identification information acquired by the acquisition unit 208 is identical to the identification information detected from the marker MK attached to the item BP determined to have been released from the warehouse, the inventory / return determination unit 207 determines that the item BP corresponding to the identification information has been re-entered into the warehouse.

[0156] Therefore, it is possible to determine whether an item BP taken out of the refrigerator 1 is re-stored in the refrigerator.

[0157] (Other embodiments) As described above, the above-mentioned first embodiment has been described as an example disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above-mentioned first embodiment to create new embodiments. Therefore, other embodiments will be exemplified below.

[0158] For example, when the detection device 2 cuts out the cut-out range KH from the recorded image 215 to generate the article image BPG, the detection device 2 may transmit the generated article image BPG to the email address of the user P as the communication information 214. This allows the article image BPG to be transmitted to the terminal device 3 without going through the article management server 4, thereby shortening the time until the article image BPG is displayed on the terminal device 3.

[0159] Furthermore, the item management server 4 may also manage other items such as the remaining amount of item BP, the expiration date of item BP, and the best-before date of item BP, in addition to the storage start date and time. The item management server 4 sets a notification date to be notified to the user P based on the expiration date. The notification date may be, for example, the same day as the expiration date or the day before the expiration date. Alternatively, the user P may operate the terminal device 3 to set a notification date for each item BP. If the food product BP has not been removed from the refrigerator 1 by the set notification date, the product management server 4 notifies the detection device 2 or the terminal device 3 that there is a product BP whose notification date has arrived.

[0160] In the above-described embodiment, the person presence determination unit 202 determines whether or not a person is present around the refrigerator 1 based on the detection value of the human presence sensor 26. However, the person presence determination unit 202 may make its determination based on the detection value of an illuminance sensor or an electrostatic sensor provided on the grip of the refrigerator 1 instead of the human presence sensor 26. In the case of this electrostatic sensor, the refrigerator 1 and the detection device 2 are capable of communicating, and the refrigerator 1 transmits the detection value of the electrostatic sensor to the detection device 2.

[0161] In the above-described embodiment, the entry and exit of the item BP into and from the refrigerator 1 is determined, but it is also possible to determine from the recorded image 215 whether the item BP has been stored in the fresh freezing compartment 13, the freezer compartment 14, the vegetable compartment 15, etc. Furthermore, by changing the installation position of the cameras and the number of cameras installed, it is also possible to improve the accuracy of determining whether the item BP has been stored in the fresh freezing compartment 13, the freezer compartment 14, or the vegetable compartment 15.

[0162] In another embodiment, the record R may be stored in the terminal device 3. In this configuration, the application execution unit 300 appropriately updates the contents of the record R stored in the terminal device 3 in the same manner as the above-described item management server 4. This eliminates the need for the item management system 1000 to include the item management server 4, simplifying the system configuration.

[0163] In other embodiments, the types of compartments formed in main box 10 of refrigerator 1 may include storage compartments other than refrigerator compartment 11, ice making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15, or at least one of ice making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15 may be absent. Also, in other embodiments, the number of doors provided at the opening on the front of refrigerator compartment 11 may be one.

[0164] For example, the detection device processor 20, the terminal processor 30, and the server processor 40 may be configured by a single processor or multiple processors. The detection device processor 20, the terminal processor 30, and the server processor 40 may be hardware programmed to realize corresponding functional units. That is, the detection device processor 20, the terminal processor 30, and the server processor 40 may be configured by, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0165] The configuration of the detection device 2 shown in FIG. 4 and the configurations of the terminal device 3 and the item management server 4 shown in FIG. 6 are merely examples, and the specific implementation form is not particularly limited. In other words, it is not necessarily necessary to implement hardware corresponding to each unit individually, and it is of course possible to configure the functions of each unit by having a single processor execute a program. Furthermore, some of the functions realized by software in the above-described embodiments may be implemented as hardware, or some of the functions realized by hardware may be implemented as software. In addition, the specific detailed configurations of each of the other units of the detection device 2, the terminal device 3, and the item management server 4 may also be changed as desired without departing from the spirit of the present disclosure.

[0166] The step units of the operations shown in Figures 8, 14, and 15 are divided according to the main processing content to make the operations easier to understand, and the operation is not limited by the way the processing units are divided or the names of the processing units. The operations may be divided into more step units depending on the processing content. Furthermore, one step unit may be divided so that it includes more processing. Furthermore, the order of the steps may be changed as appropriate within the scope that does not interfere with the purpose of this disclosure.

[0167] The control program 211 executed by the detection device processor 20 can also be realized in a state in which the control program 211 is recorded on a portable information recording medium. Examples of the information recording medium include magnetic recording media such as hard disks, optical recording media such as CDs, and semiconductor storage devices such as USB (Universal Serial Bus) memories and SSDs (Solid State Drives), but other recording media can also be used.

[0168] It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents. [Industrial Applicability]

[0169] The detection device, the control method for the detection device, and the program of the present disclosure can be used to detect items entering and leaving a refrigerator. [Explanation of symbols]

[0170] 1 refrigerator 2. Detection device 5. Communications equipment 11 Refrigerator 11C Door 20 Detector Processor 27 Refrigerator Camera 29 Speaker 202 Person presence determination unit 203 Open / close determination unit 204 Shooting control unit 206 Detection unit 207 Incoming / Outgoing Determination Unit 208 Notification Department 211 Control Program BP Goods MK Marker MKA background area MKB pattern image TA detection area

Claims

1. A detection system including: a marker image attached to at least one of a food item stored in a refrigerator and a container for storing the food; and a detection device for detecting the marker image, The detection device includes: an imaging unit that images an area including at least a part of the interior of the refrigerator from above the interior when the door of the refrigerator is in an open state; a detection unit that performs image analysis on an image of a range corresponding to a detection area provided in front of a chilled compartment of the refrigerator, which is a bottom surface of a refrigerating compartment of the refrigerator, among the captured images captured by the photographing unit, and detects the marker image attached to the item; a determination unit that determines whether the item to which the marker image is attached has been stored in the storage compartment or whether the item has been removed from the storage compartment based on a trajectory of movement of the marker image detected from an image in a range corresponding to the detection area; Equipped with the marker image includes a rectangular background image and a pattern image that is superimposed on the background image and includes identification information that identifies the marker image; the detection unit determines a range to be cut out from the photographed image by adding a preset setting value to four vertices of the background image detected from the photographed image, and cuts out an image of the determined range from the photographed image; The determination unit determines whether the item has entered the warehouse or whether the item has left the warehouse based on a trajectory of the marker image included in the image of the range cut out by the detection unit.

2. the determination unit detects an edge of a door pocket of the refrigerator from the captured image, and identifies a storage area corresponding to the door pocket based on the detected edge; The detection system of claim 1, wherein the marker image is detected from an image in a range corresponding to the detection area, and when movement of the detected marker image to the storage area is detected, it is determined that the item has been stored in the storage area of ​​the door pocket.

3. the determination unit detects edges of left and right door pockets of the refrigerator from the captured image, sets a second area based on the detected edge of the left door pocket, sets a third area based on the detected edge of the right door pocket, sets a predetermined area corresponding to a refrigerating compartment of the refrigerator as a first area, and sets an area other than the first area, the second area, and the third area as a fourth area; 2. The detection system according to claim 1, wherein the marker image is detected from an image in a range corresponding to the detection area, and when movement of the detected marker image to the fourth area is detected, it is determined that the item has been removed from the refrigerator.

4. The detection system according to claim 3 , wherein the determination unit detects the marker image from an image in a range corresponding to the detection area, and determines that the item has been placed in the first area when movement of the detected marker image to the first area is detected.

5. the detection device includes a storage unit that stores communication information; The detection system according to claim 1 , wherein the determination unit transmits the image cut out from the captured image by the detection unit to a destination indicated by the communication information.

6. an acquisition unit that acquires identification information corresponding to the pattern image detected by the detection unit, 6. The detection system according to claim 5, wherein the determination unit determines that the item with the detected marker image has been stored in the warehouse again if the acquisition unit acquires the same identification information within the preset time period.

7. an acquisition unit that acquires identification information corresponding to the pattern image detected by the detection unit, 6. The detection system according to claim 5, wherein the determination unit determines that the item corresponding to the identification information has been re-entered into the warehouse when the identification information acquired by the acquisition unit is identical to the identification information detected from the marker image attached to the item determined to have been released.

8. A computer mounted on a detection device that detects a marker image attached to at least one of a food item stored in a refrigerator and a container that stores the food item, A process of acquiring a photographed image of an area including at least a part of the interior of the refrigerator from above the interior of the refrigerator when the door of the refrigerator is in an open state, by an imaging unit; a process of performing image analysis on a range of the acquired photographed image, which corresponds to a detection area provided in front of a chilled compartment on a bottom surface of a refrigerating compartment of the refrigerator, as a detection area, and detecting the marker image attached to the item; and executing a process of determining whether the item to which the marker image is attached has been stored in the warehouse or whether the item has been removed from the warehouse, based on a trajectory of movement of the marker image detected from an image in a range corresponding to the detection area; the marker image includes a rectangular background image and a pattern image that is superimposed on the background image and includes identification information that identifies the marker image; The process of detecting includes adding a preset value to four vertices of the background image detected from the photographed image to determine a range to be cut out from the photographed image, and cutting out an image of the determined range from the photographed image; The determination process determines whether the item has entered the warehouse or whether the item has left the warehouse based on a trajectory of the marker image included in the image of the range extracted by the detection process.

9. A processor mounted on a detection device that detects a marker image attached to at least one of a food item stored in a refrigerator and a container that stores the food item, a step of acquiring a photographed image of an area including at least a part of the interior of the refrigerator from above the interior of the refrigerator when the door of the refrigerator is in an open state; a step of performing image analysis on a range of the acquired photographed image, which corresponds to a detection area provided in front of a chilled compartment on a bottom surface of a refrigerating compartment of the refrigerator, as a detection area, to detect the marker image attached to the item; and determining whether the item to which the marker image is attached has been stored in the warehouse or whether the item has been removed from the warehouse, based on a trajectory of movement of the marker image detected from an image in a range corresponding to the detection area; the marker image includes a rectangular background image and a pattern image that is superimposed on the background image and includes identification information that identifies the marker image; the detecting step includes adding a preset value to four vertices of the background image detected from the photographed image to determine a range to be cut out from the photographed image, and cutting out an image of the determined range from the photographed image; The determining step determines whether the item has entered the warehouse or whether the item has left the warehouse based on a trajectory of the marker image included in the image of the range extracted by the detection process.

Images