Information processing system and imaging apparatus

By introducing a registration and selection unit in the information processing system, using the ranging sensor and the camera control circuit board, the difficulty in selecting images caused by different storage models in the prior art is solved, and the situation in the storage warehouse is realized. Accurately reflected.

CN118974497APending Publication Date: 2024-11-15PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380024365.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-02
Filing Date
2023-02-06
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The prior art cannot properly select and capture images according to the model of the storage library, resulting in the inability to effectively reflect the situation in the storage library.

Method used

An information processing system and a shooting device are designed to select appropriate shooting images from the images captured by the camera by registering the specified opening and closing body of the storage library, including a registration unit and a selection unit, and a distance measuring sensor and a camera are used. Control the circuit board to realize reference shooting of the storage library.

Benefits of technology

It can accurately select and reflect the shooting images independent of the model of the storage warehouse, improving the adaptability and accuracy of shooting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118974497A_ABST
    Figure CN118974497A_ABST
Patent Text Reader

Abstract

The invention provides an information processing system and an imaging device, which are capable of rotating a captured image in which the condition in a storage is appropriately reflected from an image captured by the imaging device without depending on the type of the storage captured by the imaging device. An information processing system includes an imaging device and a terminal device that communicates with the imaging device, the imaging device including: a camera that captures an image of a storage chamber having a box body in which a storage chamber is formed and an opening / closing body that opens / closes an opening of the storage chamber; a registration unit that registers a predetermined open state of the opening / closing body; and a selection unit that selects, from among images captured by the camera, a captured image when the opening / closing body is in an open state, based on the predetermined open state registered by the registration unit.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an information processing system and a photographing device. Background Art

[0002] Patent Document 1 discloses a technology in which a camera is installed on the upper part of a refrigerator, and when an object passing through an opening of the refrigerator is detected, the camera takes an image, and the food stored in the refrigerator is managed based on the imaged result.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-070476 Summary of the invention

[0006] Problems to be solved by the invention

[0007] The present invention provides an information processing system and an imaging device capable of selecting an image that appropriately reflects the internal conditions of a storage warehouse from images captured by a camera, regardless of the model of the storage warehouse captured by the imaging device.

[0008] Methods for solving problems

[0009] The information processing system of the present invention includes a photographing device and a terminal device that communicates with the photographing device, the photographing device including: a camera that photographs a storage warehouse, the storage warehouse having a box body that forms a storage room and an opening and closing body that opens and closes an opening of the storage room; a registration unit that registers a specified opening state of the opening and closing body; and a selection unit that selects a photographed image in which the opening and closing body is in an open state from images photographed by the camera based on the specified opening state registered by the registration unit.

[0010] In addition, the photographing device of the present invention includes: a camera for photographing a storage warehouse, wherein the storage warehouse has a box body forming a storage room and an opening and closing body that opens and closes an opening of the storage room; a registration unit for registering a specified opening state of the opening and closing body; and a selection unit, which selects a photographed image in which the opening and closing body is in an open state from images photographed by the camera based on the specified opening state registered by the registration unit.

[0011] In addition, this specification includes the entire contents of Japanese Patent Application No. 2022-031920 filed on March 2, 2022.

[0012] Effects of the Invention

[0013] The information processing system and the camera device of the present invention select the image captured when the opening and closing body is in the open state based on the registered specified open state, so that the captured image can be selected based on the storage captured by the camera. Therefore, it is possible to select a captured image that appropriately reflects the internal conditions of the storage from the images captured by the camera, regardless of the type of storage captured by the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a diagram showing the structure of the article management system in the first embodiment.

[0015] Figure 2 This is a perspective view of the refrigerator in Embodiment 1.

[0016] Figure 3 This is a plan view of the imaging device installed in the cold storage in Embodiment 1 as viewed from the right.

[0017] Figure 4 This is a perspective view of the restricting member in contact with the front end edge of the housing in Embodiment 1 as viewed from the left front.

[0018] Figure 5 This is a plan view of the cold storage and the imaging device in Embodiment 1 as viewed from the front.

[0019] Figure 6 This is a plan view of the cold storage and the imaging device in Embodiment 1 as viewed from the right.

[0020] Figure 7 This is a plan view of the imaging device installed in the cold storage in Embodiment 1 as viewed from above.

[0021] Figure 8 This is a plan view of the imaging device installed in the cold storage in Embodiment 1 as viewed from above.

[0022] Fig. 9 This is a block diagram showing the structure of the imaging device in Embodiment 1.

[0023] Fig.10 This is a block diagram showing the configuration of a terminal device and an item management server in the first embodiment.

[0024] Fig.11 This is a flowchart showing the operation of the information processing system in the first embodiment.

[0025] Fig.12 This is a flowchart showing the operation of the information processing system in the first embodiment.

[0026] Fig.13 This is a diagram showing an example of the first guide image in the first embodiment.

[0027] Fig.14 This is a flowchart showing the operation of the information processing system in the first embodiment.

[0028] Fig.15 This is a diagram showing an example of the second guide image in the first embodiment.

[0029] Fig.16 This is a flowchart showing the operation of the imaging device in the first embodiment.

[0030] Fig.17 This is a flowchart showing the operation of the article management system in the first embodiment.

[0031] Fig.18 This is a flowchart showing the operation of the article management system in the first embodiment.

[0032] Fig.19 This is a diagram showing the structure of the article management system in Implementation Example 2.

[0033] Fig. 20 It is a plan view showing the cold storage in Embodiment 2 when viewed from the right.

[0034] Fig.21 This is a block diagram showing the structure of a cold storage in the second embodiment.

[0035] Fig. 22 This is a flowchart showing the operation of the article management system in the second embodiment. DETAILED DESCRIPTION

[0036] (Knowledge and insights that form the basis of this disclosure, etc.)

[0037] When the inventor of the present invention thought of the present invention, there was a technology that performed shooting when an object passing through the opening of the cold storage was detected, and managed the food stored in the cold storage based on the shooting results. However, depending on the model of the storage room, the size of the opening and closing body that opens and closes the opening of the storage room, the range that the opening and closing body can open and close, etc. may be different. Therefore, it is desired that the basis for selecting the captured image is the record corresponding to the model of the storage room. However, in the prior art, this desire was not taken into account, and the inventor discovered a technical problem that a captured image that does not properly reflect the internal conditions of the storage room may be selected, and the subject of the present invention is formed to solve this technical problem.

[0038] Therefore, the present invention provides an information processing system and an imaging device capable of selecting an image that appropriately reflects the internal conditions of a storage warehouse from images captured by a camera, regardless of the model of the storage warehouse captured by the imaging device.

[0039] The following describes the implementation in detail with reference to the accompanying drawings. However, sometimes more detailed descriptions than necessary are omitted. For example, there are cases where detailed descriptions of already known matters or repeated descriptions of substantially the same structures are omitted.

[0040] In addition, the drawings and the following description are provided for those skilled in the art to fully understand the present invention, and are not intended to limit the subject matter described in the claims by these descriptions.

[0041] (Implementation Method 1)

[0042] [1-1. Structure]

[0043] [1-1-1. Structure of the item management system]

[0044] Figure 1 This is a diagram showing the structure of the article management system 1A in the first embodiment.

[0045] The article management system 1A is a system for managing articles stored in the refrigerator 2A. The article management system 1A of the present embodiment manages an image showing articles stored in the refrigerator 2A as an example of article management.

[0046] Refrigerator 2A corresponds to the "storage" of the present invention.

[0047] In this embodiment, the articles include food, food materials, containers for storing food, etc. A predetermined mark is given to the articles. As an example of the mark, a code such as a QR code (registered trademark) can be cited.

[0048] Article management system 1A includes cold storage 2A.

[0049] The cold storage 2A includes a box body 20 with an opening on the front surface. A cold storage room 21, an ice making room 22, a fresh freezing room 23, a freezing room 24 and a vegetable room 25 are formed in the box body 20. These storage rooms are cooled by cold air. An ice temperature preservation room 26 with a lower internal temperature than the cold storage room 21 is formed in the cold storage room 21. A drawer 26A capable of storing items is provided in the ice temperature preservation room 26. A revolving door 21A is provided at the opening of the front surface of the cold storage room 21. The door 21A is a double-door (opposite door) type door, including a revolving left door 21B and a revolving right door 21C. Drawers 22A, 23A, 24A, and 25A capable of storing items are respectively provided in the ice making room 22, the fresh freezing room 23, the freezing room 24 and the vegetable room 25. The drawers 22A to 26A are each opened on the upper surface when they are set.

[0050] Refrigerating room 21 corresponds to the "first storage room" of the present invention.

[0051] In the following description, when the refrigerating chamber 21, the ice making chamber 22, the fresh freezing chamber 23, the freezing chamber 24, and the vegetable chamber 25 are not distinguished, they are referred to as "storage chambers" and are given the reference numeral "27".

[0052] In the following description, when the ice making chamber 22, the fresh freezing chamber 23, the freezing chamber 24, the vegetable chamber 25, and the ice temperature preservation chamber 26 are not distinguished, they are referred to as "drawer chambers" and given the reference numeral "28". In the present embodiment, the drawer chamber 28 is equivalent to the "second storage chamber" of the present invention.

[0053] In the following description, when the drawers 22A, 23A, 24A, 25A, and 26A are not distinguished, they are referred to as "drawer 28A".

[0054] In the following description, when the door 21A and the drawer 28A are not distinguished, they are referred to as an "opening and closing body" and are given the reference numeral "29A".

[0055] Article management system 1A includes imaging device 3. Imaging device 3 is provided on upper surface 20A of housing 20 of refrigerator 2A. Imaging device 3 captures an image of refrigerator 2A.

[0056] Article management system 1A includes terminal device 4. Terminal device 4 is a portable computer such as a smartphone. Terminal device 4 has an application program for managing articles stored in refrigerator 2A installed therein, and communicates with article management server 5 using the function of the application program.

[0057] In the following description, this application is referred to as an "item management application" and is given the reference number "411".

[0058] exist Figure 1 In FIG. 1 , a user P at home is indicated by a solid line, and a user P outside the residence H is indicated by a dotted line. When the terminal device 4 is used by the user P at home, it communicates with the item management server 5 via the communication device 6 or without the communication device 6. In addition, when the terminal device 4 is used by the user P outside the residence H and cannot establish a communication connection with the communication device 6, it communicates with the item management server 5 without the communication device 6.

[0059] The communication device 6 is connected to a network NW composed of a communication line such as a public line network or a dedicated line, and communicates with the article management server 5 via the network NW. The communication device 6 functions as an interface device for connecting the cold storage 2A, the imaging device 3, and the terminal device 4 to the network NW. The communication device 6 constructs a local area network in the residence H.

[0060] The article management system 1A includes an article management server 5. The article management server 5 is a device for managing articles stored in the cold storage 2A. The article management server 5 stores images showing the articles as management of the articles. The article management server 5 is connected to the network NW. In addition, in each figure, the article management server 5 is represented by a single frame, but this does not mean that the article management server 5 is composed of a single device.

[0061] The article management system 1A includes an information processing system 7A. The information processing system 7A includes an image pickup device 3 and a terminal device 4, and performs information processing through the image pickup device 3 and the terminal device 4.

[0062] [1-1-2. Structure of cold storage]

[0063] Figure 2 It is a perspective view of refrigerator 2A in an installed state.

[0064] exist Figure 2 , the X-axis, Y-axis, and Z-axis are illustrated. The X-axis, Y-axis, and Z-axis are orthogonal to each other. The Z-axis represents the up-down direction, which is equivalent to the height direction of the cold storage 2A. The X-axis and the Y-axis are parallel to the horizontal direction. The X-axis represents the left-right direction. The left-right direction is equivalent to the width direction of the cold storage 2A. The Y-axis represents the front-back direction. The positive direction of the X-axis represents the right direction. The positive direction of the Y-axis represents the front direction. The positive direction of the Z-axis represents the upper direction.

[0065] Refrigerator 2A includes housing 20, door 21A, and drawer 28A.

[0066] The box body 20 is a box-shaped component that forms the main body of the cold storage 2A. The box body 20 includes: an outer box 201 with an open front surface; an inner box 202 located inside the outer box 201 and with an open front surface; and a heat insulating material filled in the space between the outer box 201 and the inner box 202. The outer box 201 is formed of, for example, a metal plate such as iron, a hard resin such as ABS, etc. The inner box 202 is formed of, for example, a hard resin such as ABS. The heat insulating material is, for example, hard polyurethane foam. The heat insulating material is not limited thereto, and any material can be used as long as it has heat insulating properties and can be filled in the space between the outer box 201 and the inner box 202.

[0067] The inner space of the inner box 202 is divided into a plurality of spaces by a partition member. In this embodiment, the inner space of the inner box 202 is divided into five spaces, which function as a refrigerating chamber 21, an ice making chamber 22, a fresh freezing chamber 23, a freezing chamber 24, and a vegetable chamber 25, respectively.

[0068] The left door 21B is rotatably supported by a first hinge member 203A provided on the upper surface 20A of the housing 20. The right door 21C is rotatably supported by a second hinge member 203B provided on the upper surface 20A of the housing 20.

[0069] When the refrigerator 2A is viewed from above, the left door 21B has a first notch 204A formed on the front surface of the end opposite to the end connected to the first hinge member 203A. The first notch 204A is formed to extend in the height direction of the refrigerator 2A. The first notch 204A functions as a handle for opening and closing the left door 21B.

[0070] When the refrigerator 2A is viewed from above, the right door 21C has a second notch 204B formed on the front surface of the end opposite to the end connected to the second hinge member 203B. The second notch 204B is formed to extend in the height direction of the refrigerator 2A. The second notch 204B functions as a handle for opening and closing the right door 21C.

[0071] In a plan view of the refrigerator 2A in the installed state, the first notch 204A and the second notch 204B form a recess 205 on the front surface of the closed door 21A. The recess 205 of this embodiment is formed substantially at the center of the refrigerator 2A in the width direction in a plan view of the refrigerator 2A in the installed state.

[0072] [1-1-3. Structure of the imaging device]

[0073] Reference Figure 3-Figure 8 , the photographing device 3 is described.

[0074] exist Figure 3-Figure 8 In the figure, the Figure 2 The X-axis, Y-axis, and Z-axis shown are the same as the X-axis, Y-axis, and Z-axis. The X-axis direction corresponds to the width direction of the imaging device 3. The Y-axis direction corresponds to the front-rear direction of the imaging device 3.

[0075] Figure 3 This is a plan view of the imaging device 3 installed in the cold storage 2A as seen from the right. Figure 4 This is a perspective view of the restricting member 303 in contact with the front end edge 20B of the housing 20 as viewed from the left front.

[0076] like Figure 3 and Figure 4 As shown, the photographing device 3 includes: a main body 301 disposed on the upper surface 20A of the housing 20; and a photographing member 302 extending forward from the main body 301. The photographing member 302 is disposed on the upper front portion of the main body 301.

[0077] The main body 301 includes a limiting member 303 that limits the position of the photographing member 302 in the front-to-back direction of the cold storage 2A by abutting against the front edge 20B of the box body 20. The limiting member 303 is a piece that extends downward from the bottom surface 301A of the main body 301 in the set state. The limiting member 303 extends in the width direction of the main body 301 (the width direction of the photographing device 3) on the front surface of the main body 301, and has a mark 304 at the center of the width direction of the main body 301. The mark 304 indicates the center of the width direction of the photographing device 3.

[0078] The restricting member 303 is formed to be shorter than the length from the upper surface 20A of the housing 20 to a door seal (hereinafter referred to as an “upper door seal”) provided substantially parallel to the upper surface 20A on the inner surface upper portion of the door 21A in the installed state.

[0079] Figure 5 This is a plan view of the cold storage 2A and the imaging device 3 installed in the cold storage 2A as seen from the front. Figure 5 In the illustrated refrigerator 2A, left door 21B and right door 21C are in an open state. Figure 6 This is a plan view of the cold storage 2A and the imaging device 3 installed in the cold storage 2A as seen from the right. Figure 6 Drawer 25A of refrigerator 2A is shown in an open state, that is, in a pulled-out state.

[0080] The imaging device 3 includes a wide-angle camera 305 and a narrow-angle camera 306. The wide-angle camera 305 and the narrow-angle camera 306 are provided at the front end portion of the bottom surface 302A of the imaging member 302.

[0081] In the following description, when the wide-angle camera 305 and the narrow-angle camera 306 are not distinguished, they are referred to as “cameras” and are given the reference numeral “ 307 ”.

[0082] When the door 21A is closed, the camera 307 of the imaging device 3 is installed in front of the door 21A.

[0083] The wide-angle camera 305 corresponds to the "first camera" of the present invention. The narrow-angle camera 306 corresponds to the "second camera" of the present invention.

[0084] Wide-angle camera 305 is a camera having a wider angle than narrow-angle camera 306. In a state where imaging device 3 is installed in refrigerator 2A, wide-angle camera 305 captures refrigerating room 21 from the upper front of refrigerator 2A.

[0085] For example, the shooting range of the wide-angle camera 305 includes the following when viewed from the front of the cold storage 2A: Figure 5The range A1 is the range including the door shelves 211 of the left door 21B and the right door 21C and the opening of the refrigerator 21 when the left door 21B and the right door 21C are opened to the maximum degree. In addition, the shooting range of the wide-angle camera 305 includes the range of the door shelves 211 of the left door 21B and the right door 21C and the opening of the refrigerator 21 when the left door 21B and the right door 21C are opened to the maximum degree. Figure 6 Range A3 is a range including the front end of the left door 21B and the right door 21C in the open state to the front end of the shelf 212 in the front-rear direction, and includes the opening of the refrigerator compartment 21 in the top-bottom direction.

[0086] Narrow-angle camera 306 is a camera having a narrower angle than wide-angle camera 305. Narrow-angle camera 306 captures drawer 28A from the upper front of housing 20 when imaging device 3 is installed in refrigerator 2A.

[0087] For example, the shooting range of the narrow-angle camera 306 includes the following when viewed from the front of the cold storage 2A: Figure 5 Range A2 is a range including drawer 28A when viewed from the front of refrigerator 2A. For example, the shooting range of narrow-angle camera 306 includes drawer 28A when viewed from the right of refrigerator 2A. Figure 6 The range A4 is a range including the drawer 28A in the state of being pulled out to the maximum extent in the front-rear direction.

[0088] The imaging device 3 includes a left door distance measuring sensor 308, a right door distance measuring sensor 309, a drawer distance measuring sensor 310, and a human sensing sensor 311. The left door distance measuring sensor 308 and the right door distance measuring sensor 309 are provided on the main body 301. The drawer distance measuring sensor 310 and the human sensing sensor 311 are provided at the front end of the bottom surface 302A of the imaging component 302. The left door distance measuring sensor 308, the right door distance measuring sensor 309, and the drawer distance measuring sensor 310 in this embodiment are exemplified as optical sensors, but the sensing method is not limited.

[0089] In the following description, when the left door distance measuring sensor 308 and the right door distance measuring sensor 309 are not distinguished, they are referred to as “distance measuring sensors” and are given the reference numeral “ 312 ”.

[0090] The left door distance measuring sensor 308 detects the distance between the left door 21B and itself, and outputs a detection value indicating the detected distance. The sensing object of the left door distance measuring sensor 308 is the portion of the left door 21B above the upper door seal.

[0091] The right door distance measuring sensor 309 detects the distance between the right door 21C and itself, and outputs a detection value indicating the detected distance. The right door distance measuring sensor 309 senses a portion of the right door 21C above the upper door seal.

[0092] The drawer distance measuring sensor 310 detects the distance between the drawer 28A and itself, and outputs a detection value indicating the detected distance.

[0093] The human sensing sensor 311 may be an infrared sensor or an ultrasonic sensor.

[0094] The arrangement of the camera 307 will be described.

[0095] Figure 7 This is a plan view when imaging device 3 installed in cold storage 2A is viewed from above.

[0096] like Figure 7 As shown, the wide-angle camera 305, the narrow-angle camera 306, the drawer distance sensor 310, and the human sensing sensor 311 are arranged in a row in the width direction of the camera 3 at the front end portion of the bottom surface 302A of the camera member 302. The wide-angle camera 305 and the narrow-angle camera 306 are arranged on the central side of the drawer distance sensor 310 and the human sensing sensor 311 in the width direction of the camera 3. The length of the camera 3 in the width direction in this embodiment is shorter than the length of the recess 205 in the left-right direction. Therefore, when the camera 3 is arranged in the refrigerator 2A in a state where the mark 304 is located in the approximate center in the width direction of the refrigerator 2A, the wide-angle camera 305 and the narrow-angle camera 306 are located at a position overlapping the recess 205 in the width direction of the refrigerator 2A. In addition, in this case, the drawer distance sensor 310 and the human sensing sensor 311 are also located at a position overlapping the recess 205 in the width direction of the refrigerator 2A.

[0097] Furthermore, the length of the imaging device 3 in the width direction may be greater than the length of the recess 205. In this configuration, the wide-angle camera 305 and the narrow-angle camera 306 are provided at a distance shorter than the length of the recess 205 in the left-right direction at the front end of the bottom surface 302A.

[0098] The arrangement of the distance measuring sensor 312 will be described.

[0099] Figure 8 This is a plan view when imaging device 3 installed in cold storage 2A is viewed from above.

[0100] The distance measuring sensor 312 is provided at the front lower portion of the main body 301 .

[0101] The left door distance measuring sensor 308 is provided so that the irradiation surface for irradiating light and the light receiving surface for receiving reflected light face the left front of the camera 3. That is, the left door distance measuring sensor 308 is provided on the main body 301 so that the sensing direction faces the first hinge member 203A side in the width direction of the camera 3, in other words, the rotation axis side of the left door 21B, when the camera 3 is provided in the cold storage 2A.

[0102] Specifically, the left door distance measuring sensor 308 is installed in the main body 301 in such a manner that when the camera 3 is installed in the cold storage 2A and the camera 3 is viewed from above, the optical axis AX1 indicating the sensing direction intersects the inner surface 21B1 of the left door 21B opened at the maximum opening. The installation angle is appropriately determined by prior testing, simulation, etc.

[0103] In addition, the left door distance measuring sensor 308 is arranged so that the irradiation surface of the irradiated light is closer to the left door 21B side than the light receiving surface of the reflected light. That is, the left door distance measuring sensor 308 is arranged so that the irradiation surface is closer to the positive direction of the Y axis (the front side when the cold storage is installed) than the light receiving surface. As a result, light can easily irradiate the surface of the left door 21B, and the detection range of the left door distance measuring sensor 308 can be wide.

[0104] The right door distance measuring sensor 309 is provided so that the irradiation surface and the light receiving surface face the right front of the camera 3. That is, the right door distance measuring sensor 309 is provided on the main body 301 so that the sensing direction faces the second hinge member 203B side in the width direction of the camera 3, in other words, the rotation axis side of the right door 21C, when the camera 3 is provided in the cold storage 2A.

[0105] Specifically, the right door distance measuring sensor 309 is installed in the main body 301 in such a manner that when the camera 3 is installed in the cold storage 2A and viewed from above, the optical axis AX2 indicating the sensing direction intersects the inner surface 21C1 of the right door 21C opened at the maximum opening. The installation angle is appropriately determined by prior testing, simulation, etc.

[0106] In addition, the right door distance measuring sensor 309 is arranged so that the irradiation surface of the irradiated light is closer to the right door 21C side than the light receiving surface of the reflected light. That is, the right door distance measuring sensor 309 is arranged so that the irradiation surface is located in the positive direction of the Y axis (the front side when the cold storage is installed) than the light receiving surface. As a result, light can easily irradiate the surface of the right door 21C, and the detection range of the right door distance measuring sensor 309 can be wide.

[0107] Fig. 9 It is a block diagram showing the structure of the imaging device 3.

[0108] like Fig. 9As shown, the photographing device 3 includes a sensor control circuit board 313 , a distance measuring sensor for a left door 308 , a distance measuring sensor for a right door 309 , a distance measuring sensor for a drawer 310 , and a human sensing sensor 311 .

[0109] The sensor control circuit board 313 includes a first processor 320 such as a CPU (Central Processing Unit), a first memory 330 , a first communication module 340 , and an interface circuit for connecting to other devices and sensors. The sensor control circuit board 313 is provided on the bottom surface 301A of the main body 301 .

[0110] The first memory 330 is a storage device that non-volatilely stores programs and data executed by the first processor 320. The first memory 330 is composed of a non-volatile storage device. In addition, the first memory 330 may also include a RAM (Random Access Memory) that constitutes a work area of ​​the first processor 320. The first memory 330 stores data processed by the first processor 320 and a control program 331 executed by the first processor 320.

[0111] The first communication module 340 includes communication hardware such as a communication line and a connector that conform to a prescribed wired communication standard, and communicates with the camera control circuit board 314 via a cable.

[0112] The left door distance measuring sensor 308, the right door distance measuring sensor 309, the drawer distance measuring sensor 310, and the human sensing sensor 311 are connected to the sensor control circuit board 313. These sensors output detection values ​​to the sensor control circuit board 313, respectively.

[0113] The first processor 320 functions as a first communication control unit 321 by reading and executing the control program 331 stored in the first memory 330 .

[0114] The first communication control unit 321 communicates with the camera control circuit board 314 via the first communication module 340. The first communication control unit 321 transmits the detection values ​​output by each sensor connected to the sensor control circuit board 313 to the camera control circuit board 314.

[0115] The imaging device 3 includes a camera control circuit board 314 , a third communication module 315 , a fourth communication module 316 , a wide-angle camera 305 , a narrow-angle camera 306 , a first LED (Light Emitting Diode) 317 , and a second LED 318 .

[0116] The camera control circuit board 314 includes a second processor 350 such as a CPU, a second memory 370 , a second communication module 380 , and an interface circuit for connecting to other devices and sensors. The camera control circuit board 314 is provided on the bottom surface 301A of the main body 301 .

[0117] The second memory 370 is a storage device that stores programs and data executed by the second processor 350 in a nonvolatile manner. The second memory 370 is composed of a nonvolatile storage device. In addition, the second memory 370 may also include a RAM that constitutes a work area of ​​the second processor 350. The second memory 370 stores data processed by the second processor 350, a control program 371 executed by the second processor 350, an image capture device ID (Identification) 372 that uniquely identifies the image capture device 3, and setting data 373.

[0118] The setting data 373 is data related to the setting of the imaging device 3 , and includes first registration data 373A, second registration data 373B, and third registration data 373C.

[0119] The first registration data 373A is data in which the prescribed open state of the door 21A is registered. The so-called prescribed open state is an open state that serves as a reference in the selection of the selection unit 358 described later. In the present embodiment, the so-called prescribed open state of the door 21A is a state in which the left door 21B is opened at 90°, and a state in which the right door 21C is opened at 90°. In the first registration data 373A, the detection value of the distance sensor 312 when the door 21A is in the prescribed open state is recorded. In addition, in the first registration data 373A, the detection value of the distance sensor 312 when the door 21A is in the closed state is recorded. Among them, regarding the angle of the left door 21B, it is set to 0° when the left door 21B is in the closed state, and the rotation direction of the left door 21B in the open state is set to be positive. In addition, regarding the angle of the right door 21C, it is set to 0° when the right door 21C is in the closed state, and the rotation direction of the right door 21C in the open state is set to be positive.

[0120] The second registration data 373B is data for registering the specified open state of the drawer 28A. In the present embodiment, the so-called specified open state of the drawer 28A is a state in which the drawer 28A is pulled out to the maximum extent. The position of the front edge of the drawer 28A when the drawer 28A is in the specified open state is recorded in the second registration data 373B.

[0121] The third registration data 373C is data recording the SSID and password of the local area network constructed by the communication device 6.

[0122] The second communication module 380 includes communication hardware such as a communication line and a connector that comply with a prescribed communication standard, and communicates with the sensor control circuit board 313 under the control of the second processor 350 .

[0123] The camera control circuit board 314 is connected to the third communication module 315 , the fourth communication module 316 , the wide-angle camera 305 , the narrow-angle camera 306 , the first LED 317 , and the second LED 318 .

[0124] The third communication module 315 includes a communication line and a connector that conform to a predetermined communication standard, and communicates with the article management server 5 via the communication device 6 and the network NW. The third communication module 315 is provided on the bottom surface 302A of the imaging unit 302.

[0125] The fourth communication module 316 includes a communication line and a connector that conform to a prescribed short-range wireless communication standard, and communicates with the terminal device 4. The fourth communication module 316 may also include a communication line and a connector that conform to a prescribed communication standard, and communicate with the terminal device 4 via the communication device 6 and the network NW. The fourth communication module 316 is provided on the bottom surface 302A of the imaging component 302.

[0126] The third communication module 315 and the fourth communication module 316 are provided at positions spaced apart from each other by a predetermined distance or more. This can suppress interference of radio waves between the communication modules. The predetermined distance is, for example, 100 mm.

[0127] In addition, the third communication module 315 and the fourth communication module 316 may be configured to communicate with the terminal device 4 or the item management server 5 as one communication module.

[0128] The wide-angle camera 305 and the narrow-angle camera 306 each output the shooting result to the camera control circuit board 314. The shooting result is a dynamic image. The shooting result is not limited to a dynamic image, but can also be an image. In addition, it can also be a plurality of images shot continuously. In addition, a dynamic image can also include the concept of a plurality of still images.

[0129] The second processor 350 functions as the second communication control unit 351, the third communication control unit 352, the fourth communication control unit 353, the person presence judgment unit 354, the opening and closing judgment unit 355, the shooting control unit 356, the recording control unit 357, the selection unit 358, the generation unit 359, and the registration unit 360 by reading and executing the control program 371 stored in the second memory 370.

[0130] The second communication control unit 351 communicates with the sensor control circuit board 313 via the second communication module 380. The second communication control unit 351 receives the detection values ​​of various sensors connected to the sensor control circuit board 313 from the sensor control circuit board 313. The second communication control unit 351 outputs the detection value of the human sensing sensor 311 received from the sensor control circuit board 313 to the person presence determination unit 354. The second communication control unit 351 outputs the detection values ​​of each distance measuring sensor received from the sensor control circuit board 313 to the opening and closing determination unit 355, the video recording control unit 357, and the registration unit 360.

[0131] The third communication control unit 352 communicates with the item management server 5 via the third communication module 315 .

[0132] The fourth communication control unit 353 communicates with the terminal device 4 via the fourth communication module 316 .

[0133] The person presence determination unit 354 determines whether there is a person around the cold storage 2A based on the detection value of the human sensing sensor 311. For example, the person presence determination unit 354 determines that there is a person around the cold storage 2A when the detection value of the human sensing sensor 311 is greater than a specified value, and determines that there is no person around the cold storage 2A when the detection value of the human sensing sensor 311 is less than the specified value.

[0134] The opening and closing determination unit 355 determines whether the left door 21B is in an open state or a closed state based on the first registration data 373A and the detection value of the left door distance measuring sensor 308. When the left door distance measuring sensor 308 outputs a detection value different from the detection value corresponding to the closed state, the opening and closing determination unit 355 determines that the left door 21B is in an open state. When the left door distance measuring sensor 308 outputs a detection value corresponding to the closed state, the opening and closing determination unit 355 determines that the left door 21B is in a closed state.

[0135] The opening and closing determination unit 355 determines whether the right door 21C is in an open state or a closed state based on the first registration data 373A and the detection value of the right door distance measuring sensor 309. The opening and closing determination unit 355 determines that the right door 21C is in an open state when the right door distance measuring sensor 309 outputs a detection value different from the detection value corresponding to the closed state. The opening and closing determination unit 355 determines that the right door 21C is in a closed state when the right door distance measuring sensor 309 outputs a detection value corresponding to the closed state.

[0136] The opening and closing judgment unit 355 judges whether the drawer 28A is in an open state or a closed state according to the distance represented by the detection value of the drawer distance measuring sensor 310. In addition, the opening and closing judgment unit 355 can also judge whether the drawer 28A is in an open state or a closed state by performing image analysis on the dynamic image captured by the narrow-angle camera 306.

[0137] When the person presence determination unit 354 determines that there is a person around the cold storage 2A, the shooting control unit 356 causes the wide-angle camera 305 and the narrow-angle camera 306 to start shooting. When the wide-angle camera 305 and the narrow-angle camera 306 start shooting, the shooting control unit 356 turns on the first LED 317 to notify the user P that shooting has started. After the wide-angle camera 305 and the narrow-angle camera 306 start shooting, and when the person presence determination unit 354 determines that there is no person around the cold storage 2A, the shooting control unit 356 causes the wide-angle camera 305 and the narrow-angle camera 306 to stop shooting. When the wide-angle camera 305 and the narrow-angle camera 306 stop shooting, the shooting control unit 356 turns off the first LED 317.

[0138] When the opening / closing determination unit 355 determines that the opening / closing body 29A has become an open state, the recording control unit 357 starts recording the shooting results of the wide-angle camera 305 and the narrow-angle camera 306. Specifically, the recording control unit 357 starts recording the first recording data 374 indicating the shooting results of the wide-angle camera 305 in the second memory 370. In addition, the recording control unit 357 starts recording the second recording data 375 indicating the shooting results of the narrow-angle camera 306 in the second memory 370. In addition, when the opening / closing determination unit 355 determines that the opening / closing body 29A has become an open state, the recording control unit 357 turns on the second LED 318 to illuminate the inside of the storage room 27. When the opening / closing determination unit 355 determines that both the opening / closing bodies 29A have become closed states, the recording control unit 357 ends recording the shooting results of the wide-angle camera 305 and the narrow-angle camera 306. Furthermore, when the opening / closing determination unit 355 determines that both the opening / closing bodies 29A are in the closed state, the recording control unit 357 turns off the second LED 318 .

[0139] Furthermore, the recording control unit 357 may start recording the image capturing result of the wide-angle camera 305 when the opening / closing determination unit 355 determines that at least one of the left door 21B and the right door 21C is in the open state. Specifically, the recording control unit 357 starts recording the first image capturing data 374 indicating the image capturing result of the wide-angle camera 305 in the second memory 370. In this case, the recording of the image capturing result of the narrow-angle camera 306 may be started at the same time. Furthermore, the recording control unit 357 may light up the second LED 318 when the opening / closing determination unit 355 determines that at least one of the left door 21B and the right door 21C is in the open state. Alternatively, the recording control unit 357 may end recording the image capturing result of the wide-angle camera 305 when the opening / closing determination unit 355 determines that both the left door 21B and the right door 21C are in the closed state. In this case, the recording of the image capturing result of the narrow-angle camera 306 may be ended at the same time. Furthermore, the recording control unit 357 may turn off the second LED 318 when the opening / closing determination unit 355 determines that at least one of the left door 21B and the right door 21C is in the closed state.

[0140] Furthermore, the recording control unit 357 may start recording the shooting result of the narrow-angle camera 306 when the opening / closing determination unit 355 determines that the drawer 28A is in the open state. Specifically, the recording control unit 357 starts recording the second recording data 375 indicating the shooting result of the narrow-angle camera 306 in the second memory 370. In this case, the recording of the shooting result of the wide-angle camera 305 may be started at the same time. Furthermore, the recording control unit 357 may turn on the second LED 318 when the opening / closing determination unit 355 determines that the drawer 28A is in the open state. Alternatively, the recording control unit 357 may end recording the shooting result of the narrow-angle camera 306 when the opening / closing determination unit 355 determines that all the drawers 28A are in the closed state. In this case, the recording of the shooting result of the wide-angle camera 305 may be ended at the same time. Furthermore, the recording control unit 357 may turn off the second LED 318 when the opening / closing determination unit 355 determines that the drawer 28A is in the closed state.

[0141] During recording, the detection value of the distance sensor 312 is input to the recording control unit 357. The recording control unit 357 associates the detection value of the distance sensor 308 for the left door and the detection value of the distance sensor 309 for the right door with the shooting result of the wide-angle camera 305 during recording and the shooting result of the narrow-angle camera 306 during recording at intervals of each frame or at intervals of a predetermined number of frames. In addition, during recording, the recording control unit may associate the pull-out degree with the shooting result of the wide-angle camera 305 during recording and the shooting result of the narrow-angle camera 306 during recording.

[0142] The selection unit 358 selects a captured image from the dynamic image captured by the camera 307. The selection unit 358 includes a first selection unit 358A and a second selection unit 358B. The first selection unit 358A selects a first captured image showing the refrigerating chamber 21 with the door 21A in an open state from the dynamic image represented by the first video data 374. The second selection unit 358B selects a second captured image from the dynamic image represented by the second video data 375. The second captured image of the present embodiment is a captured image showing the drawer 28A in an open state. Details of the first captured image and the second captured image will be described later.

[0143] The generation unit 359 generates upload data. The upload data is data uploaded to the article management server 5. The generation unit 359 outputs the generated upload data to the third communication control unit 352.

[0144] The registration unit 360 registers the predetermined open state of the door 21A by generating the first registration data 373A and causing the second memory 370 to store the first registration data 373A. In addition, the registration unit 360 registers the predetermined open state of the drawer 28A by generating the second registration data 373B and causing the second memory 370 to store the second registration data 373B. In addition, the registration unit 360 registers the information for connecting to the local area network by generating the third registration data 373C and causing the second memory 370 to store the third registration data 373C.

[0145] [1-1-4. Structure of the item management server]

[0146] Fig.10 It is a block diagram showing the configuration of the terminal device 4 and the item management server 5.

[0147] The article management server 5 includes a server processor 50 which is a processor such as a CPU, a server memory 51, and a server communication module 52. The server processor 50 is connected to the server memory 51 and the server communication module 52.

[0148] The server memory 51 is a storage device that non-volatilely stores programs and data executed by the server processor 50. The server memory 51 is composed of a non-volatile storage device. In addition, the server memory 51 may also include a RAM that constitutes a work area of ​​the server processor 50. The server memory 51 stores data processed by the server processor 50, a control program 511 executed by the server processor 50, and an item management DB (database) 512.

[0149] The article management DB 512 has one record for each imaging device 3. This one record includes the imaging device ID 372, the image data of the first captured image, and the image data of the second captured image.

[0150] Server communication module 52 includes a communication line and a connector that conform to a predetermined communication standard, and communicates with refrigerator 2A, imaging device 3, and terminal device 4 via network NW.

[0151] The server processor 50 functions as a server communication control unit 501 and a server processing unit 502 by reading and executing a control program 511 stored in the server memory 51 .

[0152] The server communication control unit 501 communicates with the refrigerator 2A, the imaging device 3 , and the terminal device 4 via the server communication module 52 .

[0153] Server processing unit 502 performs various information processing using refrigerator 2A, imaging device 3, and terminal device 4 as clients.

[0154] [1-1-5. Configuration of terminal device]

[0155] Next, the configuration of the terminal device 4 will be described.

[0156] The terminal device 4 includes a terminal processor 40 which is a processor such as a CPU, a terminal memory 41 , a terminal communication module 42 , and a touch panel 43 .

[0157] The terminal memory 41 is a storage device that non-volatilely stores programs and data executed by the terminal processor 40. The terminal memory 41 is composed of a non-volatile storage device. In addition, the terminal memory 41 may also include a RAM that constitutes a work area of ​​the terminal processor 40. The terminal memory 41 stores data processed by the terminal processor 40, a control program executed by the terminal processor 40, and an item management application 411.

[0158] The terminal communication module 42 includes a communication line, a connector, etc. that comply with a prescribed communication standard, and communicates with the item management server 5 under the control of the terminal processor 40 .

[0159] The touch panel 43 includes a display panel and a touch sensor superimposed on or provided integrally with the display panel.

[0160] As described above, the terminal processor 40 functions as the application execution unit 400. The application execution unit 400 functions as a terminal communication control unit 401, a display control unit 402, and a reception unit 403.

[0161] The terminal communication control unit 401 communicates with the item management server 5 via the terminal communication module 42 .

[0162] The display control unit 402 causes the touch panel 43 to display a user interface for managing items stored in the cold storage 2A. In the following description, this user interface is referred to as an "application UI." The display control unit 402 of this embodiment displays a guide image that guides the opening and closing body 29A to a specified open state by displaying the application UI.

[0163] The receiving unit 403 receives various inputs from the user P via the application UI.

[0164] The terminal communication control unit 401 corresponds to the “communication control unit” of the present invention.

[0165] [1-2. Action]

[0166] Next, the operation of each part of the article management system 1A will be described.

[0167] For example, after purchasing the imaging device 3 , the user P installs the imaging device 3 in the refrigerator 2A and performs various settings of the imaging device 3 .

[0168] [1-2-1. Adjustment of the camera]

[0169] When installing the camera 3, the user P adjusts the position of the camera 3 in the width direction of the refrigerator 2A. The user P can adjust the position of the camera 3 using at least one of the first adjustment method, the second adjustment method, the third adjustment method, the fourth adjustment method, and the fifth adjustment method.

[0170] The first adjustment method is an adjustment method using an application UI. In the first adjustment method, the application UI displays the image capturing result of either the wide-angle camera 305 or the narrow-angle camera 306, and the adjustment is performed based on the displayed image capturing result of the user P.

[0171] The second adjustment method is an adjustment method using recording paper. The recording paper records a mark indicating the left end of the cold storage 2A, a mark indicating the right end of the cold storage 2A, and a mark indicating the center of the cold storage 2A in the left-right direction. In the second adjustment method, the user P can grasp where the center of the cold storage 2A in the width direction is located by recording paper. In the second adjustment method, the user P adjusts the position of the camera 3 so that the mark 304 of the limiting component 303 is located in the grasped center.

[0172] The third adjustment method is a method of using a mark given to the cold storage 2A. The mark is a mark indicating the center in the width direction of the cold storage 2A. In the third adjustment method, the user P adjusts the position of the camera 3 so that the position of the mark given to the cold storage 2A coincides with the position of the mark 304 of the limiting member 303.

[0173] The fourth adjustment method is a method of using a protrusion provided on the upper surface 20A of the cold storage 2A. The protrusion is provided in accordance with the shape of the camera 3. For example, at least a protrusion parallel to the X-axis and a protrusion parallel to the Y-axis are provided on the upper surface 20A of the cold storage 2A. In the fourth adjustment method, the user P adjusts the position of the camera 3 by placing the camera 3 in a manner matching the protrusion provided on the upper surface 20A of the cold storage 2A. In addition, the position of the camera 3 may be adjusted by placing the camera 3 in a recessed portion provided in accordance with the shape of the camera 3 on the upper surface 20A of the cold storage 2A.

[0174] The fifth adjustment method is a method using a claw provided on the camera 3 and a hole provided on the upper surface 20A of the cold storage 2A. The camera 3 is placed so that the claw and the hole are aligned, and the position of the camera 3 is adjusted. Alternatively, the hole may be provided on the camera 3, and the claw may be provided on the upper surface 20A of the cold storage 2A.

[0175] [1-2-2. Camera settings]

[0176] In the settings of the imaging device 3, a predetermined open state of the door 21A is registered. In addition, in the settings of the imaging device 3, a predetermined open state of the drawer 28A is registered. In addition, in the settings of the imaging device 3, information for connecting to a local area network is registered.

[0177] [1-2-2-1. Registration of information for connecting to a local area network]

[0178] First, registration of information for connection to a local area network will be described.

[0179] Fig.11 Flowchart FA is a flowchart showing the operation of the terminal device 4 , and flowchart FB is a flowchart showing the operation of the imaging device 3 .

[0180] exist Fig.11 At the start time of the flowchart shown, the terminal device 4 and the imaging device 3 are communicatively connected to each other and can perform short-range wireless communication.

[0181] As shown in flowchart FA, the receiving unit 403 receives input of the SSID and password of the local area network constructed by the communication device 6 via the application UI (step SA1). Next, the terminal communication control unit 401 transmits the SSID and password received in step SA1 to the imaging device 3 (step SA2).

[0182] As shown in the flowchart FB, the fourth communication control unit 353 receives the SSID and password from the terminal device 4 (step SB1). Next, the registration unit 360 registers the SSID and password received in step SB1 (step SB2). That is, in step SB2, the registration unit 360 generates the third registration data 373C recording the SSID and password received in step SB1, and causes the second memory 370 to store the generated third registration data 373C.

[0183] When the registration unit 360 causes the second memory 370 to store the third registration data 373C, the image pickup device 3 connects to the local area network using the SSID and password recorded in the third registration data 373C.

[0184] [1-2-2-2. Registration of the prescribed open state of the door]

[0185] Next, the registration of the predetermined open state of door 21A will be described.

[0186] Fig.12 Flowchart FC shows the operation of the terminal device 4 , and flowchart FD shows the operation of the imaging device 3 .

[0187] exist Fig.12 In the flowchart shown, the terminal device 4 and the photographing device 3 communicate with each other, and the photographing device 3 sends the photographing result of the wide-angle camera 305 to the terminal device 4 .

[0188] As shown in flowchart FC, display control unit 402 causes touch panel 43 to display information urging door 21A to close (step SC1 ).

[0189] Next, accepting unit 403 determines whether input for closing door 21A is received from user P (step SC2 ). If accepting unit 403 determines that input for closing door 21A is not received (step SC2 : No), it performs step SC2 determination again.

[0190] On the other hand, when the receiving unit 403 determines that the input for the closed state is received (step SC2: Yes), the terminal communication control unit 401 transmits instruction information to the imaging device 3 (step SC3). The instruction information is information that instructs the second memory 370 to record the detection value.

[0191] As shown in flowchart FD, the fourth communication control unit 353 receives instruction information from the terminal device 4 (step SD1).

[0192] Next, the registration unit 360 records the detection value currently output by the distance measuring sensor 312 in the second memory 370 (step SD2 ).

[0193] As shown in flowchart FD, the fourth communication control unit 353 transmits completion information indicating that recording of the detection value has been completed to the terminal device 4 (step SD3).

[0194] As shown in the flowchart FC, the terminal communication control unit 401 receives the completion information from the imaging device 3 (step SC4 ). Next, the display control unit 402 displays the first guide image AG1 superimposed on the imaging result of the wide-angle camera 305 (step SC5 ).

[0195] Fig.13 1 is a diagram showing an example of the first guide image AG1 displayed superimposed on the image capturing result of the wide-angle camera 305 .

[0196] The first guide image AG1 is a guide image for guiding the door 21A to a predetermined open state, and has a plurality of lines L. As shown in FIG. Fig.13 The first guide image AG1 shown has 12 lines L, namely, lines L1 to L12. Lines L1, L2, and L3 are lines L corresponding to the edge of the shelf 212 in the refrigerator compartment 21. Lines L4, L5, and L6 are lines L corresponding to the edge of the front surface of the cabinet 20. Line L4 is a line L corresponding to the left edge. Line L5 is a line L corresponding to the right edge. Line L6 is a line L corresponding to the lower edge. Lines L7 to L9 are lines L corresponding to the edge of the door shelf 211 provided in the left door 21B. Lines L10 to L12 are lines L corresponding to the edge of the door shelf 211 provided in the right door 21C.

[0197] The lines L included in the first guide image AG1 are displayed in a manner corresponding to the edges of the door 21A when it is in a predetermined open state. The display positions of the lines L included in the first guide image AG1 vary depending on the model of the cold storage 2A in which the camera 3 is installed. The display control unit 402 receives image data of the first guide image AG1 corresponding to the model of the cold storage 2A from the article management server 5 in advance, and displays the first guide image AG1 represented by the received image data.

[0198] The user P changes the rotation angle of the door 21A so that the lines L7 to L12 included in the first guide image AG1 coincide with the edge of the door shelf 211 shown in the photographing result displayed on the terminal device 4 .

[0199] Return to Fig.12 As shown in the flowchart, the receiving unit 403 determines whether an input for a predetermined open state is received from the user P (step SC6). When the receiving unit 403 determines that an input for a predetermined open state is not received (step SC6: No), the determination of step SC6 is repeated.

[0200] When the receiving unit 403 determines that the input for setting the predetermined open state is received (step SC6 : Yes), the terminal communication control unit 401 transmits the instruction information to the imaging device 3 (step SC7 ).

[0201] As shown in flowchart FD, the fourth communication control unit 353 receives instruction information from the terminal device 4 (step SD4).

[0202] Next, the registration unit 360 records the detection value currently output by the distance measuring sensor 312 in the second memory 370 (step SD5 ).

[0203] Next, the registration unit 360 generates the first registration data 373A (step SD6 ).

[0204] Step SD6 will be described in detail.

[0205] The registration unit 360 determines the detection value of the distance measuring sensor 312 recorded in step SD2 as the detection value of the distance measuring sensor 312 when the rotation angle of the door 21A is 0°, in other words, when the door 21A is in a closed state. Next, the registration unit 360 determines the detection value of the distance measuring sensor 312 recorded in step SD5 as the detection value of the distance measuring sensor 312 when the door 21A is in a predetermined open state. Next, the registration unit 360 generates the first registration data 373A in which the angle information indicating 0° is associated (corresponded) with the detection value of the distance measuring sensor 312 determined to be 0°, and the angle information indicating 90° is associated with the detection value of the distance measuring sensor 312 when the door 21A is in a predetermined open state.

[0206] The registration unit 360 stores the first registration data 373A generated in step SD6 in the second memory 370 (step SD7 ).

[0207] [1-2-2-3. Registration of a predetermined open state of a drawer]

[0208] Next, the registration of the predetermined open state of the drawer 28A will be described. In this description, the drawer 25A is exemplified as the drawer 28A.

[0209] Fig.14 7A is a flowchart showing the operation of the information processing system 7A. Fig.14 , flowchart FE represents the operation of the terminal device 4, and flowchart FF represents the operation of the photographing device 3.

[0210] exist Fig.14 In the flowchart shown, the terminal device 4 and the photographing device 3 communicate with each other, and the photographing device 3 sends the photographing result of the narrow-angle camera 306 to the terminal device 4 .

[0211] As shown in the flowchart FE, the display control unit 402 displays the second guide image AG2 superimposed on the image capturing result of the narrow-angle camera 306 (step SE1 ).

[0212] Fig.15 3 is a diagram showing an example of the second guide image AG2 displayed superimposed on the image capturing result of the narrow-angle camera 306 .

[0213] The second guide image AG2 is a guide image for guiding the drawer 25A to a predetermined open state. In addition, the second guide image AG2 is also an image for guiding the storage box included in the drawer 25A to a state in which the storage box is pulled out to the maximum extent. Fig.15 The second guide image AG2 shown has five lines L, namely, lines L13 to L17. Lines L13 to L16 are lines L corresponding to the outer edge of the drawer 25A. Line L17 is a line L corresponding to the front edge of the storage box.

[0214] The lines L13 to L16 included in the second guide image AG2 are displayed in a manner corresponding to the edges of the drawer 25A when it is in a predetermined open state. In addition, the line L17 included in the second guide image AG2 is displayed in a manner corresponding to the front edge of the storage box of the drawer 25A when the storage box is pulled out to the maximum extent. The display position of the line L included in the second guide image AG2 varies depending on the model of the cold storage 2A in which the camera 3 is installed. The display control unit 402 receives the image data of the second guide image AG2 corresponding to the model of the cold storage 2A from the article management server 5 in advance, and displays the second guide image AG2 represented by the received image data.

[0215] The user P pulls out the drawer 28A so that the line L15 included in the second guide image AG2 coincides with the front edge of the drawer 25A included in the photographing result displayed on the terminal device 4 .

[0216] return Fig.14 As shown in the flowchart, the receiving unit 403 determines whether an input for a predetermined open state is received from the user P (step SE2). When the receiving unit 403 determines that an input for a predetermined open state is not received (step SE2: No), the determination of step SE2 is repeated.

[0217] When the receiving unit 403 determines that the input for forming the specified open state is received (step SE2: Yes), the terminal communication control unit 401 sends the registration request information to the imaging device 3 (step SE3). The registration request information is information requesting the registration of the specified open state, and records the type of the drawer 28A to be registered. In this description, since the drawer 25A is exemplified, the drawer 25A is recorded as the type of the drawer 28A to be registered in the registration request information sent in step SE3.

[0218] As shown in flowchart FF, the fourth communication control unit 353 receives registration request information from the terminal device 4 (step SF1).

[0219] Next, the registration unit 360 generates the second registration data 373B (step SF2 ).

[0220] In step SF2, the registration unit 360 recognizes the position of the front edge of the drawer 25A in the most recent image captured by the narrow-angle camera 306. The registration unit 360 performs image analysis such as shape recognition and color difference recognition on the captured image to recognize the position of the front edge of the drawer 25A in the captured image. Next, the registration unit 360 generates the second registration data 373B recording the recognized position.

[0221] Among them, the drawer 28A of the registration object is recorded as drawer 25A in the second registration data 373B. The registration unit 360 records the type of the registration object indicated by the registration request information in the second registration data 373B. In this description, since drawer 25A is exemplified, drawer 25A is recorded as the type of drawer 28A of the registration object in the second registration data 373B generated in step SF2.

[0222] The registration unit 360 causes the second memory 370 to store the second registration data 373B generated in step SF2 (step SF3 ).

[0223] In addition, Fig.14 In the description of , the case where drawer 25A is registered as drawer 28A is exemplified, and drawers 28A other than drawer 25A are also registered in the same manner. In this case, the second registration data 373B of each drawer 28A is stored in the second memory 360 .

[0224] In addition, the above-mentioned various registrations can also be performed after starting to use the imaging device 3. In this case, the registration unit 360 updates the corresponding registration data to the latest data every time the registration is performed.

[0225] Next, the operation of each part of the article management system 1A after the installation and setting of the imaging device 3 are performed will be described.

[0226] [1-2-3. About video recording]

[0227] First, the operation of the camera 3 regarding video recording will be described.

[0228] Fig.16 This is a flowchart FG showing the operation of the imaging device 3 .

[0229] exist Fig.16In the operation shown, each functional unit of the second processor 350 performs processing using the detection values ​​of each sensor connected to the sensor control circuit board 313.

[0230] Human presence determination unit 354 determines whether there is a human being around refrigerator 2A (step SG1 ). When human presence determination unit 354 determines that there is no human being around refrigerator 2A (step SG1 : No), it performs the determination of step SG1 again.

[0231] On the other hand, when human presence determination unit 354 determines that a person exists around refrigerator 2A (step SG1 : Yes), imaging control unit 356 causes wide-angle camera 305 and narrow-angle camera 306 to start imaging (step SG2 ).

[0232] The opening / closing determination unit 355 determines whether the opening / closing body 29A is in the open state (step SG3 ). When the opening / closing determination unit 355 determines that the opening / closing body 29A is not in the open state (step SG3 : No), the second processor 350 performs the process of step SG7 .

[0233] On the other hand, when the opening and closing determination unit 355 determines that the opening and closing body 29A is in the open state (step SG3 : Yes), the recording control unit 357 starts recording the imaging results of the wide-angle camera 305 and the narrow-angle camera 306 (step SG4 ).

[0234] Next, the opening / closing determination unit 355 determines whether the opening / closing body 29A is in the closed state (step SG5 ). When the opening / closing determination unit 355 determines that the opening / closing body 29A is not in the closed state (step SG5 : No), the determination in step SG5 is performed again.

[0235] On the other hand, when it is determined that the opening and closing body 29A has become a closed state (step SG5: Yes), the recording control unit 357 ends the recording of the shooting result (step SG6). As a result, the first recording data 374 and the second recording data 375 can be generated in the second memory 370. In addition, the type of the drawer 28A shown in the dynamic image represented by the second recording data 275 is recorded in the second recording data 375. The type can be determined by the opening and closing determination unit 355 based on the detection value of the drawer distance sensor 310, the image analysis of the dynamic image captured by the narrow-angle camera 306, and other methods. The recording control unit 357 records the type determined by the opening and closing determination unit 355 in the header of the second recording data 275.

[0236] Human presence determination unit 354 determines whether a human is present around refrigerator 2A (step SG7). If human presence determination unit 354 determines that a human is present around refrigerator 2A (step SG7: Yes), second processor 350 executes the processes after step SG3 again.

[0237] On the other hand, when human presence determination unit 354 determines that there is no human around refrigerator 2A (step SG7 : No), imaging control unit 356 causes wide-angle camera 305 and narrow-angle camera 306 to end imaging (step SG8 ).

[0238] [1-2-4. Operation regarding uploading to the article management server]

[0239] Next, the operation of uploading to the article management server 5 will be described.

[0240] Fig.17 This is a flowchart showing the operation of the article management system 1A.

[0241] exist Fig.17 , flowchart FH represents the operation of the imaging device 3 , and flowchart FI represents the operation of the article management server 5 .

[0242] The generator 359 determines whether any of the first recorded data 374 and the second recorded data 375 is stored in the second memory 370 (step SH1). When the generator 359 determines that no recorded data is stored (step SH1: No), the determination of step SH1 is repeated.

[0243] On the other hand, when the generator 359 determines that the recorded data is stored (step SH1: Yes), it determines whether the recorded data is the first recorded data 374 or the second recorded data 375 (step SH2). The determination in step SH2 is performed based on the type information of the recorded data recorded in the header.

[0244] When the generating unit 359 determines that it is the first recorded data 374 (step SH2: first recorded data), the first selecting unit 358A selects the first captured image from the moving images represented by the first recorded data 374 stored in the second memory 370 (step SH3).

[0245] Step SH3 will be described in detail.

[0246] The first captured image is a captured image when the left door 21B and the right door 21C are opened at a rotation angle of α°≤θ≤β°, and is a captured image in which the user P is least reflected. α° is, for example, 60°, and β° is, for example, 100°.

[0247] First selection unit 358A converts the detection value of left door distance measuring sensor 308 associated with first video data 374 into the rotation angle of left door 21B, and converts the detection value of right door distance measuring sensor 309 associated with first video data 374 into the rotation angle of right door 21C.

[0248] The first selection unit 358A refers to the first registration data 373A and converts the detection value of the left door distance sensor 308 associated with the first video data 374 into the rotation angle of the left door 21B, for example, as described below. The first selection unit 358A obtains the detection value of the left door distance sensor 308 corresponding to 0° and the detection value of the left door distance sensor 308 corresponding to 90° from the first registration data 373A. Next, the first selection unit 358A identifies which rotation angle the detection value of the left door distance sensor 308 associated with the first video data 374 corresponds to, based on the two obtained detection values. The identification method is, for example, a method of obtaining a change degree of the detection value of the left door distance sensor 308 with respect to the rotation angle of the left door 21B based on the detection value of 0° and the detection value of 90°, and identifying the rotation angle based on the obtained change degree. That is, at least two detection values ​​are used to obtain the degree of change of the detection value of the left door distance sensor 308 with respect to the rotation angle of the left door 21B, and the rotation angle is identified based on the obtained degree of change. The detection values ​​of the distance sensor may be three or more. Furthermore, the first selection unit 358A converts the detection value of the left door distance sensor 308 associated with the first video data 374 into the identified rotation angle. The first selection unit 358A similarly converts the detection value of the right door distance sensor 309 into the rotation angle of the right door 21C.

[0249] Next, the first selection unit 358A refers to the converted rotation angle and selects, from the first video recording data 374, a captured image in which the left door 21B and the right door 21C are opened at a rotation angle of α°≤θ≤β°.

[0250] Next, the first selection unit 358A selects the captured image with the least reflection of the user P from among the selected captured images as the first captured image. The first selection unit 358A recognizes the user P reflected in the captured image by using a method such as pattern matching, and selects the captured image with the smallest area of ​​the recognized image of the user P as the captured image with the least reflection of the user P.

[0251] The first selection unit 358A determines whether the first captured image can be selected (step SH4 ). When the first selection unit 358A determines that the first captured image cannot be selected (step SH4 : No), the second processor 350 performs the process of step SH11 .

[0252] On the other hand, when the first selection unit 358A determines that the first captured image is selected (step SH4: Yes), the generation unit 359 generates the first upload data (step SH5). The first upload data includes the image data of the first captured image selected by the first selection unit 358A and the camera ID 372 stored in the second memory 370.

[0253] Next, the third communication control unit 352 transmits the first upload data generated by the generating unit 359 to the item management server 5 (step SH6).

[0254] Returning to the description of step SH2, when the generation unit 359 determines that it is the second video data 375 (step SH2: second video data), the second selection unit 358B selects the second captured image from the dynamic image represented by the second video data 375 stored in the second memory 370 (step SH7).

[0255] Step SH7 will be described in detail.

[0256] The second captured image is a captured image of the drawer 28A that is pulled out to a degree of γ% or more, when the pulled-out degree in the closed state is 0% and the pulled-out degree in the predetermined open state is 100%. The pulled-out degree is the degree of being pulled out. In addition, the second captured image is a captured image in which the user P is least reflected. γ% is, for example, 70%.

[0257] First, the second selection unit 358B reads out the second registration data 373B corresponding to the type of the drawer 28A recorded in the second video data 375 to be processed from the second memory 370. Then, the second selection unit 358B recognizes the position of the front edge of the drawer 28A recorded in the read second registration data 373B as the position when the pull-out degree is 100%, and recognizes the position of the upper end of the captured image captured by the narrow-angle camera 306 as the position of the front edge of the drawer 28A when the pull-out degree is 0%. In this recognition, it is assumed that the direction of pulling out the drawer 28A in the captured image of the narrow-angle camera 306 is the direction from the upper end to the lower end of the captured image. Moreover, the second selection unit 358B selects the captured image in which the front edge of the drawer 28A is located at a position greater than γ% based on the two recognized positions. Next, the second selection unit 358B selects the captured image in which the user P is least reflected from the selected captured images as the second captured image.

[0258] The second selection unit 358B determines whether the second captured image has been selected (step SH8 ). When the second selection unit 358B determines that the second captured image has not been selected (step SH8 : No), the second processor 350 performs the process of step SH11 .

[0259] On the other hand, when the second selection unit 358B determines that the second captured image is selected (step SH8: Yes), the generation unit 359 generates second upload data (step SH9). The second upload data includes the image data of the second captured image selected by the second selection unit 358B and the imaging device ID 372 stored in the second memory 370.

[0260] Next, the third communication control unit 352 transmits the second upload data generated by the generation unit 359 to the item management server 5 (step SH10).

[0261] After step SH2, the generator 359 deletes the recorded data to be processed from the second memory 370 (step SH11). If the transmission in step SH6 is performed, the generator 359 deletes the first recorded data 374 from the second memory 370 in step SH11. If the transmission in step SH10 is performed, the generator 359 deletes the second recorded data 375 from the second memory 370 in step SH11.

[0262] As shown in the flowchart FI, the server communication control unit 501 determines whether the first upload data or the second upload data is received from the camera 3 (step SI1). When the server communication control unit 501 determines that the first upload data or the second upload data is received (step SI1: yes), the server processing unit 502 updates the item management DB 512 based on the upload data received by the server communication control unit 501 (step SI2).

[0263] Step SI2 will be described in detail.

[0264] The server processing unit 502 identifies the record including the imaging device ID 372 included in the first uploaded data from the article management DB 512 , and updates the image data of the first captured image of the identified record to the image data of the first captured image included in the first uploaded data.

[0265] The server processing unit 502 identifies the record including the imaging device ID 372 included in the second uploaded data from the article management DB 512 , and updates the image data of the second captured image of the identified record to the image data of the second captured image included in the second uploaded data.

[0266] [1-2-5. About the operation of application UI]

[0267] Next, the operation of the item management system 1A regarding the application UI will be described.

[0268] Fig.18 1A is a flowchart showing the operation of the article management system 1A. Fig.18, flowchart FJ represents the operation of the terminal device 4, and flowchart FK represents the operation of the item management server 5.

[0269] As shown in flowchart FJ, the terminal communication control unit 401 determines whether to request image data stored in the article management DB 512 from the article management server 5 (step SJ1). For example, when the article management application 411 is activated, the terminal communication control unit 401 makes an affirmative determination in step SJ1.

[0270] When the terminal communication control unit 401 determines that the image data stored in the article management DB 512 is requested (step SJ1: Yes), the terminal communication control unit 401 sends request data for requesting the image data to the article management server 5 (step SJ2). The request data includes the camera ID 372 stored in the terminal memory 41. In addition, the terminal 4 obtains the camera ID 372 from the camera 3 at the time (timing) when the terminal 4 establishes a communication connection with the camera 3.

[0271] As shown in flowchart FK, the server communication control unit 501 receives the request data (step SK1). Next, the server processing unit 502 obtains the image data associated with the camera ID 372 included in the request data received by the server communication control unit 501 from the item management DB 512 (step SK2). Next, the server communication control unit 501 sends the image data obtained by the server processing unit 502 to the terminal device 4 as a response to the request data (step SK3).

[0272] Referring to flowchart FJ, terminal communication control unit 401 receives image data from item management server 5 (step SJ3). Next, display control unit 402 causes touch panel 43 to display an application UI including an image represented by the image data received in step SJ3 (step SJ4).

[0273] [1-3. Effects, etc.]

[0274] As described above, the photographing device 3 includes: a wide-angle camera 305 for photographing the refrigerating chamber 21 of the cold storage 2A; and a first selection unit 358A, which selects, based on the rotation angle of the rotating door 21A that opens and closes the opening of the refrigerating chamber 21, a first photographed image of the refrigerating chamber 21 showing the door 21A in an open state from the dynamic image photographed by the wide-angle camera 305.

[0275] Thus, since the first captured image is selected based on the rotation angle of door 21A, the first captured image that appropriately shows refrigerating room 21 can be selected from the dynamic image captured by wide-angle camera 305. Therefore, the captured image that appropriately shows the internal conditions of refrigerating room 2A can be selected from the dynamic image captured by camera 307.

[0276] Imaging device 3 includes main body 301 provided on upper surface 20A of housing 20 of refrigerator 2A, and main body 301 includes distance sensor 312 for measuring the distance to door 21A. First selection unit 358A selects a first captured image based on the rotation angle of door 21A corresponding to the detection value of distance sensor 312.

[0277] Thus, the range-finding sensor 312 can be used to select a captured image that appropriately reflects the internal conditions of the cold storage 2A. In addition, since the main body 301 provided on the upper surface 20A of the box body 20 has the range-finding sensor 312, the range-finding sensor 312 can stably and correctly measure the distance to the door 21A. Therefore, the range-finding sensor 312 can be used to select a captured image that appropriately reflects the internal conditions of the cold storage 2A with high accuracy.

[0278] The distance measuring sensor 312 is provided on the main body 301 so that the sensing direction faces the rotation axis side of the door 21A when the main body 301 is provided in the housing 20 .

[0279] Thereby, distance measuring sensor 312 can appropriately measure the distance to door 21A according to the rotation of door 21A. Therefore, using distance measuring sensor 312, it is possible to select a captured image that appropriately reflects the internal condition of refrigerator 2A with high accuracy.

[0280] Image capturing device 3 includes: a main body 301 provided on upper surface 20A of housing 20 of refrigerator 2A; and an image capturing unit 302 having a wide-angle camera 305 extending from main body 301 toward the front of housing 20 .

[0281] Thus, since the wide-angle camera 305 can shoot from above the cold storage 2A, the image captured by the wide-angle camera 305 can include the interior of the cold storage room 21 and the inner surface of the door 21A in the open state. Therefore, a captured image that more appropriately reflects the internal conditions of the cold storage 2A can be selected.

[0282] The photographing device 3 includes a narrow-angle camera 306 for photographing the drawer 28A of the cold storage 2A when the drawer chamber 28 is opened and closed. The cold storage chamber 21 is arranged at a position above the drawer chamber 28. A recess 205 extending in the height direction of the cold storage 2A is formed on the front surface of the door 21A in the closed state. The wide-angle camera 305 and the narrow-angle camera 306 are arranged side by side with the photographing component 302 in the width direction of the photographing device 3 at a distance shorter than the length of the recess 205 in the width direction of the cold storage 2A.

[0283] Thus, the opening of the drawer 28A can be photographed even when the door 21A is closed. In addition, since the photographing device 3 can be designed so that the photographing component 302 of the photographing device 3 does not protrude as far as possible in front of the cold storage 2A, the photographing device 3 can be miniaturized for the cold storage 2A in which the recess 205 is formed.

[0284] The photographing device 3 includes: a narrow-angle camera 306 for photographing the drawer 28A of the drawer chamber 28 of the cold storage 2A when the drawer 28 is opened and closed; and a second selection unit 358B, which selects a second captured image showing the drawer 28A in an open state from the dynamic image photographed by the narrow-angle camera 306 based on the extent to which the drawer 28A is pulled out.

[0285] Thus, since the second captured image is selected based on the degree to which drawer 28A is pulled out, the second captured image that appropriately shows drawer 28A can be selected from the dynamic image captured by narrow-angle camera 306. Therefore, a captured image that appropriately shows the internal conditions of cold storage 2A can be selected from the dynamic image captured by camera 307.

[0286] Image capturing device 3 includes: main body 301 provided on upper surface 20A of housing 20 of refrigerator 2A; and image capturing unit 302 having a narrow-angle camera 306 extending from main body 301 toward the front of housing 20 .

[0287] Thus, since narrow-angle camera 306 can capture images from above refrigerator 2A, the opening of drawer 28A can be included in the captured image captured by narrow-angle camera 306. Therefore, a captured image that more appropriately reflects the interior of refrigerator 2A can be selected.

[0288] The camera 3 includes a wide-angle camera 305 for photographing the refrigerating chamber 21 of the cold storage 2A. The refrigerating chamber 21 is arranged at a position above the drawer chamber 28. A recess 205 extending in the height direction of the cold storage 2A is formed on the front surface of the door 21A in the closed state. The wide-angle camera 305 and the narrow-angle camera 306 are arranged side by side with the camera component 302 in the width direction of the camera 3 at a distance shorter than the length of the recess 205 in the width direction of the cold storage 2A.

[0289] Thus, the opening of the drawer 28A can be photographed even when the door 21A is closed. In addition, since the photographing device 3 can be designed so that the photographing component 302 of the photographing device 3 does not protrude as far as possible in front of the cold storage 2A, the photographing device 3 can be miniaturized for the cold storage 2A in which the recess 205 is formed.

[0290] The information processing system 7A includes an imaging device 3 and a terminal device 4 that communicates with the imaging device 3. The imaging device 3 includes: a camera 307 that captures a cold storage 2A having a housing 20 forming a storage room 27 and an opening and closing body 29A that opens and closes an opening of the storage room 27; a registration unit 360 that registers a predetermined open state of the opening and closing body 29A; and a selection unit 358 that selects a captured image when the opening and closing body 29A is in an open state from images captured by the camera 307 based on the predetermined open state registered by the registration unit 360.

[0291] Thus, since the captured image when the opening and closing body 29A is in the open state is selected based on the registered predetermined open state, the captured image can be selected based on the reference corresponding to the cold storage 2A captured by the camera 3. Therefore, the captured image that appropriately reflects the internal conditions of the cold storage 2A can be selected from the images captured by the camera 307 without depending on the model of the cold storage 2A captured by the camera 3.

[0292] The terminal device 4 includes: a display control unit 402 that displays a guide image for guiding the state of the opening and closing body 29A to a specified open state; and a receiving unit 403 that receives input for making the state of the opening and closing body 29A a specified open state. When the receiving unit 403 receives the input, the registration unit 360 registers the current state of the opening and closing body 29A as the specified open state.

[0293] Thus, the user P can easily set the state of the opening and closing body 29A to the specified open state by viewing the guide image displayed on the terminal device 4. In addition, the registration unit 360 can correctly register the specified open state of the opening and closing body 29A. Therefore, the convenience of the user in registering the specified open state can be improved, and a captured image that more appropriately reflects the internal conditions of the cold storage 2A can be selected.

[0294] The terminal device 4 includes a terminal communication control unit 401 that receives a moving image captured by the camera 307 from the imaging device 3. The display control unit 402 displays the captured image represented by the image data received by the terminal communication control unit 401 and the guide image AG in a superimposed manner.

[0295] Thus, since the user P can intuitively set the opening and closing body 29A to the predetermined open state, the convenience of the user P in registering the predetermined open state can be further improved.

[0296] The opening and closing body 29A is a swing-type door 21A. The display control unit 402 displays a first guidance image AG1 as a guidance image for guiding the state of the door 21A to a predetermined open state. The first guidance image AG1 includes a line L corresponding to the edge of the door frame 211 of the door 21A in the predetermined open state.

[0297] Thus, since the reference for guiding door 21A to the predetermined open state is displayed, user P can more intuitively and accurately set door 21A to the predetermined open state.

[0298] The opening and closing body 29A is a drawer 28A. The display control unit 402 displays a second guide image AG2 as a guide image for guiding the state of the drawer 28A to a predetermined open state. The second guide image AG2 includes a line L corresponding to the front edge of the drawer 28A in the predetermined open state.

[0299] Thus, since the reference for guiding the drawer 28A to the predetermined open state is displayed, the user P can more intuitively and accurately set the drawer 28A to the predetermined open state.

[0300] The display position of line L in the guide image differs depending on the model of refrigerator 2A captured by camera 307 .

[0301] Thus, user P can more intuitively and accurately set opening and closing body 29A to a predetermined open state regardless of the model of refrigerator 2A.

[0302] The photographing device 3 includes: a camera 307 for photographing a cold storage 2A having a box body 20 forming a storage room 27 and an opening and closing body 29A for opening and closing an opening of the storage room 27; a registration unit 360 for registering a specified open state of the opening and closing body 29A; and a selection unit 358 for selecting a photographed image in which the opening and closing body 29A is in an open state from images photographed by the camera 307 based on the specified open state registered by the registration unit 360.

[0303] Thereby, the same effects as those of the above-mentioned information processing system 7A can be obtained.

[0304] (Implementation Method 2)

[0305] Next, Embodiment 2 will be described. In the description of Embodiment 2, the same components as those of the respective parts of the article management system 1A of Embodiment 1 are denoted by the same reference numerals, and detailed descriptions thereof will be appropriately omitted.

[0306] [2-1. Structure]

[0307] [2-1-1. Structure of the item management system]

[0308] Fig.19 This is a diagram showing the structure of an article management system 1B according to the second embodiment.

[0309] Compared with the article management system 1A of the first embodiment, the article management system 1B of the present embodiment does not include the camera 3 and the cold storage 2A, but includes the cold storage 2B instead of them. In addition, compared with the information processing system 7A of the first embodiment, the information processing system 7B of the present embodiment includes the cold storage 2B instead of the camera 3. In addition, the cold storage 2B of the present embodiment may be a cold storage having the recess 205 formed in the door 21A, or may be a cold storage having no recess 205 formed therein.

[0310] In the present embodiment, the ice and fresh-keeping compartment 26 formed in the refrigerating compartment 21 of the refrigerator 2B corresponds to the "second storage compartment" of the present invention.

[0311] [2-1-2. Structure of cold storage]

[0312] Fig. 20 This is a plan view of cold storage 2B as viewed from the right. Fig.21 2B is a block diagram showing the structure of the cold storage 2B. Fig. 20 In, with Figure 2 The X-axis, Y-axis, and Z-axis are also shown.

[0313] like Fig. 20 As shown, the cold storage 2B includes a wide-angle camera 305 and a narrow-angle camera 306. These two cameras are connected to the camera control circuit board 314 via a harness arranged between the outer box 201 and the inner box 202. The wide-angle camera 305 and the narrow-angle camera 306 of the second embodiment are arranged at the front end of the upper surface 21D in the cold storage room 21, and shoot the lower side from the upper surface 21D. Fig. 20 In the figure, the wide-angle camera 305 and the narrow-angle camera 306 are arranged in the front-to-back direction of the cold storage 2B, but this is only to show that the wide-angle camera 305 and the narrow-angle camera 306 are set in the cold storage room 21, and the setting method is not limited to Fig. 20 For example, the wide-angle camera and the narrow-angle camera 306 may be arranged in the left-right direction of the cold storage 2B.

[0314] The shooting range of the wide-angle camera 305 of Embodiment 2 includes the following range when viewed from the front of the cold storage 2B, for example. When the left door 21B and the right door 21C are in an open state, this range includes the door shelves 211 of the left door 21B and the right door 21C and the opening of the cold storage room 21. In addition, the shooting range of the wide-angle camera 305 of Embodiment 2 includes the following range when viewed from the right of the cold storage 2B. This range includes the range from the front end of the door 21A in the open state to the front end of the shelf 212 in the front-to-back direction of the cold storage 2B, and includes the opening of the cold storage room 21 in the up-down direction.

[0315] The shooting range of the narrow-angle camera 306 of Embodiment 2 includes the following range when viewed from the front of the cold storage 2B, for example. This range includes the drawer 26A when viewed from the front of the cold storage 2B. In addition, the shooting range of the narrow-angle camera 306 of Embodiment 2 includes the following range when viewed from the right of the cold storage 2B, for example. This range includes the drawer 26A in the state of being pulled out to the maximum extent in the front-back direction of the cold storage 2B.

[0316] like Fig. 20 and Fig.21 As shown, refrigerator 2B includes left door distance measuring sensor 308, right door distance measuring sensor 309, and human sensing sensor 311. These sensors are connected to sensor control circuit board 313.

[0317] The distance measuring sensor 312 is provided on the upper surface 20A of the box body 20. The arrangement of the distance measuring sensor 308 for the left door and the distance measuring sensor 309 for the right door is the same as that of the first embodiment. Fig. 20 The diagram shows the arrangement of the left door distance measuring sensor 308 and the right door distance measuring sensor 309 in the front-rear direction of the cold storage 2B. However, this merely shows that the distance measuring sensor 312 is arranged on the upper surface 20A. The arrangement of the left door distance measuring sensor 308 and the right door distance measuring sensor 309 is not limited to the arrangement of the left door distance measuring sensor 308 and the right door distance measuring sensor 309. Fig. 20 .

[0318] The human sensing sensor 311 is provided on the upper front surface of the door 21A.

[0319] like Fig. 20 and Fig.21 As shown, the cold storage 2B includes a heater 222. The heater 222 heats the lenses of the wide-angle camera 305 and the narrow-angle camera 306.

[0320] The heater 222 corresponds to the “heating unit” of the present invention.

[0321] like Fig.21 As shown, the refrigerator 2B includes a sensor control circuit board 313 and a camera control circuit board 314. The sensor control circuit board 313 and the camera control circuit board 314 are stored in a storage space of a recessed portion formed in the upper surface 20A.

[0322] The camera control circuit board 314 is connected to the third communication module 315 , the fourth communication module 316 , the wide-angle camera 305 , the narrow-angle camera 306 , and the heater 222 included in the refrigerator 2B.

[0323] The second memory 370 according to the second embodiment includes first registration data 373A, third registration data 373C, and fourth registration data 373D.

[0324] The fourth registration data 373D is data for registering the specified open state of the drawer 26A. In the present embodiment, the specified open state of the drawer 26A is the state when the drawer 26A is pulled out with the maximum amplitude. In the fourth registration data 373D, the position of the front edge of the drawer 26A when the drawer 26A is in the specified open state is recorded.

[0325] The second processor 350 according to the second embodiment functions as a heating control unit 361 in addition to functioning as each functional unit of the second processor 350 according to the first embodiment.

[0326] The shooting control unit 356 of the second embodiment makes the wide-angle camera 305 and the narrow-angle camera 306 start shooting when the person presence determination unit 354 determines that there are people around the cold storage 2B. In addition, the shooting control unit 356 of the second embodiment makes the wide-angle camera 305 and the narrow-angle camera 306 stop shooting when the person presence determination unit 354 determines that there are no people around the cold storage 2B after the wide-angle camera 305 and the narrow-angle camera 306 start shooting. In addition, the shooting control unit 356 may light up the first LED 317 when making the wide-angle camera 305 and the narrow-angle camera 306 start shooting. In addition, the shooting control unit 356 may turn off the first LED 317 when the wide-angle camera 305 and the narrow-angle camera 306 stop shooting.

[0327] The recording control unit 357 of the second embodiment starts recording the image capturing result of the wide-angle camera 305 and the image capturing result of the narrow-angle camera 306 when the opening / closing determination unit 355 determines that at least one of the left door 21B and the right door 21C is in the open state. Specifically, the recording control unit 357 starts recording the image capturing file 376 in the second memory 370. The image capturing file 376 includes the third image capturing data 377 indicating the dynamic image captured by the wide-angle camera 305 and the fourth image capturing data 378 indicating the dynamic image captured by the narrow-angle camera 306. The recording control unit 357 of the second embodiment ends recording the image capturing result of the wide-angle camera 305 and the image capturing result of the narrow-angle camera 306 when the opening / closing determination unit 355 determines that the door 21A is in the closed state. Alternatively, the recording control unit 357 may light up the second LED 318 when the opening / closing determination unit 355 determines that at least one of the left door 21B and the right door 21C is in the open state. Furthermore, when the opening / closing determination unit 355 determines that the door 21A is in the closed state, the second LED 318 may be turned off. In addition, the video recording control unit 357 of the second embodiment associates the detection value of the left door distance measuring sensor 308 and the detection value of the right door distance measuring sensor 309 with the shooting results of the wide-angle camera 305 and the narrow-angle camera 306 in the video recording, as in the first embodiment. Furthermore, in the video recording, the video recording control unit may associate the pulling degree with the shooting results of the wide-angle camera 305 and the narrow-angle camera 306 in the video recording.

[0328] The first selection unit 358A of the second embodiment selects the first captured image from the moving image represented by the third recorded data 377 . The second selection unit 358B selects the second captured image from the moving image represented by the fourth recorded data 378 .

[0329] The registration unit 360 in the second embodiment registers information (e.g., SSID, password) for connecting to the network NW by generating the third registration data 373C and causing the second memory 370 to store the third registration data 373C. In addition, in the present embodiment, before the camera 3 starts to be used, the first registration data 373A and the fourth registration data 373D are stored in the second memory 370. In addition, in the present embodiment, the registration unit 360 can perform various registrations even after the cold storage 2B starts to be used. In this case, the registration unit 360 updates the corresponding registration data to the latest data each time the registration is performed.

[0330] The heating control unit 361 heats the heater 222. When the person presence determination unit 354 determines that a person exists around the cold storage 2B, the heating control unit 361 heats the heater 222 for a predetermined period. In addition, when the opening and closing determination unit 355 determines that the door 21A has become an open state, the heating control unit 361 heats the heater 222. When the opening and closing determination unit 355 determines that the door 21A has become a closed state, the heating control unit 361 ends the heating of the heater 222.

[0331] [2-1-3. Structure of the item management server]

[0332] The article management DB 512 included in the article management server 5 of the second embodiment has one record for each camera 3. The one record includes the camera ID 372, the image data of the first captured image, and the image data of the second captured image. In the second embodiment, the image data of the second captured image included in the record is different from the image data of the first embodiment. The second captured image in the second embodiment will be described later.

[0333] [2-2. Action]

[0334] Next, the operation of each part of the article management system 1B will be described.

[0335] [2-2-1. Cold storage settings]

[0336] For example, user P performs settings of refrigerator 2B after purchasing refrigerator 2B until refrigerator 2B starts to be used. In the settings of refrigerator 2B, at least information for connecting to network NW is registered.

[0337] [Registration of information for connecting to 2-2-1-1. LAN]

[0338] The information processing system 7B of the second embodiment registers information for connecting to the local area network in the same manner as the information processing system 7A of the first embodiment. That is, the cold storage 2B receives the SSID and password received by the terminal device 4 from the terminal device 4, and the registration unit 360 of the cold storage 2B generates the third registration data 373C and causes the second memory 370 to store the third registration data 373C.

[0339] [2-2-2. About video recording]

[0340] First, the operation of the refrigerator 2B regarding the recording will be described.

[0341] The cold storage 2B of the second embodiment performs the same operation as the camera 3 of the first embodiment. That is, the person presence determination unit 354 of the second embodiment performs the processing of steps SG1 and SG7. The opening and closing determination unit 355 of the second embodiment performs the processing of steps SG3 and SG5. The shooting control unit 356 of the second embodiment performs the processing of steps SG2 and SG8. In addition, the recording control unit 357 of the second embodiment performs the processing of steps SG4 and SG6.

[0342] [2-2-3. About uploading to the item management server]

[0343] Next, the operation of uploading to the article management server 5 will be described.

[0344] Fig. 22 1B is a flowchart showing the operation of the article management system 1B. Fig. 22 Flowchart FL shows the operation of refrigerator 2B, and flow chart FM shows the operation of article management server 5.

[0345] The generation unit 359 determines whether the recording file 376 is stored in the second memory 370 (step SL1). When the generation unit 359 determines that the recording file 376 is not stored (step SL1: No), the determination of step SL1 is repeated.

[0346] On the other hand, when the generation unit 359 determines that the video file 376 is stored (step SL1: Yes), the first selection unit 358A selects the first captured image from the third video data 377 included in the video file 376 stored in the second memory 370 (step SL2). The first captured image of this embodiment is the same as the first captured image described in the first embodiment. The first selection unit 358A of the second embodiment selects the first captured image in the same manner as in the first embodiment.

[0347] The first selection unit 358A determines whether the first captured image is selected (step SL3 ). When the first selection unit 358A determines that the first captured image is not selected (step SL3 : No), the second processor 350 performs the process of step SL11 .

[0348] On the other hand, when the first selection unit 358A determines that the first captured image is selected (step SL3 : Yes), the second selection unit 358B selects the second captured image from the fourth recorded data 378 included in the recorded file 376 stored in the second memory 370 (step SL4 ).

[0349] The second captured image of the present embodiment is a captured image of the drawer 26A in the following state, and is a captured image in which the user P is least reflected. The above-mentioned following state is a state in which the pulling degree is γ% or more when the pulling degree of the closed state is 0% and the pulling degree of the predetermined open state is 100%. γ% is, for example, 70%.

[0350] The second selection unit 358B recognizes the position of the front edge of the drawer 26A recorded in the fourth registration data 373D as the position when the drawer 26A is pulled out to 100%, and recognizes the position of the upper end of the captured image captured by the narrow-angle camera 306 as the position of the front edge of the drawer 26A when the drawer 26A is pulled out to 0%. Furthermore, the second selection unit 358B selects a captured image in which the front edge of the drawer 26A is located at a position greater than γ% based on the two recognized positions. Next, the second selection unit 358B selects the captured image in which the user P is least reflected from the selected captured images as the second captured image.

[0351] The second selection unit 358B determines whether the second captured image is selected (step SL5). When the second selection unit 358B determines that the second captured image is selected (step SL5: Yes), the generation unit 359 generates the third upload data (step SL6). The third upload data includes the image data of the first captured image selected in step SL1, the image data of the second captured image selected in step SL4, and the camera ID 372 stored in the second memory 370.

[0352] Next, the third communication control unit 352 transmits the third upload data generated by the generation unit 359 to the item management server 5 (step SL7).

[0353] Returning to the description of step SL5, when the second selection unit 358B determines that the second captured image has not been selected (step SL5: No), the generation unit 359 generates the fourth upload data (step SL8). The fourth upload data includes the image data of the first captured image selected in step SL1 and the camera ID 372 stored in the second memory 370.

[0354] Next, the third communication control unit 352 transmits the fourth upload data generated by the generation unit 359 to the item management server 5 (step SL9).

[0355] The generating unit 359 deletes the recorded video file 376 processed after step SL2 from the second memory 370 (step SL10 ).

[0356] As shown in flowchart FM, the server communication control unit 501 determines whether the third upload data or the fourth upload data is received from the image capture device 3 (step SM1). If the server communication control unit 501 determines that the upload data is not received (step SM1: No), the determination of step SM1 is repeated.

[0357] On the other hand, when the server communication control unit 501 determines that the third uploaded data or the fourth uploaded data has been received (step SM1: Yes), the server processing unit 502 updates the item management DB 512 based on the uploaded data received by the server communication control unit 501 (step SM2).

[0358] Step SM2 will be described in detail.

[0359] The server processing unit 502 determines the record containing the camera ID 372 included in the third uploaded data from the item management DB 512, and updates the image data of the first captured image and the image data of the second captured image of the determined record to the image data of the first captured image and the image data of the second captured image included in the third uploaded data.

[0360] The server processing unit 502 identifies the record including the imaging device ID 372 included in the fourth uploaded data from the article management DB 512 , and updates the image data of the first captured image in the identified record to the image data of the first captured image included in the fourth uploaded data.

[0361] [2-2-4. About the operation of application UI]

[0362] Next, the operation of the article management system 1B related to the application UI will be described.

[0363] Article management system 1B according to Embodiment 2 performs the same operation as article management system 1A according to Embodiment 1. Terminal device 4 according to this embodiment displays an image of refrigerating room 21 and an image of drawer 26A.

[0364] [2-3. Effects, etc.]

[0365] As described above, the cold storage 2B has a cold storage room 21. The cold storage 2B includes: a wide-angle camera 305 for photographing the cold storage room 21; and a first selection unit 358A for selecting, from the dynamic image photographed by the wide-angle camera 305, a first photographed image of the cold storage room 21 showing the door 21A in an open state, based on the rotation angle of the rotary door 21A for opening and closing the opening of the cold storage room 21.

[0366] Thereby, the same effect as that of the first embodiment can be obtained.

[0367] Refrigerator 2B includes housing 20 forming refrigerating chamber 21 and distance sensor 312 provided on upper surface 20A of housing 20 for measuring the distance to door 21A. First selection unit 358A selects the first captured image based on the rotation angle of door 21A corresponding to the detection value of distance sensor 312.

[0368] Thereby, the same effects as those of the above-mentioned first embodiment are obtained.

[0369] The distance measuring sensor 312 is provided so that the sensing direction is toward the rotation axis side of the door 21A.

[0370] Thereby, the same effects as those of the above-mentioned first embodiment are obtained.

[0371] The ice and fresh-keeping room 26 is formed inside the refrigerating room 21. The wide-angle camera 305 is disposed on the inner upper surface of the refrigerating room 21.

[0372] Thus, since the wide-angle camera 305 can shoot from above the refrigerating room 21, the image captured by the wide-angle camera 305 can include the interior of the refrigerating room 21 and the inner surface of the door 21A in the open state. Therefore, a captured image that more appropriately reflects the internal conditions of the refrigerating room 2B can be selected.

[0373] The refrigerator compartment 21 is cooled by cold air. A heater 222 for heating the lens of the wide-angle camera 305 is included.

[0374] Thus, since the lens blur of wide-angle camera 305 can be suppressed when door 21A is in the open state, the blur reflected in the lens can be suppressed in the captured image captured by wide-angle camera 305. Therefore, a captured image that more appropriately reflects the internal conditions of cold storage 2B can be selected.

[0375] The cold storage 2B has a cold storage room 26. The cold storage 2B includes: a narrow-angle camera 306 for photographing the drawer 26A of the cold storage room 26, and a second selection unit 358B for selecting a second photographed image showing the drawer 26A in an open state from the dynamic image photographed by the narrow-angle camera 306 based on the degree to which the drawer 26A is pulled out.

[0376] Thereby, the same effect as that of the first embodiment can be obtained.

[0377] The ice and fresh-keeping room 26 is formed inside the refrigerating room 21. The narrow-angle camera 306 is disposed on the inner upper surface of the refrigerating room 21.

[0378] Thus, since narrow-angle camera 306 can capture images from above refrigerating room 21, the opening of drawer 26A can be included in the captured image captured by narrow-angle camera 306. Therefore, a captured image that more appropriately captures the interior conditions of refrigerator 2B can be selected.

[0379] The refrigerating chamber 21 is cooled by cold air. A heater 222 for heating the lens of the narrow-angle camera 306 is included.

[0380] Thus, since the lens blur of narrow-angle camera 306 can be suppressed when door 21A is in the open state, the blur reflected in the lens can be suppressed in the captured image captured by narrow-angle camera 306. Therefore, a captured image that more appropriately reflects the internal conditions of cold storage 2B can be selected.

[0381] (Other embodiments)

[0382] As described above, the above-mentioned embodiments 1 and 2 are described as examples recorded in this specification. However, the technology of the present invention is not limited to this, and can also be applied to embodiments that have been changed, replaced, added, omitted, etc. In addition, the various components described in the above-mentioned embodiments 1 and 2 can also be combined to form a new embodiment. Therefore, other embodiments are exemplified below.

[0383] In the above-mentioned Embodiment 1 and Embodiment 2, the double-opening door 21A is exemplified. However, the door 21A is not limited to the double-opening door 21A. The widths of the left door 21B and the right door 21C may be different.

[0384] In the above-mentioned Embodiment 1 and Embodiment 2, the ice-warm fresh-keeping room 26 formed in the refrigerator compartment 21 and having a lower internal temperature than the refrigerator compartment 21 is exemplified. However, the drawer formed in the refrigerator compartment 21 is not limited to the ice-warm fresh-keeping room 26. That is, a storage box such as a vegetable box may also be formed in the refrigerator compartment 21.

[0385] In the above-mentioned Embodiments 1 and 2, the first LED 317 and the second LED 318 are exemplified. However, the first LED 317 and the second LED 318 may be configured as one LED. Thus, the user P can be notified of the start of shooting with one LED, and the storage room 27 can be illuminated. In addition, the number of components can be reduced.

[0386] In the above-mentioned Embodiment 1 and Embodiment 2, the wide-angle camera 305 and the narrow-angle camera 306 are illustrated. However, the wide-angle camera 305 and the narrow-angle camera 306 may be configured as one camera. In this case, the shooting range of the camera may be changed based on the opening and closing state of the opening and closing body 29A. In addition, the camera may be switched to a wide angle or a narrow angle based on the opening and closing state of the opening and closing body 29A. Thus, the number of components can be reduced.

[0387] In other embodiments, the cold storage 2A may include a communication module that can be connected to a local area network. In the case of adopting this structure, the communication module is arranged in a superimposed manner with the first hinge member 203A or the second hinge member 203B. In addition, in the case of adopting this structure, a component that can shield radio waves can be arranged between the position of the first hinge member 203A or the second hinge member 203B and the position of the set shooting device 3. For example, the communication module of the shooting device 3 can be surrounded by metal. In addition, the side of the shooting device 3 can be composed of metal. In addition, in the case of adopting this, the terminal device 4 can also send the SSID and password handed over to the shooting device 3 to the communication module of the cold storage 2A. The main control circuit board of the cold storage 2A receives the SSID and password through the communication module of the cold storage 2A, and uses the received SSID and password to connect the cold storage 2A to the network NW. In addition, in the case where the main control circuit board of the cold storage 2A stores the SSID and password, the SSID and password can also be sent to the shooting device 3. The image capturing device 3 is connected to the network NW using the received SSID and password. The main body control circuit board of the refrigerator 2A is installed in a storage space of a recessed portion formed in the upper surface of the refrigerator 2A.

[0388] In addition, in other embodiments, the cold storage 2B may include a communication module that can be connected to a local area network. In the case of adopting this structure, the communication module is arranged to overlap with the first hinge member 203A or the second hinge member 203B. In addition, in the case of adopting this structure, the third upload data and the fourth upload data may also be sent to the article management server 5 by the communication module.

[0389] In other embodiments, the communication device 6 may receive input of the SSID and password of the local area network to be established via an application, and the imaging device 3 may be connected to the local area network.

[0390] In other embodiments, the model information of the refrigerator may be registered in advance via the application. The model information may include, for example, the number of layers of the refrigerator compartment, the number of layers of the door shelves, the number of doors, the number of layers of the storage boxes of the vegetable compartment and the freezer compartment.

[0391] In other embodiments, the heating temperature of heater 222 may be different depending on the temperature and humidity outside refrigerator 2B. In this case, a sensor for measuring temperature and humidity is provided in refrigerator 2B, and heating control unit 361 recognizes the temperature and humidity outside refrigerator 2B using the sensor.

[0392] In the above-mentioned embodiment, a distance measuring sensor is exemplified as a sensor for detecting the opening and closing of the door 21A and the drawer 28A. However, the sensor for detecting the opening and closing of the door 21A may be other sensors such as a Hall IC, an angle sensor, a camera, a temperature sensor, etc. In addition, the sensor for detecting the opening and closing of the drawer 28A may be other sensors such as a Hall IC, an acceleration sensor, a mechanical switch, an angle sensor, a camera, a temperature sensor, etc.

[0393] In the above-mentioned embodiment, the state in which the left door 21B and the right door 21C are opened at 90° is exemplified as the prescribed open state of the door 21A. However, the prescribed open state of the door 21A is not limited to the state in which it is opened at 90°, and it may be, for example, a state in which it is opened at any angle such as 60° or 100°. In addition, the prescribed open state of the door 21A may be a state in which the left door 21B and the right door 21C are fully open. The so-called fully open state is, for example, a state in which the door 21B and the right door 21C are opened at 150°. In this case, the first guide image AG1 may not be displayed. The same is true for the prescribed open state of the drawer 28A, and the prescribed open state of the drawer 28A is not limited to a state in which it is pulled out to the maximum extent.

[0394] In other embodiments, in addition to the functional units of the second processor 350 in the first embodiment or the second embodiment, the second processor 350 further functions as a detection unit. In the case where the opening and closing angle of at least one of the left door 21B and the right door 21C reaches an angle (for example, 45° or more) at which food can be taken out from the video recording control unit 357 starting until the end of the video recording of the camera 307, the detection unit may capture an image of the object being put into or taken out of the refrigerating chamber 21 from the dynamic image. In addition, in the case where a mark is given to the object, the detection unit may detect the marked object.

[0395] In other embodiments, in addition to the functional units of the second processor 350 in the first embodiment or the second embodiment, the second processor 350 also functions as an entry and exit judgment unit. In this case, the entry and exit judgment unit changes the boundary line used for the entry and exit judgment according to the opening and closing angles of the left door 21B and the right door 21C to judge the entry and exit of the article. In addition, the entry and exit judgment unit may change the boundary line used for the entry and exit judgment according to the pulling degree of the drawer 28A to judge the entry and exit of the article.

[0396] In other embodiments, the first captured image may be corrected and uploaded to the article management server 5. As an example of image correction, correction by viewpoint conversion can be cited. The correction value of the image correction may be a value corresponding to the rotation angle of the door 21A or a fixed value.

[0397] In the above-mentioned embodiment, the case where the first captured image and the second captured image are used for display purposes of the terminal device 4 is exemplified. However, the first captured image and the second captured image may be used for other purposes such as detecting articles stored in the cold storages 2A and 2B.

[0398] In other embodiments, it is also possible to select a captured image of a storage box included in the drawer 28A from the dynamic image captured by the narrow-angle camera 306, and upload it to the article management server 5. In this case, the prescribed open state of the storage box is registered by the registration unit 360. In addition, when registering the state of the storage box, the terminal device 4 displays the second guide image AG2. Since the second guide image AG2 includes the line L17, the user P can easily set the state of the storage box to the prescribed open state.

[0399] In the above-mentioned embodiment, the refrigerators 2A and 2B are exemplified as the "storage" of the present invention. However, the "storage" of the present invention is not limited to the refrigerators 2A and 2B, and may be other storages such as a freezer and a room temperature storage.

[0400] In other embodiments, the storage room formed in the box 20 of the cold storage 2A, 2B may be a storage room other than the above-mentioned five types, and at least one of the drawer rooms 28 may not be provided. In addition, in other embodiments, the number of doors provided in the opening portion of the front surface of the cold storage room 21 is only one. In this case, the distance sensor 312 is provided on the side of the hinge side of the pivotally supported door in the side surface of the main body 301. Thus, the opening and closing angle of the door can be detected. In addition, the distance sensor 312 is provided on the upper surface 20A of the box 20 in a manner facing the hinge of the pivotally supported door. Thus, the opening and closing angle of the door can be detected.

[0401] The first processor 320, the second processor 350, the server processor 50, and the terminal processor 40 may be composed of a single processor or a plurality of processors. These processors may also be hardware programmed to implement corresponding functional units. That is, these processors may be composed of, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0402] Fig. 9 , Fig.10 and Fig.21 The structure of the camera 3, the cold storage 2B, the terminal device 4 and the article management server 5 shown is an example, and the specific implementation is not particularly limited. That is, it is not necessary to install hardware corresponding to each part separately, and a structure in which the functions of each part are realized by executing a program by a processor can also be adopted. In addition, in the above-mentioned implementation, part of the function realized by software can be used as hardware, or part of the function realized by hardware can be realized by software.

[0403] Fig.11 , Fig.12 , Fig.14 , Fig.16 , Fig.17 , Fig.18 and Fig. 22 The step units of the actions shown are formed by dividing according to the main processing content in order to make the actions easier to understand. The actions are not limited by the division method and name of the processing units. It is also possible to divide more step units according to the processing content. In addition, it is also possible to divide in a way that one step unit contains more processing. In addition, the order of the steps can be appropriately exchanged within the scope that does not hinder the main purpose of the present invention.

[0404] Furthermore, the above-described embodiments are modes for illustrating the technology of the present invention, and therefore various changes, substitutions, additions, omissions, etc. can be made within the scope of the claims or their equivalents.

[0405] Industrial Applicability

[0406] As described above, the information processing system and the imaging device of the present invention can be used for applications using a captured image showing the internal conditions of a storage.

[0407] Description of Reference Numerals

[0408] 1A, 1B item management system

[0409] 2A, 2B cold storage (storage)

[0410] 3. Shooting device

[0411] 4 Terminal Devices

[0412] 5 Item Management Server

[0413] 7A, 7B Information Processing System

[0414] 20 cabinets

[0415] 20A upper surface

[0416] 21 Refrigerator (Storage Room 1)

[0417] Door 21A

[0418] 26 Ice and temperature preservation room (Second storage room)

[0419] 26A Drawer

[0420] 27Storage Room

[0421] 28 Drawer Room (Second Storage Room)

[0422] 28A Drawer

[0423] 29A opening and closing body

[0424] 205 concave part

[0425] 211 Door Shelf (Storage Parts)

[0426] 222 heater (heating unit)

[0427] 301 Main Body

[0428] 302 shooting parts

[0429] 305 wide-angle camera (first camera)

[0430] 306 narrow angle camera (second camera)

[0431] 307 Cameras

[0432] 358 Selection Department

[0433] 358A 1st Selection

[0434] 358B Second Option

[0435] 360 Registration Department

[0436] 401 Terminal Communication Control Unit (Communication Control Unit)

[0437] 402 Display Control Unit

[0438] AG1 1st guide image

[0439] AG2 2nd guide image

[0440] L line.

Claims

1. An information processing system, characterized in that: comprising a photographing device and a terminal device capable of communicating with the photographing device, The photographing device comprises: A camera for photographing a storage room, wherein the storage room has a box body forming a storage room and an opening and closing body for opening and closing an opening of the storage room; a registering unit for registering a predetermined open state of the opening and closing body; and A selection unit selects, based on the predetermined open state registered by the registration unit, an image captured when the opening and closing body is in the open state from among the images captured by the camera.

2. The information processing system according to claim 1, wherein: The terminal device comprises: a display control unit that displays a guide image for guiding the state of the opening and closing body to the prescribed open state; and a receiving unit that receives an input for changing the state of the opening and closing body to the prescribed open state, The registration unit registers the current state of the opening and closing body as the predetermined open state when the receiving unit receives the input.

3. The information processing system according to claim 2, wherein: The terminal device includes: a communication control unit that receives the dynamic image captured by the camera from the imaging device, The display control unit displays the imaging result of the camera received by the communication control unit in a superimposed manner with the guide image.

4. The information processing system according to claim 3, wherein: The opening and closing body is a rotating door. the display control unit displays a first guide image as the guide image for guiding the state of the door to the predetermined open state, The first guide image includes a line corresponding to an edge of a door shelf of the door in the predetermined open state.

5. The information processing system according to claim 3, wherein: The opening and closing body is a drawer, the display control unit displays a second guide image as the guide image for guiding the state of the drawer to the predetermined open state, The second guide image includes a line corresponding to a front edge of the drawer in the predetermined open state.

6. The information processing system according to claim 4 or 5, characterized in that: The display position of the line in the guide image differs depending on the model of the storage captured by the camera.

7. A photographing device, characterized in that: include: A camera for photographing a storage room, wherein the storage room has a box body forming a storage room and an opening and closing body for opening and closing an opening of the storage room; a registering unit for registering a predetermined open state of the opening and closing body; and A selection unit selects, based on the predetermined open state registered by the registration unit, an image captured when the opening and closing body is in the open state from among the images captured by the camera.