Program, terminal device, method for controlling the terminal device, and display system

Abstract

To provide a program capable of allowing a user to intuitively comprehend the situation inside storage.SOLUTION: The program causes a terminal processor to function as a display control unit for displaying, on a display, a first front view image that is an image generated from an image of the storage room picked up from above the front of the storage room of the storage, which is an image of the storage room viewed from a position on the front side of the storage room.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a program, a terminal device, a method for controlling the terminal device, and a display system.

Background Art

[0002] Patent Document 1 discloses a technique in which a camera is provided above a refrigerator, photographing is executed when an object passing through an opening of the refrigerator is detected, and food stored in the refrigerator is managed based on the photographing result.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a program, a terminal device, a method for controlling the terminal device, and a display system that enable a user to intuitively grasp the situation inside a storage.

Means for Solving the Problems

[0005] The program in the present disclosure causes a processor of a terminal device to function as a display control unit that causes a display unit to display a first front view image that is an image generated from a photographed image of the storage chamber taken from the front upper side of the storage chamber of the storage and is an image of the storage chamber viewed from a position on the front side of the storage chamber, and a reception unit that receives an operation from a user. When the reception unit receives a downward swipe operation from the user, the display control unit gradually displays, side by side with the first front view image below the first front view image, a photographed image of a chilled chamber taken from the front upper side of the storage chamber in a manner that moves along with the movement of an operator in the downward swipe operation and moves from above downward on the display unit. A marker is placed on the drawer of the chilled compartment, and the image of the chilled compartment is an image taken when the distance between the back of the chilled compartment and the drawer of the chilled compartment is greater than a predetermined value. 。

[0006] Furthermore, the terminal device in this disclosure includes a display control unit that displays a first front view image on a display unit, which is an image generated from an image of the storage room taken from the front and above of the storage room, and is an image of the storage room viewed from a position on the front side of the storage room; and a reception unit that receives operations from a user. When the reception unit receives a downward swipe operation from the user, the display control unit gradually displays an image of the chilled compartment taken from the front and above of the storage room on the display unit below the first front view image, alongside the first front view image, in accordance with the movement of the operator in the downward swipe operation and in a manner that moves from top to bottom. A marker is placed on the drawer of the chilled compartment, and the image of the chilled compartment is an image taken when the distance between the back of the chilled compartment and the drawer of the chilled compartment is greater than a predetermined value. .

[0007] Furthermore, the control method for the terminal device in this disclosure includes the step of displaying on the display unit a first front view image which is an image generated from an image of the storage room taken from the front and above of the storage room, and is an image of the storage room viewed from a position on the front side of the storage room, and when a downward swipe operation is received from the user, the step includes gradually displaying on the display unit below the first front view image and alongside the first front view image, in accordance with the movement of the operator in the downward swipe operation and in a manner that moves from top to bottom, an image of the chilled compartment taken from the front and above of the storage room, in accordance with the movement of the operator in the downward swipe operation and in a manner that moves from top to bottom A marker is placed on the drawer of the chilled compartment, and the image of the chilled compartment is an image taken when the distance between the back of the chilled compartment and the drawer of the chilled compartment is greater than a predetermined value. .

[0008] Furthermore, the display system in this disclosure comprises a generation unit that generates a first front view image of the storage room of a storage facility, viewed from a position on the front side of the storage room, from an image of the storage room taken from above and in front of the storage room, and a terminal device that displays the first front view image generated by the generation unit, wherein when the terminal device receives a downward swipe operation from the user, it gradually displays an image of the chilled compartment taken from above and in front of the storage room, in accordance with the movement of the operator in the downward swipe operation and in a manner that moves from top to bottom, below the first front view image and alongside the first front view image. A marker is placed on the drawer of the chilled compartment, and the image of the chilled compartment is an image taken when the distance between the back of the chilled compartment and the drawer of the chilled compartment is greater than a predetermined value. . [Effects of the Invention]

[0009] The program, terminal device, control method for the terminal device, and display system described in this disclosure allow the user to intuitively understand the conditions inside the storage facility. [Brief explanation of the drawing]

[0010] [Figure 1] Diagram showing the configuration of the display system [Figure 2] Front view of the imaging device, seen from the front. [Figure 3] Plan view of the imaging device from the right side. [Figure 4] Block diagram showing the configuration of the imaging device. [Figure 5] Block diagram showing the configuration of terminal equipment and image management server. [Figure 6] A diagram showing an example of records stored in the image management database. [Figure 7] Flowchart showing the operation of the imaging device [Figure 8] Flowchart illustrating the operation of the display system [Figure 9] An explanatory diagram illustrating the correction of distortion aberration in a specific frame. [Figure 10] An explanatory diagram illustrating the extraction of images of the refrigerator compartment, left door, and right door. [Figure 11] An explanatory diagram illustrating the viewpoint transformation in images of the refrigerator compartment. [Figure 12] An explanatory diagram illustrating the perspective shift in the door image. [Figure 13] Flowchart illustrating the operation of the display system [Figure 14] A diagram showing an example of an app UI. [Modes for carrying out the invention]

[0011] (Knowledge and other information that formed the basis of this disclosure) When the inventors came up with the present disclosure, there was a technology in which a terminal device displayed a list of foods stored in a refrigerator. However, the inventors discovered that simply displaying a list of foods, as in the prior art, presented the problem that users could not intuitively grasp the situation inside a storage device such as a refrigerator. In order to solve this problem, the inventors arrived at the subject matter of the present disclosure. Therefore, the present disclosure provides a program, a terminal device, a control method for the terminal device, and a display system that enable users to intuitively grasp the situation inside a storage device.

[0012] Hereinafter, embodiments will be described in detail with reference to the drawings. However, there may be cases where more detailed descriptions than necessary are omitted. For example, there may be cases where detailed descriptions of well-known matters or redundant descriptions of substantially identical configurations are omitted. Note that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0013] (Embodiment 1) Embodiment 1 will be described. [1. Configuration] [1-1. Configuration of Display System] FIG. 1 is a diagram showing the configuration of a display system 1000. The display system 1000 is a system that displays an image inside the refrigerator 1. The refrigerator 1 corresponds to the "storage device" of the present disclosure.

[0014] The display system 1000 includes the refrigerator 1. Refrigerator 1 comprises a main box 10 with an open front. The main box 10 includes a refrigerator compartment 11, an ice-making compartment 12, a fresh freezing compartment 13, a freezer compartment 14, and a vegetable compartment 15. A chilled compartment 11D is provided inside the refrigerator compartment 11. The chilled compartment 11D is a storage compartment with a lower internal temperature than the refrigerator compartment 11. The internal temperature of the chilled compartment 11D is set, for example, between 0°C and 3°C. The chilled compartment 11D is provided with a drawer 11D1 capable of storing items. In this embodiment, the items are food or containers for storing food. The front opening of the refrigerator compartment 11 is provided with a rotating left door 11A and a rotating right door 11B. Inside the left door 11A is a left door shelf 11A1 with a door pocket 111. Inside the right door 11B is a right door shelf 11B1 with a door pocket 111. Each of the ice-making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15 is provided with drawers 12A, 13A, 14A, and 15A, respectively, capable of storing items. The refrigerated room 11 corresponds to the “storage room” in this disclosure.

[0015] In the following description, if the left door 11A and the right door 11B are not distinguished, they will be referred to as "door" and designated as "11C". Also, in the following description, if the left door shelf 11A1 and the right door shelf 11B1 are not distinguished, they will be referred to as "door shelf" and designated as "11C1". Also, in the following description, if the drawers 12A, 13A, 14A, and 15A are not distinguished, they will be referred to as "drawer" and designated as "16A".

[0016] The display system 1000 includes a camera 2. The camera 2 is installed on the upper surface 10A of the main casing 10 of the refrigerator 1. The camera 2 photographs the refrigerator 1 from the front and above.

[0017] The display system 1000 includes a terminal device 3. The terminal device 3 is, for example, a smartphone or a tablet. The terminal device 3 has a display application 311 installed. The display application 311 is an installable application program that has the function of displaying images of the inside of the refrigerator 1. The terminal device 3 communicates with the image management server 4 through the functions of the display application 311. The display application 311 corresponds to the “program” in this disclosure.

[0018] In Figure 1, a solid line represents a user P who is at home, and a dotted line represents a user P who has left home H. When terminal device 3 is used by user P who is at home, it communicates with the image management server 4 via the communication device 5, or without using the communication device 5, through the functions of the display application 311. Furthermore, when terminal device 3 is used by user P who has left home H and cannot establish a communication connection with the communication device 5, it communicates with the image management server 4 without using the communication device 5, through the functions of the display application 311.

[0019] The communication device 5 connects to the communication network NW and communicates with the image management server 4 via the communication network NW. The communication network NW is a network consisting of a public telephone network, a dedicated line, other communication lines, and various communication equipment, and its specific configuration is not limited. For example, the communication network NW may be a wide-area network. The communication network NW may also include at least one of a wireless communication circuit and a wired communication circuit. The communication device 5 functions as an interface device for connecting the imaging device 2 and the terminal device 3 to the communication network NW.

[0020] The display system 1000 includes an image management server 4. The image management server 4 is a server device that manages the image data of images displayed by the functions of the display application 311. The image management server 4 is connected to a communication network NW. In each figure, the image management server 4 is represented by a single block, but this does not necessarily mean that the image management server 4 is composed of a single server device. For example, the image management server 4 may be composed of multiple server devices with different processing functions.

[0021] [1-2. Configuration of the imaging device] The imaging device 2 will be described with reference to Figures 2, 3, and 4. Figures 2 and 3 illustrate the X, Y, and Z axes. The X, Y, and Z axes are orthogonal to each other. The Z axis indicates the vertical direction. The X and Y axes are parallel to the horizontal direction. The X axis indicates the left-right direction. The Y axis indicates the front-back direction. The positive direction of the X axis is to the right. The positive direction of the Y axis is forward. The positive direction of the Z axis is upward.

[0022] Figure 2 is a front view of the camera device 2 installed in the refrigerator 1, viewed from the front. In Figure 2, the left door 11A and the right door 11B are open. Figure 3 is a top view of the camera device 2 installed in the refrigerator 1, viewed from the right. In Figure 3, the drawer 15A is open, i.e., pulled out.

[0023] The imaging device 2 includes a camera 27 for the refrigerator compartment and a camera 28 for the drawer. The refrigerator compartment camera 27 is a wider-angle camera than the drawer camera 28. When the camera device 2 is installed in the refrigerator 1 and the door 11C1 is open, the refrigerator compartment camera 27 photographs the refrigerator 1 from the front and above the refrigerator compartment 11. When the door 11C1 is closed, the refrigerator compartment camera 27 photographs the refrigerator 1 from the front and above the refrigerator compartment 1. The image captured by the refrigerator compartment camera 27 is a moving image.

[0024] For example, the shooting range of the refrigerator compartment camera 27, viewed from the front of the refrigerator 1, includes the range A1 shown in Figure 2. Range A1 includes the door shelf 11C1 and the opening of the refrigerator compartment 11 when the left door 11A and the right door 11B are open. The opening of the refrigerator compartment 11 is the opening through which items are loaded and unloaded. Furthermore, the shooting range of the refrigerator compartment camera 27, viewed from the right side of the refrigerator 1, includes the range A3 shown in Figure 3. Range A3 includes, in the front-to-back direction, the area from the front end of the open door 11C to the front end of the shelf 112, and in the up-to-down direction, the opening of the refrigerator compartment 11.

[0025] The drawer camera 28 photographs the refrigerator 1 from the front and above. The result of the drawer camera 28 is a moving image. For example, the shooting range of the drawer camera 28 includes the range A2 shown in Figure 2, when viewed from the front of the refrigerator 1. Range A2 includes the chiller compartment 11D and the drawer 16A, when viewed from the front of the refrigerator 1. Also, for example, the shooting range of the drawer camera 28 includes the range A4 shown in Figure 3, when viewed from the right side of the refrigerator 1. Range A4 includes the drawer 16A in its fully extended state and the front end of the chiller compartment 11D when the drawer 11D1 is not extended.

[0026] The imaging device 2 includes a distance measuring sensor 23 for the left door, a distance measuring sensor 24 for the right door, and a distance measuring sensor 25 for the pull-out mechanism. The left door distance measuring sensor 23 is a distance measuring sensor. The left door distance measuring sensor 23 detects the distance between the left door 11A and the left door distance measuring sensor 23 at a predetermined period and outputs a detected value indicating the detected distance to the imaging device processor 20. The right door distance measuring sensor 24 is a distance measuring sensor. The right door distance measuring sensor 24 detects the distance between the right door 11B and the right door distance measuring sensor 24 at a predetermined period and outputs a detected value indicating the detected distance to the imaging device processor 20. The extraction distance measuring sensor 25 is a distance measuring sensor. The extraction distance measuring sensor 25 detects the distance between the extraction unit 16A and the extraction distance measuring sensor 25 at a predetermined period and outputs a detected value indicating the detected distance to the imaging device processor 20.

[0027] The distance measuring sensor 23 for the left door, the distance measuring sensor 24 for the right door, and the distance measuring sensor 25 for the pull-out door may be optical distance measuring sensors or ultrasonic distance measuring sensors.

[0028] The imaging device 2 is equipped with a human presence sensor 26. The human presence sensor 26 outputs a detected value to the imaging device processor 20 at a predetermined interval. The human presence sensor 26 may be an infrared sensor, an ultrasonic sensor, or a visible light sensor.

[0029] Note that the arrangement and installation positions of the cameras and sensors in Figures 2 and 3 are illustrated for the convenience of explaining the imaging device 2, and the arrangement and installation positions of the cameras and sensors are not limited to those shown in Figures 2 and 3.

[0030] Figure 4 is a block diagram showing the configuration of the imaging device 2. The imaging device 2 includes an imaging device processor 20, which is a processor such as a CPU (Central Processing Unit) or MPC (Micro-processing unit), and an imaging device memory 21, which is a memory for storing programs and data.

[0031] The imaging device processor 20 functions as the imaging device communication control unit 201, the person presence detection unit 202, the opening / closing detection unit 203, the imaging control unit 204, the recording control unit 205, the image generation unit 206, and the upload data generation unit 207 by reading and executing the control program 211 stored in the imaging device memory 21. The image generation unit 206 corresponds to the "generation unit" in this disclosure.

[0032] The imaging device memory 21 stores programs executed by the imaging device processor 20 and data processed by the imaging device processor 20. The imaging device memory 21 stores control programs 211 executed by the imaging device processor 20, imaging device ID (Identification) 212, and various other data. The imaging device memory 21 also stores recorded data 213 until it is erased by the imaging device processor 20. The imaging device memory 21 has a non-volatile storage area. Alternatively, the imaging device memory 21 may also have a volatile storage area and constitute the work area of ​​the imaging device processor 20.

[0033] The imaging device ID 212 is the identification information for imaging device 2. Examples of imaging device ID 212 include the serial number and manufacturing number. The recorded data 213 includes first recorded data 213A and second recorded data 213B. The first recorded data 213A is data showing the shooting results of the refrigerator camera 27. The second recorded data 213B is data showing the shooting results of the drawer camera 28.

[0034] The imaging device processor 20 is connected to the imaging device communication unit 22, the left door distance sensor 23, the right door distance sensor 24, the drawer distance sensor 25, the human presence sensor 26, the refrigerator compartment camera 27, and the drawer camera 28. Note that other devices may also be connected to the imaging device processor 20.

[0035] The imaging device communication unit 22 is a communication interface having communication circuits and connectors in accordance with a predetermined communication standard, and communicates with the image management server 4 according to the control of the imaging device processor 20. In this embodiment, the communication standard used by the imaging device communication unit 22 is exemplified as a wireless communication standard, but a wired communication standard may also be used.

[0036] As described above, the imaging device processor 20 functions as an imaging device communication control unit 201, a person presence detection unit 202, an open / close detection unit 203, an imaging control unit 204, a recording control unit 205, an image generation unit 206, and an upload data generation unit 207.

[0037] The imaging device communication control unit 201 communicates with the image management server 4 via the imaging device communication unit 22.

[0038] The person presence detection unit 202 determines whether or not a person is present around the refrigerator 1 based on the detection value output by the human presence sensor 26. For example, if the human presence detection unit 202 detects that a person is present around the refrigerator 1 when the human presence sensor 26 outputs a detection value equal to or greater than a predetermined value, it determines that there is no person present around the refrigerator 1 when the human presence detection unit 202 outputs a detection value equal to or greater than the predetermined value. If the person presence detection unit 202 determines that a person is present around the refrigerator 1, it outputs first presence information indicating the presence of a person to the shooting control unit 204. If the person presence detection unit 202 determines that there is no person present around the refrigerator 1, it outputs second presence information indicating the absence of a person to the shooting control unit 204.

[0039] The opening / closing determination unit 203 determines whether the left door 11A is open or closed based on the detection value of the left door distance sensor 23. The opening / closing determination unit 203 determines that the left door 11A is open if the left door distance sensor 23 outputs a detection value that is greater than the detection value corresponding to the closed state of the left door 11A. The opening / closing determination unit 203 determines that the left door 11A is closed if the left door distance sensor 23 outputs a detection value that corresponds to the closed state of the left door 11A. The opening / closing determination unit 203 determines whether the right door 11B is open or closed based on the detection value of the right door distance sensor 24. The opening / closing determination unit 203 determines that the right door 11B is open if the right door distance sensor 24 outputs a detection value that is greater than the detection value corresponding to the closed state of the right door 11B. The opening / closing determination unit 203 determines that the right door 11B is closed if the right door distance sensor 24 outputs a detection value that corresponds to the closed state of the right door 11B. The open / closed determination unit 203 determines whether the drawer 16A is in an open or closed state according to the distance detected by the drawer distance sensor 25.

[0040] If the opening / closing determination unit 203 determines that at least one of the left door 11A and the right door 11B is open, it outputs first door opening / closing information indicating that door 11C is open to the recording control unit 205. If the opening / closing determination unit 203 determines that both the left door 11A and the right door 11B are closed, it outputs second door opening / closing information indicating that both the left door 11A and the right door 11B are closed to the recording control unit 205. If the open / close determination unit 203 determines that any of the drawers 16A are in the open state, it outputs first drawer open / close information indicating that any of the drawers 16A are in the open state to the recording control unit 205. If the open / close determination unit 203 determines that any of the drawers 16A are in the closed state, it outputs second drawer open / close information indicating that any of the drawers 16A are in the closed state to the recording control unit 205.

[0041] If the person presence detection unit 202 outputs first presence information, the shooting control unit 204 causes the refrigerator camera 27 and the drawer camera 28 to start shooting. After the shooting of the refrigerator camera 27 and the drawer camera 28 has started, if the person presence detection unit 202 outputs second presence information, the shooting control unit 204 causes the refrigerator camera 27 and the drawer camera 28 to stop shooting.

[0042] The recording control unit 205 starts recording the images captured by the refrigerator camera 27 and the drawer camera 28 when the opening / closing determination unit 203 outputs first door opening / closing information. In other words, when the opening / closing determination unit 203 outputs first door opening / closing information, the recording control unit 205 starts recording recording data 213 to the recording device memory 21. The recording control unit 205 also starts recording the images captured by the refrigerator camera 27 and the drawer camera 28 when the opening / closing determination unit 203 outputs first drawer opening / closing information. The recording control unit 205 also starts recording the images captured by the refrigerator camera 27 and the drawer camera 28 when the opening / closing determination unit 203 outputs first door opening / closing information and first drawer opening / closing information. The recording control unit 205 ends recording the images captured by the refrigerator camera 27 and the drawer camera 28 when the recording determination unit 203 outputs second door opening / closing information and second drawer opening / closing information. Furthermore, the recording control unit 205 may be configured to terminate recording of the shooting results of the refrigerator compartment camera 27 and the drawer camera 28 when the opening / closing determination unit 203 outputs information about the opening / closing of the second door. In other words, the recording control unit 205 terminates recording of the recording data 213 to the shooting device memory 21 when the opening / closing determination unit 203 outputs information about the opening / closing of the second door. Also, the recording control unit 205 may be configured to terminate recording of the shooting results that were started when the opening / closing determination unit 203 outputs information about the opening / closing of the second drawer.

[0043] During recording, the recording control unit 205 receives detection values ​​from the door distance sensor 23 and the right door distance sensor 24. The recording control unit 205 associates the detection values ​​from the left door distance sensor 23 and the right door distance sensor 24 with the shooting results from the refrigerator camera 27 and the drawer camera 28 during recording, frame by frame, or at intervals of a predetermined number of frames.

[0044] The image generation unit 206 generates images that are displayed according to the functions of the application UI 330. Details of the images generated by the image generation unit 206 will be described later.

[0045] The upload data generation unit 207 generates upload data. Upload data is the data to be sent to the image management server 4. The upload data generation unit 207 outputs the generated upload data to the imaging device communication control unit 201.

[0046] [1-3. Image Management Server Configuration] Figure 5 is a block diagram showing the configuration of the terminal device 3 and the image management server 4. First, let's explain the configuration of image management server 4. The image management server 4 includes a server processor 40, which is a processor such as a CPU or MPC; server memory 41, which is memory for storing programs and data; and a server communication unit 42.

[0047] The server processor 40 functions as a server communication control unit 401 and a server processing unit 402 by reading and executing a control program 411 stored in the server memory 41.

[0048] The server memory 41 is a memory that stores programs executed by the server processor 40 and data processed by the server processor 40. The server memory 41 stores control programs 411 executed by the server processor 40, an image management database 412, and various other data. The server memory 41 has a non-volatile storage area. Alternatively, the server memory 41 may also have a volatile storage area and constitute the work area of ​​the server processor 40.

[0049] Figure 6 shows an example of a record R stored in the image management DB412. A single record R stored in the image management DB 412 contains account ID 4121, imaging device ID 212, image data table 4122, and chilled room image data table 4123.

[0050] Account ID 4121 is identification information that identifies the account assigned to user P who uses the display application 311.

[0051] Image data table 4122 stores multiple image data sets for image set KG. Image set KG is an image that includes a front view refrigerator compartment image RFG1, a front view left door image LDG1, and a front view right door image RDG1. The front view refrigerator compartment image RFG1, the front view left door image LDG1, and the front view right door image RDG1 will be described later. Image data table 4122 has a first field F1, a second field F2, a third field F3, a fourth field F4, and a fifth field F5.

[0052] The first field F1 stores the image data of the latest image set KG. The second field F2 stores the image data of the latest image set KG, following the image data stored in the first field F1. The third field F3 stores the image data of the latest image set KG, following the image data stored in the second field F2. The fourth field F4 stores the image data of the latest image set KG, following the image data stored in the third field F3. The fifth field F5 stores the image data of the latest image set KG, following the image data stored in the fourth field F4.

[0053] The chilled compartment image data table 4123 stores multiple chilled compartment image data. The chilled compartment image data is image data of the chilled compartment image CG, which is a captured image of the chilled compartment 11D. The chilled compartment image data table 4123 has a sixth field F6, a seventh field F7, an eighth field F8, a ninth field F9, and a tenth field F10.

[0054] Field 6, F6, stores the latest chilled compartment image data. The seventh field, F7, stores the chilled compartment image data that was stored in the sixth field, F6, during the last update of the chilled image data table 4123. The eighth field F8 stores the chilled compartment image data that was stored in the seventh field F7 during the last update of the chilled image data table 4123. The ninth field, F9, stores the chilled compartment image data that was stored in the eighth field, F8, during the last update of the chilled image data table 4123. The 10th field F10 stores the chilled compartment image data that was stored in the 9th field F9 during the last update of the chilled image data table 4123.

[0055] Returning to the explanation of Figure 5, the server communication unit 42 is a communication interface equipped with a communication circuit and the like that conforms to a predetermined communication standard, and communicates with devices connected to the communication network NW in accordance with the predetermined communication standard. In this embodiment, the server communication unit 42 communicates with the imaging device 2 and the terminal device 3.

[0056] As described above, the server processor 40 functions as a server communication control unit 401 and a server processing unit 402.

[0057] The server communication control unit 401 communicates with the imaging device 2 and the terminal device 3 via the server communication unit 42.

[0058] The server processing unit 402 performs various information processing tasks.

[0059] [1-4. Terminal Device Configuration] Next, we will describe the functional configuration of terminal device 3. The terminal device 3 includes a terminal processor 30, which is a processor such as a CPU or MPC; a terminal memory 31, which is a memory for storing programs and data; a terminal communication unit 32; and a touch panel 33. The terminal processor 30 corresponds to the “processor” in this disclosure. The touch panel 33 corresponds to the “display unit” in this disclosure.

[0060] The terminal processor 30 functions as an application execution unit 300 by reading and executing the display application 311 stored in the terminal memory 31.

[0061] The terminal memory 31 stores programs executed by the terminal processor 30 and data processed by the terminal processor 30. The terminal memory 31 also stores the display 311, account ID 4121, and various other data executed by the terminal processor 30. The terminal memory 31 has a non-volatile storage area. Alternatively, the terminal memory 31 may also have a volatile storage area and constitute the work area of ​​the terminal processor 30.

[0062] The terminal communication unit 32 is a communication interface having communication circuits and connectors in accordance with a predetermined communication standard, and communicates with the image management server 4 according to the control of the terminal processor 30. In this embodiment, the communication standard used by the terminal communication unit 32 is exemplified as a wireless communication standard, but a wired communication standard may also be used.

[0063] The touch panel 33 comprises a display panel such as a liquid crystal display panel, and a touch sensor that is superimposed on or integrated with the display panel.

[0064] As described above, the terminal processor 30 functions as an application execution unit 300. The application execution unit 300 functions as a terminal communication control unit 301, a display control unit 302, and a reception unit 303.

[0065] The terminal communication control unit 301 communicates with the image management server 4 via the terminal communication unit 32.

[0066] The display control unit 302 displays a user interface on the touch panel 33 that shows an image of the inside of the refrigerator 1. In the following description, this user interface will be referred to as the "app UI" and denoted by the code "330".

[0067] The reception unit 303 receives various operations from user P via the application UI 330 displayed on the touch panel 33.

[0068] [2. Operation] Next, the operation of each part of the display system 1000 will be described. [2-1. Actions related to recording] First, let's explain the operations related to recording. Figure 7 is a flowchart FA showing the operation of the imaging device 2.

[0069] The person presence detection unit 202 determines whether or not a person is present around the refrigerator 1 (step SA1).

[0070] If the person presence detection unit 202 determines that there is no person around the refrigerator 1 (step SA1: NO), it performs the determination in step SA1 again.

[0071] On the other hand, if the person presence detection unit 202 determines that a person is present around the refrigerator 1 (step SA1: YES), the shooting control unit 204 instructs the refrigerator compartment camera 27 and the drawer camera 28 to start shooting (step SA2).

[0072] Next, the opening / closing determination unit 203 determines whether at least one of the left door 11A and the right door 11B is in the open state (step SA3).

[0073] If the opening / closing determination unit 203 determines that the left door 11A and the right door 11B are not in the open state (step SA3: NO), the imaging device processor 20 performs the process in step SA7.

[0074] If the open / close determination unit 203 determines that at least one of the left door 11A and the right door 11B is open (step SA3: YES), the recording control unit 205 starts recording the images captured by the refrigerator compartment camera 27 and the drawer camera 28 (step SA4).

[0075] The opening / closing determination unit 203 determines whether the open door 11C has closed (step SA5).

[0076] If the open / closed determination unit 203 determines that the open door 11C is not in the closed state (step SA5: NO), it performs the process of step SA5 again.

[0077] On the other hand, if the open / close determination unit 203 determines that the open door 11C has closed (step SA5: YES), the recording control unit 205 terminates recording of the images captured by the refrigerator compartment camera 27 and the drawer camera 28 (step SA6).

[0078] Next, the person presence determination unit 202 determines whether or not a person is present around the refrigerator 1 (step SA7).

[0079] If the person presence detection unit 202 determines that a person is present around the refrigerator 1 (step SA7: YES), the imaging device processor 20 moves the process to step SA3.

[0080] On the other hand, if the person presence detection unit 202 determines that there is no person around the refrigerator 1 (step SA7: NO), the shooting control unit 204 tells the refrigerator compartment camera 27 and the drawer camera 28 to stop shooting (step SA8).

[0081] When the imaging device processor 20 completes the processing in step SA8, it moves the processing to step SA1 and then performs the processing from step SA1 onward again.

[0082] The recorded video data 213 stored in the camera memory 21 by the operation shown in Figure 7 has, for each frame or at intervals of a predetermined number of frames, the detection values ​​of the left door distance sensor 23 and the right door distance sensor 24 output during the recording of the shooting results from the refrigerator compartment camera 27 and the drawer camera 28, respectively.

[0083] [2-2. Actions related to sending uploaded data] Next, the operation of the display system 1000 related to the transmission of uploaded data will be described. Figure 8 is a flowchart showing the operation of the display system 1000. In Figure 8, flowchart FB shows the operation of the imaging device 2, and flowchart FC shows the operation of the image management server 4.

[0084] As shown in the flowchart FB, the upload data generation unit 207 of the imaging device 2 determines whether or not the recording of the imaging results from the refrigerator camera 27 and the drawer camera 28 has finished (step SB1).

[0085] If the upload data generation unit 207 determines that recording of the shooting results from the refrigerator camera 27 and the drawer camera 28 has finished (step SB1: YES), it obtains a specific frame FL1 from the first recording data 213A included in the newly recorded recording data 213 in the recording device memory 21 (step SB2).

[0086] A specific frame FL1 is a frame in which the left door 11A and the right door 11B are visible, both open at an angle of α°≦θ≦β°, and in which the user P is least visible. α° is, for example, 60°. β° is, for example, 100°. For the left door 11A, α° and β° are defined as 0° when the left door 11A is closed, and positive when the rotation direction is such that the left door 11A is open. Similarly, for the right door 11B, α° and β° are defined as 0° when the right door 11B is closed, and positive when the rotation direction is such that the right door 11B is open.

[0087] The image generation unit 206 identifies frames from the first recorded data 213A that correspond to the detection value of the right door distance measuring sensor 24, which indicates a separation distance corresponding to an angle α°≦θ≦β°, and the detection value of the left door distance measuring sensor 23, which also indicates a separation distance corresponding to an angle α°≦θ≦β°. There is a correlation between the separation distance indicated by the detection value of the left door distance measuring sensor 23 and the angle of the left door 11A, where the larger the separation distance indicated by the detection value of the left door distance measuring sensor 23, the larger the angle of the left door 11A relative to the refrigerator compartment 11. A similar correlation exists between the separation distance indicated by the detection value of the right door distance measuring sensor 24 and the angle of the right door 11B. When the image generation unit 206 identifies a frame, it acquires the frame with the least amount of user P reflected in it from the identified frame as identified frame FL1. The image generation unit 206 recognizes user P in the identified frame by a predetermined method such as pattern matching, and selects the frame with the smallest area of ​​the recognized user P's image as the frame with the least amount of user P reflected in it.

[0088] The image generation unit 206 determines whether or not it was able to obtain a specific frame FL1 (step SB3).

[0089] If the image generation unit 206 determines that it could not acquire the specific frame FL1 (step SB3: NO), the imaging device processor 20 performs the process in step SB12.

[0090] On the other hand, if the image generation unit 206 determines that it has been able to acquire a specific frame FL1 (step SB3: YES), it generates a corrected frame FL2 by correcting the distortion aberration of the acquired specific frame FL1 (step SB4).

[0091] Figure 9 is an explanatory diagram illustrating the correction of distortion aberration in a specific frame, FL1. The lens of the refrigerator camera 27 in this embodiment exhibits convex (so-called barrel) distortion, where the center of the frame bulges out. The image generation unit 206 corrects the convex distortion occurring in a specific frame FL1 based on a predetermined algorithm and generates a corrected frame FL2. The correction coefficient for correcting the distortion is predetermined by prior tests and simulations, and is recorded as data in the imaging device memory 21. For example, the image generation unit 206 corrects the distortion so that the edges of the shelf 112 in the specific frame FL1 become straight lines.

[0092] When the image generation unit 206 generates a corrected frame FL2, it extracts the refrigerator compartment image RFG, the left door image LDG, and the right door image RDG from the generated corrected frame FL2 (step SB5).

[0093] The refrigerator compartment image RFG is an image of the refrigerator compartment 11 taken by the refrigerator compartment camera 27 from the front and above of the refrigerator compartment 11. The left door image LDG is an image of the left door 11A taken by the refrigerator compartment camera 27 from the front and above of the refrigerator compartment 11, and shows the left door shelf 11A1. The right door image RDG is an image of the right door 11B taken by the refrigerator compartment camera 27 from the front and above of the refrigerator compartment 11, and shows the right door shelf 11B1. In the following explanation, when the left door image (LDG) and the right door image (RDG) are not distinguished, they are referred to as "door images" and denoted with the symbol "DG".

[0094] Figure 10 is an explanatory diagram illustrating the extraction of the refrigerator compartment image RFG, the left door image LDG, and the right door image RDG.

[0095] In the example shown in Figure 10, the image generation unit 206 extracts an image corresponding to a rectangular area with points P1, P2, P3, and P4 as the four corners in the correction frame FL2, and uses this image as the left door image LDG.

[0096] For example, the image generation unit 206 recognizes the edge of the left door 11A as seen in the correction frame FL2 based on feature quantities such as shape and color difference, and determines points P1, P2, P3, and P4 in the correction frame FL2 such that the lines connecting the points overlap with the recognized edge. Also, for example, the image generation unit 206 recognizes a rectangular area in the correction frame FL2 that corresponds to the detection value of the left door distance measuring sensor 23, which is associated with the correction frame FL2, as the area to be used for the left door image LDG. The rectangular area corresponding to the detection value of the left door distance measuring sensor 23 is predetermined by prior tests or simulations, and is recorded as data in the imaging device memory 21.

[0097] In the example shown in Figure 10, the image generation unit 206 extracts an image corresponding to a rectangular area with points P5, P6, P7, and P8 as the four corners in the correction frame FL2, and uses this image as the right door image RDG.

[0098] For example, the image generation unit 206 recognizes the edge of the right door 11B as seen in the correction frame FL2 based on feature quantities such as shape and color difference, and determines points P5, P6, P7, and P8 in the correction frame FL2 such that the lines connecting the points overlap with the recognized edge. Also, for example, the image generation unit 206 recognizes a rectangular area in the correction frame FL2 that corresponds to the detection value of the right door distance measuring sensor 24, which is associated with the correction frame FL2, as the area to be used for the right door image RDG. The rectangular area corresponding to the detection value of the right door distance measuring sensor 24 is predetermined by prior tests or simulations, and is recorded as data in the imaging device memory 21.

[0099] In the example shown in Figure 10, the image generation unit 206 extracts an image corresponding to a rectangular area with points P9, P10, P11, and P12 as the four corners in the correction frame FL2, and uses this image as the refrigerator room image RFG.

[0100] For example, the image generation unit 206 recognizes the boundary between the shelf 112 and the side wall inside the refrigerator compartment 11 as seen in the correction frame FL2, based on shape, color difference, etc., and determines points P9, P10, P11, and P12 such that the lines connecting the points overlap with the recognized boundary. Alternatively, for example, the image generation unit 206 recognizes a rectangular area predetermined by prior tests or simulations as the area to be used for the refrigerator compartment image RFG in the correction frame FL2.

[0101] The image generation unit 206 extracts the refrigerator compartment image RFG, the left door image LDG, and the right door image RDG, and then performs a viewpoint transformation on each of these extracted images (step SB6).

[0102] In step SB6, the image generation unit 206 converts the extracted refrigerator compartment image RFG into an image of the refrigerator compartment 11 viewed from a position on the front side of the refrigerator compartment 11. More specifically, the image generation unit 206 converts it into an image of the refrigerator compartment 11 viewed from a predetermined viewpoint at a position on the front side of the refrigerator compartment 11. The position on the front side of the refrigerator compartment 11 is a position in front of the refrigerator compartment 11.

[0103] Furthermore, in step SB6, the image generation unit 206 converts the extracted left door image LDG into an image of the left door shelf 11A1 viewed from a position on the front side of the left door shelf 11A1. More specifically, the image generation unit 206 converts it into an image of the left door shelf 11A1 viewed from a predetermined viewpoint at a position on the front side of the left door shelf 11A1. The position on the front side of the left door shelf 11A1 is the position facing the left door shelf 11A1, and includes positions different from the front side of the refrigerator compartment 11, as well as positions that are the same as the front side of the refrigerator compartment 11.

[0104] Furthermore, in step SB6, the image generation unit 206 converts the extracted right door image RDG into an image of the right door shelf 11B1 viewed from a position on the front side of the right door shelf 11B1. More specifically, the image generation unit 206 converts it into an image of the right door shelf 11B1 viewed from a predetermined viewpoint at a position on the front side of the right door shelf 11B1. The position on the front side of the right door shelf 11B1 is the position facing the right door shelf 11B1, and includes positions different from the front side of the refrigerator compartment 11, as well as positions that are the same as the front side of the refrigerator compartment 11.

[0105] Figure 11 is an explanatory diagram illustrating the viewpoint transformation of the refrigerated room image (RFG). Figure 11 shows the case where the extracted refrigerator room image RFG undergoes a viewpoint transformation in a predetermined coordinate system.

[0106] The image generation unit 206 rotates the extracted refrigerator room image RFG by 180° in a predetermined coordinate system. Next, the image generation unit 206 converts the refrigerator room image RFG into a rectangular image by changing the positions of the four corners of the rotated refrigerator room image RFG to positions set in the predetermined coordinate system. In the case of Figure 11, the image generation unit 206 changes the position of corner C1 of the rotated refrigerator room image RFG to position I3, the position of corner C2 of the rotated refrigerator room image RFG to position I4, the position of corner C3 of the rotated refrigerator room image RFG to position I1, and the position of corner C4 of the rotated refrigerator room image RFG to position I2. As a result, in the case of Figure 11, the image generation unit 206 converts the refrigerator room image RFG into a rectangular image by stretching it. Positions I1, I2, I3, and I4 are the positions that constitute the four corners of the rectangle and are set in predetermined coordinates.

[0107] Figure 12 is an explanatory diagram illustrating the line-of-sight transformation of the door image DG. Figure 12 shows the case where the left door image LDG is transformed into a door image DG in a predetermined coordinate system.

[0108] The image generation unit 206 rotates the extracted left door image LDG 90° counterclockwise in a predetermined coordinate system. Next, the image generation unit 206 converts the left door image LDG into a rectangular image by changing the positions of the four corners of the rotated left door image LDG to positions set in the predetermined coordinate system. In the case of Figure 12, the image generation unit 206 changes the position of corner C5 of the rotated left door image LDG to position I6, the position of corner C6 of the rotated left door image LDG to position I7, the position of corner C7 of the rotated left door image LDG to position I8, and the position of corner C8 of the rotated left door image LDG to position I5. As a result, in the case of Figure 12, the image generation unit 206 converts the left door image LDG into a rectangular image by stretching it. Positions I5, I6, I7, and I8 are the positions that constitute the four corners of the rectangle and are set in predetermined coordinates.

[0109] Furthermore, with respect to the right door image RDG, the image generation unit 206 rotates the right door image RDG, which was extracted in a predetermined coordinate system, 90° clockwise, and changes the positions of the four corners in the same way as the left door image LDG, thereby converting the right door image RDG into a rectangular image.

[0110] In the following explanation, the refrigerator compartment image RFG, which has been transformed in step SB6, will be referred to as the "front view refrigerator compartment image" and will be denoted as "RFG1". Also, in the following explanation, the left door image LDG, which has been transformed in step SB6, will be referred to as the "front view left door image" and will be denoted as "LDG1". Also, in the following explanation, the right door image RDG, which has been transformed in step SB6, will be referred to as the "front view right door image" and will be denoted as "RDG1". Also, in the following explanation, if the front view left door image LDG1 and the front view right door image RDG1 are not distinguished, they will be referred to as the "front view door image" and will be denoted as "DG1". The front view image of the refrigerator compartment RFG1 corresponds to the "first front view image" in this disclosure. The front view image of the door DG1 corresponds to the "second front view image" in this disclosure.

[0111] After performing the conversion in step SB6, the image generation unit 206 generates the composite image KG (step SB7). The composite image KG includes a front view left door image LDG1, a front view refrigerator compartment image RFG1, and a front view right door image RDG1. The front view left door image LDG1, the front view refrigerator compartment image RFG1, and the front view right door image RDG1 are arranged adjacent to each other in this order in the left-right direction within the image area of ​​the composite image KG.

[0112] Next, the image generation unit 206 analyzes the video image shown by the second video data 213B included in the newly recorded video data 213 and determines whether or not the drawer 11D1 of the chiller compartment 11D has been pulled out (step SB8). The image generation unit 206 recognizes the drawer 11D1 based on feature quantities such as shape and color and makes the determination in step SB8.

[0113] If the image generation unit 206 determines that the drawer 11D1 of the chilled compartment 11D has not been pulled out (step SB8: NO), it proceeds to step SB10.

[0114] On the other hand, if the image generation unit 206 determines that the drawer 11D1 of the chilled compartment 11D has been pulled out (step SB8: YES), it extracts an image CG of the chilled compartment in the state where the drawer 11D1 has been pulled out from the second recording data 213B included in the newly recorded recording data 213 (step SB9). The extraction range in step SB9 is predetermined.

[0115] In step SB10, the upload data generation unit 207 generates upload data (step SB10).

[0116] The upload data generated in step SB10 includes the image data of the composite image KG generated in step SB7 and the camera ID 212 stored in the camera memory 21. If the processing in step SB9 is performed, the upload data generated in step SB10 further includes chilled room image data 4123 showing the extracted chilled room image CG.

[0117] The imaging device communication control unit 201 sends the uploaded data generated in step SB10 to the image management server 4 (step SB11).

[0118] Next, the upload data generation unit 207 erases the recorded data 213 from the recording device memory 21 (step SB12).

[0119] As shown in flowchart FC, the server communication control unit 401 of the image management server 4 determines whether or not it has received uploaded data from the imaging device 2 (step SC1).

[0120] If the server communication control unit 401 determines that it has not received the uploaded data (step SC1: NO), it performs the determination in step SC1 again.

[0121] On the other hand, if the server communication control unit 401 determines that it has received the uploaded data (step SC1: YES), the server processing unit 402 updates the image management DB 412 based on the uploaded data received by the server communication control unit 401 (step SC2).

[0122] Step SC2 will be described in detail. The server processing unit 402 identifies record R containing the imaging device ID 212 from the image management DB 412. Next, for the identified record R, the server processing unit 402 records the image data of the fourth field F4 in the fifth field F5, the image data of the third field F3 in the fourth field F4, the image data of the second field F2 in the third field F3, and the image data of the first field F1 in the second field F2. Next, for the identified record R, the server processing unit 402 records the image data contained in the uploaded data in the first field F1.

[0123] Furthermore, step SC2 will be described in detail. The server processing unit 402 records the chilled compartment image data from the 9th field F9 in the 10th field F10, the chilled compartment image data from the 8th field F8 in the 9th field F9, the chilled compartment image data from the 7th field F7 in the 8th field F8, and the chilled compartment image data from the 6th field F6 in the 7th field F7 for the identified record R. Next, if the uploaded data contains chilled compartment image data, the server processing unit 402 records the chilled compartment image data contained in the uploaded data in the 6th field F6. On the other hand, if the uploaded data does not contain chilled compartment image data, the server processing unit 402 records the same image data as the chilled compartment image data recorded in the 7th field F7 in the 6th field F6.

[0124] [2-3. Actions related to the app UI] Next, we will explain the operation of the display system 1000 related to the application UI 330. Figure 13 is a flowchart showing the operation of the display system 1000. In Figure 13, flowchart FD shows the operation of terminal device 3, and flowchart FE shows the operation of image management server 4.

[0125] The terminal communication control unit 301 of the terminal device 3 determines whether or not to request image data stored in the image management DB 412 from the image management server 4 (step SD1).

[0126] For example, the terminal communication control unit 301 makes a positive determination in step SD1 when the display application 311 is launched.

[0127] If the terminal communication control unit 301 determines that it needs to request image data stored in the image management DB 412 (step SD1: YES), it sends request data requesting the image data to the image management server 4 (step SD2). The request data sent in step SD2 includes the account ID 4121 stored in the terminal memory 31.

[0128] As shown in flowchart FE, the server communication control unit 401 of the image management server 4 receives the request data (step SE1).

[0129] Next, the server processing unit 402 retrieves the image data table 4122 and the chilled compartment image data table 4123, which correspond to the account ID 4121 included in the request data received by the server communication control unit 401, from the image management DB 412 (step SE2).

[0130] Next, the server communication control unit 401 transmits the image data table 4122 and the chilled compartment image data table 4123 acquired by the server processing unit 402 to the terminal device 3 as a response to the requested data (step SE3).

[0131] Referring to flowchart FD, the terminal communication control unit 301 of terminal device 3 receives the image data table 4122 and the chilled compartment image data table 4123 from the image management server 4 (step SD3).

[0132] Next, the display control unit 302 displays the application UI 330 on the touch panel 33 based on the image data table 4122 and the chilled compartment image data table 4123 received in step SD3 (step SD4).

[0133] Figure 14 shows an example of the application UI 330. In Figure 14, the upward direction is indicated by the symbol UP, the downward direction is indicated by the symbol DW, the leftward direction is indicated by the symbol L, and the rightward direction is indicated by the symbol R.

[0134] The application UI330 has a first area UA1 and a second area UA2. The first area UA1 and the second area UA2 are arranged vertically within the application UI330.

[0135] The first area UA1 displays the image set KG. The image set KG displayed by the first area UA1 is the image set KG indicated by the image data stored in the image data table 4122 received in step SD3. In the default application UI 330, the first area UA1 displays the image set KG indicated by the image data stored in the first field F1. The reception unit 303 accepts a swipe operation in the left or right direction in the first area UA1. A swipe operation in the left or right direction corresponds to the "first operation" in this disclosure. A swipe operation is an operation in which an operator such as a finger is moved while in contact with the touch panel 33. When the reception unit 303 accepts a swipe operation in the left or right direction in the first area UA1, the display control unit 302 switches the image set KG displayed by the first area UA1 to another image set KG.

[0136] For example, if the reception unit 303 receives a swipe operation to the left while displaying the composite image KG shown by the image data in the first field F1, the display control unit 302 switches the composite image KG displayed in the first area UA1 to the composite image KG shown by the image data stored in the second field F2. Furthermore, for example, if the reception unit 303 receives a swipe operation to the left while displaying the composite image KG shown by the image data in the second field F2, the display control unit 302 switches the composite image KG displayed by the first area UA1 to the composite image KG shown by the image data stored in the third field F3. Furthermore, for example, if the reception unit 303 receives a swipe operation to the left while displaying the composite image KG shown by the image data in the third field F3, the display control unit 302 switches the composite image KG displayed in the first area UA1 to the composite image KG shown by the image data stored in the fourth field F4. Furthermore, for example, if the reception unit 303 receives a swipe operation to the left while displaying the composite image KG shown by the image data in the fourth field F4, the display control unit 302 switches the composite image KG displayed by the first area UA1 to the composite image KG shown by the image data stored in the fifth field F5. Furthermore, for example, if the reception unit 303 receives a swipe operation to the left while displaying the composite image KG shown by the image data in the fifth field F5, the display control unit 302 switches the composite image KG displayed by the first area UA1 to the composite image KG shown by the image data stored in the first field F1.

[0137] For example, if the reception unit 303 receives a swipe operation to the right while displaying the composite image KG shown by the image data in the first field F1, the display control unit 302 switches the composite image KG displayed in the first area UA1 to the composite image KG shown by the image data stored in the fifth field F5. Furthermore, for example, if the reception unit 303 receives a swipe operation to the right while displaying the composite image KG shown by the image data in the fifth field F5, the display control unit 302 switches the composite image KG displayed by the first area UA1 to the composite image KG shown by the image data stored in the fourth field F4. Furthermore, for example, if the reception unit 303 receives a swipe operation to the right while displaying the composite image KG shown by the image data in the fourth field F4, the display control unit 302 switches the composite image KG displayed in the first area UA1 to the composite image KG shown by the image data stored in the third field F3. Furthermore, for example, if the reception unit 303 receives a swipe operation to the right while displaying the composite image KG shown by the image data in the third field F3, the display control unit 302 switches the composite image KG displayed by the first area UA1 to the composite image KG shown by the image data stored in the second field F2. Furthermore, for example, if the reception unit 303 receives a swipe operation to the right while displaying the composite image KG shown by the image data in the second field F2, the display control unit 302 switches the composite image KG displayed by the first area UA1 to the composite image KG shown by the image data stored in the first field F1.

[0138] The second area UA2 displays the chilled compartment image CG. The second area UA2 displays the chilled compartment image CG indicated by the chilled compartment image data 4123 received in step SD3. The reception unit 303 accepts downward swipe operations and upward swipe operations in the second area UA2. A downward swipe operation corresponds to the "second operation" in this disclosure. When the reception unit 303 accepts a downward swipe operation in the second area UA2 which is not displaying the chilled compartment image CG, the display control unit 302 causes the chilled compartment image CG to be displayed in the second area UA2. The display control unit 302 gradually displays the chilled compartment image CG in the second area UA2 in a manner that moves from top to bottom in accordance with the movement of the operator in the downward swipe operation. When the reception unit 303 receives an upward swipe operation in the second area UA2 where the chilled compartment image CG is displayed, the display control unit 302 hides the chilled compartment image CG in the second area UA2. The display control unit 302 gradually hides the chilled compartment image CG in the second area UA2 in a manner that moves from bottom to top as the operator moves during the upward swipe operation. Furthermore, if the reception unit 303 receives a swipe operation in the left or right direction in the second area UA2, the display control unit 302 may switch the chilled compartment image CG displayed in the second area UA2 to another chilled compartment image CG.

[0139] [3. Effects, etc.] As explained above, the display application 311 causes the terminal processor 30 of the terminal device 3 to function as a display control unit 302 that displays a front-view refrigerator compartment image RFG1 on the touch panel 33. RFG1 is an image generated from an image of the refrigerator compartment 11 taken from the front and above of the refrigerator compartment 11, and is an image of the refrigerator compartment 11 viewed from the front side of the refrigerator compartment 11.

[0140] This displays an image of the refrigerator compartment 11 as seen from the front of the refrigerator compartment 11. Therefore, user P can intuitively understand the situation inside the refrigerator 1.

[0141] The display control unit 302 further displays a front view door image DG1 on the touch panel 33, which is an image generated from an image of the door 11C that opens and closes the refrigerator compartment 11, taken from the front and above of the refrigerator compartment 11, and is an image of the door shelf 11C1 viewed from the front side of the door shelf 11C1 provided on the door 11C.

[0142] As a result, an image of the door shelf 11C1 viewed from the front of the door shelf 11C1 is also displayed, allowing user P to intuitively understand the situation of the door shelf 11C1 in addition to the situation of the refrigerator compartment 11. Therefore, user P can intuitively understand more about the situation inside the refrigerator 1.

[0143] The display control unit 302 displays the front view refrigerator compartment image RFG1 and the front view door image DG1 side by side.

[0144] This makes it possible to display an image that corresponds to the configuration of the refrigerator compartment 11 and the door shelf 11C1 when the door 11C is open, allowing user P to more intuitively understand the status of the refrigerator compartment 11 and the door shelf 11C1. Therefore, user P can more intuitively understand the status inside the refrigerator 1.

[0145] The display application 311 causes the terminal processor 30 to function as a reception unit 303 that receives operations from user P. When the reception unit 303 receives a swipe operation in the left or right direction from user P, the display control unit 302 switches the displayed front view refrigerator compartment image RFG1 to another front view refrigerator compartment image RFG1, and also switches the displayed front view door image DG1 to another front view door image DG1.

[0146] This allows user P to easily switch to other front-view refrigerator compartment images RFG1 and front-view door images DG1, thereby improving convenience for user P.

[0147] The display control unit 302 further displays the chilled compartment image CG, which is an image of the chilled compartment 11D taken from the front and above of the refrigerator compartment 11.

[0148] This allows for the display of additional images that show the status of the chiller compartment 11D, enabling user P to intuitively understand the status of the chiller compartment 11D in addition to the status of the refrigerator compartment 11. Thus, user P can intuitively understand more about the interior of the refrigerator 1.

[0149] The display application 311 causes the terminal processor 30 to function as a reception unit 303 that receives operations from user P. When the reception unit 303 receives a downward swipe operation from user P, the display control unit 302 displays the chilled compartment image CG alongside the front view refrigerator compartment image RFG1.

[0150] This allows the chiller compartment image CG to be displayed in a way that corresponds to the actual use of the chiller compartment 11D, enabling user P to more intuitively understand the status of the chiller compartment 11D. Therefore, user P can more intuitively understand the status inside the refrigerator 1.

[0151] The display application 311 is an application program that can be installed on the terminal device 3.

[0152] This allows a terminal device 3 that does not have the function to allow user P to understand the status inside the refrigerator 1 to be transformed into a terminal device 3 that has this function by installing the display application 311.

[0153] The terminal device 3 includes a touch panel 33 and a display control unit 302 that displays a front view image of the refrigerator compartment RFG1 on the touch panel 33.

[0154] This produces the same effect as the display application 311 described above.

[0155] The control method for terminal device 3 includes the step of displaying a front view image of the refrigerator compartment RFG1.

[0156] This produces the same effect as the display application 311 described above.

[0157] The display system 1000 includes an image generation unit 206 that generates an image of the refrigerator compartment 11 from a front view image RFG1 taken from the front of the refrigerator compartment 11 and an image of the refrigerator compartment 11 taken from above and in front of the refrigerator compartment 11, and a terminal device 3 that displays the front view image RFG1 generated by the image generation unit 206.

[0158] This produces the same effect as the display application 311 described above.

[0159] (Other embodiments) As described above, Embodiment 1 has been presented as an example disclosed in this application. However, the technology in this disclosure is not limited to this and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in Embodiment 1. Therefore, other embodiments are illustrated below.

[0160] In the embodiments described above, a refrigerator 1 was used as an example of the "storage facility" in this disclosure. However, the "storage facility" in this disclosure is not limited to a refrigerator 1, and may be other storage facilities such as a freezer or a room-temperature storage facility.

[0161] In the embodiments described above, a refrigerator 11 was given as an example of the "storage room" in this disclosure. However, the "storage room" in this disclosure is not limited to a refrigerator 11, and may be the interior of a freezer, a room-temperature storage room, or other type of storage facility.

[0162] In the embodiment described above, the composite image KG is generated from the shooting results of the refrigerator compartment camera 27. However, in other embodiments, the composite image KG may be generated from the shooting results of the refrigerator compartment camera 27 and the shooting results of the drawer camera 28. This allows the images constituting the composite image KG to include the shooting results of a camera with a narrower angle than the refrigerator compartment camera 27, thereby generating a clearer composite image KG.

[0163] In other embodiments, the various images displayed in the app UI 330 may be changeable depending on the model and layout of the refrigerator 1. In this configuration, the image generation unit 206 adjusts the image size and eye-tracking correction values ​​according to the width of the refrigerator 1, the number of compartments inside the refrigerator 1, the height of the refrigerator 1, and the number of doors provided in the opening of the refrigerator 1.

[0164] In other embodiments, the determination of whether or not the drawer 11D1 of the chilled compartment 11D has been pulled out may be made by the following method. For example, a marker may be provided on the drawer 11D1 of the chilled compartment 11D, and the image generation unit 206 may determine that the drawer 11D1 has been pulled out if the marker is captured in the image taken by the refrigerator camera 27 or the drawer camera 28. Alternatively, for example, the drawer 11D1 may consist of a chilled compartment door that rotates vertically around the front upper part of the chilled compartment 11D as an axis, and a chilled compartment case for storing items, and a marker may be provided on the chilled compartment door of the drawer 11D1, and the image generation unit 206 may determine that the drawer 11D1 has been pulled out if the marker is captured in the image taken by the refrigerator camera 27 or the drawer camera 28. Alternatively, for example, a marker may be provided on the bottom surface of the refrigerator compartment 11 and in front of the chilled compartment 11D, and the image generation unit 206 may determine that the drawer 11D1 has been pulled out when the marker is no longer visible in the image captured by the refrigerator compartment camera 27 or the drawer camera 28. Alternatively, for example, a distance measuring sensor may be provided on the back of the chilled compartment 11D to measure the distance from the drawer 11D1, and the image generation unit 206 may determine that the drawer 11D1 has been pulled out when the detected value of the distance measuring sensor is greater than a predetermined value. Alternatively, for example, a marker may be provided on the top surface of the drawer 11D1, and the image generation unit 206 may determine that the drawer 11D1 has been pulled out when the marker is visible in the image captured by the refrigerator compartment camera 27 or the drawer camera 28.

[0165] In the above-described embodiment 1, the imaging device processor 20 functions as the image generation unit 206. However, in another embodiment, the server processor 40 may function as the image generation unit 206. In this configuration, the imaging device 2 transmits the recorded data 213 to the image management server 4. The image generation unit 206 of the server processor 40 of the image management server 4 then refers to the recorded data 213 received from the imaging device 2 and performs the processing in steps SB2, SB3, SB4, SB5, SB6, SB7, SB8, and SB9.

[0166] In the above-described embodiment 1, the refrigerator compartment image RFG extracted from the image captured by the refrigerator compartment camera 27 is converted to a viewpoint and displayed on the application UI 330. In another embodiment, the application UI 330 may display an image captured by a camera mounted on the inside of the door 11C instead of the front-view refrigerator compartment image RFG1.

[0167] In the above-described embodiment 1, the chilled compartment image CG is extracted from the image captured by the drawer camera 28. In other embodiments, the chilled compartment image CG may be extracted from the image captured by the refrigerator compartment camera 27.

[0168] In the above-described embodiment 1, the combined image KG is displayed in the first region UA1. In other embodiments, the default first region UA1 may be configured to display the front view refrigerator compartment image RFG1 from the combined image KG. In this configuration, when the reception unit 303 receives a left or right swipe operation in the first region UA1, the display control unit 302 displays either the front view left door image LDG1 or the front view right door image RDG1, which constitutes the combined image KG, based on the direction and amount of movement of the operator in the swipe operation.

[0169] In the above-described embodiment 1, a configuration was illustrated in which the person presence determination unit 202 determines whether or not a person is present around the refrigerator 1 based on the detection value of the human presence sensor 26. However, the person presence determination unit 202 may also make the determination based on the detection value of an illuminance sensor or an electrostatic sensor provided on the gripping part of the refrigerator 1, instead of the human presence sensor 26. In the case of this electrostatic sensor, the refrigerator 1 and the camera 2 can communicate with each other, and the refrigerator 1 transmits the detection value of the electrostatic sensor to the camera 2.

[0170] In the first embodiment described above, the touch panel 33 provided on the terminal device 3 was given as an example of the "display unit" in this disclosure. In other embodiments, the "display unit" in this disclosure may be a display device separate from the terminal device 3.

[0171] In other embodiments, the terminal device 3 may store record R. In this configuration, the application execution unit 300 updates the contents of record R stored by the terminal device 3 as appropriate, similar to the image management server 4 described above. In this configuration, the shooting device 2 directly transmits the uploaded data to the terminal device 3. This simplifies the system configuration as the display system 1000 does not need to have an image management server 4.

[0172] In other embodiments, the main body 10 of the refrigerator 1 may have storage compartments other than the refrigerator compartment 11, ice-making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15, or at least one of the ice-making compartment 12, fresh freezing compartment 13, freezer compartment 14, and vegetable compartment 15 may be omitted. Also, in other embodiments, the number of doors provided at the front opening of the refrigerator compartment 11 may be just one.

[0173] For example, the imaging device processor 20, the terminal processor 30, and the server processor 40 may be composed of a single processor or multiple processors. The imaging device processor 20, the terminal processor 30, and the server processor 40 may also be hardware programmed to implement the corresponding functional units. That is, the imaging device processor 20, the terminal processor 30, and the server processor 40 may be composed of, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0174] The configurations of the imaging device 2, terminal device 3, and image management server 4 shown in Figures 4 and 5 are examples, and the specific implementation is not particularly limited. In other words, it is not necessarily required that hardware corresponding to each part be implemented individually, and it is certainly possible to configure the system so that a single processor executes a program to realize the functions of each part. Furthermore, some of the functions realized by software in the above-described embodiment may be implemented by hardware, or some of the functions realized by hardware may be realized by software. In addition, the specific detailed configurations of other parts of the imaging device 2, terminal device 3, and image management server 4 can also be arbitrarily changed without departing from the spirit of this disclosure.

[0175] The operational step units shown in Figures 7, 8, and 12 are divided according to the main processing content to facilitate understanding of the operation, and the operation is not limited by the way the processing units are divided or the names of the processing units. Depending on the processing content, it may be further divided into more step units. Alternatively, it may be divided so that one step unit includes even more processing. Furthermore, the order of the steps may be changed as appropriate, as long as it does not impede the intent of this disclosure.

[0176] The control program 211 executed by the imaging device processor 20 can also be implemented by recording the control program 211 on a portable information recording medium. Examples of information recording media include magnetic recording media such as hard disks, optical recording media such as CDs, and semiconductor storage devices such as USB (Universal Serial Bus) memory and SSDs (Solid State Drives), but other recording media can also be used.

[0177] Since the embodiments described above are for illustrative purposes of the technology described herein, various modifications, substitutions, additions, omissions, etc., can be made within the claims or their equivalents. [Industrial applicability]

[0178] As described above, the program, terminal device, control method for the terminal device, and display system according to the present invention can be used for displaying the conditions inside a storage facility. [Explanation of symbols]

[0179] 1. Refrigerator (storage room) 2. Imaging device 3 Terminal devices 11. Refrigerated room (storage room) 11A Left Door 11A1 Left door shelf 11B Right Door 11B1 Right door shelf 11C Door 11C1 Door shelf 11D Chilled compartment 27. Camera for the refrigerator compartment 28 Drawer Camera 30 Terminal Processors (Processors) 33. Touch panel (display unit) 302 Display Control Unit Room 303 Reception 311 Display applications (programs, application programs) 411 Control Program CG chilled compartment image DG1 Front view door image (second front view image) LDG1 Front view left door image RDG1 Front view, right door image RFG1 Front view image of the refrigerated compartment (first front view image)

Claims

1. The processor of the terminal device, A display control unit that displays a first front view image on a display unit, which is an image generated from an image of the storage room of the storage facility taken from the front and above of the storage room, and which is an image of the storage room viewed from a position on the front side of the storage room. It will function as a reception unit that receives operations from users. The display control unit, When the reception unit receives a downward swipe operation from the user, As the operator moves in the downward swipe operation, and in a manner moving from top to bottom, the image of the chilled compartment taken from the front upper part of the storage compartment is gradually displayed on the display unit below the first front view image, alongside the first front view image. A marker is placed on the drawer of the chilled compartment, and the image of the chilled compartment is an image taken when the distance between the back of the chilled compartment and the drawer of the chilled compartment is greater than a predetermined value. program.

2. The display control unit, A second front view image is further displayed on the display unit, which is an image generated from an image of the door that opens and closes the storage room, taken from the front and above of the storage room, and is an image of the door shelf provided on the door, viewed from the front side of the door shelf. The program according to claim 1.

3. The display control unit, The first front view image and the second front view image are displayed side by side. The program according to claim 2.

4. When the reception unit receives a first operation from a user, the display control unit switches the displayed first front view image to another first front view image, and also switches the displayed second front view image to another second front view image. The program according to claim 3.

5. An application program that can be installed on the aforementioned terminal device, The program according to any one of claims 1 to 4.

6. The aforementioned storage facility is a refrigerator, The aforementioned storage room is a refrigerated room. The program according to any one of claims 1 to 5.

7. A display control unit that displays a first front view image on a display unit, which is an image generated from an image of the storage room taken from the front and above of the storage room of the storage facility, and which is an image of the storage room viewed from a position on the front side of the storage room. It includes a reception unit that receives operations from the user, The display control unit, When the reception unit receives a downward swipe operation from the user, As the operator moves in the downward swipe operation, and in a manner moving from top to bottom, the image of the chilled compartment taken from the front upper part of the storage compartment is gradually displayed on the display unit below the first front view image, alongside the first front view image. A marker is placed on the drawer of the chilled compartment, and the image of the chilled compartment is an image taken when the distance between the back of the chilled compartment and the drawer of the chilled compartment is greater than a predetermined value. Terminal device.

8. The process includes the step of displaying a first front view image on a display unit, which is an image generated from a photograph of the storage room taken from the front and above of the storage room of the storage facility, and which is an image of the storage room viewed from a position on the front side of the storage room. The aforementioned step is, When a downward swipe gesture is received from the user, As the operator moves in the downward swipe operation, and in a manner moving from top to bottom, the image of the chilled compartment taken from the front upper part of the storage compartment is gradually displayed on the display unit below the first front view image, alongside the first front view image. A marker is placed on the drawer of the chilled compartment, and the image of the chilled compartment is an image taken when the distance between the back of the chilled compartment and the drawer of the chilled compartment is greater than a predetermined value. A method for controlling terminal devices.

9. A generation unit generates a first front view image of the storage room of a storage facility, taken from a position on the front side of the storage room, from an image of the storage room taken from above and in front of the storage room. The system includes a terminal device that displays the first front view image generated by the generation unit, The aforementioned terminal device is When the user initiates a downward swipe gesture, As the operator moves in the downward swipe operation, and in a manner that moves from top to bottom, the image of the chilled compartment taken from the front upper part of the storage compartment is gradually displayed below the first front view image, alongside the first front view image. A marker is placed on the drawer of the chilled compartment, and the image of the chilled compartment is an image taken when the distance between the back of the chilled compartment and the drawer of the chilled compartment is greater than a predetermined value. Display system.

Images