Setting method, imaging system, server device, imaging device, terminal device, and program

Abstract

An imaging system (10) is provided with: an imaging device (30) which is disposed on an upper part of a refrigerator (20), is capable of imaging a lower part of an opening in a first storage chamber in a state in which a door is opened, and is capable of imaging the inside of a second storage chamber in a state in which the second storage chamber is pulled out; an acquisition part (44) which acquires a first image obtained by imaging the lower part of the opening and a second image obtained by imaging the inside of the second storage chamber; a detection unit (45) which detects the position of an imaging target provided to a non-movable part of the lower part of the opening in the first storage chamber in the first image; a calculation unit (46) for calculating a designated position of the non-movable part in the first image by using refrigerator (20) structure information; a specification unit (47) which specifies the positional relationship between the position of the imaging target and the designated position of the non-movable part; and a setting unit (48) that sets a cropping position when the inside area of the second storage chamber is cropped from the second image on the basis of the positional relationship.

Description

Setting method, imaging system, server device, imaging device, terminal device, and program

[0001] The present disclosure relates to a method for setting a position for cutting out an interior area of ​​a refrigerator in an image captured from above the refrigerator, an imaging system, a server device, an imaging device, a terminal device, and a program.

[0002] Conventionally, a wide-angle camera installed on top of a refrigerator capable of capturing images of the entire interior of the refrigerator has been used to capture images of the interior of the refrigerator. However, wide-angle cameras have poor resolution for distant subjects, which poses a problem in terms of visibility of the interior of distant drawer compartments. Patent Document 1 discloses a photographing device that uses a narrow-angle camera to capture images of the interior of drawer compartments such as the freezer compartment and vegetable compartment, and can select from the captured images an image that adequately captures the interior conditions.

[0003] Japanese Patent Application Laid-Open No. 2023-127929

[0004] With conventional imaging devices, a telephoto, narrow-angle camera installed on top of the refrigerator captures the interior of the drawer compartment, which is located far below, and then the required area is cut out and displayed from the captured image. However, the cutout area does not always capture the appropriate area, as the background around the interior is sometimes reflected in the cutout area.

[0005] The present disclosure provides an imaging system and the like that can extract an appropriate area from an image captured of the interior of a drawer compartment of a refrigerator.

[0006] A position setting method according to one aspect of the present disclosure is a method for setting a cut-out position when cutting out an interior area of ​​a refrigerator that has a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment, and includes the steps of: acquiring a first image of a portion of the refrigerator that is captured with the door open; detecting a position of an object to be photographed that is installed in a non-movable portion below the opening in the first image; calculating a designated position of the non-movable portion in the first image using structural information of the refrigerator; specifying a positional relationship between the position of the object to be photographed and the designated position of the non-movable portion; and setting a cut-out position when cutting out an interior area of ​​the second storage compartment from a second image that is captured with the second storage compartment pulled out, based on the positional relationship.

[0007] an imaging system according to another aspect of the present disclosure that is capable of capturing images from above a refrigerator having a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment, and includes: an imaging device that is disposed above the refrigerator and is capable of capturing an image of an area below the opening of the first storage compartment when the door is open, and an image of the interior of the second storage compartment when the second storage compartment is pulled out; a memory that stores structural information of the refrigerator; an acquisition unit that acquires a first image captured by the imaging device of the area below the opening and a second image captured by the imaging device of the interior of the second storage compartment; a detection unit that detects, in the first image, a position of an object to be photographed that is installed in a non-movable part below the opening of the first storage compartment; a calculation unit that calculates, using the structural information, a designated position of the non-movable part in the first image; an identification unit that identifies a positional relationship between the position of the object to be photographed and the designated position of the non-movable part; and a setting unit that sets a cut-out position when cutting out an area inside the second storage compartment from the second image based on the positional relationship.

[0008] Furthermore, a server device according to one aspect of the present disclosure is a server device that processes images captured from above of a refrigerator having a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment, and includes: a memory unit that stores structural information of the refrigerator; an acquisition unit that acquires, by an imaging device arranged above the refrigerator, a first image captured of a lower part of an opening of the first storage compartment with the door open and a second image captured of the interior of the second storage compartment with the second storage compartment pulled out; a detection unit that detects, in the first image, a position of an object to be photographed that is installed in a non-movable part below the opening of the first storage compartment; a calculation unit that calculates, using the structural information, a designated position of the non-movable part in the first image; an identification unit that identifies a positional relationship between the position of the object to be photographed and the designated position of the non-movable part; and a setting unit that sets a cut-out position when cutting out an interior area of ​​the second storage compartment from the second image based on the positional relationship.

[0009] An imaging device according to one aspect of the present disclosure is an imaging device capable of photographing a refrigerator from above, the refrigerator having a first storage compartment with a door and a second storage compartment in a drawer shape below the first storage compartment, the imaging device including a camera unit disposed above the refrigerator and capable of photographing a lower portion of an opening of the first storage compartment when the door is open and an interior of the second storage compartment when the second storage compartment is pulled out, a memory unit that stores structural information of the refrigerator, and a first image of the lower portion of the opening of the first storage compartment photographed by the camera unit disposed above the refrigerator when the door is open. an acquisition unit that acquires a first image and a second image of the interior of the second storage compartment when the second storage compartment is pulled out; a detection unit that detects, in the first image, a position of an object to be photographed that is placed in a non-movable part below the opening of the first storage compartment; a calculation unit that calculates a designated position of the non-movable part in the first image using the structural information; an identification unit that identifies a positional relationship between the position of the object to be photographed and the designated position of the non-movable part; and a setting unit that sets a cut-out position when cutting out an interior area of ​​the second storage compartment from the second image based on the positional relationship.

[0010] According to the present disclosure, it is possible to extract an appropriate area from an image captured of the interior of a drawer compartment of a refrigerator.

[0011] FIG. 1 is a block diagram showing the functional configuration of an imaging system. FIG. 2 is an external view of a refrigerator and an imaging device included in the imaging system. FIG. 3 is a front view of the refrigerator and the imaging device as seen from the front. FIG. 4 is a side view of the refrigerator and the imaging device as seen from the right. FIG. 5 is a plan view of the imaging device installed in the refrigerator as seen from above. FIG. 6 is a side view of the imaging device as seen from the right. FIG. 7 is a diagram showing an example of the imaging state of a narrow-angle camera. FIG. 8 is a diagram showing an example of a marker placed in a refrigerator compartment. FIG. 9 is a diagram showing an example of a perspective view of a hinge cover. FIG. 10 is a diagram showing an example of a UI displayed on the display unit of an information terminal at the time of initial setup. FIG. 11 is a diagram showing an example of a UI displayed on the display unit of an information terminal at the time of initial setup. FIG. 12 is a diagram showing an example of a method for calculating a coordinate difference from a first image at the time of initial setup. FIG. 13 is a diagram showing an example of a method for setting a position for cutting out the interior area of ​​the drawer compartment in a second image using the coordinate difference calculated at the time of initial setup. FIG. 14 is a flowchart showing the operation of the imaging system. FIG. 15 is a block diagram showing a functional configuration of the imaging system different from that shown in FIG. 1.

[0012] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0013] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0014] In the following description, images and videos may be collectively referred to as images.

[0015] (Knowledge, etc., that Forms the Basis of the Present Disclosure) At the time the inventors arrived at the present disclosure, they were considering a system that would capture an image of a drawer compartment located far below from above a refrigerator and manage food inventory from the captured image. The inventors noted that when a telephoto, narrow-angle camera installed at the top of the refrigerator captures an image of the interior of a drawer compartment located far below, the necessary area is cropped and displayed from the captured image, assuming that there is no misalignment in the camera's installation position, no variation in the construction of the housing, or no individual variation in the camera's angle of view. However, in reality, these deviations and variations exist, and the inventors discovered a problem in that the quality of the image is not ensured in terms of inventory visibility, such as the interior of the drawer compartment being cut out or the background surrounding the interior being reflected. The subject matter of the present disclosure was created to solve this problem.

[0016] The present disclosure provides an imaging system and the like that can extract an appropriate area from an image of the interior of a refrigerator drawer. This stabilizes the extraction display range regardless of variations in installation, construction, and individual units, increases the allowable variations in installation, construction, and individual units, and improves installation constraints and yield.

[0017] (Embodiment) [Configuration] First, the configuration of an imaging system according to an embodiment will be described. Fig. 1 is a block diagram showing the functional configuration of an imaging system according to an embodiment.

[0018] The imaging system 10 shown in Fig. 1 is a system for capturing images of the inside of a refrigerator 20 and displaying the captured images, which show food stored in the refrigerator 20, on a display unit 52 of an information terminal 50, which is a terminal device. Specifically, the imaging system 10 includes an imaging device 30 and a server device 40. The imaging system 10 may also be referred to as the imaging system, including one or both of the refrigerator 20 and the information terminal 50. The imaging system 10 is also used to display, on the display unit 52 of the information terminal 50, a screen that presents the user with the processes that the user must perform for various settings, for example, during the initial setup stage, when the imaging device 30 is attached to the top of the refrigerator 20 or when the position of the imaging device 30 is adjusted.

[0019] The refrigerator 20 is an example of a storage unit that can cool stored items, and is installed in a user's home or the like to refrigerate food.

[0020] Server device 40 is a computer located outside the facility where refrigerator 20 is installed, and specifically, is a cloud server. Server device 40 performs information processing to display, for example, an image of the interior of refrigerator 20 showing food stored therein on information terminal 50. Server device 40 includes a communication unit 41, an information processing unit 42, and a storage unit 43.

[0021] The communication unit 41 is a communication circuit that enables the server device 40 to communicate with the imaging device 30 and the information terminal 50 via the wide area communication network 60. The communication unit 41 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard of the communication performed by the communication unit 41.

[0022] The information processing unit 42 performs information processing to display, for example, an image of the interior of the refrigerator 20 showing the food stored therein, or an instruction screen for the user during initial setup, on the information terminal 50. The information processing unit 42 is implemented, for example, by a microcomputer, but may also be implemented by a processor or a dedicated circuit. The information processing unit 42 has, as functional components, an acquisition unit 44, a detection unit 45, a calculation unit 46, an identification unit 47, and a setting unit 48. The functions of the acquisition unit 44, the detection unit 45, the calculation unit 46, the identification unit 47, and the setting unit 48 are implemented, for example, by the microcomputer or the like constituting the information processing unit 42 executing a computer program stored in the storage unit 43. The functions of the acquisition unit 44, the detection unit 45, the calculation unit 46, the identification unit 47, and the setting unit 48 will be described in detail below.

[0023] The storage unit 43 is a storage device that stores the computer programs executed by the information processing unit 42 and various information required for the information processing. The storage unit 43 is realized by, for example, a semiconductor memory.

[0024] The information terminal 50 is an information terminal owned by a user. The user uses the information terminal 50, for example, to attach the imaging device 30 to the refrigerator 20 and perform initial settings, or to check the food stored in the refrigerator 20 while away from home. The information terminal 50 is, for example, a portable information terminal such as a smartphone or a tablet terminal, but may also be a stationary information terminal such as a personal computer. The information terminal 50 includes a communication unit 51 and a display unit 52.

[0025] The communication unit 51 is a communication circuit that enables the information terminal 50 to communicate with the imaging device 30 and the server device 40 via the wide area communication network 60. The communication unit 51 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard of the communication performed by the communication unit 51.

[0026] 10 and 11, for example, are displayed on the display unit 52. The display unit 52 is realized by a display panel such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel.

[0027] FIG. 2 is an external view of the refrigerator 20 and the imaging device 30 included in the imaging system 10.

[0028] The refrigerator 20 is an example of a storage unit that can cool stored items, and is installed in a user's home or the like to refrigerate food.

[0029] Imaging device 30 captures an image of refrigerator 20 with a drawer (e.g., a vegetable drawer) pulled out from above refrigerator 20. That is, imaging device 30 captures, for example, an image of the drawer of refrigerator 20 viewed from above refrigerator 20. Imaging device 30 is realized, for example, by a camera having a wide-angle lens and a telephoto lens.

[0030] Fig. 3 is a front view of the refrigerator 20 and the image capture device 30. The refrigerator 20 shown in Fig. 3 has both its left and right doors open. Fig. 4 is a side view of the refrigerator 20 and the image capture device 30 seen from the right. The refrigerator 20 shown in Fig. 4 has its drawer 25A in an open state, i.e., pulled out.

[0031] The interior space of refrigerator 20 is divided into multiple spaces by partition members. In this embodiment, this interior space is divided into five spaces, which respectively function as refrigerator compartment 21, ice-making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25. In the following description, when there is no need to distinguish between refrigerator compartment 21, ice-making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, they may be referred to as "storage compartments" and assigned the reference number "26."

[0032] In the following description, when there is no need to distinguish between the drawer-shaped ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, they may be referred to as "drawer compartments." The drawer compartments correspond to the "second storage compartments" in this disclosure.

[0033] Drawers 22A, 23A, 24A, and 25A capable of accommodating food are installed in ice-making compartment 22, fresh-freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively. Each of drawers 22A to 25A has an open top when installed.

[0034] In the following description, when drawers 22A, 23A, 24A, and 25A are referred to collectively without distinction, they will be designated by the symbol "26A" and will be referred to as "drawer 26A."

[0035] FIG. 3 illustrates the X-axis, Y-axis, and Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to one another. The Z-axis indicates the up-down direction, which corresponds to the height direction of the refrigerator 20. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction. The left-right direction corresponds to the width direction of the refrigerator 20. The Y-axis indicates the front-to-back direction. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the front direction. The positive direction of the Z-axis indicates the upward direction. The X-axis, Y-axis, and Z-axis are also illustrated in each of FIGS. 4 to 9.

[0036] The imaging device 30 includes a main body 32 and an imaging member 31 extending forward from the main body 32. The imaging member 31 is provided above the front of the main body 32. The main body 32 includes a restricting member 34. The restricting member 34 restricts the position of the imaging member 31 in the front-to-rear direction of the refrigerator 20.

[0037] Imaging device 30 includes wide-angle camera 301 and narrow-angle camera 302. Wide-angle camera 301 and narrow-angle camera 302 are installed at the front end of the bottom surface of imaging member 31. When imaging device 30 is attached to refrigerator 20 and the door is closed, wide-angle camera 301 and narrow-angle camera 302 are located in front of the door. Wide-angle camera 301 has a wider angle than narrow-angle camera 302. When imaging device 30 is attached to refrigerator 20, wide-angle camera 301 captures images of refrigeration compartment 21 from above and in front of refrigerator 20.

[0038] For example, the imaging range of wide-angle camera 301 includes range A1 when viewed from the front of refrigerator 20. Range A1 is a range that includes door pockets 21E of right door 21B and left door 21C and opening 201 of refrigerator compartment 21 when right door 21B and left door 21C are fully open.

[0039] 4 when viewed from the right of refrigerator 20. Range A3 is a range that includes, in the front-to-back direction, the front ends of right door 21B and left door 21C in the open state to the front end of shelf 21F, and is a range that includes, in the up-down direction, opening 201 of refrigerator compartment 21. Wide-angle camera 301 corresponds to an example of a "camera" in the present disclosure.

[0040] Narrow-angle camera 302 has a narrower angle than wide-angle camera 301. In other words, the imaging angle of narrow-angle camera 302 in the left-right direction, i.e., the X-axis direction, is smaller than the imaging angle of wide-angle camera 301 in the left-right direction, i.e., the X-axis direction. Also, the imaging angle of narrow-angle camera 302 in the front-to-back direction, i.e., the Y-axis direction, is smaller than the imaging angle of wide-angle camera 301 in the front-to-back direction, i.e., the Y-axis direction.

[0041] With the imaging device 30 attached to the refrigerator 20, the narrow-angle camera 302 captures an image of the drawer 26A from above the front of the refrigerator 20.

[0042] For example, the imaging range of narrow-angle camera 302 includes range A2 shown in FIG. 3 when viewed from the front of refrigerator 20. Range A2 is a range that includes drawer 26A when viewed from the front of refrigerator 20. Also, for example, the imaging range of narrow-angle camera 302 includes range A4 shown in FIG. 4 when viewed from the right of refrigerator 20. Range A4 is a range that includes drawer 26A when it is pulled out to the maximum in the front-to-back direction. Note that the imaging range of narrow-angle camera 302 may also be a range that includes drawers provided in refrigerator compartment 21. Narrow-angle camera 302 corresponds to an example of a "camera" in the present disclosure.

[0043] FIG. 5 is a plan view of the imaging device 30 installed in the refrigerator 20 as seen from above.

[0044] The left door 21C is rotatably supported by a first hinge member 203A installed on the top surface 20A of the refrigerator 20. The right door 21B is rotatably supported by a second hinge member 203B installed on the top surface 20A of the refrigerator 20.

[0045] When viewed from above, refrigerator 20 is in the installed state, and recessed portions 205 are formed on the front surfaces of left door 21C and right door 21B in the closed state. Recessed portions 205 are formed in the approximate center of refrigerator 20 in the width direction when viewed from above.

[0046] FIG. 6 is a side view of the imaging device 30 as seen from the right.

[0047] The refrigerator 20 has a main box 27 with an open front. The main box 27 has a refrigerator compartment 21, an ice-making compartment 22, a fresh-freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25 formed therein. These storage compartments are cooled with cold air. A revolving door is installed in the opening 201 on the front of the refrigerator compartment 21. The door opens and closes the opening 201. The door includes a revolving right door 21B and a revolving left door 21C.

[0048] The imaging device 30 includes a main body 32 installed on the upper surface 20A of the main box 27, and a photographing member 31 extending forward from the main body 32. The photographing member 31 is provided above the front of the main body 32.

[0049] The main body 32 includes a regulating member 34. The regulating member 34 abuts against the front edge of the opening 201 of the main box body 27, thereby regulating the position of the imaging member 31 in the front-to-rear direction of the refrigerator 20. The regulating member 34 is a plate-like member extending downward from the bottom surface 30A of the main body 32. The regulating member 34 extends in the width direction of the main body 32 (the width direction of the imaging device 30) on the front surface of the main body 32. The length of the regulating member 34 extending from the bottom surface 30A is shorter than the length from the top surface 20A of the main box body 27 to a door gasket that is provided at the upper part of the inner surface of the door, generally parallel to the top surface 20A.

[0050] 7 is a diagram showing an example of the image capturing state of the narrow-angle camera. As shown in FIG. 7, narrow-angle camera 302 constitutes part of image capturing system 10. Image capturing system 10 includes narrow-angle camera 302, a first marker MK51, a second marker MK52, a third marker MK41, and a fourth marker MK42 (the first marker MK51 to the fourth marker MK42 may be collectively referred to as markers MK). In other words, image capturing system 10 includes refrigerator 20 equipped with first marker MK51, second marker MK52, third marker MK41, and fourth marker MK42, and image capturing device 30 equipped with narrow-angle camera 302.

[0051] The first marker MK51, the second marker MK52, the third marker MK41, and the fourth marker MK42 will be described with reference to FIGS.

[0052] The photographing state of the narrow-angle camera 302 will be described with reference to FIG.

[0053] 7 illustrates the same X-axis, Y-axis, and Z-axis as those shown in FIG. 3. The X-axis direction corresponds to the width direction and left-right direction of the refrigerator 20. The Y-axis direction corresponds to the front-rear direction of the refrigerator 20. The Z-axis direction corresponds to the up-down direction of the refrigerator 20.

[0054] Fig. 7 is a diagram showing an example of the state of image capture by narrow-angle camera 302. As shown in Fig. 7, narrow-angle camera 302 is placed on the top surface of main box body 27 of refrigerator 20. In other words, as described with reference to Fig. 6, narrow-angle camera 302 is installed at the front end of bottom surface 31A of imaging member 31 that constitutes imaging device 30. Also, as shown in Fig. 7, narrow-angle camera 302 captures an image of drawer 26A from the upper front of refrigerator 20.

[0055] As shown in Figure 7, narrow-angle camera 302 captures an image of an object placed within a capture range at a capture angle θ2. The capture range is a range along the front of refrigerator 20. The capture range extends generally parallel to the Z axis. Capture angle θ2 is, for example, 25 degrees. Narrow-angle camera 302 captures an image of food stored inside drawer 26A. Drawer 26A refers to drawers 22A, 23A, 24A, and 25A. Drawers 22A, 23A, 24A, and 25A capable of storing food are installed in ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively.

[0056] In FIG. 7, a case where the drawer 26A is the drawer 25A arranged in the vegetable compartment 25 will be described.

[0057] As shown by the solid line in FIG. 7, when drawer 25A is pulled out to a position where first marker MK51 is not blocked by drawer 24A, narrow-angle camera 302 can photograph first marker MK51.

[0058] In Fig. 7, the drawer 25A is at the maximum open position. In other words, in Fig. 7, the drawer 25A is at the maximum open position.

[0059] Also, as shown in FIG. 7, when the drawer 24A is at the maximum open position, the first marker MK51 is located at a predetermined position.

[0060] Therefore, for example, when drawer 25A transitions from a closed state to an open state and reaches the maximum open position shown in FIG. 7, recording of the images captured by wide-angle camera 301 and narrow-angle camera 302 begins. At this time, imaging device 30 may output a buzzer sound. In this case, it is possible to notify the user that first marker MK51 is in a predetermined position. Note that recording of the images captured by wide-angle camera 301 and narrow-angle camera 302 may also begin when first marker MK51 or second marker MK2 is detected.

[0061] 7 illustrates the case where drawer 26A is drawer 25A arranged in vegetable compartment 25, but the present disclosure is not limited to this. Drawer 26A may be any of drawers 22A, 23A, 24A, and 25A arranged in ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively.

[0062] 7, the drawer 26A of the vegetable compartment 25 has a storage case 25B. The storage case 25B is located in the upper part of the drawer 26A and is configured to be removable. The second marker MK52 is located on the upper surface of the front side of the storage case 25B.

[0063] 7, marker MK1 is placed in refrigerator compartment 21. Narrow-angle camera 302 can capture an image of marker MK1 when the door is open.

[0064] Drawer 24A has storage case 24B, which is configured to be able to be pulled out, in the upper section of drawer 24A. Storage case 24B has an opening / closing body that opens and closes the opening of storage case 24B. The opening / closing body has two plate-shaped opening / closing members that are arranged to be able to slide left and right in the storage case 24B. A fourth marker MK42 is arranged on the top surface of the opening / closing member.

[0065] Fig. 8 is a diagram showing an example of a marker MK1 placed on a non-movable portion of the opening of the refrigerator compartment 21 near the bottom of the right door 21B. As shown in Fig. 8, the marker MK1 is placed on the hinge cover AS. Fig. 8 shows a state in which the hinge cover AS is attached to the right door hinge member 204. In Fig. 8, a part of the right door hinge member 204 that is not covered by the hinge cover AS is exposed. The right door hinge member 204 rotatably supports the right door 21B.

[0066] The hinge cover AS is formed in an L-shape in a plan view, and an imaging surface PF2, which is the object to be imaged, is formed on the upper surface. A marker MK1 containing identification information indicating the refrigeration compartment 21 is formed on the imaging surface PF2. For example, a sticker on which the marker MK1 is formed is affixed to the imaging surface PF2.

[0067] Figure 9 is a perspective view of the hinge cover AS. As shown in Figure 9, a notch AS1 is formed in the hinge cover AS. The notch AS1 fits into the upper part of the frame of the drawer 23A (e.g., the right door hinge member 204). The hinge cover AS is made of, for example, resin. The hinge cover AS may be the same color as the main box body 27 and the hinge, or may be a different color. The hinge cover AS is made of, for example, white.

[0068] 8 and 9, the case where the marker MK1 is placed on the right door 21B of the refrigerator compartment 21 has been described, but the present disclosure is not limited to this. The marker MK1 may also be placed on the left door 21C of the refrigerator compartment 21, for example.

[0069] Fig. 10 shows an example of a screen display on the user's information terminal 50 for showing the user the procedure for attaching a marker MK to the refrigerator 20. Fig. 11 shows an example of a screen display on the user's information terminal 50 for showing the user the procedure for checking the attachment status of the marker MK.

[0070] When the user attaches imaging device 30 to the specified position on top of refrigerator 20, at least one of the pieces of information, such as the model name, product number, and serial number of imaging device 30 to be attached, is registered in server device 40 as basic information of imaging device 30, and at least one of the pieces of information, such as the model name, product number, and serial number of refrigerator 20 to which imaging device 30 is attached, is registered as basic information of refrigerator 20 (attached refrigerator device registration complete). Server device 40 can extract structural information, such as dimensions of each position of registered refrigerator 20, from the basic information of registered refrigerator 20.

[0071] Once the basic information has been registered, a screen (a) prompting the user to install a marker MK is displayed on the user's information terminal 50. When the user presses the installation start button displayed on the screen (a), a screen (b) instructing the installation position of the marker MK in each storage compartment is displayed. The user confirms the installation position of the marker MK and prepares the marker MK to be installed according to the instructions on the screen (b). The storage compartments in which the marker MK will be installed are the refrigerator compartment 21, the vegetable compartment 25, and the freezer compartment 24. The installation order is arbitrary, but here, the user is instructed to set the markers in the order of the refrigerator compartment 21, the vegetable compartment 25, and the freezer compartment 24.

[0072] When the "Next" button displayed on screen (b) is pressed, screen (c) appears, instructing the user to place a marker MK in refrigerator compartment 21. The user follows the instructions on screen (c) to place markers MK at two specified attachment positions near left door 21C and right door 21B of refrigerator compartment 21. The attachment positions of markers MK near left door 21C and right door 21B are non-movable portions below opening 201 of refrigerator compartment 21, for example, on the top surfaces of the covers of the hinge members that rotatably support left door 21C and right door 21B, as shown in Figures 8 and 9. Here, it is assumed that marker MK1 is placed in the non-movable portion near right door 21B.

[0073] After placing the marker MK at the specified position in the refrigerator compartment 21, the user presses the "Next" button displayed on screen (c). When the user presses the "Next" button, screen (d) is displayed, instructing the user to set the marker MK in the vegetable compartment 25. The user follows the instructions on screen (d) to place the first marker MK51 at the specified installation position in the vegetable compartment 25. In addition to placing the first marker MK51 at the top of the front part of the drawer of the vegetable compartment 25, depending on the model of the refrigerator 20, the user may also be instructed to place the second marker MK52 at the front of the storage case 25B of the vegetable compartment 25.

[0074] After setting the marker MK at the specified position in the vegetable compartment 25, the user presses the "Next" button displayed on screen (d). When the user presses the "Next" button, screen (e) is displayed, instructing the user to set the marker MK in the freezer compartment 24. The user follows the instructions on screen (e) to place the third marker MK41 at the specified installation position in the freezer compartment 24. In addition to placing the third marker MK41 at the top of the front part of the drawer of the freezer compartment 24, depending on the model of the refrigerator 20, the user may also be instructed to place the fourth marker MK42 at the front of the storage case 24B of the freezer compartment 24.

[0075] After placing the marker MK at the specified position in the freezer compartment 24, the user presses the "Next" button displayed on the screen (e). When the "Next" button is pressed by the user, the process proceeds to a step of checking whether the marker MK is properly attached, as shown in FIG.

[0076] First, a screen (f) prompting the user to start an automatic check of the attachment status of the marker MK1 for the refrigerator compartment 21 is displayed on the screen of the user's information terminal 50. When the user presses the "Start Automatic Check" button, the imaging system 10 automatically checks the attachment status of the marker MK1 for the refrigerator compartment 21. Here, the marker MK1 for the refrigerator compartment 21 is used to determine the cut-out position when the imaging device 30 captures an image of the refrigerator interior when the vegetable compartment 25, which is a drawer compartment, is pulled out during normal use and cuts out the interior area from that image. Therefore, the automatic check checks whether the marker MK1 for the refrigerator compartment 21 is properly detected in the image captured by the imaging device 30, and also adjusts the position of the area to be cut out from the image as the interior area of ​​the vegetable compartment 25. The method for adjusting and determining the cut-out position will be described in detail below.

[0077] Once the automatic check of the attachment status of marker MK1 in refrigerator compartment 21 is complete, a screen (g) prompting the user to start an automatic check of the attachment status of first marker MK51 in vegetable compartment 25 is displayed on the screen of information terminal 50. In this automatic check, it is checked whether first marker MK51 in vegetable compartment 25 is properly detected in the image captured by imaging device 30. If second marker MK52 is attached to storage case 25B of vegetable compartment 25, it is also checked whether second marker MK52 is properly detected.

[0078] Once the automatic check of the attachment status of the marker MK in the vegetable compartment 25 is complete, a screen (h) prompting the user to start an automatic check of the attachment status of the third marker MK41 in the freezer compartment 24 is displayed on the screen of the user's information terminal 50. In this automatic check, it is checked whether the third marker MK41 in the freezer compartment 24 is properly detected in the image captured by the imaging device 30. If the fourth marker MK42 is attached to the storage case 24B of the freezer compartment 24, it is also checked whether the fourth marker MK42 is properly detected.

[0079] Once the user has completed the above steps, the marker installation is complete.

[0080] FIG. 12 is a diagram showing an example of a method for calculating a coordinate difference from a first image captured by the imaging device 30 of the lower part of the opening 201 of the refrigerator compartment 21 during initial setup, such as immediately after the imaging device 30 and the marker MK are installed in the refrigerator 20.

[0081] Figure 13 is a diagram showing an example of a method for appropriately setting the position of the area to be cut out from the image as the interior area of ​​the vegetable compartment 25 in the second image captured by the imaging device 30 of the vegetable compartment 25, which is a drawer compartment, using the coordinate difference calculated during initial setup.

[0082] The processes in Figures 12 and 13 are performed at the stage (f) in Figure 11. In addition to the initial setup, they may also be performed when adjusting the position of the imaging device again during normal use.

[0083] (1) in FIG. 12 shows the designated position of a non-movable part (hinge) calculated from the dimensions and other structural information of the refrigerator. (2) in FIG. 12 shows the position of a non-movable part (hinge) or a marker MK placed on a non-movable part detected from an image captured by an imaging device (camera) as the object to be photographed. (3) in FIG. 12 shows, with an arrow, the state in which the positional relationship between the designated position calculated in (1) and the position of the object to be photographed detected in (2). Specifically, the difference between the coordinates of the designated position in (1) and the coordinates of the position of the object to be photographed in (2) is calculated, and this difference is used to adjust and determine the reference position for cutting out the fridge interior image. If there is a marker MK near the left door 21C and a marker MK near the right door 21B, the positional relationship (coordinate difference) near the left door 21C and the positional relationship (coordinate difference) near the right door 21B may be calculated separately, and the average of these differences may be used.

[0084] FIG. 13 compares the cropped area (cropped area in FIG. 13 ) cropped from the second image captured by the imaging device 30 of the vegetable compartment 25 during normal use, with the cropped area (cropped area) cropped from the second image captured during normal use, with the cropped area (cropped area) cropped from the second image captured by the imaging device 30 during normal use, with the cropped area (cropped area) cropped from the second image captured during normal use. (4) in FIG. 13 shows the cropped area (cropped area) in the pre-calibration state, where the cropped position is not adjusted and is calculated based on the dimensions and other structural information of the refrigerator 20. (5) in FIG. 13 shows the cropped area adjusted (calibrated) to the appropriate position by applying the specified positional relationship (coordinate difference) determined in (3) to the state of (4). Note that the positional relationship (coordinate difference) is adjusted depending on the distance from the imaging device 30 to the vegetable compartment 25 drawer.

[0085] As can be seen from Fig. 13, by adjusting the cutout position during initial setup, etc., it is possible to set a more appropriate cutout area during normal use. For example, as shown in (5) of Fig. 13, during normal use, it is possible to cut out an area from the captured image that matches the pullout area more than when no adjustment is made.

[0086] 14 is a flowchart showing the process when the imaging system 10 automatically checks the attachment state of the marker MK1 in the refrigerator compartment 21. This is executed when the user presses the "Start automatic check" button on the screen (f) in FIG.

[0087] When the user presses the "Start automatic check" button, it is confirmed whether the basic information of the refrigerator 20 has been registered (specified) by the user (S01). If the basic information has been registered by the user, it is determined that the basic information has been specified (YES in S01), and the process proceeds to S03. If the basic information has not been registered by the user (NO in S01), a screen prompting the user to input the basic information is displayed (notified) on the information terminal 50 (S02). If the user has registered the basic information, YES is determined in S01, and the process proceeds to S03.

[0088] Once the basic information of the refrigerator has been specified, it is detected whether both doors of the refrigerator compartment 21 are open (S03). If it is detected that both doors are open (YES in S03), the process proceeds to S05. If both doors are not open (NO in S03), a screen prompting the user to fully open both doors is displayed (notified) on the information terminal 50 (S04). If both doors have been opened by the user and this state is detected, the result in S03 is YES, and the process proceeds to S05.

[0089] If it is detected that both doors of the refrigerator are open, imaging device 30 captures an image of the lower part of opening 201 of refrigeration compartment 21 (S05), and server device 40 acquires the captured first image (S06).

[0090] Once the first image is acquired, the server device 40 checks whether the marker MK is present in the first image (S07). If the marker MK is detected in the first image (YES in S07), the process proceeds to S09. If the marker MK is not detected in the first image (NO in S07), the information terminal 50 displays (notifies) a screen prompting the user to check whether the image capture device 30 and the marker MK1 in the refrigerator compartment 21 are properly installed, and if they are not properly installed, to correct them to the appropriate status (S08). If the user checks the image capture device 30 and the marker MK1, corrects them to the appropriate status, and the marker MK1 is detected, the result in S07 becomes YES, and the process proceeds to S09.

[0091] If marker MK1 of refrigerator compartment 21 is detected, server device 40 performs process (1) shown in FIG. 12, i.e., calculates the designated position of the non-movable part (hinge) from the dimensions and other structural information corresponding to the basic information of the specified refrigerator 20 (S09).

[0092] Once the designated position of the non-movable part is calculated, the server device 40 performs the processes (2) and (3) shown in Fig. 12 (S10). That is, as in (2), the server device 40 acquires the position of the non-movable part (hinge) or the marker MK1 placed on the non-movable part, which is detected from the first image captured by the imaging device 30, as the object to be photographed, and as in (3), it specifies the positional relationship between the designated position calculated in (1) and the position of the object to be photographed detected in (2). Specifically, it calculates the difference between the coordinates of the designated position in (1) and the coordinates of the position of the object to be photographed in (2).

[0093] Next, the difference is used to adjust and determine the reference position for cutting out the interior area (S11). If there are markers MK near the left door 21C and markers MK near the right door 21B, the positional relationship (coordinate difference) near the left door 21C and the positional relationship (coordinate difference) near the right door 21B may be found in S10 and the average calculated, and the average of these differences may be used in S11.

[0094] Once the cut-out position of the interior area of ​​vegetable compartment 25 has been set, the user is notified that the automatic check of the attachment status of marker MK1 in refrigerator compartment 21, such as the setting of the cut-out position, has been completed (S12). This notification may be made by playing a melody from imaging device 30 or by displaying a message on the screen of the user's information terminal 50. This notification may be made by playing a voice message or by displaying a message on the screen of the user's information terminal 50 to prompt the user to close both open doors (S12).

[0095] When the completion of the automatic check of the attachment state of the marker MK1 in the refrigerator compartment 21 is notified in S12, the screen (g) shown in FIG. 11 is displayed on the information terminal 50 of the user.

[0096] [Modification] In the above embodiment, an embodiment has been described in which, in the imaging system 10 shown in FIG. 1 , an image captured by the imaging device 30 is transmitted to the server device 40, and the main processing in FIG. 14 is performed by the server device 40. However, the present disclosure is not limited to the above embodiment.

[0097] For example, the main processing in Fig. 14 may be executed in the imaging device 30. In this case, the imaging system 10 may have a configuration as shown in Fig. 15, for example. That is, similar to the storage unit 43 included in the server device 40 in Fig. 1, the imaging device 30 in Fig. 15 includes a storage unit 73 that stores basic information about the refrigerator. Furthermore, in Fig. 15, the imaging device 30 includes a communication unit 71 and an information processing unit 72 that includes an acquisition unit 74, a detection unit 75, a calculation unit 76, an identification unit 77, and a setting unit 78 so as to have the same functions as the communication unit 41 and the acquisition unit 44, the detection unit 45, the calculation unit 46, the identification unit 47, and the setting unit 48 included in the information processing unit 42 of the server device 40 in Fig. 1, respectively.

[0098] [Other Modifications] In the above-described embodiment and modification examples, refrigerator 20 may be a refrigerator for general household use, a refrigerator used for product display in a retail store such as a convenience store, or a refrigerator for other commercial use.

[0099] Furthermore, the refrigerator 20 is an example of a storage cabinet, and the present disclosure can also be realized as other storage cabinets that have a function of cooling stored items, such as a freezer. Furthermore, the items (i.e., objects) stored in the storage cabinet are not limited to food, and may be other items. Cases in which items other than food are stored in the refrigerator 20 are also conceivable.

[0100] [Effects, etc.] Hereinafter, examples of techniques that can be obtained from the disclosure of this specification will be given, and effects, etc. that can be obtained from these techniques will be described.

[0101] Technique 1 is a method for setting a cut-out position when cutting out an interior area of ​​a refrigerator that has a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment, the setting method including: an acquisition step of acquiring a first image that captures an image of a lower part of an opening of the first storage compartment with the door open; a detection step of detecting, in the first image, the position of an object to be photographed that is installed in a non-movable part below the opening; a calculation step of calculating a designated position of the non-movable part in the first image using structural information of the refrigerator; an identification step of identifying a positional relationship between the position of the object to be photographed and the designated position of the non-movable part; and a setting step of setting, based on the positional relationship, a cut-out position when cutting out an interior area of ​​the second storage compartment from a second image that is obtained by photographing the second storage compartment with the second storage compartment pulled out.

[0102] This setting method makes it possible to set an appropriate cut-out position, so that an appropriate area can be cut out from an image of the interior of the drawer compartment of the refrigerator.

[0103] Technique 2 is the setting method according to Technique 1, in which the acquiring step is performed after attaching an imaging device to the top of the refrigerator or adjusting the position of the imaging device.

[0104] By using this setting method, an appropriate cut-out position is set at the initial setting stage, making it possible to cut out an appropriate area from an image of the interior of the drawer compartment of the refrigerator.

[0105] Technique 3 is the setting method described in Technique 1 or 2, further comprising a notification step of notifying a user to check the installation state of an imaging device used to capture the first image and the second image or the state of the object to be photographed, if the position of the object to be photographed is not detected in the detection step.

[0106] This setting method allows the user to be prompted to correct the physical state of the imaging device or the object to be photographed when the position of the object to be photographed is not detected, and makes it possible to extract an appropriate area from an image of the interior of the refrigerator drawer compartment.

[0107] Technique 4 is the setting method according to any one of techniques 1 to 3, wherein the structural information is specified in accordance with basic information of the refrigerator designated by a user.

[0108] This setting method makes it possible to identify appropriate structural information linked to basic information, so that an appropriate cut-out position can be set using the appropriate structural information, and an appropriate area can be cut out from an image taken of the interior of a refrigerator drawer compartment.

[0109] Technique 5 is the setting method according to any one of techniques 1 to 4, wherein the positional relationship is a difference value between coordinates representing the position of the object and coordinates representing the designated position of the non-movable part.

[0110] This setting method makes it possible to set the cut-out position with higher accuracy, and to cut out an appropriate area from an image of the interior of the drawer compartment of a refrigerator.

[0111] Technique 6 is the setting method according to any one of techniques 1 to 5, wherein the object to be photographed is a cover of a hinge member that pivotally supports the door in a rotatable manner at a lower part of the opening portion.

[0112] This setting method allows the reference object to be a firmly fixed member, making it possible to set the cut-out position with greater precision and cut out an appropriate area from an image of the interior of the refrigerator drawer compartment.

[0113] Technique 7 is the setting method according to technique 6, in which the position of the object to be photographed is detected based on a marker placed on the top surface of the cover.

[0114] This setting method makes it possible to reliably detect the position of the object to be photographed from the image of the marker that is easy to detect, and therefore makes it possible to extract an appropriate area from an image of the interior of the drawer compartment of the refrigerator.

[0115] Technology 8 is an imaging system capable of photographing from above a refrigerator having a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment, the imaging system including: an imaging device disposed above the refrigerator and capable of photographing an area below the opening of the first storage compartment when the door is open, and an area inside the second storage compartment when the second storage compartment is pulled out; a storage unit that stores structural information of the refrigerator; an acquisition unit that acquires a first image captured by the imaging device of the area below the opening and a second image captured by the imaging device of the interior of the second storage compartment; a detection unit that detects, in the first image, a position of an object to be photographed that is installed in a non-movable part below the opening of the first storage compartment; a calculation unit that calculates, using the structural information, a designated position of the non-movable part in the first image; an identification unit that identifies a positional relationship between the position of the object to be photographed and the designated position of the non-movable part; and a setting unit that sets a cut-out position when cutting out an area inside the second storage compartment from the second image based on the positional relationship.

[0116] Such an imaging system makes it possible to set an appropriate cut-out position, so that an appropriate area can be cut out from an image of the interior of the drawer compartment of a refrigerator.

[0117] Technology 9 is a server device that processes images taken from above of a refrigerator having a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment, the server device including: a memory unit that stores structural information of the refrigerator; an acquisition unit that acquires, by an imaging device arranged above the refrigerator, a first image taken of a lower part of an opening of the first storage compartment with the door open and a second image taken of the interior of the second storage compartment with the second storage compartment pulled out; a detection unit that detects, in the first image, a position of an object to be photographed that is installed in a non-movable part below the opening of the first storage compartment; a calculation unit that calculates, using the structural information, a designated position of the non-movable part in the first image; an identification unit that identifies a positional relationship between the position of the object to be photographed and the designated position of the non-movable part; and a setting unit that sets a cut-out position when cutting out an interior area of ​​the second storage compartment from the second image based on the positional relationship.

[0118] Such a server device makes it possible to set an appropriate cut-out position, so that an appropriate area can be cut out from an image of the interior of the drawer compartment of a refrigerator.

[0119] Technology 10 is an imaging device capable of photographing a refrigerator from above, the refrigerator having a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment, the imaging device including a camera unit disposed on top of the refrigerator and capable of photographing the bottom of the opening of the first storage compartment when the door is open and the interior of the second storage compartment when the second storage compartment is pulled out, a memory unit that stores structural information of the refrigerator, and a first image taken by the camera unit disposed on top of the refrigerator of the bottom of the opening of the first storage compartment when the door is open, and a second image taken by the camera unit disposed on top of the refrigerator. The imaging device includes an acquisition unit that acquires a second image of the interior of the second storage compartment when the storage compartment is pulled out; a detection unit that detects, in the first image, the position of an object to be photographed that is installed in a non-movable part below the opening of the first storage compartment; a calculation unit that calculates a designated position of the non-movable part in the first image using the structural information; an identification unit that identifies the positional relationship between the position of the object to be photographed and the designated position of the non-movable part; and a setting unit that sets a cut-out position when cutting out an interior area of ​​the second storage compartment from the second image based on the positional relationship.

[0120] Such an imaging device makes it possible to set an appropriate cut-out position, so that an appropriate area can be cut out from an image of the interior of the drawer compartment of a refrigerator.

[0121] Technology 11 is a terminal device that can be connected to the imaging system described in Technology 8 via a network, and the terminal device has a display unit that displays a screen that prompts a user to check the installation status of the imaging device or the status of the object to be photographed when the detection unit does not detect the position of the object to be photographed.

[0122] Such a terminal device can prompt the user to correct the physical state of the imaging device or the object to be photographed when the position of the object to be photographed is not detected, and enables the imaging system to extract an appropriate area from an image of the interior of the refrigerator drawer compartment.

[0123] Technology 12 is the terminal device according to Technology 11, wherein the display unit displays a screen for prompting input of basic information of the refrigerator to which the imaging device is attached, and the terminal device further includes a communication unit that transmits the basic information to the imaging system so that the calculation unit can identify the structural information using the basic information.

[0124] Such a terminal device allows the imaging system to identify appropriate structural information linked to basic information, thereby enabling the appropriate cut-out position to be set using the appropriate structural information, and making it possible to cut out an appropriate area from an image captured of the interior of a refrigerator drawer compartment.

[0125] Technique 13 is a program for causing a computer to execute the setting method according to any one of techniques 1 to 7.

[0126] Such a program enables a computer to extract an appropriate area from an image of the interior of a drawer compartment of a refrigerator.

[0127] Technology 14 is an information processing system that processes images of a refrigerator photographed from above, the refrigerator having a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment. The information processing system includes: a memory unit that stores structural information of the refrigerator; an acquisition unit that acquires, by an imaging device arranged above the refrigerator, a first image of a lower part of an opening of the first storage compartment with the door open and a second image of the interior of the second storage compartment with the second storage compartment pulled out; a detection unit that detects, in the first image, a position of an object to be photographed that is installed in a non-movable part below the opening of the first storage compartment; a calculation unit that calculates, using the structural information, a designated position of the non-movable part in the first image; an identification unit that identifies a positional relationship between the position of the object to be photographed and the designated position of the non-movable part; and a setting unit that sets a cut-out position when cutting out an interior area of ​​the second storage compartment from the second image based on the positional relationship.

[0128] Technique 15 is the information processing system of technique 14, further comprising a server device, wherein the storage unit, the acquisition unit, the detection unit, the calculation unit, the identification unit, and the setting unit are included in the server device.

[0129] Technique 16 is an information processing system according to technique 14, including the imaging device.

[0130] Technique 17 is the information processing system of technique 16, wherein the storage unit, the acquisition unit, the detection unit, the calculation unit, the identification unit, and the setting unit are included in the imaging device.

[0131] (Other Embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0132] For example, in the above embodiment, the imaging system is realized by multiple devices, but it may also be realized by a single device. For example, the imaging system may be realized as a single device corresponding to a server device or an imaging device. When the imaging system is realized by multiple devices, the components (especially functional components) of the imaging system may be distributed in any way among the multiple devices. For example, some or all of the processing described as being executed by the server device in the above embodiment may also be executed by an information terminal.

[0133] Furthermore, for example, the method of communication between the devices in the above-described embodiment is not particularly limited, and a relay device (not shown) may be involved in the communication between the devices.

[0134] For example, in the above embodiment, a process executed by a specific processing unit may be executed by another processing unit. Also, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0135] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0136] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0137] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0138] For example, the present disclosure may be realized as a method for setting a cutout position executed by a computer, or as a program for causing a computer to execute the setting method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

[0139] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.

[0140] The position setting method, imaging system, server device, imaging device, terminal device, and program disclosed herein are useful as a system that can extract an appropriate area from an image captured of the interior of a drawer compartment of a refrigerator.

[0141] REFRIGERATED SYMBOLS 10 Imaging system 20 Refrigerator 21 Refrigerating compartment (first storage compartment) 22 Ice making compartment (second storage compartment) 23 Fresh freezing compartment (second storage compartment) 24 Freezer compartment (second storage compartment) 25 Vegetable compartment (second storage compartment) 26 Storage compartment (second storage compartment) 27 Main box 30 Imaging device 31 Imaging member 32 Main body 34 Restricting member 40 Server device 41 Communication unit 42 Information processing unit 43 Memory unit 44 Acquisition unit 45 Detection unit 46 Calculation unit 47 Identification unit 48 Setting unit 50 Information terminal (terminal device) 51 Communication unit 52 Display unit 60 Wide area communication network (network) 71 Communication unit 72 Information processing unit 73 Memory unit 74 Acquisition unit 75 Detection unit 76 Calculation unit 77 Identification unit 78 DESCRIPTION OF SYMBOLS Setting section 201 Opening (opening portion) 204 Right door hinge member (hinge member) 205 Recess 301 Wide-angle camera 302 Narrow-angle camera 313 First wireless communication module 314 Metal plate 203A First hinge member 203B Second hinge member 20A Top surface (upper portion) 21B Right door (door) 21C Left door (door) 21E Door pocket 21F Shelf 22A Drawer (drawer-shaped) 23A Drawer (drawer-shaped) 24A Drawer (drawer-shaped) 24B Storage case 25A Drawer (drawer-shaped) 25B Storage case 26A Drawer (drawer-shaped) 30A Bottom surface 31A Bottom surface A1 Range A2 Range A3 Range A4 Range AS Hinge cover (photographed object, cover) AS1 Notch MK Marker MK1 Marker MK41 Third marker MK42 Fourth marker MK51 First marker MK52 Second marker PF2 Photographed surface (top surface of cover) θ2 Photographed angle

Claims

1. A method for setting a cut-out position when cutting out an interior area of ​​a refrigerator having a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment, comprising: an acquisition step of acquiring a first image of the lower part of the opening of the first storage compartment with the door open; a detection step of detecting the position of an object to be photographed installed in a non-movable part below the opening in the first image; a calculation step of calculating a designated position of the non-movable part in the first image using structural information of the refrigerator; an identification step of identifying the positional relationship between the position of the object to be photographed and the designated position of the non-movable part; and a setting step of setting a cut-out position when cutting out the interior area of ​​the second storage compartment from a second image obtained by photographing the second storage compartment with the second storage compartment pulled out, based on the positional relationship.

2. The setting method according to claim 1, wherein the acquiring step is performed after attaching an imaging device to the top of the refrigerator or adjusting the position of the imaging device.

3. The setting method according to claim 1, further comprising a notification step of notifying a user, if the position of the object to be photographed is not detected in the detection step, to prompt the user to check the installation status of the imaging device used to photograph the first image and the second image, or the status of the object to be photographed.

4. The setting method according to claim 1, wherein the structural information is specified in accordance with basic information of the refrigerator designated by the user.

5. The setting method according to claim 1, wherein the positional relationship is a difference value between coordinates representing the position of the object to be photographed and coordinates representing the designated position of the non-movable part.

6. The setting method according to claim 1, wherein the object to be photographed is a cover for a hinge member that rotatably supports the door at the bottom of the opening portion.

7. The setting method according to claim 6, wherein the position of the object to be photographed is detected based on a marker placed on the top surface of the cover.

8. An imaging system capable of photographing from above a refrigerator having a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment, comprising: an imaging device disposed above the refrigerator and capable of photographing the area below the opening of the first storage compartment when the door is open, and the interior of the second storage compartment when the second storage compartment is pulled out; a memory unit that stores structural information of the refrigerator; an acquisition unit that acquires a first image taken by the imaging device of the area below the opening and a second image taken by the imaging device of the interior of the second storage compartment; a detection unit that detects the position of an object to be photographed placed in a non-movable part below the opening of the first storage compartment in the first image; a calculation unit that calculates a designated position of the non-movable part in the first image using the structural information; an identification unit that identifies the positional relationship between the position of the object to be photographed and the designated position of the non-movable part; and a setting unit that sets a cut-out position when cutting out the interior area of ​​the second storage compartment from the second image based on the positional relationship.

9. A server device that processes images taken from above of a refrigerator having a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment, the server device comprising: a memory unit that stores structural information of the refrigerator; an acquisition unit that acquires, by an imaging device located above the refrigerator, a first image taken of the lower part of the opening of the first storage compartment with the door open and a second image taken of the interior of the second storage compartment with the second storage compartment pulled out; a detection unit that detects the position of an object to be photographed installed in a non-movable part below the opening of the first storage compartment in the first image; a calculation unit that calculates a designated position of the non-movable part in the first image using the structural information; an identification unit that identifies the positional relationship between the position of the object to be photographed and the designated position of the non-movable part; and a setting unit that sets a cut-out position when cutting out the interior area of ​​the second storage compartment from the second image based on the positional relationship.

10. An imaging device capable of photographing a refrigerator from above, the refrigerator having a first storage compartment with a door and a drawer-like second storage compartment below the first storage compartment, comprising: a camera unit arranged on top of the refrigerator and capable of photographing the area below the opening of the first storage compartment when the door is open, and the interior of the second storage compartment when the second storage compartment is pulled out; a memory unit that stores structural information of the refrigerator; an acquisition unit that acquires, by the camera unit arranged on top of the refrigerator, a first image taken of the area below the opening of the first storage compartment when the door is open, and a second image taken of the interior of the second storage compartment when the second storage compartment is pulled out; a detection unit that detects the position of an object to be photographed, which is placed in a non-movable part below the opening of the first storage compartment, in the first image; a calculation unit that calculates a designated position of the non-movable part in the first image using the structural information; and an identification unit that identifies the positional relationship between the position of the object to be photographed and the designated position of the non-movable part. a setting unit that sets an extraction position when extracting an interior area of ​​the second storage compartment from the second image based on the positional relationship.

11. A terminal device connectable to the imaging system of claim 8 via a network, said terminal device having a display unit that displays a screen prompting the user to check the installation status of said imaging device or the status of said object when said detection unit does not detect the position of said object.

12. The terminal device according to claim 11, wherein the display unit displays a screen for inputting basic information about the refrigerator to which the imaging device is attached, and the terminal device further has a communication unit that transmits the basic information to the imaging system so that the calculation unit can identify the structural information using the basic information.

13. A program for causing a computer to execute the setting method according to claim 1.

Images