Correction method, imaging system, imaging device, terminal device, adjustment method, server device, and program

Abstract

An imaging system (10) is provided with: an imaging unit (34) that is disposed on the upper-front portion of a refrigerator (20) and that generates a first image by imaging the inside of the refrigerator in a door-open state; a processing unit (35) that processes the first image to generate a second image for display on an information terminal (50); a display unit (54) that displays the second image; an adjustment unit (58) that has a function for adjusting the inclination of a subject in the second image, and generates an adjustment value corresponding to the adjusted inclination; a storage unit (43) that stores the adjustment value; a calculation unit (36) that uses the adjustment value stored in the storage unit (43) to calculate a correction value for correcting a first image newly captured and generated by the imaging unit (34); and a correction unit (37) that uses the correction value to correct the new first image captured and generated by the imaging unit (34).

Description

Correction method, imaging system, imaging device, terminal device, adjustment method, server device, and program

[0001] The present disclosure relates to a correction method for an image taken from the upper front of a refrigerator having a storage compartment with a door, an imaging system, an imaging device, a terminal device, an adjustment method, a server device, and a program.

[0002] Conventionally, a wide-angle camera installed on top of a refrigerator capable of capturing an image of the entire interior of the refrigerator has been used to capture images of the interior of the refrigerator. Patent Document 1 discloses a display method in which a fisheye camera or a panoramic camera is used as the wide-angle camera to capture an image of the refrigerator compartment and an interior area is cut out from the captured image.

[0003] Japanese Patent Application Laid-Open No. 2023-122162

[0004] Conventional imaging devices have a wide-angle camera with large distortion, which is problematic from the perspective of visibility of in-warehouse inventory.

[0005] The present disclosure provides an imaging system and the like that uses a wide-angle camera to capture images of the inside of a refrigerator compartment and obtain images with little distortion even when the image is cropped from the interior area.

[0006] A correction method according to one aspect of the present disclosure is a method for correcting an image taken from the upper front of a refrigerator having a storage compartment with a door, the method including: a first acquisition step of acquiring a first image taken of the storage compartment with the door open; a processing step of using the first image to generate a second image by cutting out a predetermined area of ​​the storage compartment; a display step of displaying the second image on a terminal device; an adjustment step of adjusting the tilt of a subject appearing in the second image on the terminal device and generating an adjustment value corresponding to the adjusted tilt; a calculation step of calculating a correction value for correcting the newly acquired first image using the adjustment value; and, after the calculation step, a second acquisition step of acquiring a new first image taken of the storage compartment with the door open; and a correction step of correcting the new first image using the correction value.

[0007] Moreover, an imaging system according to one aspect of the present disclosure is an imaging system capable of photographing an image of a storage compartment having a door from the upper front of a refrigerator having the storage compartment, and includes: an imaging unit disposed at the upper front of the refrigerator and configured to photograph the interior of the storage compartment with the door open and generate a first image; a processing unit that processes the first image and generates a second image to be displayed on a terminal device; a display unit that displays the second image; an adjustment unit that has a function of adjusting the tilt of a subject appearing in the second image and generates an adjustment value corresponding to the adjusted tilt; a memory unit that stores the adjustment value; a calculation unit that uses the adjustment value stored in the memory unit to calculate a correction value for correcting a first image that is newly photographed and generated by the imaging unit; and a correction unit that uses the correction value to correct the new first image that is newly photographed and generated by the imaging unit.

[0008] Moreover, an imaging device according to one aspect of the present disclosure is an imaging device capable of photographing a storage compartment having a door from an upper front portion of a refrigerator, the imaging device including: an imaging unit disposed at the upper front portion of the refrigerator, which photographs the interior of the storage compartment with the door open to generate a first image; a processing unit which processes the first image to generate a second image to be displayed on a terminal device connectable via a network; a first communication unit which transmits the second image to the terminal device or a server device connectable via a network, and receives an adjustment value generated corresponding to the adjusted tilt after an inclination of a subject appearing in the second image is adjusted in the terminal device; a calculation unit which calculates a correction value using the adjustment value received by the first communication unit to correct a first image newly captured and generated by the imaging unit; and a correction unit which corrects the new first image newly captured and generated by the imaging unit using the correction value.

[0009] Furthermore, a correction method according to one aspect of the present disclosure is a correction method for correcting an image of a storage compartment captured by an imaging device disposed at an upper front portion of a refrigerator having a door, the correction method including: an imaging step of capturing an image of the interior of the storage compartment with the door open to generate a first image; a processing step of processing the first image to generate a second image to be displayed on a terminal device connectable via a network; a transmission step of transmitting the second image to the terminal device or a server device connectable via a network; a reception step of receiving, in the terminal device, an adjustment value generated corresponding to the adjusted tilt after an inclination of a subject appearing in the second image is adjusted; a calculation step of calculating a correction value for correcting a newly captured and generated first image using the received adjustment value; and a correction step of correcting the newly captured and generated first image using the correction value.

[0010] Furthermore, a terminal device according to one aspect of the present disclosure is a terminal device that can be connected to the imaging device via a network, and includes a second communication unit that receives the second image transmitted from the imaging device, a display unit that displays the received second image, and an adjustment unit that has a function of adjusting the inclination of the subject appearing in the second image and generates an adjustment value corresponding to the adjusted inclination.

[0011] Furthermore, an adjustment method according to one aspect of the present disclosure is a method for adjusting an image in the terminal device, and includes a receiving step of receiving the second image generated by the imaging device, a display step of displaying the received second image, an adjustment step of allowing a user to adjust the inclination of a subject appearing in the second image, and a generation step of generating an adjustment value corresponding to the adjusted inclination.

[0012] Furthermore, a server device according to one aspect of the present disclosure is a server device in the above-described imaging system, and the server device has a communication unit that receives the first image and transmits the second image to the terminal device, the processing unit, the memory unit, the calculation unit, and the correction unit, wherein the correction unit generates the corrected first image by correcting the newly generated first image, the processing unit processes the generated corrected first image to generate a new second image, and the communication unit transmits the new second image to the terminal device.

[0013] According to the present disclosure, it is possible to obtain an image with little distortion even when the interior of a refrigerator is photographed using a wide-angle camera and the image is cut out from the interior area.

[0014] FIG. 1 is a block diagram showing the functional configuration of an imaging system. FIG. 2 is an external view of a refrigerator and an imaging device provided in the imaging system. FIG. 3 is a front view of the refrigerator and the imaging device as seen from the front. FIG. 4 is a side view of the refrigerator and the imaging device as seen from the right. FIG. 5 is a plan view of the imaging device installed in the refrigerator as seen from above. FIG. 6 is a side view of the imaging device as seen from the right. FIG. 7 is a diagram showing an example of the imaging state of a narrow-angle camera. FIG. 8 is a diagram showing the transition of a distortion-corrected image comparing cases with and without optical axis deviation correction. FIG. 9 is a conceptual diagram showing an image of optical axis deviation. FIG. 10 is a diagram showing the sequence from when an image is captured until an image with distortion correction is displayed. FIG. 11 is a block diagram showing a functional configuration different from that shown in FIG. 1 of the imaging system. FIG. 12 is a diagram showing a sequence different from that shown in FIG. 10 from when an image is captured until an image with distortion correction is displayed. FIG. 13 is a diagram showing that images of the left and right doors are also corrected by optical axis deviation correction.

[0015] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components.

[0016] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0017] In the following description, images and videos may be collectively referred to as images.

[0018] (Knowledge that formed the basis of the present disclosure) At the time the inventors came up with the idea for the present disclosure, a system was being considered in which an imaging device having a camera would be used to photograph the interior of the refrigerator compartment from the upper front of the refrigerator, and the inventory of food ingredients would be determined from the photographed image.

[0019] To capture a wide field of view, such as the interior of a refrigerator, a wide-angle camera using a fisheye lens is generally used. Images captured using a wide-angle lens are circular images (fisheye images) with significant distortion, so correction is performed to cut out a specific area from the fisheye image in an appropriate viewing state, and a cut-out image of the desired interior area with little distortion is obtained.

[0020] However, the present inventors have discovered that there is a problem in that the subject in the cut-out image obtained by such correction appears tilted.

[0021] If a wide-angle camera, which produces images that appear tilted as one type of distortion, were directly applied to a system for capturing images of the inside of a refrigerator, it would be difficult to ensure sufficient quality in terms of inventory visibility. For this reason, if imaging devices equipped with such wide-angle cameras were mass-produced during manufacturing, the yield would decrease, which could have a significant impact on product costs.

[0022] Therefore, in order to solve this problem, the inventors made extensive efforts and discovered that the tilt of the subject in such a cropped image is caused by the positional relationship between the image sensor and lens in a wide-angle camera. They found that by improving this point, it is possible to reduce the tilt of the subject even in such a cropped image, which constitutes the subject of the present disclosure.

[0023] The present disclosure makes it possible to obtain a good cut-out image by using a wide-angle camera to photograph the interior of a refrigerator compartment and cutting out the image of the interior area, reducing not only the inherent distortion that wide-angle camera images have, but also the distortion caused by the tilt of the cut-out image.

[0024] (Embodiment) [Configuration] First, the configuration of an imaging system according to an embodiment will be described. Fig. 1 is a block diagram showing the functional configuration of an imaging system according to an embodiment.

[0025] The imaging system 10 shown in FIG. 1 is a system for capturing images of the inside of a refrigerator 20 and displaying the captured images, which show food stored in the refrigerator 20, on a display unit 54 of an information terminal 50, which is a terminal device. Specifically, the imaging system 10 includes an imaging device 30, a server device 40, and an information terminal 50. The imaging system 10 may also be referred to as an imaging system, including the refrigerator 20. The imaging device may also be referred to as a camera unit. The imaging system 10 is also used to display a screen on the display unit 54 of the information terminal 50 that presents the user with the processes that the user must perform for various settings, for example, during the initial setup stage, when the imaging device 30 is attached to the upper front of the refrigerator 20 or when the position of the imaging device 30 is adjusted.

[0026] The refrigerator 20 is an example of a storage unit capable of refrigerating stored items, and is installed in a user's home or the like to refrigerate food. The refrigerator 20 has at least one storage compartment with one or two doors and one or more drawer-like storage compartments.

[0027] Imaging device 30 is disposed at the upper front of the refrigerator and captures images of the interior of the refrigerator when the storage compartment door is open or the storage compartment drawer is pulled out. Imaging device 30, for example, starts capturing images when the user opens the refrigerator door, stops capturing images when the user closes the door, and transmits the image of the interior immediately before that to server device 40. Alternatively, imaging device 30 starts capturing images when the user pulls out the storage compartment drawer, stops capturing images when the user closes the drawer, and transmits the image of the interior immediately before that to server device 40. Imaging device 30 includes a communication unit 31, an information processing unit 32, a storage unit 33, and an imaging unit 34.

[0028] The communication unit 31 is a communication circuit that enables the imaging device 30 to communicate with the server device 40 and the information terminal 50 via the wide area communication network 60. The communication unit 31 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard used for communication by the communication unit 31.

[0029] The information processing unit 32 performs information processing such as processing and correction of data of the fridge interior image captured by the imaging unit 34. The information processing unit 32 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 32 includes, as functional components, a processing unit 35, a calculation unit 36, and a correction unit 37. The various functions of the information processing unit 32 are realized, for example, by the microcomputer or the like constituting the information processing unit 32 executing a computer program stored in the storage unit 33. The functions of the processing unit 35, the calculation unit 36, and the correction unit 37 will be described in detail below.

[0030] The storage unit 33 is a storage device that stores the computer program executed by the information processing unit 32 and various information necessary for the information processing. For example, the storage unit 33 may receive and store various information from the server device 40. The storage unit 33 is realized by, for example, a semiconductor memory.

[0031] Server device 40 is a computer located outside the facility where refrigerator 20 is installed, and specifically, is a cloud server. Server device 40 performs information processing to display, for example, an image of the interior of refrigerator 20 showing food stored therein on information terminal 50. Server device 40 includes a communication unit 41, an information processing unit 42, and a storage unit 43.

[0032] The communication unit 41 is a communication circuit that enables the server device 40 to communicate with the imaging device 30 and the information terminal 50 via the wide area communication network 60. The communication unit 41 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard of the communication performed by the communication unit 41.

[0033] The information processing unit 42 performs information processing to display, for example, an image of the interior of the refrigerator 20 showing the food stored therein, or an instruction screen for the user during initial setup, on the information terminal 50. The information processing unit 42 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The various functions of the information processing unit 42 are realized, for example, by the microcomputer or the like constituting the information processing unit 42 executing a computer program stored in the storage unit 43.

[0034] The storage unit 43 is a storage device that stores the computer programs executed by the information processing unit 42 and various information required for the information processing. The storage unit 43 is realized by, for example, a semiconductor memory.

[0035] The information terminal 50 is an information terminal owned by a user. The user uses the information terminal 50, for example, to attach the imaging device 30 to the refrigerator 20 and perform initial settings, or to check the food stored in the refrigerator 20 while away from home. The information terminal 50 is, for example, a portable information terminal such as a smartphone or a tablet terminal, but may also be a stationary information terminal such as a personal computer. The information terminal 50 includes a communication unit 51, an information processing unit 52, a storage unit 53, and a display unit 54.

[0036] The communication unit 51 is a communication circuit that enables the information terminal 50 to communicate with the imaging device 30 and the server device 40 via the wide area communication network 60. The communication unit 51 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard of the communication performed by the communication unit 51.

[0037] The information processing unit 52 performs information processing to display, for example, an image of the fridge interior or an instruction screen for the user during initial setup on the display unit 54. The information processing unit 52 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 52 has an adjustment unit 58 as a functional component. The various functions of the information processing unit 52 are realized, for example, by the microcomputer or the like constituting the information processing unit 52 executing a computer program stored in the storage unit 53. The functions of the adjustment unit 58 will be described in detail below.

[0038] The storage unit 53 is a storage device that stores the computer program executed by the information processing unit 52 and various information necessary for the information processing. For example, the necessary computer program or various information may be downloaded from the server device 40 and stored therein. The storage unit 43 is realized by, for example, a semiconductor memory.

[0039] 10 and 12, for example, are displayed on the display unit 54. The display unit 54 is realized by a display panel such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel.

[0040] FIG. 2 is an external view of the refrigerator 20 and the imaging device 30 included in the imaging system 10.

[0041] The refrigerator 20 is an example of a storage unit capable of refrigerating stored items, and is installed in a user's home or the like to refrigerate food. The refrigerator 20 has at least one storage compartment with one or two doors and one or more drawer-like storage compartments.

[0042] The imaging device 30 includes two cameras, one with a wide-angle lens and the other with a telephoto lens. The imaging device 30 is positioned at the upper front of the refrigerator and captures images of the interior of the refrigerator when the storage compartment door is open or the storage compartment drawer is pulled out. For example, the imaging device 30 starts capturing images when the user opens the refrigerator door and stops capturing images when the user closes the door, transmitting the image of the interior immediately before that to the server device 40. Alternatively, the imaging device 30 starts capturing images when the user pulls out a storage compartment drawer and stops capturing images when the user closes the drawer, transmitting the image of the interior immediately before that to the server device 40. For example, the imaging device 30 captures an image of a storage compartment drawer (e.g., a vegetable drawer) of the refrigerator 20 pulled out from above the refrigerator 20 using the camera with a telephoto lens. In other words, the imaging device 30 captures an image of the drawer of the refrigerator 20 viewed from above the refrigerator 20, for example, looking down on the refrigerator 20. Furthermore, for example, the imaging device 30 captures an image of the refrigerator 20 from above with a camera having a wide-angle lens, with the storage compartment door (for example, the refrigerator compartment door) open.

[0043] Fig. 3 is a front view of the refrigerator 20 and the image capture device 30. The refrigerator 20 shown in Fig. 3 has both its left and right doors open. Fig. 4 is a side view of the refrigerator 20 and the image capture device 30 seen from the right. The refrigerator 20 shown in Fig. 4 has its drawer 25A in an open state, i.e., pulled out.

[0044] The interior space of refrigerator 20 is divided into multiple spaces by partition members. In this embodiment, this interior space is divided into five spaces, which respectively function as refrigerator compartment 21, ice-making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25. In the following description, when there is no need to distinguish between refrigerator compartment 21, ice-making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, they may be referred to as "storage compartments" and assigned the reference number "26."

[0045] In the following description, when there is no need to distinguish between the drawer-shaped ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, they may be referred to as "drawer compartments." The drawer compartments correspond to the "second storage compartments" in this disclosure.

[0046] Drawers 22A, 23A, 24A, and 25A capable of accommodating food are installed in ice-making compartment 22, fresh-freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively. Each of drawers 22A to 25A has an open top when installed.

[0047] In the following description, when drawers 22A, 23A, 24A, and 25A are referred to collectively without distinction, they will be designated by the symbol "26A" and will be referred to as "drawer 26A."

[0048] FIG. 3 illustrates the X-axis, Y-axis, and Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to one another. The Z-axis indicates the up-down direction, which corresponds to the height direction of the refrigerator 20. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction, which corresponds to the width direction of the refrigerator 20. The Y-axis indicates the front-to-back direction. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the front direction. The positive direction of the Z-axis indicates the upward direction. The X-axis, Y-axis, and Z-axis are also illustrated in each of FIGS. 4 to 7.

[0049] Imaging device 30 includes main body 304 and imaging member 303 extending forward from main body 304. Imaging member 303 is provided above the front of main body 304. Main body 304 includes regulating member 305. Regulating member 305 regulates the position of imaging member 303 in the front-to-rear direction of refrigerator 20.

[0050] Imaging device 30 includes wide-angle camera 301 and narrow-angle camera 302. Wide-angle camera 301 and narrow-angle camera 302 are installed at the front end of the bottom surface of imaging member 303. When imaging device 30 is attached to refrigerator 20 and the door is closed, wide-angle camera 301 and narrow-angle camera 302 are located in front of the door. Wide-angle camera 301 has a wider angle than narrow-angle camera 302. When imaging device 30 is attached to refrigerator 20, wide-angle camera 301 captures an image of refrigeration compartment 21 from above and in front of refrigerator 20.

[0051] For example, the imaging range of wide-angle camera 301 includes range A1 when viewed from the front of refrigerator 20. Range A1 is a range that includes door pockets 21E of right door 21B and left door 21C and opening 201 of refrigerator compartment 21 when right door 21B and left door 21C are fully open.

[0052] 4 when viewed from the right of refrigerator 20. Range A3 is a range that includes, in the front-to-back direction, the front ends of right door 21B and left door 21C in the open state to the front end of shelf 21F, and is a range that includes, in the up-down direction, opening 201 of refrigerator compartment 21. Wide-angle camera 301 corresponds to an example of a "camera" in the present disclosure.

[0053] Narrow-angle camera 302 has a narrower angle than wide-angle camera 301. In other words, the imaging angle of narrow-angle camera 302 in the left-right direction, i.e., the X-axis direction, is smaller than the imaging angle of wide-angle camera 301 in the left-right direction, i.e., the X-axis direction. Also, the imaging angle of narrow-angle camera 302 in the front-to-back direction, i.e., the Y-axis direction, is smaller than the imaging angle of wide-angle camera 301 in the front-to-back direction, i.e., the Y-axis direction.

[0054] With the imaging device 30 attached to the refrigerator 20, the narrow-angle camera 302 captures an image of the drawer 26A from above the front of the refrigerator 20.

[0055] For example, the imaging range of narrow-angle camera 302 includes range A2 shown in FIG. 3 when viewed from the front of refrigerator 20. Range A2 is a range that includes drawer 26A when viewed from the front of refrigerator 20. Also, for example, the imaging range of narrow-angle camera 302 includes range A4 shown in FIG. 4 when viewed from the right of refrigerator 20. Range A4 is a range that includes drawer 26A when it is pulled out to the maximum in the front-to-back direction. Note that the imaging range of narrow-angle camera 302 may also be a range that includes drawers provided in refrigerator compartment 21. Narrow-angle camera 302 corresponds to an example of a "camera" in the present disclosure.

[0056] FIG. 5 is a plan view of the imaging device 30 installed in the refrigerator 20 as seen from above.

[0057] The left door 21C is rotatably supported by a first hinge member 203A installed on the top surface 20A of the refrigerator 20. The right door 21B is rotatably supported by a second hinge member 203B installed on the top surface 20A of the refrigerator 20.

[0058] When viewed from above, refrigerator 20 is in the installed state, and recessed portions 205 are formed on the front surfaces of left door 21C and right door 21B in the closed state. Recessed portions 205 are formed in the approximate center of refrigerator 20 in the width direction when viewed from above.

[0059] FIG. 6 is a side view of the imaging device 30 as seen from the right.

[0060] The refrigerator 20 has a main box 27 with an open front. The main box 27 has a refrigerator compartment 21, an ice-making compartment 22, a fresh-freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25 formed therein. These storage compartments are cooled with cold air. A revolving door is installed in the opening 201 on the front of the refrigerator compartment 21. The door opens and closes the opening 201. The door includes a revolving right door 21B and a revolving left door 21C.

[0061] The imaging device 30 includes a main body 304 installed on the top surface 20A of the main box 27, and an imaging member 303 extending forward from the main body 304. The imaging member 303 is provided above the front of the main body 304.

[0062] Main body 304 includes regulating member 305. Regulating member 305 abuts against the front edge of opening 201 of main box body 27, thereby regulating the position of imaging member 303 in the front-to-rear direction of refrigerator 20. Regulating member 305 is a plate-like member extending downward from bottom surface 30A of main body 304. Regulating member 305 extends in the width direction of main body 304 (width direction of imaging device 30) on the front surface of main body 304. The length of regulating member 305 extending from bottom surface 30A is shorter than the length from top surface 20A of main box body 27 to a door gasket that is provided at the upper part of the inner surface of the door, generally parallel to top surface 20A.

[0063] A first wireless communication module 313 is installed inside the imaging member 303. The first wireless communication module 313 is a device equipped with hardware such as an antenna and a wireless communication circuit. The first wireless communication module 313 is installed in the imaging device 30 so as to be located forward of the opening 201 of the main box 27 when the imaging device 30 is attached to the refrigerator 20. More specifically, the first wireless communication module 313 is installed forward of the restricting member 305 in the imaging device 30. If multiple wireless communication modules are installed in the imaging device 30 and the refrigerator 20 when the imaging device 30 is attached to the refrigerator 20, a metal plate 314 is installed between the multiple wireless communication modules.

[0064] 7 is a diagram showing an example of the image capturing state of narrow-angle camera 302. As shown in FIG. 7, narrow-angle camera 302 constitutes part of image capturing system 10. Image capturing system 10 includes narrow-angle camera 302, a first marker MK51, a second marker MK52, a third marker MK41, and a fourth marker MK42 (the first marker MK51 to the fourth marker MK42 may be collectively referred to as markers MK). In other words, image capturing system 10 includes refrigerator 20 equipped with first marker MK51, second marker MK52, third marker MK41, and fourth marker MK42, and image capturing device 30 equipped with narrow-angle camera 302.

[0065] The photographing state of the narrow-angle camera 302 will be described with reference to FIG.

[0066] 7 illustrates the same X-axis, Y-axis, and Z-axis as those shown in FIG. 3. The X-axis direction corresponds to the width direction and left-right direction of the refrigerator 20. The Y-axis direction corresponds to the front-rear direction of the refrigerator 20. The Z-axis direction corresponds to the up-down direction of the refrigerator 20.

[0067] Fig. 7 is a diagram showing an example of the state of image capture by narrow-angle camera 302. As shown in Fig. 7, narrow-angle camera 302 is placed on the top surface of main box body 27 of refrigerator 20. In other words, as described with reference to Fig. 6, narrow-angle camera 302 is installed at the front end of bottom surface 31A of imaging member 303 that constitutes imaging device 30. Also, as shown in Fig. 7, narrow-angle camera 302 captures an image of drawer 26A from the upper front of refrigerator 20.

[0068] As shown in Figure 7, narrow-angle camera 302 captures an image of an object placed within a capture range at a capture angle θ2. The capture range is a range along the front of refrigerator 20. The capture range extends generally parallel to the Z axis. Capture angle θ2 is, for example, 25 degrees. Narrow-angle camera 302 captures an image of food stored inside drawer 26A. Drawer 26A refers to drawers 22A, 23A, 24A, and 25A. Drawers 22A, 23A, 24A, and 25A capable of storing food are installed in ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively.

[0069] In FIG. 7, a case where the drawer 26A is the drawer 25A arranged in the vegetable compartment 25 will be described.

[0070] As shown by the solid line in FIG. 7, when drawer 25A is pulled out to a position where first marker MK51 is not blocked by drawer 24A, narrow-angle camera 302 can photograph first marker MK51.

[0071] In Fig. 7, the drawer 25A is at the maximum open position. In other words, in Fig. 7, the drawer 25A is at the maximum open position.

[0072] Also, as shown in FIG. 7, when the drawer 25A is at the maximum open position, the first marker MK51 is located at a predetermined position.

[0073] Therefore, for example, when drawer 25A transitions from a closed state to an open state and reaches the maximum open position shown in FIG. 7, recording of the images captured by wide-angle camera 301 and narrow-angle camera 302 begins. At this time, imaging device 30 may output a buzzer sound. In this case, it is possible to notify the user that first marker MK51 is in a predetermined position. Note that recording of the images captured by wide-angle camera 301 and narrow-angle camera 302 may also begin when first marker MK51 or second marker MK52 is detected.

[0074] 7 illustrates the case where drawer 26A is drawer 25A arranged in vegetable compartment 25, but the present disclosure is not limited to this. Drawer 26A may be any of drawers 22A, 23A, 24A, and 25A arranged in ice making compartment 22, fresh freezing compartment 23, freezer compartment 24, and vegetable compartment 25, respectively.

[0075] 7, the drawer 26A of the vegetable compartment 25 has a storage case 25B. The storage case 25B is located in the upper part of the drawer 26A and is configured to be removable. The second marker MK52 is located on the upper surface of the front side of the storage case 25B.

[0076] 7, marker MK1 is placed in refrigerator compartment 21. Narrow-angle camera 302 can capture an image of marker MK1 when the door is open.

[0077] Drawer 24A has storage case 24B, which is configured to be able to be pulled out, in the upper section of drawer 24A. Storage case 24B has an opening / closing body that opens and closes the opening of storage case 24B. The opening / closing body has two plate-shaped opening / closing members that are arranged to be able to slide left and right in the storage case 24B. A fourth marker MK42 is arranged on the top surface of the opening / closing member.

[0078] Here, we will show examples of images in which optical axis misalignment occurs in the wide-angle camera 301, and examples of images in which the optical axis misalignment has been corrected, and explain how the inventors came to discover a correction method to solve this problem.

[0079] A camera for acquiring image data converts light captured through an optical lens into an electrical signal using an image sensor, converting the state of a subject into digital data and generating an image or video. As mentioned above, images and videos are collectively referred to as images. Images captured and generated by a camera contain various distortions.

[0080] FIG. 8 is a diagram showing the transition of a distortion-corrected image, comparing the case where optical axis misalignment correction is performed with the case where it is performed. To capture a wide field of view, a wide-angle camera using a fisheye lens is generally used. Images captured using a wide-angle lens, if left as is, become circular images (fisheye images) with significant distortion, as shown in FIG. 8( a). The following method is one method for correcting a specific area from this fisheye image in an appropriate viewing state (hereinafter referred to as distortion correction).

[0081] The circular fisheye image shown in Fig. 8(a) is converted into a horizontally stretched image shown in Fig. 8(b) (fisheye image correction, hereinafter also referred to as fisheye correction). Then, from the fisheye-corrected image shown in Fig. 8(b), a necessary area is cropped into a trapezoid shape and converted into a rectangle to obtain the cropped image shown in Fig. 8(c).

[0082] However, in an image obtained by such distortion correction, the subject may appear tilted, as shown in (c).

[0083] Distortion correction is performed to obtain a second image, which is a cropped image obtained by cropping a necessary region from a first image, which is a fisheye image obtained by a wide-angle camera. That is, image (d) is obtained by applying a correction value to first image (a), which is a fisheye image, to correct the optical axis shift. Image (e) is obtained by performing fisheye image correction on image (d) after optical axis shift correction. A necessary region is cropped from image (e) after fisheye image correction and converted into a rectangle, resulting in second image (f), which is a cropped image.

[0084] FIG. 9 is a conceptual diagram illustrating optical axis misalignment. The inventors discovered that the tilt of a subject described above is caused by the positional relationship between the image sensor and the lens, as shown in FIG. 9 . The distortion correction described above is typically performed under the assumption that the center of the image sensor coincides with the line passing through the center of the lens, i.e., the optical axis. However, the inventors discovered that when distortion correction based on this assumption is applied directly to a camera whose optical axis is offset from the center of the image sensor, an image is generated in which the subject appears tilted after distortion correction due to the difference between the center of the distortion correction (the center of the image sensor) and the actual center of distortion (the center of the lens). Hereinafter, this misalignment between the center of the lens (the optical axis) and the center of the image sensor is referred to as optical axis misalignment.

[0085] If a wide-angle camera that generates images in which the subject appears tilted due to optical axis misalignment is directly applied to a system for capturing images of the inside of a refrigerator, it will be difficult to ensure sufficient quality in terms of inventory visibility.For this reason, if imaging devices equipped with wide-angle cameras with optical axis misalignment are mass-produced during manufacturing, the yield will decrease, which could have a significant impact on product costs.

[0086] The inventors have therefore discovered that such tilt can be reduced by calculating the amount of optical axis misalignment from the magnitude of such tilt and performing a correction (hereinafter referred to as optical axis misalignment correction) to shift the center of distortion correction to match this amount of optical axis misalignment.

[0087] The magnitude of distortion in fisheye image correction depends on the distance from the center of the lens. However, since the reference for distortion correction is usually the center of the image sensor, if there is misalignment of the optical axis, a difference occurs between the distortion correction process and the actual distortion of the lens, and the subject in the image after distortion correction appears tilted. Hereinafter, the tilt of the subject in the image may also be referred to as image tilt. The magnitude of this tilt in the image after distortion correction is proportional to the amount of optical axis misalignment.

[0088] Optical axis deviation correction can reduce the tilt of the subject in the image and keep it within a certain range by shifting the center of the fisheye image to match the amount of optical axis deviation calculated from the magnitude of the tilt.

[0089] The amount of optical axis misalignment varies from camera to camera. In other words, because there is variation between cameras, optical axis misalignment correction must be performed for each camera. Here, since the magnitude of the tilt of the image after distortion correction is proportional to the amount of optical axis misalignment, if the tilt of the image can be adjusted to determine the magnitude of the tilt, the amount of optical axis misalignment can be determined, and this can be used to perform optical axis misalignment correction.

[0090] In this imaging system, the user adjusts the terminal device on which the image is displayed so that the tilt of the image on the terminal device screen is zero, thereby determining the magnitude of tilt. While the system may be designed to precisely set the tilt of the image to zero, this is not a requirement; it is sufficient if the tilt is adjusted to a degree that the user does not perceive. Alternatively, the system may be designed to allow adjustment whenever the user perceives tilt. Alternatively, the system may be trained in advance using a learning model, and then the image tilt may be adjusted to zero using machine learning such as AI.

[0091] The distortion correction may be performed in the imaging device 30 or in the server device 40. An example of the imaging system 10 in Fig. 1 will be described below assuming that the distortion correction is performed in the imaging device 30.

[0092] The following describes the flow of processing from when an image of the interior of the refrigerator is captured by the imaging device 30 to when the image with distortion correction reflected is displayed on the information terminal 50.

[0093] FIG. 10 is a diagram showing a sequence from when an image is captured to when the image with distortion correction reflected is displayed.

[0094] (0) Acquiring a fridge interior image First, the imaging device 30 captures an image of the interior of the refrigerator compartment 21 using the wide-angle camera 301 of the imaging unit 34 to generate a first image (a), which is a fisheye image. The imaging device 30 then performs distortion correction on the generated first image (a) in the processing unit 35. Specifically, optical axis misalignment correction is first performed on the first image (a). If the user has previously adjusted the image tilt, the correction value is calculated using the adjustment value at that time, and optical axis misalignment correction is performed. Here, it is assumed that the user has not previously adjusted the image tilt, such as when capturing an image for the first time after the imaging device 30 is installed. In this case, the adjustment value is provisionally set to zero by default. However, if the wide-angle camera 301 of the imaging device 30 has optical axis misalignment, fisheye correction is performed on the first image (a) without actually performing optical axis misalignment correction, and image (b) is generated. A desired area is then cropped from image (b), and a cropped image, second image (c), is generated. Therefore, the subject in the second image (c) remains tilted, and the second image (c) generated by the processing unit 35, together with the equipment ID and adjustment value information (zero) of the imaging device 30, is transmitted from the communication unit 31 of the imaging device 30 to the server device 40 via the wide area communication network 60.

[0095] When the server device 40 receives information such as the second image (c) from the imaging device 30 via the communication unit 41, the server device 40 stores the received information in the storage unit 43. Furthermore, the server device 40 transmits this second image (c) together with the device ID and adjustment value information of the imaging device 30 from the communication unit 41 via the wide area communication network 60 to the information terminal 50, which is a terminal device, through processing by the information processing unit 42.

[0096] "(1) Display refrigerator compartment image and press adjustment button" When the information terminal 50 receives the second image (c) from the server device 40 via the communication unit 51, the information terminal 50 displays the received second image (c) on the display unit 54 on the screen. The second image (c) may be stored in the storage unit 53. The tilt adjustment button is displayed on the screen together with the second image (c).

[0097] If the user feels that the subject in the second image (c) displayed on the screen is tilted, the user can adjust the tilt of the second image (c). If the user wants to adjust the tilt of the second image (c), the user presses the tilt adjustment button displayed on the screen of the information terminal 50.

[0098] (2) Transition to tilt adjustment screen When the user presses the tilt adjustment button displayed on the screen of information terminal 50, the screen transitions to a screen for tilt adjustment of second image (c). On the tilt adjustment screen, the second image (c) is displayed together with left / right rotation buttons and an OK button for executing the functions of adjustment unit 58.

[0099] "(3) Use the left / right rotation button to level the shelf and press OK" In Figure 10, two left / right rotation buttons are displayed on the tilt adjustment screen: one to rotate the subject transferred to the second image (c) to the right and one to rotate it to the left. The user can adjust the tilt of the subject in the second image (c) displayed on the screen by operating either the left rotation button or the right rotation button. Note that the left / right rotation button may be a single button that can operate both right and left rotation. Alternatively, the left / right rotation button may not be displayed, and the user may rotate the second image (c) left / right by directly touching it on the screen.

[0100] The user rotates the subject (here, a refrigerator shelf) shown in the second image (c) by pressing either the left rotation button or the right rotation button, and stops the rotation operation when the shelf appears to be horizontal, i.e., when the tilt of the shelf appears to be zero. An image (g) of the subject, i.e., the shelf, appears when the user feels that the tilt of the shelf appears to be zero. The user then presses the OK button.

[0101] When the user presses the OK button, the information terminal 50 detects the angle adjusted by the user in the adjustment unit 58 and generates an adjustment value X corresponding to the detected angle. The information terminal 50 transmits the adjustment value X generated by the adjustment unit 58 together with the device ID of the imaging device 30 from the communication unit 51 to the server device 40 via the wide area communication network 60.

[0102] The server device 40 stores the received device ID and adjustment value X in the storage unit 43 .

[0103] (4) Displaying a warning that the results will not be reflected immediately In this imaging system 10, the results of the tilt adjustment made by the user will not be reflected immediately, and the information terminal 50 displays a warning to that effect on the screen of the display unit 54. An OK button is displayed on the information terminal 50 along with the warning message. When the user presses the OK button, the screen transitions to the next screen.

[0104] Here, the reason why the result of the tilt adjustment made by the user is not reflected immediately is as follows. In order for the adjustment value X to be transmitted from the server device 40, in which the adjustment value X is stored, to the imaging device 30 that performs the optical axis misalignment correction, a transmission request for transmitting the adjustment value X must be transmitted from the imaging device 30 to the server device 40, and the server device 40 must receive the transmission request. However, at the time the user adjusts the tilt of the second image (c) on the information terminal 50, the imaging device 30 has not yet sent the transmission request for the adjustment value X to the server device 40, and the imaging device 30 has not yet acquired the adjustment value X. The next time the refrigerator door is opened or closed, the imaging device 30 can transmit the transmission request for the adjustment value X to the server device 40 and acquire the adjustment value X from the server device 40.

[0105] "(5) Return to refrigerator compartment image display screen" The second image (c) before tilt adjustment is displayed on the screen of the display unit 54 of the information terminal 50. The user's operation to adjust the tilt of the image ends here. After this, in order to display the image with the adjusted tilt, the user performs the operations from (6) onwards.

[0106] (6) Take a photo of the interior of the refrigerator from any door. Obtain the adjustment value in response to the image upload communication. In order for the imaging device 30 to obtain the adjustment value X, the user opens one of the refrigerator doors and has the imaging device 30 take a photo of the interior. The imaging device 30 uses the wide-angle camera 301 to take a photo of the interior of the refrigerator compartment 21 and generate a first image (a), which is a fisheye image. The imaging device 30 then performs distortion correction on the generated first image (a). Specifically, optical axis misalignment correction is first performed on the first image (a). Here, since the adjustment value X has not yet been obtained, optical axis misalignment correction is performed with the adjustment value set to zero by default (effectively, no optical axis misalignment correction is performed), and fisheye correction is performed on the first image (a), generating image (b). The desired area is then cropped from image (b), and a second image (c), which is the cropped image, is generated. Therefore, the subject in the second image (c) remains tilted, and the second image (c) is transmitted from the imaging device 30 to the server device 40 via the wide area communication network 60 together with the device ID of the imaging device 30 and the adjustment value information (zero).

[0107] Upon receiving the device ID and the second image (c) from the imaging device 30, the server device 40 transmits them to the information terminal 50 via the wide area communication network 60, and also transmits the adjustment value X corresponding to the device ID stored in the storage unit 43 to the imaging device 30 corresponding to the device ID. This allows the imaging device 30 to acquire the adjustment value X, which is then converted into a correction value for correcting the optical axis misalignment using the adjustment value X in the calculation unit 36. This enables the imaging device 30 to perform optical axis misalignment correction using this correction value in the correction unit 37. Note that the adjustment value X acquired by the imaging device 30 here is stored in the storage unit 33 of the imaging device 30 until the next time the user performs a tilt adjustment and a new adjustment value generated therefor is newly transmitted from the server device 40 and received by the imaging device 30. Therefore, optical axis misalignment correction using this adjustment value X can be continuously performed.

[0108] (7) Open the door again and photograph the fridge interior If the user wants to display an image on the information terminal 50 in which the tilt of the subject is not perceived, the user opens one of the doors again and has the image capture device 30 capture the fridge interior. The image capture device 30 performs optical axis misalignment correction on the first image (a) to generate image (d), which is a corrected fisheye image. Fisheye correction is then performed on image (d), generating image (e). A desired area is then cropped from image (e), generating second image (f), which is a cropped image. The second image (f) does not appear to be tilted because the optical axis misalignment correction was performed using the adjustment value X obtained by adjusting the subject so that the user does not perceive any tilt. The second image (f) is transmitted from the image capture device 30 to the server device 40 along with the device ID and adjustment value X of the image capture device 30.

[0109] When the server device 40 receives the device ID and the second image (f) from the imaging device 30 , it transmits them to the information terminal 50 .

[0110] The information terminal 50 displays the received second image (f). The tilt of the subject that the user initially perceived has been updated in the displayed second image (f), and the subject is displayed in a state where the tilt is not perceived by the user.

[0111] When an image of the fridge interior is captured thereafter, the image capture device 30 can correct the optical axis misalignment of the first image using the acquired adjustment value X until the adjustment value is updated.

[0112] It should be noted that the above processing flow is not essential, and it is also possible for the information terminal 50 to transmit an instruction signal to the imaging device 30, when the adjustment value X is transmitted from the information terminal 50 to the server device 40, causing the imaging device 30 to issue a transmission request for the adjustment value X to the server device 40, and for the imaging device 30 to transmit a transmission request to the server device 40 based on the instruction signal, so that the imaging device 30 receives the adjustment value X from the server device 40, performs optical axis misalignment correction on the first image (a) captured initially, generates a cropped image from the corrected image, transmits the corrected cropped image to the information terminal 50 via the server device 40, and the information terminal 50 receives and displays the corrected cropped image. However, since such processing would increase the processing load on the information terminal 50 and the like, in this embodiment, the processing is performed according to the flow of FIG. 10.

[0113] [Variations] In the above embodiment, in the imaging system 10 shown in FIG. 1, as shown in FIG. 10, a configuration has been described in which the imaging device 30 processes an image captured by the imaging device 30 to generate a cut-out image, and then transmits the image to the information terminal 50 via the server device 40. However, the present disclosure is not limited to the above configuration.

[0114] For example, the server device 40 may perform the processing up to processing the captured image and generating the cut-out image.

[0115] FIG. 11 is a block diagram showing a functional configuration of the imaging system different from that of FIG. 1, and is a block diagram showing the functional configuration when the server device 40 performs the processing from processing the captured image to generating the extracted image.

[0116] The imaging system 10 shown in FIG. 11 is a system for capturing images of the inside of a refrigerator 20 and displaying the captured images, which show food stored in the refrigerator 20, on a display unit 54 of an information terminal 50, which is a terminal device. Specifically, the imaging system 10 includes an imaging device 30, a server device 40, and an information terminal 50. The imaging system 10 may also be referred to as an imaging system, including the refrigerator 20. The imaging device may also be referred to as a camera unit. The imaging system 10 may also be used to display, on the display unit 54 of the information terminal 50, a screen that presents the user with the processes that the user must perform for various settings, for example, during the initial setup stage, when the imaging device 30 is attached to the upper front of the refrigerator 20 or when the position of the imaging device 30 is adjusted.

[0117] Imaging device 30 is disposed at the upper front of the refrigerator and captures images of the interior of the refrigerator when the storage compartment door is open or the storage compartment drawer is pulled out. Imaging device 30, for example, starts capturing images when the user opens the refrigerator door, stops capturing images when the user closes the door, and transmits the image of the interior immediately before that to server device 40. Alternatively, imaging device 30 starts capturing images when the user pulls out the storage compartment drawer, stops capturing images when the user closes the drawer, and transmits the image of the interior immediately before that to server device 40. Imaging device 30 includes a communication unit 31, an information processing unit 32, a storage unit 33, and an imaging unit 34.

[0118] The communication unit 31 is a communication circuit that enables the imaging device 30 to communicate with the server device 40 and the information terminal 50 via the wide area communication network 60. The communication unit 31 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard used for communication by the communication unit 31.

[0119] The information processing unit 32 performs information processing such as adding an appliance ID to the data of the fridge interior image captured by the imaging unit 34. The information processing unit 32 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The various functions of the information processing unit 32 are realized, for example, by the microcomputer or the like constituting the information processing unit 32 executing a computer program stored in the storage unit 33.

[0120] The storage unit 33 is a storage device that stores the computer program executed by the information processing unit 32 and various information necessary for the information processing. For example, the storage unit 33 may receive and store various information from the server device 40. The storage unit 33 is realized by, for example, a semiconductor memory.

[0121] Server device 40 is a computer located outside the facility where refrigerator 20 is installed, and specifically, is a cloud server. Server device 40 performs information processing to display, for example, an image of the interior of refrigerator 20 showing food stored therein on information terminal 50. Server device 40 includes a communication unit 41, an information processing unit 42, and a storage unit 43.

[0122] The communication unit 41 is a communication circuit that enables the server device 40 to communicate with the imaging device 30 and the information terminal 50 via the wide area communication network 60. The communication unit 41 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard of the communication performed by the communication unit 41.

[0123] The information processing unit 42 performs information processing, for example, to process and correct data of images of the fridge interior captured by the imaging device 30, or to display an instruction screen for the user on the information terminal 50. The information processing unit 42 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 42 includes, as functional components, a processing unit 45, a calculation unit 46, and a correction unit 47. The various functions of the information processing unit 42 are realized, for example, by the microcomputer or the like constituting the information processing unit 42 executing a computer program stored in the storage unit 43. The functions of the processing unit 45, the calculation unit 46, and the correction unit 47 are similar to the functions of the processing unit 35, the calculation unit 36, and the correction unit 37 shown in FIG. 1.

[0124] The storage unit 43 is a storage device that stores the computer programs executed by the information processing unit 42 and various information required for the information processing. The storage unit 43 is realized by, for example, a semiconductor memory.

[0125] The information terminal 50 is an information terminal owned by a user. The user uses the information terminal 50, for example, to attach the imaging device 30 to the refrigerator 20 and perform initial setup, to adjust the display of the interior image, or to check the food stored in the refrigerator 20 while away from home. The information terminal 50 is a portable information terminal such as a smartphone or tablet, but may also be a stationary information terminal such as a personal computer. The information terminal 50 includes a communication unit 51, an information processing unit 52, a storage unit 53, and a display unit 54. The information processing unit 52 includes an adjustment unit 58 as a functional component. The functions of the communication unit 51, the information processing unit 52, the storage unit 53, the display unit 54, and the adjustment unit 58 are similar to the functions of the communication unit 51, the information processing unit 52, the storage unit 53, the display unit 54, and the adjustment unit 58 shown in FIG. 1 .

[0126] Figure 12 is a diagram showing a different sequence from that shown in Figure 10 from when an image is captured until the image with distortion correction reflected is displayed, and is a diagram showing a sequence in which an image captured by an imaging device 30 is received by a server device 40, the received image is processed by the server device 40 to generate a cut-out image, and the cut-out image is then sent to an information terminal 50 for display.

[0127] (0) Obtaining a photographed image of the refrigerator interior First, imaging device 30 uses wide-angle camera 301 of imaging unit 34 to photograph the interior of refrigerator compartment 21 and generate a first image (a), which is a fisheye image. Imaging device 30 then transmits first image (a) as is, together with the device ID of imaging device 30, from communication unit 31 via wide area communication network 60 to server device 40.

[0128] When the server device 40 receives information such as the first image (a) from the imaging device 30 via the communication unit 41, the server device 40 stores the received information in the storage unit 43, and the processing unit 45 performs distortion correction on the generated first image (a). Specifically, optical axis misalignment correction is first performed on the first image (a). If the user has previously adjusted the image tilt, the correction value is calculated using the adjustment value at that time, and optical axis misalignment correction is performed. Here, it is assumed that the user has not previously adjusted the image tilt, such as during the first image capture after the imaging device 30 is installed. In this case, the adjustment value is provisionally set to zero by default. However, if the wide-angle camera 301 of the imaging device 30 has optical axis misalignment, fisheye correction is performed on the first image (a) without actually performing optical axis misalignment correction, and image (b) is generated. A desired region is then cropped from image (b), and a cropped image, a second image (c), is generated. Therefore, the subject in the second image (c) remains tilted, and the second image (c) generated by the processing unit 45 is transmitted from the communication unit 41 to the information terminal 50, which is a terminal device, via the wide area communication network 60, together with the equipment ID and adjustment value information (zero) of the imaging device 30.

[0129] "(1) Display refrigerator compartment image and press adjustment button" When the information terminal 50 receives the second image (c) from the server device 40 via the communication unit 51, the information terminal 50 displays the received second image (c) on the display unit 54 on the screen. The second image (c) may be stored in the storage unit 53. The tilt adjustment button is displayed on the screen together with the second image (c).

[0130] If the user feels that the subject in the second image (c) displayed on the screen is tilted, the user can adjust the tilt of the second image (c). If the user wants to adjust the tilt of the second image (c), the user presses the tilt adjustment button displayed on the screen of the information terminal 50.

[0131] (2) Transition to tilt adjustment screen When the user presses the tilt adjustment button displayed on the screen of information terminal 50, the screen transitions to a screen for tilt adjustment of second image (c). On the tilt adjustment screen, the second image (c) is displayed together with left / right rotation buttons and an OK button for executing the functions of adjustment unit 58.

[0132] "(3) Use the left / right rotation button to level the shelf and press OK" In Figure 12, two left / right rotation buttons are displayed on the tilt adjustment screen: one to rotate the subject displayed in the second image (c) to the right and one to rotate it to the left. The user can adjust the tilt of the subject displayed in the second image (c) displayed on the screen by operating either the left rotation button or the right rotation button. Note that the left / right rotation button may be a single button that can operate both right and left rotation. Alternatively, the left / right rotation button may not be displayed, and the user may rotate the second image (c) to the left or right by directly touching it on the screen.

[0133] The user rotates the subject (here, a refrigerator shelf) shown in the second image (c) by pressing either the left rotation button or the right rotation button, and stops the rotation operation when the shelf appears to be horizontal, i.e., when the tilt of the shelf appears to be zero. An image (g) of the subject, i.e., the shelf, appears when the user feels that the tilt of the shelf appears to be zero. The user then presses the OK button.

[0134] When the user presses the OK button, the information terminal 50 detects the angle adjusted by the user in the adjustment unit 58 and generates an adjustment value X corresponding to the detected angle. The information terminal 50 transmits the adjustment value X generated by the adjustment unit 58 together with the device ID of the imaging device 30 from the communication unit 51 to the server device 40 via the wide area communication network 60.

[0135] The server device 40 stores the received device ID and adjustment value X in the storage unit 43. Furthermore, since the server device 40 has acquired the adjustment value X, the calculation unit 46 converts it into a correction value for correcting the optical axis misalignment using the adjustment value X, and stores this correction value in the storage unit 43. This enables the server device 40 to perform optical axis misalignment correction using this correction value in the correction unit 47.

[0136] The adjustment value X acquired by the server device 40 and the generated correction value are stored in the memory unit 43 of the server device 40 until the next time the user performs tilt adjustment on the information terminal 50 and the new adjustment value generated there is transmitted from the information terminal 50 and received by the server device 40, so that optical axis misalignment correction can be continuously performed using this stored correction value.

[0137] (4) Displaying a warning that the results will not be reflected immediately In this imaging system 10, the results of the tilt adjustment made by the user will not be reflected immediately, and the information terminal 50 displays a warning to that effect on the screen of the display unit 54. An OK button is displayed on the information terminal 50 along with the warning message. When the user presses the OK button, the screen transitions to the next screen.

[0138] Although the results of the tilt adjustment made by the user are not immediately reflected here, when the information terminal 50 transmits the adjustment value X to the server device 40, it may also transmit an instruction signal to the server device 40 to perform distortion correction, including optical axis misalignment correction, on the second image (c) and return the corrected image to the information terminal 50. In this case, the information terminal 50 can immediately reflect and display an image without tilt. However, since such processing increases the processing load on the information terminal 50 and the like, in this modified example, the processing is performed according to the flow of FIG. 12.

[0139] "(5) Return to refrigerator compartment image display screen" The second image (c) before tilt adjustment is displayed on the screen of the display unit 54 of the information terminal 50. The user's operation to adjust the tilt of the image is now complete. After this, in order to display the image with the tilt adjusted, the user performs operation (6).

[0140] (6) Open the door again and photograph the interior If the user wants to display an image on the information terminal 50 in which the tilt of the subject is not apparent, the user opens one of the doors again and has the imaging device 30 photograph the interior. The imaging device 30 uses the wide-angle camera 301 of the imaging unit 34 to photograph the interior of the refrigerator compartment 21 and generate a first image (a), which is a fisheye image. The imaging device 30 transmits the first image (a) as is, together with the device ID of the imaging device 30, from the communication unit 31 via the wide area communication network 60 to the server device 40.

[0141] When the server device 40 receives information such as the first image (a) from the imaging device 30 via the communication unit 41, the server device 40 stores the received information in the storage unit 43, and the processing unit 45 performs distortion correction on the generated first image (a). Specifically, optical axis deviation correction is first performed on the first image (a). The server device 40 stores the adjustment value X and the correction value, so it is able to perform optical axis deviation correction. The server device 40 performs optical axis deviation correction on the first image (a) to generate image (d), which is a corrected fisheye image. Fisheye correction is then performed on image (d), generating image (e). A desired region is then cropped from image (e), generating a cropped image, a second image (f). The second image (f) has undergone optical axis deviation correction using a correction value calculated using the adjustment value X obtained by adjusting the captured subject so that the user does not perceive any tilt, and therefore does not appear to be tilted. The second image (f) is transmitted from the server device 40 to the information terminal 50 together with the device ID of the imaging device 30 and the adjustment value X.

[0142] The information terminal 50 displays the received second image (f). The tilt of the subject that the user initially perceived has been updated in the displayed second image (f), and the subject is displayed in a state where the tilt is not perceived by the user.

[0143] When an image of the fridge interior is captured thereafter, the server device 40 can correct the optical axis misalignment of the first image using the acquired adjustment value X until the adjustment value is updated.

[0144] [Correction of Door Images] FIG. 13 is a diagram showing an example in which the images of the left and right doors included in the fridge interior image are also corrected by performing the optical axis misalignment correction described in the above embodiment and modified examples.

[0145] 8 has been described, the process by which a second image, a cropped image, generated by cropping the interior of refrigerator compartment 21 from a fisheye image, which is a first image captured by wide-angle camera 301, is obtained. As can be seen from Fig. 8, because the fisheye image is an image captured at a wide angle, it also captures door pocket 21E of right door 21B and door pocket 21E of left door 21C. Therefore, when the areas of the door pocket of the right door and the door pocket of the left door are set as the cropped areas, if the cropped image of door pocket 21E of right door 21B and the cropped image of door pocket 21E of left door 21C are displayed side by side, the heights of the left and right doors will appear to be misaligned, as shown in the upper right diagram of Fig. 13, unless optical axis misalignment correction is performed.

[0146] On the other hand, if the cropped image of door pocket 21E of right door 21B and the cropped image of door pocket 21E of left door 21C are displayed side by side after optical axis misalignment is corrected on the fisheye image, the tilt adjustment is also performed on each of these cropped areas in the same way as the tilt adjustment performed on the image inside refrigerator compartment 21. Therefore, the cropped images of each door pocket obtained after distortion correction also have their tilts appropriately adjusted.

[0147] Therefore, as shown in Figure 13, not only is the tilt of each cut-out image adjusted, but when an image of the door pocket 21E of the right door 21B and an image of the door pocket 21E of the left door 21C are displayed side by side, the heights of the right door 21B and the left door 21C can be displayed to be the same height, as shown in the image on the lower right side of Figure 13.

[0148] In other words, by performing distortion correction including optical axis misalignment correction according to the present disclosure, not only is the tilt of the image of the interior of the refrigerator compartment adjusted in the cut-out image, but the tilt of the left and right door pockets is also adjusted, and the height of the left and right doors is also automatically adjusted, making it possible to display the cut-out image on the screen of the terminal device.

[0149] [Other Modifications] In the above-described embodiment and modification examples, refrigerator 20 may be a refrigerator for general household use, a refrigerator used for product display in a retail store such as a convenience store, or a refrigerator for other commercial use.

[0150] Furthermore, the refrigerator 20 is an example of a storage cabinet, and the present disclosure can also be realized as other storage cabinets that have a function of cooling stored items, such as a freezer. Furthermore, the items (i.e., objects) stored in the storage cabinet are not limited to food, and may be other items. Cases in which items other than food are stored in the refrigerator 20 are also conceivable.

[0151] [Effects, etc.] Hereinafter, examples of techniques that can be obtained from the disclosure of this specification will be given, and effects, etc. that can be obtained from these techniques will be described.

[0152] Technique 1 is a method for correcting an image taken from the upper front of a refrigerator having a storage compartment with a door, the correction method including: a first acquisition step of acquiring a first image taken of the storage compartment with the door open; a processing step of generating a second image by using the first image, the second image being a cutout of a predetermined area of ​​the storage compartment; a display step of displaying the second image on a terminal device; an adjustment step of adjusting a tilt of a subject appearing in the second image on the terminal device and generating an adjustment value corresponding to the adjusted tilt; a calculation step of calculating a correction value for correcting the newly acquired first image using the adjustment value; and, after the calculation step, a second acquisition step of acquiring a new first image taken of the storage compartment with the door open; and a correction step of correcting the new first image using the correction value.

[0153] This correction method allows the adjustment value obtained by adjusting the image on the terminal device to eliminate tilt of the subject, to be converted into a correction value for correcting distortion, and this correction value can be reflected in the newly acquired image of the interior of the refrigerator.As a result, it is possible to obtain an image with little distortion even when photographing the inside of the refrigerator and cutting out the interior area.

[0154] Technique 2 is the correction method described in Technique 1, in which in the adjustment step, a display is displayed on the screen of the terminal device prompting the user to adjust the tilt of the subject in the second image, and when the user adjusts the tilt on the terminal device, the angle adjusted by the user is generated as the adjustment value.

[0155] This correction method converts the adjustment value obtained by adjusting the terminal device so that the user does not perceive the tilt of the subject in the image into a correction value for correcting distortion, and this correction value can be reflected in the newly acquired image of the interior of the refrigerator.As a result, it is possible to obtain an image with little distortion even when photographing the inside of the refrigerator and cutting out the interior area.

[0156] Technology 3 is an imaging system capable of photographing an image of a storage compartment having a door from an upper front portion of a refrigerator having the storage compartment, the imaging system including: an imaging unit that is disposed at the upper front portion of the refrigerator and photographs the interior of the storage compartment with the door open to generate a first image; a processing unit that processes the first image to generate a second image to be displayed on a terminal device; a display unit that displays the second image; an adjustment unit that has a function of adjusting the tilt of a subject shown in the second image and generates an adjustment value corresponding to the adjusted tilt; a memory unit that saves the adjustment value; a calculation unit that uses the adjustment value saved in the memory unit to calculate a correction value for correcting a first image that is newly photographed and generated by the imaging unit; and a correction unit that corrects the new first image that is newly photographed and generated by the imaging unit, using the correction value.

[0157] With this type of imaging system, the adjustment value obtained by adjusting the terminal device to eliminate tilt of the subject in the image can be converted into a correction value for correcting distortion, and this correction value can be reflected in the newly acquired image of the interior of the refrigerator.As a result, it is possible to obtain an image with little distortion even when photographing the inside of the refrigerator and cutting out the interior area.

[0158] Technology 4 relates to an imaging device capable of photographing a storage compartment having a door from an upper front portion of a refrigerator, the imaging device including: an imaging unit that is disposed at the upper front portion of the refrigerator and photographs the interior of the storage compartment with the door open to generate a first image; a processing unit that processes the first image to generate a second image to be displayed on a terminal device connectable via a network; a first communication unit that transmits the second image to the terminal device or a server device connectable via a network, and receives an adjustment value generated corresponding to the adjusted tilt after the tilt of a subject shown in the second image is adjusted in the terminal device; a calculation unit that calculates a correction value using the adjustment value received by the first communication unit to correct a first image that is newly photographed and generated by the imaging unit; and a correction unit that corrects the new first image that is newly photographed and generated by the imaging unit using the correction value.

[0159] With such an imaging device, the adjustment value obtained by adjusting the image on the terminal device to eliminate tilt of the subject in the image can be converted into a correction value for correcting distortion, and this correction value can be reflected in the newly acquired image of the interior of the refrigerator.As a result, it is possible to obtain an image with little distortion even when photographing the inside of the refrigerator and cutting out the interior area.

[0160] Technique 5 is a correction method for correcting an image of a storage compartment captured by an imaging device disposed at an upper front portion of a refrigerator having a door, the correction method including: an imaging step of capturing an image of the interior of the storage compartment with the door open to generate a first image; a processing step of processing the first image to generate a second image to be displayed on a terminal device connectable via a network; a transmission step of transmitting the second image to the terminal device or a server device connectable via a network; a reception step of receiving, in the terminal device, an adjustment value generated corresponding to the adjusted tilt after an inclination of a subject captured in the second image is adjusted; a calculation step of calculating a correction value for correcting a new first image that is newly captured and generated using the received adjustment value; and a correction step of correcting the new first image that is newly captured and generated using the correction value.

[0161] This correction method allows the adjustment value obtained by adjusting the image on the terminal device to eliminate tilt of the subject, to be converted into a correction value for correcting distortion, and this correction value can be reflected in the newly acquired image of the interior of the refrigerator.As a result, it is possible to obtain an image with little distortion even when photographing the inside of the refrigerator and cutting out the interior area.

[0162] The sixth technique is a program for causing a computer to execute the correction method described in the fifth technique.

[0163] Such a program makes it possible to convert the adjustment value obtained by adjusting the image on the terminal device to eliminate tilt of the subject in the image into a correction value for correcting distortion, and to reflect this correction value in the newly acquired image of the interior of the refrigerator.As a result, it is possible to obtain an image with little distortion even when photographing the inside of the refrigerator and cutting out the interior area.

[0164] Technology 7 is a terminal device connectable to the imaging device described in Technology 4 via a network, and includes a second communication unit that receives the second image transmitted from the imaging device, a display unit that displays the received second image, and an adjustment unit that has a function of adjusting the tilt of a subject appearing in the second image and generates an adjustment value corresponding to the adjusted tilt.

[0165] Such a terminal device adjusts the image on the display unit, i.e., the screen, so that the tilt of the subject is eliminated, and transmits the obtained adjustment value to the imaging device. The image capturing device then converts the adjustment value into a correction value for correcting distortion, and this correction value can be reflected in the newly acquired image of the interior of the refrigerator. This makes it possible to obtain an image with little distortion, even when the interior of the refrigerator is photographed and the image is cut out from the image.

[0166] Technology 8 is the terminal device described in Technology 7, in which the terminal device is connectable to a server device via a network, and the second communication unit receives the second image transmitted from the imaging device via the server device.

[0167] With such a terminal device, when a user wants to check an image of the interior of a refrigerator, the image can be received from the server device and displayed on the display unit, i.e., on the screen. When the image is received, the image is adjusted so that the subject in the image is not tilted, and the obtained adjustment value is sent to the imaging device. The image is converted into a correction value for correcting distortion in the imaging device, and this correction value can be reflected in the newly acquired image of the interior of the refrigerator. Therefore, even when an image of the interior of the refrigerator is captured and the interior area is cut out, an image with little distortion can be obtained.

[0168] Technology 9 is the terminal device described in Technology 7, in which the terminal device is connectable to a server device via a network, the server device has a memory unit that stores the adjustment values ​​and has a function of transmitting the adjustment values ​​stored in the memory unit to the imaging device, and the second communication unit transmits the adjustment values ​​generated by the adjustment unit to the server device.

[0169] Such a terminal device not only makes it possible to photograph the interior of the refrigerator compartment and obtain images with less distortion when cutting out the interior area, but also makes it possible to send the adjustment values ​​obtained when adjusting the tilt to the imaging device via the server device.The adjustment values ​​can be stored in the memory of the server device and sent from the server device to the imaging device when necessary, thereby reducing the processing load on the terminal device.

[0170] Technique 10 is the terminal device according to technique 7, further comprising a storage unit that stores the adjustment value, and the second communication unit transmits the adjustment value stored in the storage unit to the imaging device.

[0171] With such a terminal device, images of the interior of the refrigerator compartment are captured and images of the interior area are obtained with little distortion. In addition, by storing the adjustment values ​​obtained when adjusting the tilt in the terminal device's own memory, the adjustment values ​​can be sent from the terminal device to the imaging device when necessary. This reduces the time it takes for the adjustment values ​​to be received by the imaging device and improves responsiveness until the interior image is displayed on the terminal device.

[0172] Technology 11 is an adjustment method for adjusting an image in a terminal device connectable to the imaging device described in Technology 4 via a network, the adjustment method including a receiving step of receiving the second image generated by the imaging device, a display step of displaying the received second image, an adjustment step of allowing a user to adjust the tilt of a subject appearing in the second image, and a generation step of generating an adjustment value corresponding to the adjusted tilt.

[0173] With this adjustment method, an adjustment value obtained by adjusting the image on the display unit, i.e., screen, of the terminal device so that the tilt of the subject is eliminated is transmitted to the imaging device, where it is converted into a correction value for correcting distortion. This correction value can then be reflected in the newly acquired image of the interior of the refrigerator, making it possible to obtain an image with little distortion even when photographing the interior of the refrigerator and cutting out the interior area.

[0174] Technique 12 is a program for causing a computer to execute the adjustment method described in Technique 11.

[0175] With this kind of program, an adjustment value obtained by adjusting the image on the display unit, i.e., screen, of the terminal device so that the tilt of the subject in the image is eliminated is transmitted to the imaging device, where it is converted into a correction value for correcting distortion, and this correction value can be reflected in the newly acquired image of the interior of the refrigerator.As a result, it is possible to obtain an image with little distortion even when photographing the inside of the refrigerator and cutting out the interior area.

[0176] Technology 13 is a server device in the imaging system described in Technology 3, the server device having a communication unit that receives the first image and transmits the second image to the terminal device, the processing unit, the storage unit, the calculation unit, and the correction unit, wherein the correction unit generates the corrected first image by correcting the newly generated first image, the processing unit processes the generated corrected first image to generate the new second image, and the communication unit transmits the new second image to the terminal device.

[0177] Such a server device can receive adjustment values ​​obtained by adjusting the terminal device to eliminate tilt of the subject in the image, convert them into correction values ​​for correcting distortion, and reflect these correction values ​​in a new interior image acquired by the imaging device and send them to the terminal device.As a result, it is possible to obtain images with little distortion even when photographing the inside of the refrigerator and cutting out the interior area.

[0178] (Other Embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0179] For example, in the above embodiment, the imaging system is realized by multiple devices, but it may also be realized by a single device. For example, the imaging system may be realized as a single device corresponding to a server device or an imaging device. When the imaging system is realized by multiple devices, the components (especially functional components) of the imaging system may be distributed in any way among the multiple devices. For example, some or all of the processing described as being executed by the server device in the above embodiment may also be executed by an information terminal.

[0180] Furthermore, for example, the method of communication between the devices in the above-described embodiment is not particularly limited, and a relay device (not shown) may be involved in the communication between the devices.

[0181] For example, in the above embodiment, a process executed by a specific processing unit may be executed by another processing unit, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0182] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0183] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0184] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0185] For example, the present disclosure may be realized as a method for setting a cutout position executed by a computer, or as a program for causing a computer to execute the setting method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

[0186] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.

[0187] The image correction method, imaging system, imaging device, terminal device, adjustment method, server device, and program disclosed herein are useful as a system that can extract an appropriate area from an image of the interior of a drawer compartment of a refrigerator.

[0188] DESCRIPTION OF SYMBOLS 10 Imaging system 20 Refrigerator 21 Refrigerating compartment (first storage compartment) 22 Ice making compartment (second storage compartment) 23 Fresh freezing compartment (second storage compartment) 24 Freezer compartment (second storage compartment) 25 Vegetable compartment (second storage compartment) 26 Storage compartment (second storage compartment) 27 Main box 30 Imaging device 31 Communication unit 32 Information processing unit 33 Memory unit 34 Imaging unit 35 Processing unit 36 ​​Calculation unit 37 Correction unit 40 Server device 41 Communication unit 42 Information processing unit 43 Memory unit 45 Processing unit 46 Calculation unit 47 Correction unit 50 Information terminal (terminal device) 51 Communication unit 52 Information processing unit 53 Memory unit 54 Display unit 58 Adjustment unit 60 Wide area communication network (network) 201 Opening (opening portion) 205 Recess 301 Wide-angle camera 302 Narrow-angle camera 303 Photography member 304 Main body 305 Restriction member 313 First wireless communication module 314 Metal plate 203A First hinge member 203B Second hinge member 20A Top surface (upper part) 21B Right door (door) 21C Left door (door) 21E Door pocket 21F Shelf 22A Drawer (drawer-shaped) 23A Drawer (drawer-shaped) 24A Drawer (drawer-shaped) 24B Storage case 25A Drawer (drawer-shaped) 25B Storage case 26A Drawer (drawer-shaped) 30A Bottom surface 31A Bottom surface A1 Range A2 Range A3 Range A4 Range MK Marker MK1 Marker MK41 3rd marker MK42 4th marker MK51 1st marker MK52 2nd marker θ2 Shooting angle

Claims

1. A method for correcting an image taken from the upper front of a refrigerator having a storage compartment with a door, the method comprising: a first acquisition step of acquiring a first image taken of the storage compartment with the door open; a processing step of using the first image to generate a second image by cutting out a predetermined area of ​​the storage compartment; a display step of displaying the second image on a terminal device; an adjustment step of adjusting the tilt of a subject appearing in the second image on the terminal device and generating an adjustment value corresponding to the adjusted tilt; a calculation step of calculating a correction value for correcting the newly acquired first image using the adjustment value; a second acquisition step, after the calculation step, of acquiring a new first image taken of the storage compartment with the door open; and a correction step of correcting the new first image using the correction value.

2. The correction method according to claim 1, wherein in the adjustment step, a display is displayed on the screen of the terminal device prompting the user to adjust the tilt of the subject in the second image, and when the user adjusts the tilt on the terminal device, the angle adjusted by the user is generated as the adjustment value.

3. An imaging system capable of photographing a storage compartment having a door from the upper front of a refrigerator, the imaging system comprising: an imaging unit disposed at the upper front of the refrigerator, which photographs the interior of the storage compartment with the door open and generates a first image; a processing unit which processes the first image and generates a second image to be displayed on a terminal device; a display unit which displays the second image; an adjustment unit which has a function of adjusting the tilt of a subject appearing in the second image and generates an adjustment value corresponding to the adjusted tilt; a memory unit which stores the adjustment value; a calculation unit which uses the adjustment value stored in the memory unit to calculate a correction value for correcting a first image newly photographed and generated by the imaging unit; and a correction unit which uses the correction value to correct the new first image newly photographed and generated by the imaging unit.

4. An imaging device capable of photographing a storage compartment having a door from the upper front of a refrigerator, the imaging device comprising: an imaging unit located at the upper front of the refrigerator, which photographs the interior of the storage compartment with the door open and generates a first image; a processing unit which processes the first image and generates a second image to be displayed on a terminal device connectable via a network; a first communication unit which transmits the second image to the terminal device or a server device connectable via a network, and receives an adjustment value generated corresponding to the adjusted tilt after the tilt of a subject appearing in the second image is adjusted on the terminal device; a calculation unit which uses the adjustment value received by the first communication unit to calculate a correction value for correcting the first image newly photographed and generated by the imaging unit; and a correction unit which uses the correction value to correct the new first image newly photographed and generated by the imaging unit.

5. A correction method for correcting an image of a storage compartment captured by an imaging device located at the upper front of a refrigerator having a storage compartment with a door, the correction method comprising: an imaging step of capturing an image of the interior of the storage compartment with the door open to generate a first image; a processing step of processing the first image to generate a second image to be displayed on a terminal device connectable via a network; a transmission step of transmitting the second image to the terminal device or a server device connectable via a network; a reception step of receiving an adjustment value generated in the terminal device after adjusting the tilt of a subject appearing in the second image so as to correspond to the adjusted tilt; a calculation step of calculating a correction value for correcting a newly captured and generated first image using the received adjustment value; and a correction step of correcting the newly captured and generated first image using the correction value.

6. A program for causing a computer to execute the correction method according to claim 5.

7. A terminal device connectable to the imaging device described in claim 4 via a network, comprising: a second communication unit that receives the second image transmitted from the imaging device; a display unit that displays the received second image; and an adjustment unit that has a function of adjusting the tilt of a subject appearing in the second image and generates an adjustment value corresponding to the adjusted tilt.

8. The terminal device according to claim 7, wherein the terminal device is connectable to a server device via a network, and the second communication unit receives the second image transmitted from the imaging device via the server device.

9. The terminal device according to claim 7, wherein the terminal device is connectable to a server device via a network, the server device has a memory unit that stores the adjustment values ​​and has a function of transmitting the adjustment values ​​stored in the memory unit to the imaging device, and the second communication unit transmits the adjustment values ​​generated by the adjustment unit to the server device.

10. The terminal device according to claim 7, further comprising a storage unit that stores the adjustment value, and the second communication unit transmits the adjustment value stored in the storage unit to the imaging device.

11. An adjustment method for adjusting an image in a terminal device connectable to the imaging device described in claim 4 via a network, comprising: a receiving step for receiving the second image generated by the imaging device; a display step for displaying the received second image; an adjustment step for allowing a user to adjust the tilt of a subject appearing in the second image; and a generation step for generating an adjustment value corresponding to the adjusted tilt.

12. A program for causing a computer to execute the adjustment method according to claim 11.

13. A server device in the imaging system described in claim 3, the server device having: a communication unit that receives the first image and transmits the second image to the terminal device; the processing unit; the memory unit; the calculation unit; and the correction unit that generates the corrected first image by correcting the newly generated first image; the processing unit that processes the generated corrected first image to generate the new second image; and the communication unit that transmits the new second image to the terminal device.