Door opening angle calculation method and storage
Patent Information
- Application Number
- JP2022009288
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-01-25
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2042-01-25
Smart Images

Figure 0007578631000001 
Figure 0007578631000002 
Figure 0007578631000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a door opening angle calculation method and the like. [Background technology]
[0002] As a method for calculating the opening angle of a refrigerator door, for example, Patent Document 1 describes a method in which "in a refrigerator equipped with a volume resistor attached to the hinge of the refrigerator compartment...the opening / closing angle of the refrigerator compartment door is detected by the volume resistor." Furthermore, Patent Document 2 describes that "a new image is created using data of a plurality of images captured by the imaging unit from different angles based on the opening and closing angle of the door." [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2005-164196 A [Patent Document 2] Patent No. 6323279 Summary of the Invention [Problem to be solved by the invention]
[0004] The technology described in Patent Document 1 requires a volume resistor to be provided on the hinge of the refrigerator door, which increases manufacturing costs. Also, the technology described in Patent Document 2 takes images of the interior of the refrigerator from various angles to create a three-dimensional composite image, but there is room for simplification of the processing. [Means for solving the problem]
[0005] The door opening angle calculation method according to the present disclosure includes an opening angle calculation step of calculating, when a hinged door provided on a housing is opened, an opening angle of the door based on the number of pixels between a first edge which is an edge of the housing on an image captured by an imaging means, and a second edge which is an edge of the door on the captured image, and an opening angle correction step of correcting the opening angle based on the positional relationship between a predetermined reference line or reference position in the captured image and the first edge. [Brief description of the drawings]
[0006] [Figure 1] FIG. 1 is a front view of a refrigerator according to a first embodiment. [Diagram 2] FIG. 1 is a side view of a refrigerator according to a first embodiment. [Diagram 3] FIG. 1 is a front view of a refrigerator according to a first embodiment with left and right refrigerator compartment doors open. [Figure 4] FIG. 2 is a front view of the refrigerator according to the first embodiment with the doors, shelves, etc. removed. [Diagram 5] FIG. 1 is a plan view of a refrigerator according to a first embodiment with left and right refrigerator compartment doors open. [Figure 6] 1 is a perspective view of a camera unit of a refrigerator according to a first embodiment when viewed from diagonally below. [Figure 7A] FIG. 2 is a front view of a camera unit of the refrigerator according to the first embodiment. [Figure 7B] 7B is a cross-sectional view taken along line II-II of FIG. 7A in the refrigerator according to the first embodiment. [Figure 8] FIG. 1 is a system configuration diagram including a camera unit of a refrigerator according to a first embodiment. [Figure 9] 1 is an example of a result of photographing by a camera unit of a refrigerator according to a first embodiment, in a state where no image processing has been performed. [Figure 10] 1 is an example of a result of photographing by a camera unit of a refrigerator according to the first embodiment, after predetermined image processing has been performed. [Figure 11]4 is a flowchart of a process executed by a control unit of the refrigerator according to the first embodiment. [Figure 12A] FIG. 2 is an explanatory diagram relating to extraction of feature points on a captured image of the refrigerator according to the first embodiment. [Figure 12B] FIG. 2 is an explanatory diagram showing a series of feature points on a captured image of the refrigerator according to the first embodiment. [Figure 12C] 1 is an explanatory diagram showing a trapezoidal portion included in an area on a captured image of a refrigerator according to a first embodiment. FIG. [Figure 13] 4 is an explanatory diagram relating to a trapezoidal portion between a first edge and a second edge in the refrigerator according to the first embodiment. FIG. [Figure 14] FIG. 2 is an explanatory diagram showing the relationship between the area of a trapezoidal portion in the refrigerator according to the first embodiment and the opening angle of the refrigerator compartment door. [Figure 15A] FIG. 2 is an explanatory diagram relating to an image captured when the camera unit is not tilted in the refrigerator according to the first embodiment. [Figure 15B] FIG. 2 is an explanatory diagram relating to an image captured when the camera unit is tilted to the left in the refrigerator according to the first embodiment. [Figure 15C] FIG. 2 is an explanatory diagram relating to an image captured when the camera unit is tilted to the right in the refrigerator according to the first embodiment. [Figure 16] 13 is a flowchart of a process executed by a control unit of a refrigerator according to a second embodiment. [Figure 17] FIG. 11 is an explanatory diagram showing an image captured when the camera unit is not tilted in the refrigerator according to the second embodiment. [Figure 18] FIG. 11 is an explanatory diagram showing an image captured when the camera unit is not tilted in the refrigerator according to the third embodiment. [Figure 19] FIG. 13 is an explanatory diagram showing an image captured when the camera unit is not tilted in the refrigerator according to the fourth embodiment. [Figure 20A] FIG. 13 is an explanatory diagram showing an allowable range of a first edge in a captured image in a refrigerator according to a fifth embodiment. [Figure 20B]FIG. 13 is an explanatory diagram showing an allowable range of a second edge in a captured image in a refrigerator according to a fifth embodiment. [Figure 21A] 13 is a flowchart of a process executed by a control unit of a refrigerator according to a fifth embodiment. [Figure 21B] 13 is a flowchart of a process executed by a control unit of a refrigerator according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0007] First Embodiment FIG. 1 is a front view of a refrigerator 100 according to the first embodiment. Refrigerator 100 (storage compartment) is an appliance for cooling food and the like, and includes a housing 1, doors such as refrigerator compartment doors 211, 212, and a camera unit 3 (photographing means). Housing 1 has multiple storage compartments therein. In the example of Fig. 1, refrigerator 100 has, from the top, a refrigerator compartment 21, an ice-making compartment 22, an upper freezer compartment 23, a vegetable compartment 24, and a lower freezer compartment 25 arranged side by side as storage compartments.
[0008] The housing 1 is configured such that a space between a resin inner box 12 (see FIG. 4) forming a storage compartment and an outer box 11 made of a steel plate is filled with a heat insulating material such as urethane foam (not shown). As shown in FIG. 1, the refrigerator 100 is provided with left and right refrigerator compartment doors 211, 212 as hinged doors forming the refrigerator compartment 21 together with the housing 1. The left refrigerator compartment door 211 (door, left door) is rotatable about the axis of a hinge 211a (see FIG. 5). The same is true for the right refrigerator compartment door 212 (door, right door). In this way, the left and right refrigerator compartment doors 211, 212 are French-style doors. The refrigerator 100 is provided with an ice-making compartment door 221, an upper freezer compartment door 231, a vegetable compartment door 241, and a lower freezer compartment door 251 as drawer-type doors.
[0009] Although not shown, the refrigerator 100 includes a compressor, a radiator (condenser), a capillary tube (throttling mechanism), and a cooler (evaporator). A refrigerant circulates through the compressor, the radiator, the capillary tube, and the cooler in this order, and the air in the storage compartment is cooled by heat exchange with the refrigerant flowing through the cooler. The camera unit 3 shown in Fig. 1 captures images of the storage compartment and refrigerator compartment doors 211 and 212 of the refrigerator 100, and is installed on the top surface of the housing 1. Note that outline arrows 71 and 72 shown in Fig. 1 will be described later.
[0010] FIG. 2 is a side view of the refrigerator 100. 2, the camera unit 3 includes a main body 31 and a support 32. The main body 31 captures images of the storage compartment and refrigerator compartment doors 211, 212 of the refrigerator 100. The support 32 supports the main body 31 and is installed on the top surface of the housing 1.
[0011] A lens 31a (see also FIG. 6) is provided near the front end of the main body 31. The lens 31a is an optical element that refracts and focuses light. For example, a fisheye lens is used as the lens 31a. When the refrigerator compartment doors 211, 212 (see also FIG. 3) are opened, the lens 31a faces downward so that the camera unit 3 can capture images of the refrigerator compartment 21 (see FIG. 3) and the refrigerator compartment doors 211, 212.
[0012] As shown in FIG. 2, lens 31a is located forward of the front end of housing 1 (i.e., opening 10a). More preferably, lens 31a is located further forward of the front surface of refrigerator compartment doors 211, 212 (see FIG. 1) in the closed state. This makes it easier for refrigerator compartment 21 to be included in the field of view of lens 31a when refrigerator compartment doors 211, 212 are opened, for example. It is also possible for camera unit 3 to capture images of vegetable compartment 24, etc. (see FIG. 1) when a drawer-type door such as vegetable compartment door 241 is open. In this way, lens 31a is provided in a position that allows each storage compartment to be viewed from above.
[0013] FIG. 3 is a front view of refrigerator 100 with left and right refrigerator compartment doors 211, 212 open. As shown in Fig. 3, refrigerator compartment 21 is provided with a plurality of shelves 213 that divides refrigerator compartment 21 into predetermined compartments. A plurality of door pockets 211c for storing food and the like are provided on inner panel 211b of left refrigerator compartment door 211 (similar to right refrigerator compartment door 212). When left and right refrigerator compartment doors 211, 212 are opened, foods and the like in refrigerator compartment 21 and door pockets 211c, 212c are viewed from above within the field of view of lens 31a of camera unit 3 (see also Fig. 9).
[0014] In the left refrigerator compartment door 211, a square frame-shaped packing 211d is provided near the edge of the inner plate 211b. The packing 211d is a resin member for improving the airtightness of the refrigerator compartment 21. When the refrigerator compartment door 211 is closed, the packing 211d comes into contact (closely adheres) with the periphery of the opening 10a of the housing 1 (see FIG. 4).
[0015] The side piece 211e shown in FIG. 3 is a plate-like member forming the side surface of the refrigerator door 211, and is elongated and extends vertically. The refrigerator door 211 also includes another side piece 211f (see FIG. 5) provided near the axis of the hinge 211a, in addition to the side piece 211e provided on the opposite side of the axis of the hinge 211a (see FIG. 5). Both of these side pieces 211e and 211f (see FIG. 5) extend vertically and are parallel to the axis of the hinge 211a (see FIG. 5). The side pieces 211e and 211f may be adjacent to the above-mentioned rectangular frame-shaped packing 211d (see FIG. 3), or may partially overlap the packing 211d. The same applies to the right refrigerator door 212.
[0016] Furthermore, when manually photographing the refrigerator compartment 21 using the camera unit 3, a photographing button 5 that is pressed by a user is provided on each of the left and right refrigerator compartment doors 211, 212. In the example of Fig. 3, a photographing button 5 is provided on the lower part of each of the left and right side pieces 211e, 212e.
[0017] FIG. 4 is a front view of the refrigerator 100 with the doors, shelves, etc. removed. As shown in Fig. 4, the front side (front face side) of the housing 1 is provided with an opening 10a corresponding to the refrigerator compartment 21, and openings 10b, 10c, 10d, and 10e corresponding to the other compartments. The opening 10a of the refrigerator compartment 21 is surrounded by a left side panel 1a, a right side panel 1b, a top panel 1c, and a heat-insulating partition wall 1d. The heat-insulating partition wall 1d is a heat-insulating wall that separates the refrigerator compartment 21 from the ice-making compartment 22 and also separates the refrigerator compartment 21 from the upper freezer compartment 23.
[0018] The side plates 1a, 1b and the top plate 1c are thin plate-like members forming a part of the front surface of the housing 1, and are provided near the opening 10a of the inner box 12. The left side plate 1a is provided to the left of the opening 10a of the refrigerator compartment 21, and extends parallel to the axis of the hinge 211a (see FIG. 5) (i.e., in the vertical direction). The right side plate 1b is provided to the right of the opening 10a of the refrigerator compartment 21, and extends parallel to the axis of the hinge 212a (see FIG. 5). These side plates 1a, 1b extend vertically in an elongated manner not only to the opening 10a of the refrigerator compartment 21, but also to the sides of the opening 10b of the ice-making compartment 22, the opening 10d of the vegetable compartment 24, and the opening 10e of the lower freezer compartment 25.
[0019] Top plate 1c extends horizontally in an elongated manner so as to connect the vicinity of the upper end of left side plate 1a with the vicinity of the upper end of right side plate 1b. For example, when left side refrigerator door 211 (see FIG. 3) is closed, square frame-shaped packing 211d (see FIG. 3) of refrigerator door 211 comes into close contact with left side plate 1a, top plate 1c (left half), and the front surface of heat-insulating partition wall 1d.
[0020] FIG. 5 is a plan view of the refrigerator 100 with the left and right refrigerator compartment doors 211, 212 open. As shown in Fig. 5, hinges 211a and 212a are provided on the top surface of housing 1. Hinge 211a on the left side is a member that axially supports refrigerator compartment door 211 so that it can rotate freely (the same is true for hinge 212a on the right side). Camera unit 3 has the function of photographing refrigerator compartment 21 (see Fig. 3) and refrigerator compartment doors 211, 212, and is provided on the outside of housing 1. Camera unit 3 is positioned so that the gap between refrigerator compartment door 211, 212 and housing 1 comes into view when refrigerator compartment door 211, 212 is opened.
[0021] Camera unit 3 is installed near the center in the left-right direction on the upper side of housing 1. In the example of Fig. 5, camera unit 3 is arranged slightly left of the center of housing 1 of refrigerator 100 in the left-right direction. To explain in more detail, camera unit 3 is arranged directly above the joint of refrigerator compartment doors 211, 212 in the closed state (see also Fig. 1). This makes it easier for food etc. in door pockets 211c, 212c to come into the field of view of camera unit 3 when refrigerator compartment doors 211, 212 are opened.
[0022] FIG. 6 is a perspective view of the camera unit 3 as viewed obliquely from below. The main body 31 of the camera unit 3 shown in Fig. 6 includes the lens 31a (see also Fig. 2) described above, as well as a case 31b and a cover 31c. The case 31b of the main body 31 is generally a rectangular parallelepiped that is elongated in the front-rear direction. A circular hole (not shown) is provided on the underside near the front end of the case 31b, and the lens 31a is exposed through this hole.
[0023] In case 31b, a long and narrow hole (not shown) is provided in the left-right direction on the rear side (rear side) of lens 31a, and cover 31c is fitted into this hole. Cover 31c is a light-transmitting resin member that protects camera LED 31d (see FIG. 7B).
[0024] FIG. 7A is a front view of the camera unit 3. FIG. 7A, the main body 31 of the camera unit 3 is placed on the upper side of the support part 32. Moreover, the main body 31 is placed near the center of the support part 32 in the left-right direction.
[0025] FIG. 7B is a cross-sectional view taken along line II-II of FIG. 7A. As shown in FIG. 7B, the camera unit 3 includes the lens 31a (see also FIG. 6), cover 31c (see also FIG. 6), and case 31b, as well as a camera LED 31d and a circuit board 31e.
[0026] The camera LED 31d is a light source that irradiates the refrigerator compartment 21 (see FIG. 3) and the door pockets 211c, 212c (see FIG. 3) with light so that the camera unit 3 can photograph them under appropriate brightness. A light-transmitting cover 31c (see FIG. 6) is provided below the camera LED 31d.
[0027] The circuit board 31e is a printed circuit board on which an image pickup element 31f (see FIG. 8) and a camera / communication control SoC 31g (System-on-a-Chip: see FIG. 8) are mounted. By housing the lens 31a, the camera LED 31d, and the circuit board 31e in one case 31b in this way, the camera unit 3 can be made smaller and the length of the wiring can be shortened, compared to when these are provided separately.
[0028] FIG. 8 is a diagram showing the system configuration including the camera unit 3 of the refrigerator 100. As shown in FIG. 8, in addition to the camera unit 3, the refrigerator 100 is equipped with a door sensor 4, a photographing button 5, and a control unit 6. The door sensor 4 outputs a predetermined signal indicating the open / closed state of the refrigerator compartment doors 211, 212 to the control unit 6. Although one door sensor 4 is shown in Fig. 8, in reality, a door sensor 4 corresponding to the left refrigerator compartment door 211 (see Fig. 1) and another door sensor 4 corresponding to the right refrigerator compartment door 212 (see Fig. 1) are provided. For example, when the left refrigerator compartment door 211 is opened, a predetermined open signal is output from the door sensor 4 corresponding to this refrigerator compartment door 211 to the control unit 6.
[0029] The shooting button 5 (see also FIG. 3) is a button that is pressed by the user when manually taking a picture using the camera unit 3, as described above. The control unit 6 is, for example, a microcomputer, and although not shown, is configured to include electronic circuits such as a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and various interfaces. The program stored in the ROM is read and expanded in the RAM, and the CPU executes various processes. The processes performed by the control unit 6 will be described later.
[0030] The camera unit 3 includes a lens 31a (see also FIG. 7B) and a camera LED 31d (see also FIG. 7B), as well as an imaging element 31f, a camera / communication control SoC 31g, and a wireless LAN unit 31h. The imaging element 31f is an element that photoelectrically converts light incident through the lens 31a and generates captured image data. For example, a CCD sensor (Charge Coupled Device) or a CMOS sensor (Complementary Metal Oxide Semiconductor) is used as the imaging element 31f.
[0031] The camera / communication control SoC31g is an integrated circuit designed to integrate circuits having microcomputer functions and other applied functions on a single chip and function in cooperation with each other. The camera / communication control SoC31g communicates with the control unit 6 in a predetermined manner and outputs an image capture command to the image sensor 31f. As a result, captured image data is input from the image sensor 31f to the camera / communication control SoC31g. Note that the camera / communication control SoC31g is an example of an integrated circuit, and other types of integrated circuits may be used.
[0032] The wireless LAN unit 31h transmits the captured image data output from the camera / communication control SoC 31g to the router 51. The router 51 is a communication relay device, and transmits the captured image data received from the wireless LAN unit 31h to the server 53 via the network 52. The server 53 performs a predetermined process on the captured image data and stores it in association with the identification information of the refrigerator 100. Furthermore, when the server 53 receives a request signal for an image of the interior of the refrigerator 100 from the user's mobile terminal 54, it transmits the captured image data to the mobile terminal 54. As such a mobile terminal 54, a mobile phone, a smartphone, a tablet, a wearable terminal, or the like may be used.
[0033] FIG. 9 shows an example of the result of photographing by the camera unit without image processing. In Fig. 9, the upper side of the page is the front and the lower side is the rear, so that the left and right are reversed compared to Fig. 3 and the like. In the example of Fig. 9, both the left and right refrigerator compartment doors 211, 212 are open, and in addition to the refrigerator compartment 21, the left and right door pockets 211c, 212c are also photographed. In particular, a fisheye lens is used as the lens 31a of the camera unit 3 (see Fig. 6), so that images can be captured with a wide range of angle of view. However, in a state where image processing is not performed, as shown in Fig. 9, an actually straight ridgeline appears curved, and even if the ridgeline has the same length, the number of pixels in the captured image decreases as it gets farther from the camera unit 3.
[0034] FIG. 10 shows an example of the result of photographing by the camera unit 3 after predetermined image processing has been performed. For example, predetermined image processing is performed by server 53 (see FIG. 8) based on the photographed result of FIG. 9, converting the images into images as if they were seen from the front of refrigerator compartment 21 and left and right refrigerator compartment doors 211, 212. By displaying such images on mobile terminal 54 (see FIG. 8), the user can see at a glance what foods are stored in refrigerator compartment 21 etc.
[0035] <Processing of control unit> For example, when at least one of the refrigerator doors 211, 212 (see FIG. 1) is opened and an open signal is input from the door sensor 4 (see FIG. 8), the control unit 6 (see FIG. 8) turns on the camera LED 31d and repeats the predetermined photographing by the camera unit 3. Each time such photographing is performed, photographed image data is generated, and it is desirable that the photographed image data transmitted from the camera unit 3 to the server 53 (see FIG. 8) is photographed when at least one of the refrigerator doors 211, 212 is fully opened.
[0036] Therefore, in the first embodiment, the control unit 6 uses the captured image data to detect the open angle of the refrigerator compartment doors 211, 212. The "open angle" refers to the angle at which the refrigerator compartment doors 211, 212 are open. When predetermined captured image data is input from the camera unit 3, the control unit 6 converts the captured image data (color image) into grayscale, for example, and then performs a series of processes shown in Fig. 11.
[0037] FIG. 11 is a flowchart of the process executed by the control unit of the refrigerator (see FIG. 8 as necessary). 11 mainly shows the process related to the calculation of the open angle of the refrigerator compartment doors 211, 212. In step S101, the control unit 6 performs a binarization process. For example, the control unit 6 performs the binarization process by setting the value of a pixel whose luminance is equal to or greater than a predetermined value to '1' (white) and the value of a pixel whose luminance is less than the predetermined value to '0' (black).
[0038] Next, in step S102, the control unit 6 extracts edges on the captured image. The control unit 6 extracts edges on the captured image by using, for example, a Sobel filter. Note that the method for extracting edges is not limited to the Sobel filter, and other well-known methods such as a Laplacian filter or a Canny method may be used. In step S103, the control unit 6 extracts feature points on the captured image.
[0039] FIG. 12A is an explanatory diagram regarding extraction of feature points on a captured image. In addition, Fig. 12A shows a binarized image taken with the left and right refrigerator compartment doors 211, 212 open. In Fig. 12A, the upper side of the page is the front and the lower side is the rear, so the left and right are reversed compared to Fig. 3 and the like (the same applies to Figs. 12B and 12C).
[0040] In Fig. 12A, the captured image includes inner box 12 (see also Fig. 4) and gasket 211d (see also Fig. 3) of left refrigerator door 211, as well as gasket 212d (see also Fig. 3) of right refrigerator door 212. Also, although no symbols are shown in Fig. 12A, the captured image includes left and right horizontal panels 1a, 1b (see Fig. 4), as well as left and right side pieces 211f, 212f (see Fig. 5). These symbols are not shown in Fig. 12A because the pixel values have become '0' (black) in the binarization process.
[0041] The rectangular area 7a shown in FIG. 12A is an area used to calculate the opening angle of the right-side refrigerator door 212, and its position and size on the captured image are set in advance. Specifically, this area 7a is set in advance so that the boundary between the inner box 12 (see FIG. 4) and the right-side horizontal panel 1b (see FIG. 4) passes through the area 7a on the captured image when the right-side refrigerator door 212 is open. Also, this area 7a is set in advance so that the boundary between the packing 212d (see FIG. 3) and the side piece 212f (see FIG. 5) of the right-side refrigerator door 212 passes through the area 7a on the captured image when the right-side refrigerator door 212 is open. The same is true for the other rectangular area 7b.
[0042] Feature point 8m shown in Fig. 12A is a candidate for a point included in the boundary between packing 212d (see Fig. 3) and side piece 212f (see Fig. 5) of right-side refrigerator door 212 in the captured image. For example, as indicated by the downward arrow in Fig. 12A, control unit 6 (see Fig. 8) reads pixel values of a predetermined vertical row included in area 7a of the captured image, sequentially from the top. In this case, the point where the pixel value changes from '1' (white) to '0' (black) is feature point 8m that is likely to be included in the boundary between packing 212d (see Fig. 3) and side piece 212f (see Fig. 5).
[0043] The packing 212d (see FIG. 3) and the side piece 212f (see FIG. 5) may each be painted in a predetermined color so that a luminance difference occurs between the packing 212d (see FIG. 3) and the side piece 212f (see FIG. 5) on the captured image. This makes it easier to extract the boundary between the packing 212d and the side piece 212f as a second edge 92a (see FIG. 13) described later.
[0044] Feature point 8n shown in Fig. 12A is a candidate for a point included in the boundary between inner box 12 (see Fig. 4) and side panel 1b (see Fig. 4) of housing 1 in the captured image. For example, as indicated by the downward arrow in Fig. 12A, control unit 6 (see Fig. 8) reads the pixel values of a predetermined vertical row included in area 7a of the captured image, sequentially from the top. In this case, the point where the pixel value changes from '0' (black) to '1' (white) is feature point 8n that is likely to be included in the boundary between inner box 12 (see Fig. 4) and side panel 1b (see Fig. 4).
[0045] In addition, the inner box 12 and the side board 1b may each be painted in a predetermined color so that a luminance difference occurs between the inner box 12 and the side board 1b (see FIG. 4) in the captured image. This makes it easier to extract the boundary between the inner box 12 and the side board 1b as a first edge 91a (see FIG. 13) described later.
[0046] For example, the control unit 6 extracts feature points for a given vertical column in the region 7a, and then extracts feature points for an adjacent vertical column, shifting the columns from which feature points are to be extracted horizontally one by one (see the rightward arrow in FIG. 12A). Then, the control unit 6 extracts feature points for all vertical columns included in the region 7a (S103 in FIG. 11). The same process is performed in the other region 7b.
[0047] Next, in step S104 of Fig. 11, the control unit 6 calculates approximation lines. That is, the control unit 6 specifies two approximation lines (straight lines) indicating edges on the captured image based on the multiple feature points extracted in step S103. Note that in calculating these approximation lines, a well-known method such as the least squares method may be used.
[0048] FIG. 12B is an explanatory diagram showing a series of feature points on a captured image. In the example of Fig. 12B, in the rectangular region 7a, a plurality of feature points 8n are distributed so as to be continuous at the boundary between the inner box 12 and the side panel 1b (see Fig. 4). Based on these feature points 8n, a first edge 91a (see Fig. 13), which is an edge of the housing 1 (see Fig. 4) on the captured image, is calculated as a predetermined approximation line. This first edge 91a is an edge on the captured image that corresponds to the boundary between the inner box 12 and the side panel 1b (see Fig. 4).
[0049] In the example of Fig. 12B, in the rectangular area 7a, a plurality of characteristic points 8m are distributed in a continuous manner at the boundary between packing 212d and side piece 212f (see Fig. 5) of refrigerator door 212. Based on these characteristic points 8m, a second edge 92a (see Fig. 13), which is the edge of refrigerator door 212 (door) on the captured image, is calculated as a predetermined approximation line. This second edge 92a is an edge on the captured image that corresponds to the boundary between packing 212d and side piece 212f (see Fig. 5) of refrigerator door 212.
[0050] In real space, the first edge 91a (see FIG. 13) and the second edge 92a (see FIG. 13) are parallel to the axis of the hinge 212a (see FIG. 5) of the refrigerator compartment door 212. For example, when the right refrigerator compartment door 212 is opened, most of the light irradiated near the right edge of the inner box 12 is often light from the interior light (not shown) of the refrigerator 100 or the camera LED 31d (see FIG. 7B). Therefore, light, objects, etc. around the refrigerator 100 hardly adversely affect the extraction of the first edge 91a or the second edge 92a. As a result, the control unit 6 can extract the first edge 91a or the second edge 92a with high accuracy. Next, in step S105 of FIG. 11, the control unit 6 calculates the area of the trapezoidal portion.
[0051] FIG. 12C is an explanatory diagram showing a trapezoidal portion 9a included in an area 7a on a captured image. The trapezoidal portion 9a shown in Figure 12C is the portion included in region 7a between a line segment L1 (see Figure 13) included in region 7a at a first edge 91a (see Figure 13) and another line segment L2 (see Figure 13) included in region 7a at a second edge 92a (see Figure 13).
[0052] Since the first edge 91a (see FIG. 13) is the boundary between the inner box 12 and the horizontal plate 1b (see FIG. 4), the position of the first edge 91a on the photographed image hardly changes even when the refrigerator door 212 is opened. In contrast, since the second edge 92a (see FIG. 13) is the boundary between the packing 212d of the refrigerator door 212 and the side piece 212f (see FIG. 5), the second edge 92a approaches the upper boundary line of the area 7a as the opening angle of the refrigerator door 212 increases. As a result, the area (number of pixels) of the trapezoidal portion 9a increases as the opening angle of the refrigerator door 212 increases. Therefore, in the first embodiment, the control unit 6 calculates the opening angle of the refrigerator door 212 (door) based on the area (number of pixels) of the trapezoidal portion 9a on the photographed image.
[0053] FIG. 13 is an explanatory diagram of the trapezoidal portion 9a between the first edge 91a and the second edge 92a. In addition, Fig. 13 shows the area 7a of Fig. 12C. As shown in Fig. 13, the linear first edge 91a and the linear second edge 92a intersect with a pair of opposite sides (boundaries) in the vertical direction of the rectangular area 7a. In other words, the rectangular area 7a is set in advance so that the first edge 91a and the second edge 92a pass through on the captured image.
[0054] Points 61 and 62 shown in Fig. 13 are intersections between a pair of opposite sides of the rectangular region 7a and the first edge 91a. Points 63 and 64 are intersections between a pair of opposite sides of the rectangular region 7a and the second edge 92a. As shown in Fig. 13, the length (number of pixels) of a line segment L4 connecting points 62 and 64 is shorter than the length (number of pixels) of a line segment L3 connecting points 61 and 63. This is because line segment L4 is lower in height in real space than line segment L3 and has fewer pixels in the captured image.
[0055] In the above-mentioned step S105 (see FIG. 11), the control unit 6 calculates the area of the trapezoidal portion 9a between the first edge 91a and the second edge 92a and included in the region 7a. Next, in step S106 (see FIG. 11), the control unit 6 calculates the opening angle of the refrigerator compartment door 212. That is, the control unit 6 calculates the opening angle corresponding to the area of the trapezoidal portion 9a calculated in step S105 (opening angle calculation step).
[0056] FIG. 14 is an explanatory diagram showing the relationship between the area of the trapezoidal portion and the opening angle of the refrigerator compartment door. The horizontal axis in Fig. 14 represents the area (number of pixels) of trapezoidal portion 9a (see Fig. 12C and Fig. 13) included in region 7a. The vertical axis in Fig. 14 represents the open angle of refrigerator compartment door 212. Curve C1 in Fig. 14 represents data for deriving the open angle of refrigerator compartment door 212 from the area of trapezoidal portion 9a, and is set in advance. For example, a predetermined data table or a predetermined monotonically increasing function may be used as the data for curve C1.
[0057] The control unit 6 calculates the opening angle θ1 of the refrigerator compartment door 212 based on, for example, the area S1 of the trapezoidal portion 9a (see FIG. 13) and the data of the curve C1 shown in FIG. 14 (S106: see FIG. 11). In this way, when the hinged refrigerator compartment door 212 (door) provided on the housing 1 (see FIG. 3) is opened, the control unit 6 calculates the opening angle of the refrigerator compartment door 212 based on the number of pixels between the first edge 91a (see FIG. 13), which is the edge of the housing 1 on the image captured by the camera unit 3 (imaging means: see FIG. 8), and the second edge 92a (see FIG. 13), which is the edge of the refrigerator compartment door 212 on the captured image (opening angle calculation step). The control unit 6 also calculates the opening angle of the left refrigerator compartment door 211 (see FIG. 3) in the same manner.
[0058] If camera unit 3 (see FIG. 1) is not tilted relative to housing 1 (see FIG. 1) of refrigerator 100, the error in the calculated value of the opening angle of refrigerator compartment door 212 is small enough to be negligible. However, in some cases, the heat insulating material (urethane foam or the like: not shown) filled between outer box 11 and inner box 12 may be deformed, or an assembly error may occur during the manufacture of refrigerator 100, causing a part of the top surface of outer box 11 to be slightly tilted relative to the horizontal plane.
[0059] Specifically, the left side of the camera unit 3 may be slightly lower than the right side (white arrow 71 in FIG. 1), and the right side of the camera unit 3 may be slightly lower than the left side (white arrow 72 in FIG. 1). When the camera unit 3 is tilted in this way, the area of the trapezoidal portion 9a included in the region 7a in FIG. 13 becomes larger or smaller than the area corresponding to the actual opening angle of the refrigerator compartment door 211, etc., causing an error in the calculated value of the opening angle. Therefore, in the first embodiment, the control unit 6 corrects the opening angle of the refrigerator compartment doors 211, 212 based on the positional relationship between a predetermined reference line 73 (see FIG. 15A) in the captured image and the first edges 91a, 91b (see FIG. 15A).
[0060] FIG. 15A is an explanatory diagram relating to an image captured when the camera unit is not tilted. In addition, in FIG. 15A, the upper side of the paper is the front and the lower side of the paper is the rear, so that the left and right are reversed compared to FIG. 3 etc. (The same applies to FIGS. 15B and 15C). A rectangular area 74 shown in Fig. 15A indicates an area of a captured image based on electrical signals from a large number of image pickup elements 31f (see Fig. 8) arranged vertically and horizontally in a matrix. In the rectangular area 74, the outside of a circular area 75 where light incident through the lens 31a (see Fig. 8) is reflected is actually a black image (see, for example, Fig. 12A).
[0061] One corner of the rectangular area 74 (the upper left corner of the paper in the example of FIG. 15A) is set as the origin, the horizontal direction is the x-axis, and the vertical direction is the y-axis. Each pixel in the captured image is identified based on the coordinate values of the x-axis and y-axis. The reference line 73 shown in FIG. 15A is a line that serves as a reference when the control unit 6 determines whether the camera unit 3 (see FIG. 1) is tilted or not, and is set in advance. In the example of FIG. 15A, the reference line 73 in the y-axis direction is set in advance so as to divide the rectangular area 74 equally into left and right at the center in the x-axis direction. Incidentally, even if the camera unit 3 (see FIG. 1) is tilted, the positional relationship of the reference line 73 with respect to the rectangular area 74 is kept constant.
[0062] After calculating the open angle of refrigerator compartment door 212 in step S106 in Fig. 11, control unit 6 identifies end points of first edges 91a, 91b in step S107. That is, control unit 6 identifies end point 81a on the upper side (lower left side of the paper in Fig. 15A) of first edge 91a, which is the boundary between inner box 12 and right-side horizontal panel 1b (see Fig. 4). Similarly, control unit 6 identifies end point 81b on the upper side (lower right side of the paper in Fig. 15A) of another first edge 91b, which is the boundary between inner box 12 and left-side horizontal panel 1a (see Fig. 4).
[0063] Incidentally, the second edge 92a (the edge of the refrigerator compartment door 212) moves on the captured image as the refrigerator compartment door 212 is opened or closed, but the first edge 91a (the edge of the housing 1) hardly moves. Therefore, in the first embodiment, the first edge 91a is used to detect the inclination of the camera unit 3 (see FIG. 1).
[0064] Note that the control unit 6 may perform the process of step S107 (see FIG. 11) when both the left and right refrigerator compartment doors 211, 212 are open. Alternatively, for example, when one of the left and right refrigerator compartment doors 211, 212 is open and the other is closed, the control unit 6 may perform the process of step S107 for the closed refrigerator compartment door using the first edge of the most recently captured image acquired in the past.
[0065] Next, in step S108 in Fig. 11, the control unit 6 identifies the midpoint 82 between the end points 81a and 81b. That is, the control unit 6 identifies the position (coordinate value) of the midpoint 82 of the line segment connecting the right end point 81a and the left end point 81b shown in Fig. 15A. When the camera unit 3 (see Fig. 1) is not tilted, the position of the midpoint 82 in the x-axis direction substantially coincides with the position of the reference line 73 in the x-axis direction, as shown in Fig. 15A.
[0066] In step S109 in FIG. 11, the control unit 6 judges whether or not the midpoint 82 is deviated from the reference line 73. In the example in FIG. 15A, as described above, the camera unit 3 (see FIG. 1) is not tilted, so the position of the midpoint 82 in the x-axis direction substantially coincides with the position of the reference line 73 in the x-axis direction. In this way, if the midpoint 82 is not deviated from the reference line 73 (S109: No), the control unit 6 ends the series of processes shown in FIG. 11 (END). In this case, the opening angle calculated in step S106 is used as the calculated value of the opening angle of the refrigerator compartment doors 211, 212. The series of processes shown in FIG. 11 are performed every time new captured image data is input.
[0067] FIG. 15B is an explanatory diagram relating to an image captured when the camera unit is tilted to the left. As shown by the white arrow 71 in FIG. 1, when the camera unit 3 is tilted to the left (the right side of the paper in FIG. 15B), the captured range shifts to the right (the left side of the paper in FIG. 15B) compared to when the camera unit 3 is not tilted (see FIG. 15A). Meanwhile, the positional relationship of the rectangular areas 7a and 7b with respect to the area 74 of the captured image is kept constant. As a result, the rectangular area 7b on the right side of the paper of the captured image shows a part that is higher in height (a part closer to the camera unit 3) than in FIG. 15A, so that the area of the trapezoidal portion 9b is larger than the original value. Meanwhile, the rectangular area 7a on the left side of the paper of the captured image shows a part that is lower in height (a part farther from the camera unit 3) than in FIG. 15A, so that the area of the trapezoidal portion 9a is smaller than the original value.
[0068] Furthermore, when the camera unit 3 (see FIG. 1) is tilted to the left (the right side in the plane of FIG. 15B), the positions of the upper end points 81a and 81b of the first edges 91a and 91b in the x-axis direction are shifted to the right (the left side in the plane of FIG. 15B). As a result, the position of the midpoint 82 of the end points 81a and 81b is shifted to the right of the reference line 73. In such a case, in step S109 of FIG. 11, the control unit 6 determines that the midpoint 82 is shifted from the reference line 73 (S109: Yes).
[0069] Then, in step S110, the control unit 6 corrects the opening angle of the refrigerator compartment doors 211, 212 (opening angle correction step). That is, the control unit 6 corrects the opening angle of the refrigerator compartment doors 211, 212 based on the positional relationship between a predetermined reference line 73 in the captured image and the first edges 91a, 91b. As a specific example, assume that the calculated value of the opening angle of the right refrigerator compartment door 212 is 100°, and the midpoint 82 is shifted from the reference line 73 by −5 pixels in the x-axis direction. In such a case, the control unit 6 corrects the opening angle of the refrigerator compartment door 212 based on the calculated value of the opening angle of the refrigerator compartment door 212 (e.g., 100°) and the number of pixels indicating the shift of the midpoint 82 from the reference line 73 (e.g., −5 pixels). The control unit 6 also corrects the opening angle of the left refrigerator compartment door 211 in the same manner.
[0070] It is assumed that correction values for the opening angles of the left and right refrigerator compartment doors 211, 212 are stored in advance in association with the calculated opening angle value and the number of pixels indicating the degree of deviation of the midpoint 82 from the reference line 73. In this way, the control unit 6 corrects the opening angle based on the number of pixels in the x-axis direction (horizontal direction) between the reference line 73 and the midpoint 82 of the line segment connecting the end points 81a, 81b of the first edges 91a, 91b on the right and left sides of the housing 1 (see FIG. 1). Specifically, when midpoint 82 is located on one side of reference line 73 (the left side of the paper in the example of FIG. 15B), control unit 6 corrects the opening angle of the right-side refrigerator compartment door 212 (right door) and the left-side refrigerator compartment door 211 (left door) corresponding to the one side (for example, the left side of the paper in FIG. 15B) to be larger than before correction, and corrects the opening angle of the one corresponding to the other side (for example, the right side of the paper in FIG. 15B) to be smaller than before correction. In this way, by control unit 6 correcting the opening angle of refrigerator compartment doors 211, 212, the opening angle of refrigerator compartment doors 211, 212 can be calculated with high accuracy even when camera unit 3 (see FIG. 1) is tilted.
[0071] FIG. 15C is an explanatory diagram relating to an image captured when the camera unit is tilted to the right. As shown by the hollow arrow 72 in FIG. 1, when the camera unit 3 is tilted to the right (left side of the paper in FIG. 15C), the photographed range shifts to the left (right side of the paper in FIG. 15C) compared to when the camera unit 3 is not tilted (see FIG. 15A). Therefore, the positions of the upper end points 81a and 81b of the first edge 91a in the x-axis direction shift to the left (right side of the paper in FIG. 15C). As a result, the position of the midpoint 82 between the end points 81a and 81b shifts to the left of the reference line 73. In such a case, in step S109 in FIG. 11, the control unit 6 determines that the midpoint 82 is deviated from the reference line 73 (S109: Yes). Then, in step S110, the control unit 6 corrects the opening angle of the refrigerator compartment doors 211 and 212. Specifically, control unit 6 makes the opening angle of left refrigerator door 211 larger than before the correction, and makes the opening angle of right refrigerator door 212 smaller than before the correction.
[0072] According to the first embodiment, the control unit 6 calculates the opening angle of the refrigerator compartment doors 211, 212 based on the number of pixels between the first edge 91a and the second edge 92a in the captured image (the area of the trapezoidal portion 9a: see FIG. 13). This makes it possible to easily and accurately calculate the opening angle of the refrigerator compartment doors 211, 212. Also, since there is no particular need to provide an additional member dedicated to calculating the opening angle of the refrigerator compartment doors 211, 212, the manufacturing cost of the refrigerator 100 can be reduced.
[0073] Furthermore, according to the first embodiment, even if the camera unit 3 (see FIG. 1) is tilted, the control unit 6 can calculate the open angle with high accuracy by correcting the calculated value of the open angle of the refrigerator compartment doors 211, 212. This makes it possible to properly display an image of the refrigerator compartment doors 211, 212 in an open state on the user's mobile terminal 54 (see FIG. 8).
[0074] Second Embodiment The second embodiment differs from the first embodiment in that the control unit 6 corrects the open angle of the refrigerator compartment door 212 based on the length of a line segment 83a (see FIG. 17) from an upper end point 81a (see FIG. 17) of a first edge 91a (see FIG. 17) to the boundary line of the area 7a (see FIG. 17). Note that other points (such as the configuration of the refrigerator 100) are similar to those of the first embodiment. Therefore, only the parts that differ from the first embodiment will be described, and a description of the overlapping parts will be omitted.
[0075] FIG. 16 is a flowchart of the process executed by the control unit of the refrigerator according to the second embodiment (see FIG. 8 as appropriate). Note that the processing in steps S101 to S107 in FIG. 16 is similar to the processing in steps S101 to S107 described in the first embodiment (see FIG. 11), and therefore a description thereof will be omitted. After identifying the end points 81a and 81b of the first edges 91a and 91b (see FIG. 17) in step S107, the process of the control unit 6 proceeds to step S121. In step S121, for example, for the first edge 91a (see FIG. 17) on the right side, the control unit 6 calculates the length of a line segment 83a (see FIG. 17) from its end point 81a (see FIG. 17) to the boundary line of the area 7a (see FIG. 17). Similarly, the control unit 6 calculates the length of a line segment 83b (see FIG. 17) included in the first edge 91b (see FIG. 17) on the left side.
[0076] FIG. 17 is an explanatory diagram showing an image captured when the camera unit is not tilted in the refrigerator according to the second embodiment. In FIG. 17, a line segment 83a extending from an end point 81a of the first edge 91a on the right side to the boundary line of the rectangular area 7a is indicated by a thick solid line (the same is true for a line segment 83b on the left side). After calculating the lengths of the line segments 83a and 83b in step S121 of Fig. 16, the control unit 6 determines in step S122 whether or not the lengths of the line segments 83a and 83b are different from a default value. Here, the "default value" of the length of the right line segment 83a is the value (number of pixels) of the length of the line segment 83a when the camera unit 3 (see Fig. 1) is not tilted, and is set in advance (similarly for the left line segment 83b). Note that the default values of the lengths of the right line segment 83a and the left line segment 83b may be set to the same value or different values.
[0077] In step S122, if the lengths of the line segments 83a and 83b are equal to the default values (S122: No), the control unit 6 ends the series of processes (END). In this case, the opening angle calculated in step S106 is used as the opening angle of the refrigerator compartment doors 211 and 212. In addition, in step S122, if the length of at least one of the line segments 83a and 83b is different from the default value (S122: Yes), the control unit 6 proceeds to the process in step S123.
[0078] In step S123, the control unit 6 corrects the opening angle of the refrigerator compartment doors 211, 212 (opening angle correction step). That is, the control unit 6 corrects the opening angle of the refrigerator compartment door 212 (the same applies to the left refrigerator compartment door 211) based on the length of a line segment 83a from an end point 81a of the first edge 91a (see FIG. 17) to the boundary line of the area 7a.
[0079] For example, if camera unit 3 is tilted to the right (left side on the paper in FIG. 17) (see hollow arrow 72 in FIG. 1), line segment 83a on the right side of FIG. 17 will be shorter than its original length, and line segment 83b on the left side will be longer than its original length. In other words, on the right side of refrigerator 100, the value obtained by subtracting the preset value from the actual length of line segment 83a will be negative, and on the left side of refrigerator 100, the value obtained by subtracting the preset value from the actual length of line segment 83b will be positive. In such a case, control unit 6 corrects the opening angle of right-side refrigerator compartment door 212 to be smaller than before the correction, and corrects the opening angle of left-side refrigerator compartment door 211 to be larger than before the correction.
[0080] That is, in the opening angle correction step (S123), the control unit 6 calculates the length of a line segment 83a from an end point 81a on the side (upper side) closer to the camera unit 3 (photographing means: see FIG. 1) on the first edge 91a to the boundary line of the area 7a, and if the value obtained by subtracting a preset value (predetermined value) from the length of the line segment 83a is negative, the control unit 6 makes the opening angle of the refrigerator compartment door 212 smaller than that before the correction. Also, if the value obtained by subtracting a preset value (predetermined value) from the length of the line segment 83a is positive, the control unit 6 makes the opening angle of the refrigerator compartment door 212 larger than that before the correction. Note that the corrected opening angle is stored in advance in association with the calculated value of the opening angle of the refrigerator compartment door 212 and the difference value regarding the line segment 83a. The same can be said about the correction of the opening angle of the left refrigerator compartment door 211.
[0081] According to the second embodiment, the control unit 6 corrects the opening angle of the refrigerator compartment door 212 based on the length of the line segment 83a from the end point 81a of the first edge 91a to the boundary line of the rectangular area 7a. Since the light from the interior light is more likely to be irradiated on the upper side of the first edge 91a than on the lower side on the captured image, the end point 81a can be extracted with high accuracy. Furthermore, since there is no particular need to use the midpoint between the left and right end points 81a, 81b when correcting the opening angle of the refrigerator compartment doors 211, 212 as in the first embodiment, the second embodiment can also be applied to a refrigerator equipped with a single-wing refrigerator compartment door, for example.
[0082] Third Embodiment The third embodiment differs from the first embodiment in that the control unit 6 corrects the open angle of the refrigerator compartment doors 211, 212 based on a midpoint 85 (see FIG. 18) between lower end points 84a, 84b (see FIG. 18) of first edges 91a, 91b (see FIG. 18). Other points (such as the configuration of the refrigerator 100) are similar to those of the first embodiment. Therefore, only the parts that differ from the first embodiment will be described, and a description of the overlapping parts will be omitted.
[0083] FIG. 18 is an explanatory diagram showing an image captured when the camera unit is not tilted in the refrigerator according to the third embodiment. The processing by the control unit 6 (see FIG. 8) is similar to that in the first embodiment except that the lower end points 84a, 84b of the first edges 91a, 91b are used, and will therefore be described with reference to FIG. The control unit 6 performs the same processes as steps S101 to S106 in Fig. 11 to calculate the open angles of the left and right refrigerator compartment doors 211, 212. Then, the control unit 6 specifies the lower end points 84a, 84b of the first edges 91a, 91b. That is, the control unit 6 specifies the end point 84a of the lower side (upper right side in the paper of Fig. 18) of the first edge 91a, which is the boundary between the inner box 12 (see Fig. 4) and the right side panel 1b (see Fig. 4). The end point 84b of the left first edge 91b is specified in a similar manner.
[0084] When extracting these end points 84a, 84b, the control unit 6 may extract a third edge 87 indicating the edge of the bottom 12a of the inner box 12 based on the processing in the rectangular region 7c. The control unit 6 may then specify the intersection point between the third edge 87 and the first edge 91a on the right side as end point 84a, and specify the intersection point between the third edge 87 and the first edge 91b on the left side as another end point 84b. This makes it possible to specify the lower end points 84a, 84b with high accuracy even when not much light is irradiated near the bottom 12a of the inner box 12.
[0085] After identifying the lower end points 84a, 84b of the first edges 91a, 91b in this manner, the control unit 6 identifies a midpoint 85 between the end points 84a, 84b in step S108 of Fig. 11. When the camera unit 3 (see Fig. 1) is not tilted, the position of the midpoint 85 in the x-axis direction substantially coincides with the position of the reference line 73 in the x-axis direction, as shown in Fig. 18.
[0086] On the other hand, when the position of the midpoint 85 in the x-axis direction is deviated from the reference line 73, the control unit 6 corrects the opening angle of the refrigerator compartment door 212 (the same applies to the left refrigerator compartment door 211) based on the opening angle of the refrigerator compartment door 212 and the number of pixels indicating the deviation of the midpoint 85 from the reference line 73. Note that the method of correcting the opening angle is the same as in the first embodiment, and therefore will not be described.
[0087] According to this embodiment, even if, for example, the shadow of a user appears in the captured image and the control unit 6 has difficulty identifying the upper end points of the first edges 91a, 91b, the opening angle of the refrigerator compartment doors 211, 212 can be appropriately corrected based on the lower end point 84a.
[0088] Fourth Embodiment The fourth embodiment differs from the second embodiment in that the control unit 6 corrects the open angle of the refrigerator compartment door 212 based on the length of a line segment 86a (see FIG. 19) from a lower end point 84a (see FIG. 19) to the boundary line of the area 7a (see FIG. 19) in the first edge 91a (see FIG. 19). Note that the other points are similar to those of the second embodiment. Therefore, only the parts that differ from the second embodiment will be described, and a description of the overlapping parts will be omitted.
[0089] FIG. 19 is an explanatory diagram showing an image captured when the camera unit is not tilted in the refrigerator according to the fourth embodiment. The processing by the control unit 6 (see FIG. 8) is similar to that in the second embodiment except that the lower end points 84a, 84b of the first edges 91a, 91b are used, and will therefore be described with reference to FIG.
[0090] The control unit 6 performs the same processes as steps S101 to S106 in Fig. 16 to calculate the open angle of the refrigerator compartment doors 211, 212. Next, in step S107, the control unit 6 identifies the lower end point 84a (see Fig. 19) of the first edge 91a (see Fig. 19). Next, in step S121, the control unit 6 calculates the length of a line segment 86a (see FIG. 19) from a lower end point 84a (see FIG. 19) of the first edge 91a to the boundary line of the area 7a.
[0091] In step S122, the control unit 6 determines whether the length of the line segment 86a is different from the default value. If the length of the line segment 86a is equal to the default value (S122: No), the control unit 6 ends the process (END). In this case, the camera unit 3 (see FIG. 1) is not tilted, and there is no particular need to correct the opening angle of the refrigerator compartment door 212.
[0092] Furthermore, in step S122, if the length of line segment 86a is different from the default value (S122: Yes), in step S123, control unit 6 corrects the open angle of refrigerator compartment door 212 (open angle correction step). For example, if camera unit 3 is tilted to the right (left side of the paper in FIG. 19) (see outline arrow 72 in FIG. 1), line segment 86a on the right side will be longer than the original length, and line segment 86b on the left side will be shorter than the original length. In such a case, control unit 6 corrects the open angle of right refrigerator compartment door 212 to be smaller than before the correction, and the open angle of left refrigerator compartment door 211 to be larger than before the correction.
[0093] That is, in the opening angle correction step (S123), the control unit 6 calculates the length of a line segment 86a from an end point 84a on the first edge 91a farther from the camera unit 3 (imaging means: see FIG. 1) to the boundary line of the area 7a, and if the value obtained by subtracting a preset value (predetermined value) from the length of this line segment 86a is negative, the control unit 6 makes the opening angle of the refrigerator compartment door 212 larger than before the correction. Also, if the value obtained by subtracting the preset value (predetermined value) from the length of the line segment 86a is positive, the control unit 6 makes the opening angle of the refrigerator compartment door 212 smaller than before the correction. It is assumed that the corrected opening angle is stored in advance in association with the calculated value of the opening angle of refrigerator compartment door 212 and the difference value for line segment 86a. The same can be said about the correction of the opening angle of left refrigerator compartment door 211.
[0094] According to the fourth embodiment, the opening angle of the refrigerator compartment doors 211, 212 can be appropriately corrected based on the lengths of the lower line segments 86a, 86b. Furthermore, since there is no particular need to use the midpoint between the left and right end points 81a, 81b when correcting the opening angle of the refrigerator compartment doors 211, 212, the fourth embodiment can also be applied to a refrigerator equipped with a single-wing refrigerator compartment door, for example.
[0095] Fifth embodiment The fifth embodiment differs from the first embodiment in that if the first edge 91a (see FIG. 20A) or the second edge 92a (see FIG. 20A) is out of a predetermined allowable range, the captured image at that time is not used for display on the mobile terminal 54 (see FIG. 8). Note that other points (such as the configuration of the refrigerator 100) are the same as those of the first embodiment. Therefore, only the parts that are different from the first embodiment will be described, and a description of the overlapping parts will be omitted.
[0096] FIG. 20A is an explanatory diagram showing an allowable range 93a of a first edge 91a in a captured image in the refrigerator according to the fifth embodiment. The allowable range 93a (first allowable range) shown by a dashed line in Fig. 20A is a range within which the first edge 91a is assumed to exist, and is set in advance. In the example of Fig. 20A, the allowable range 93a is set as a range between two half lines L5 and L6 extending from point P1. As described above, the first edge 91a is an edge near the edge of the inner box 12, and hardly moves regardless of the opening angle of the refrigerator compartment door 212. Therefore, the allowable range 93a of the first edge 91a is set to a range narrower than the allowable range 94a (see Fig. 21A) of the second edge 92a described next.
[0097] Note that the allowable range 93a may be set appropriately so that the first edge 91a is included in the allowable range 93a even when the camera unit 3 (see FIG. 1) is slightly tilted. In addition, although the allowable range of the first edge 91b on the left side is not shown in FIG. 20A, this allowable range is set in the same way as the right side.
[0098] FIG. 20B is an explanatory diagram showing an allowable range 94a of a second edge 92a in a captured image. Permissible range 94a (second permissible range) indicated by a dashed line in Fig. 20B is a range within which second edge 92a is expected to exist when the opening angle of refrigerator compartment door 212 is within a predetermined range (e.g., between 80° and 140°), and is set in advance. In the example of Fig. 20B, permissible range 94a is set as the range between two half lines L7 and L8 extending from point P2.
[0099] Note that the allowable range 94a may be set appropriately so that the second edge 92a is included in the allowable range 94a even if the camera unit 3 (see FIG. 1) is slightly tilted. In addition, although the allowable range of the second edge 92b on the left side is not shown in FIG. 20B, the allowable range is set in the same way as the right side.
[0100] For example, although it is a very rare case, depending on the lighting conditions, the feature points 8m and 8n (see FIG. 12B) may be erroneously extracted, and the first edge 91a and the second edge 92a may be extracted at positions shifted from their original positions on the captured image. Therefore, in the fifth embodiment, when the first edge 91a is out of a predetermined allowable range 93a (see FIG. 20A) and the second edge 92a is out of a predetermined allowable range 94a (see FIG. 20B), the captured image at that time is not used for display on the mobile terminal 54 (see FIG. 8).
[0101] 21A and 21B are flowcharts of the process executed by the control unit of the refrigerator. Note that the processing in steps S101 to S106 in FIG. 21A is similar to that described in the first embodiment (S101 to S106 in FIG. 11), and therefore a description thereof will be omitted. After calculating the open angle of refrigerator door 212 in step S106, in step S131, control unit 6 determines whether the open angle of refrigerator door 212 is within a predetermined range (for example, between 80° and 140°). This predetermined range is a criterion for determining whether or not to determine the suitability of approximation lines such as first edge 91a and second edge 92a (S134), and is set in advance.
[0102] If the opening angle of the refrigerator compartment door 212 is outside the predetermined range in step S131 (S131: Yes), the control unit 6 repeats the process of step S131. On the other hand, if the opening angle of the refrigerator compartment door 212 is within the predetermined range in step S131 (S131: Yes), the control unit 6 proceeds to the process of step S132.
[0103] In step S132, the control unit 6 determines whether or not the first edge 91a is included in the allowable range 93a (see FIG. 20A) (determination step). If the first edge 91a is included in the allowable range 93a (S132: Yes), the process by the control unit 6 proceeds to step S133. Note that in the process of step S132, the control unit 6 may determine whether or not each of the slope and intercept of the line including the first edge 91a is included in a predetermined range.
[0104] In step S133, the control unit 6 determines whether or not the second edge 92a is within the allowable range 94a (see FIG. 20B). If the second edge 92a is within the allowable range 94a (S133: Yes), the process by the control unit 6 proceeds to step S107 in FIG. 21B. Note that in the process of step S133, the control unit 6 may determine whether or not each of the slope and intercept of a line including the second edge 92a is within a predetermined range. Also, the processes of steps S107 to S110 (processes related to the correction of the opening angle) are similar to those in the first embodiment (see FIG. 11), and therefore description thereof will be omitted.
[0105] If the first edge 91a is outside the permissible range 93a (first permissible range) in step S132 (determination step) of FIG. 21A (S132: No), the process of the control unit 6 proceeds to step S134. In step S134, the control unit 6 determines that the approximation lines of the first edge 91a, etc. are inappropriate. Next, in step S135, the control unit 6 prevents the currently captured image from being used for display on the mobile terminal 54 (FIG. 8). This makes it possible to prevent an image that has been subjected to a predetermined process based on an inappropriate approximation line from being displayed on the mobile terminal 54 (see FIG. 8).
[0106] Similarly, if the second edge 92a is outside the allowable range 94a (second allowable range) in step S133 (determination step) (S133: No), the control unit 6 proceeds to the process of step S134. In this case, the control unit 6 also determines that the approximation lines of the second edge 92a, etc. are inappropriate (S134), and does not use the currently captured image for display on the mobile terminal 54 (FIG. 8) (S135). Then, the control unit 6 ends the series of processes (END) without particularly performing the processes of steps S107 to S110 in FIG. 21B (processes related to the correction of the opening angle). The same processes are performed on the first edge 91b (see FIG. 20A) and the second edge 92b (see FIG. 20B) on the left side.
[0107] Incidentally, when the camera unit 3 (see FIG. 1) takes pictures repeatedly at a predetermined frequency, the first edge 91a (see FIG. 20A) or the second edge 92a (see FIG. 20B) may be correctly extracted from the next image taken. The control unit 6 executes the processes of FIGS. 21A and 21B every time the camera unit 3 takes pictures.
[0108] According to the fifth embodiment, when the first edge 91a (see FIG. 20A) is outside the allowable range 93a or when the second edge 92a (see FIG. 20B) is outside the allowable range 94a, the control unit 6 does not use the captured image at that time for display on the mobile terminal 54 (see FIG. 8). This makes it possible to prevent a predetermined image based on an inappropriate approximation line from being displayed on the mobile terminal 54.
[0109] <<Variations>> Although the refrigerator 100 and the like according to the present disclosure have been described above with reference to the respective embodiments, the refrigerator 100 and the like are not limited to these descriptions and may be modified in various ways. For example, the method of correcting the opening angle of the refrigerator compartment doors 211, 212 is not limited to the one described in each embodiment. For example, the position of the end point 81a of the first edge 91a (see FIG. 15A) in a state where the camera unit 3 (imaging means) is correctly installed may be set as a predetermined "reference position", and in the opening angle correction step, the control unit 6 may correct the opening angle of the refrigerator compartment door 212 (door) based on the number of pixels (number of pixels in the horizontal direction) between the "reference position" and the actual position of the end point 81a of the first edge 91a. Such processing is also included in the matter of correcting the opening angle of the refrigerator compartment door 212 based on the positional relationship between the predetermined "reference position" (or reference line) in the captured image and the first edge 91a. Note that, instead of the upper end point 81a (see FIG. 15A) of the first edge 91a, the lower end point 84a (see FIG. 18) may be used. With this configuration, it is possible to correct the opening angle of the single-wing refrigerator door.
[0110] In each embodiment, the vertical reference line 73 (see FIG. 15A) is provided in the center of the captured image so as to divide the captured image evenly in the horizontal direction, but the position of the reference line 73 can be changed as appropriate depending on the configuration of the refrigerator 100. For example, the reference line 73 may be set so as to divide the captured image near the center in the horizontal direction.
[0111] In the first embodiment, a case has been described in which a corrected opening angle is read out corresponding to the calculated value of the opening angle of the refrigerator compartment doors 211, 212 and the difference in the position of the midpoint (the difference from the reference line in the x-axis direction), but this is not limiting. For example, a predetermined coefficient corresponding to the difference may be stored in advance. Then, the corrected opening angle may be calculated by multiplying the calculated value of the opening angle of the refrigerator compartment doors 211, 212 by the coefficient corresponding to the difference. The same can be said about the second to fourth embodiments.
[0112] In the second embodiment, the control unit 6 corrects the opening angle of the refrigerator compartment door 212 based on the difference between the length of the line segment 83a included in the first edge 91a (see FIG. 17) and a default value, but the present invention is not limited to this. For example, the control unit 6 may correct the opening angle of the refrigerator compartment door 212 based on the ratio between the length of the line segment 83a included in the first edge 91a on the right side (see FIG. 17) and the length of the line segment 83b included in the first edge 91b on the left side (see FIG. 17). The same can be said about the fourth embodiment.
[0113] In the fifth embodiment, the determination as to whether the first edge 91a (see FIG. 20A) is within the allowable range 93a (S132 in FIG. 21A) and the determination as to whether the second edge 92a (see FIG. 20A) is within the allowable range 94a (S133 in FIG. 21A) are both performed. However, the present invention is not limited to this. In other words, one of steps S132 and S123 in FIG. 21A may be omitted as appropriate. In the fifth embodiment, when the opening angle of the refrigerator compartment doors 211, 212 is within a predetermined range (S131), the control unit 6 sequentially performs the processes of steps S132 and S133, but this is not limiting. For example, the process of step S131 may be omitted as appropriate.
[0114] Moreover, each embodiment can be appropriately combined. For example, the first embodiment and the second embodiment can be combined, and the average value of the opening angle corrected based on the method of the first embodiment and the opening angle corrected based on the method of the second embodiment can be used as the opening angle of the refrigerator compartment doors 211, 212. In the same manner, a plurality of the first to fourth embodiments can be appropriately combined. Alternatively, for example, the second embodiment and the fifth embodiment may be combined, and when the first edge 91a and the second edge 92a are within the allowable range (fifth embodiment), the control unit 6 may correct the opening angle of the refrigerator compartment door 212 based on the method of the second embodiment.
[0115] In addition, in each embodiment, a configuration has been described in which the camera unit 3 (see FIG. 2) that generates a captured image is provided on the top surface of the housing 1, but this is not limited thereto. That is, the camera unit 3 may be provided at another predetermined position on the outside of the housing 1.
[0116] In addition, in each embodiment, the rectangular area 7a (see FIG. 13) is used to calculate the opening angle of the refrigerator compartment doors 211, 212, but this is not limiting. For example, the control unit 6 may calculate the opening angle of the refrigerator compartment doors 211, 212 based on the number of pixels between the first edge 91a and the second edge 92a without using the rectangular area 7a.
[0117] In addition, in each embodiment, the case where the edge on the captured image corresponding to the boundary between the inner box 12 (see FIG. 4) and the side panel 1b (see FIG. 4) is used as the first edge 91a has been described, but this is not limited thereto. That is, another predetermined boundary on the housing 1 may be used as the first edge 91a. In addition, in each embodiment, the case has been described in which the edge on the captured image corresponding to the boundary between the packing 212d (see FIG. 3) and the side piece 212f (see FIG. 5) of the refrigerator compartment door 212 is used as the second edge 92a, but this is not limiting. That is, another predetermined boundary on the refrigerator compartment door 212 may be used as the second edge 92a.
[0118] In addition, in each embodiment, the refrigerator 100 having the French-style refrigerator compartment doors 211, 212 (see FIG. 3) has been described, but the present invention is not limited to this. For example, each embodiment can be applied to a refrigerator having a single-opening refrigerator compartment door (not shown) or a portable refrigerator (not shown). In addition, the "storage" to which each embodiment can be applied is not limited to a refrigerator. That is, each embodiment can be applied to various types of hinged "storage". In this case, the housing of the "storage" may be configured such that there is no particular distinction between an outer box and an inner box. In addition, each embodiment can be applied to, for example, a chest-type freezer or storage. In addition, each embodiment can be applied to a storage with an upward-opening door (a flip-up door), a downward-opening door, a folding door, or the like. In addition, each embodiment can be applied to, for example, a closet with a folding door.
[0119] Also, all or part of the program for realizing the method for calculating the opening angle of the refrigerator compartment doors 211, 212 (door opening angle calculation method) described in each embodiment may be executed by one or more computers. The program may be provided via a communication line, or may be written on a recording medium such as a CD-ROM and distributed.
[0120] In addition, each embodiment has been described in detail to clearly explain the present disclosure, and is not necessarily limited to having all of the configurations described. In addition, some of the configurations of the embodiments can be added to, deleted from, or replaced with other configurations. Furthermore, the above-mentioned mechanisms and configurations are those considered necessary for the explanation, and do not necessarily show all mechanisms and configurations of the product. [Explanation of symbols]
[0121] 1 Case 3. Camera unit (photography means) 6 Control section 7a,7b area 54 Mobile Devices 73 Reference Line 81a, 81b End points (end points closer to the photographing means) 82 Midpoint 83a, 83b Line segment (line segment from the end point closer to the imaging means to the boundary line of the area) 84a, 84b End points (end points farther from the photographing means) 85 midpoint 86a, 86b Line segment (line segment from the end point farther from the photographing means to the boundary line of the area) 91a, 91b First edge 92a, 92b Second Edge 93a Tolerance range (first tolerance range) 94a Tolerance range (second tolerance range) 100 Refrigerator (storage unit) 211 Refrigerator door (door, left door) 212 Refrigerator door (door, right door) L1 line segment (first edge segment) L2 line segment (second edge line segment) S106 Opening angle calculation step S110, S123 Opening angle correction step S132, S133 Judgment steps
Claims
1. an opening angle calculation step of calculating an opening angle of a hinged door provided on a housing, based on the number of pixels between a first edge, which is an edge of the housing on an image captured by a photographing means, and a second edge, which is an edge of the door on the captured image, when the hinged door is opened; and an opening angle correcting step of correcting the opening angle based on a positional relationship between a predetermined reference line or reference position in the captured image and the first edge.
2. the door is a French door having a right door and a left door; The reference line is set so as to divide the captured image near the center in the horizontal direction, In the opening angle correcting step, the opening angle is corrected based on the number of pixels in a horizontal direction between the reference line and a midpoint of a line segment connecting end points of the first edges on the right and left sides of the housing. The door opening angle calculation method according to claim 1, further comprising:
3. In the opening angle correcting step, when the midpoint is located on one side of the reference line, the opening angle of the right door and the left door corresponding to the one side is corrected to be larger than before the correction, and the opening angle of the door corresponding to the other side is corrected to be smaller than before the correction. The door opening angle calculation method according to claim 2, further comprising:
4. The position of the end point of the first edge when the photographing means is correctly installed is set as the reference position; In the opening angle correcting step, the opening angle is corrected based on the number of pixels between the reference position and an actual position of the end point of the first edge. The door opening angle calculation method according to claim 1, further comprising:
5. a predetermined area is set on the captured image so that the first edge and the second edge pass through the predetermined area; In the opening angle calculation step, an opening angle is calculated between a line segment of the first edge included in the region and another line segment of the second edge included in the region, based on a number of pixels of a portion included in the region on the captured image; In the opening angle correction step, the opening angle is corrected based on a length of a line segment from an end point of the first edge to a boundary line of the region. The door opening angle calculation method according to claim 1, further comprising:
6. In the opening angle correction step, a length of a line segment from an end point of the first edge closer to the imaging means to a boundary line of the area is calculated, and if a value obtained by subtracting a predetermined value from the length of the line segment is negative, the opening angle is made smaller than that before the correction, and if a value obtained by subtracting the predetermined value from the length of the line segment is positive, the opening angle is made larger than that before the correction. The door opening angle calculation method according to claim 5, further comprising:
7. In the opening angle correction step, a length of a line segment from an end point of the first edge farther from the imaging means to a boundary line of the area is calculated, and if a value obtained by subtracting a predetermined value from the length of the line segment is negative, the opening angle is made larger than that before the correction, and if a value obtained by subtracting the predetermined value from the length of the line segment is positive, the opening angle is made smaller than that before the correction. The door opening angle calculation method according to claim 5, further comprising:
8. The method further includes a step of determining whether the first edge is within a first tolerance range; The opening angle calculation step is followed by the determination step, In the determining step, when the first edge is out of the first allowable range, the captured image is not used for display on the mobile terminal. The door opening angle calculation method according to any one of claims 1 to 7, further comprising:
9. The method further includes a step of determining whether the second edge is within a second tolerance range; The opening angle calculation step is followed by the determination step, In the determining step, when the second edge is out of the second allowable range, the captured image is not used for display on the mobile terminal. The door opening angle calculation method according to any one of claims 1 to 7, further comprising:
10. The door opening angle calculation method according to any one of claims 1 to 9, further comprising a control unit that executes the door opening angle calculation method according to any one of claims 1 to 9. A storage unit characterized by:
Citation Information
Patent Citations
Head position controller for disk memory device
JP1988023279A
Refrigerator
JP2005164196A
JPP7426514B
