Image processing device

Abstract

The system provides image processing that appropriately correlates the image captured by the imaging unit with the rotation angle of the door. [Solution] The image processing device is provided above a storage unit which comprises a storage chamber, a door (for example, a refrigerator door 211, 212) that rotates to open and close the opening of the storage chamber, and door pockets 211c, 212c arranged on the storage chamber side of the door, and has a processing unit that processes an image captured by an imaging unit whose imaging range includes the door pockets. The processing unit calculates the coordinates of the door pockets on the image captured by the imaging unit in relation to an origin set at a predetermined location on the image (for example, coordinates X1, X2), calculates the rotation angle of the door from the coordinates, and processes the image according to the rotation angle.

Description

[Technical Field]

[0001] The present disclosure relates to an image processing apparatus. [Background Art]

[0002] As a technique for measuring the opening angle of a refrigerator door, for example, the one described in Patent Document 1 is known. In Patent Document 1, the distance from the distance measuring unit to the right door is associated with the coordinates of a point R set on the right door, and the distance from the distance measuring unit to the left door is associated with the coordinates of a point L set on the left door. The setting unit identifies the coordinates of the point R and the point L corresponding to the measurement result of the distance measuring unit in the setting information, and identifies the opening angle of the door based on the identified coordinates. [Prior Art Literature] [Patent Literature]

[0003] [Patent Document 1] Japanese Patent No. 7482410 (paragraph 0097) [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] Patent Document 1 describes capturing an image from above the refrigerator with the door open. At this time, there are cases where it is desired to process an image captured by the imaging unit according to the opening angle of the door. In the method of Patent Document 1, it is necessary to separately perform processing by associating the opening angle obtained by the distance measuring sensor with the image captured by the imaging unit, which may make the processing complicated. [Means for Solving the Problem]

[0005] The image processing apparatus according to the present disclosure is provided above a storage unit comprising a storage chamber, a door that rotates to open and close the opening of the storage chamber, and a door pocket disposed on the storage chamber side of the door, and has a processing unit that processes an image captured by an imaging unit that includes the door pocket in its imaging range, wherein the processing unit calculates the coordinates of the door pocket on the image captured by the imaging unit in relation to an origin set at a predetermined location on the image, calculates the rotation angle of the door from the coordinates, and processes the image according to the rotation angle. [Brief explanation of the drawing]

[0006] [Figure 1] This is a front view of the refrigerator according to the embodiment. [Figure 2] This is a side view of a refrigerator according to an embodiment. [Figure 3] This is a front view of the refrigerator according to the embodiment, with the left and right refrigerator compartment doors open. [Figure 4] This is a plan view of the refrigerator according to the embodiment, with the left and right refrigerator compartment doors open. [Figure 5] This is a perspective view of the camera unit of the refrigerator according to the embodiment, viewed from diagonally below. [Figure 6A] This is a front view of the camera unit of the refrigerator according to the embodiment. [Figure 6B] This is a cross-sectional view taken along the line II-II in Figure 6A of the refrigerator according to the embodiment. [Figure 7] This is a system configuration diagram of a refrigerator according to the embodiment. [Figure 8] This is an example of a photograph taken by the camera unit of the refrigerator according to the embodiment, before any image processing has been performed. [Figure 9] This is an example of a photograph taken by the camera unit of the refrigerator according to the embodiment, after predetermined image processing has been performed. [Figure 10] This is a flowchart of the image processing of the camera unit according to the embodiment. [Figure 11]FIG. 10 is a diagram showing the relationship between the origin and the door during processing for calculating the opening angle of the door. [Figure 12A] It is an example of a door pocket detection result when the door opening angle is 140 degrees. [Figure 12B] It is an example of a door pocket detection result when the door opening angle is 90 degrees. [Figure 12C] It is an example of a door pocket detection result when the door opening angle is 60 degrees. [Figure 12D] It is an example of a door pocket detection result when the door opening angle is 10 degrees. [Figure 13A] It is an explanatory diagram of a procedure for deriving a relational expression for calculating the door opening angle. [Figure 13B] It is a measurement result of door pocket coordinates with respect to the door opening angle. [Figure 14A] It is an example of another door pocket detection result when the door opening angle is 140 degrees. [Figure 14B] It is an example of another door pocket detection result when the door opening angle is 90 degrees. [Figure 14C] It is an example of another door pocket detection result when the door opening angle is 60 degrees. [Figure 14D] It is an example of another door pocket detection result when the door opening angle is 10 degrees. [Figure 15] It is another flowchart of image processing of the camera unit according to the embodiment. [Figure 16A] FIG. 15 is a diagram showing the relationship between the origin and the pulled-out storage compartment during processing for calculating the pull-out amount. [Figure 16B] It is an example of a captured result obtained by the camera unit of the refrigerator according to the embodiment, in a state where image processing has been performed. [Figure 17] It is an explanatory diagram of position adjustment of the camera unit according to the embodiment. [Figure 18] It is an explanatory diagram of another position adjustment of the camera unit according to the embodiment. [Figure 19] It is an explanatory diagram of another example of door opening / closing detection. DESCRIPTION OF EMBODIMENTS

[0007] Embodiments for carrying out the present invention will be described in detail below with appropriate reference to the drawings. The image processing apparatus of the present embodiment is an image processing apparatus including a processing unit that is provided above a storage compartment including a storage chamber, a door that rotatably opens and closes an opening of the storage chamber, and a door pocket disposed on the storage chamber side of the door, and processes an image captured by an imaging unit whose imaging range includes the door pocket. Herein, a refrigerator 100 including French-type refrigerating compartment doors 211 and 212 (see FIG. 1) is described as the storage compartment, but the present invention is not limited thereto. For example, the embodiment can also be applied to refrigerators including a single-opening refrigerating compartment door (not shown) and portable refrigerators (not shown).

[0008] Furthermore, the "storage compartment" to which the present invention is applicable is not limited to refrigerators. That is, the present invention can be applied to various types of hinged "storage compartments" regardless of whether temperature or humidity control such as warm storage, freezing, drying or humidification is performed, or the like. In addition, for example, the present invention can also be applied to closets, storage shelves and the like provided with door pockets.

[0009] FIG. 1 is a front view of the refrigerator 100 according to the embodiment. The refrigerator 100 (storage compartment) is an apparatus for cooling food and the like, and includes, in addition to a housing 1, each door such as the refrigerating compartment doors 211 and 212, and a camera unit 3. The housing has a plurality of storage chambers therein. In the example of FIG. 1, as the storage chambers of the refrigerator 100, a refrigerating chamber 21, an ice making chamber 22 and an upper freezing chamber 23 arranged side by side, a lower freezing chamber 24, and a vegetable compartment 25 are provided in this order from top to bottom.

[0010] The enclosure 1 has a structure in which an insulating material such as foamed urethane (not shown) is filled between a resin inner box 12 (see Figure 3) that forms a storage compartment and a steel plate outer box 11. Multiple openings 10a (see Figure 2) corresponding to each compartment are provided on the front side (front side) of the enclosure 1. For example, when food or other items are placed in through the opening 10a of the refrigerator compartment 21, the refrigerator compartment doors 211 and 212 are opened. When the refrigerator compartment doors 211 and 212 are closed, the opening 10a of the refrigerator compartment 21 is sealed off. In this way, the refrigerator compartment doors 211 and 212 (doors) have the function of sealing the opening 10a of the enclosure 1.

[0011] As shown in Figure 1, the refrigerator 100 is equipped with left and right refrigerator doors 211 and 212, which together with the housing 1 form the refrigerator compartment 21 (storage compartment) as hinged doors. The left refrigerator door 211 is rotatable around the axis of the hinge 211a (see Figure 4). The same applies to the right refrigerator door 212. In addition, the refrigerator 100 is equipped with pull-out doors, including an ice maker door 221, an upper freezer door 231, a lower freezer door 241, and a vegetable compartment door 251.

[0012] The refrigerator compartment 21 is provided with multiple shelves 213 (see Figure 3) that divide the refrigerator compartment 21 into predetermined sections. On the inside of the left refrigerator compartment door 211 (door), there are multiple (multiple tiers) of door pockets 211c (see Figure 3) for storing food and other items (the same applies to the right refrigerator compartment door 212). The ice-making compartment 22 shown in Figure 1 is provided with an ice-making compartment container (not shown) that can be pulled out together with the ice-making compartment door 221. Similarly, the upper freezer compartment 23 is provided with an upper freezer compartment container (not shown), the lower freezer compartment 24 is provided with a lower freezer compartment container (not shown), and the vegetable compartment 25 is provided with a vegetable compartment container (not shown).

[0013] The refrigerator 100, although not shown in the diagram, includes a compressor, a heat sink (condenser), a capillary tube (throttling mechanism), and an evaporator. The refrigerant circulates sequentially through the compressor, heat sink, capillary tube, and evaporator, and the air in the storage compartment is cooled by heat exchange with the refrigerant flowing through the evaporator. The camera unit 3 shown in Figure 1 photographs at least the storage compartment (for example, the refrigerator compartment 21) of the refrigerator 100 and is installed on the outside of the housing 1 (on the top of the housing 1 in Figure 1).

[0014] Figure 2 is a side view of refrigerator 100. As shown in Figure 2, the camera unit 3 has a detachable configuration and comprises a main body 31 and a support part 32. The main body 31 is for photographing the storage compartment of the refrigerator 100, etc. The support part 32 supports the main body 31 and is installed on the top surface of the housing 1.

[0015] A lens 31a (see also Figure 5) is installed near the front end of the main body 31. The lens 31a is an optical element that refracts and focuses light, and is positioned to face downwards. Such a lens 31a can be a wide-angle lens, such as a fisheye lens, but is not limited to this.

[0016] As shown in Figure 2, the lens 31a is located in front of the front end of the housing 1 (i.e., the opening 10a). More preferably, the lens 31a is located even further in front of the front surface of the closed refrigerator compartment doors 211, 212 (see Figure 1). This makes it easier for the refrigerator compartment 21 to come into the field of view of the lens 31a when, for example, the refrigerator compartment doors 211, 212 are opened. It is also possible to photograph the vegetable compartment 25, etc. (see Figure 1) with the camera unit 3 when a pull-out door such as the vegetable compartment door 251 is open.

[0017] Figure 3 is a front view of the refrigerator 100 with the left and right refrigerator compartment doors 211 and 212 open. As shown in Figure 3, the inner panel 211b of the left refrigerator compartment door 211 is equipped with multiple door pockets 211c for storing food and other items (the same applies to the right refrigerator compartment door 212). When the left and right refrigerator compartment doors 211 and 212 are opened, the food and other items in the refrigerator compartment 21 and the door pockets 211c and 212c are brought into the field of view of the lens 31a of the camera unit 3 from an overhead perspective (see also Figure 8).

[0018] A rectangular gasket 211d is installed near the edge of the inner panel 211b of the left-side refrigerator compartment door 211. The gasket 211d is a resin component that enhances the airtightness of the refrigerator compartment 21. When the refrigerator compartment door 211 is closed, the gasket 211d comes into contact with (tightly adheres to) the opening 10a (see Figure 4) of the housing 1.

[0019] The side piece 211e shown in Figure 3 is a plate-like member that forms the side of the refrigerator door 211 and extends in a long, narrow shape in the vertical direction. In addition to the side piece 211e, which is provided on the opposite side of the hinge 211a (see Figure 4), the refrigerator door 211 also has another side piece 211f (see Figure 4) provided near the axis of the hinge 211a. Both of these side pieces 211e and 211f (see Figure 4) extend in the vertical direction and are provided parallel to the axis of the hinge 211a (see Figure 4). The same applies to the right-hand refrigerator door 212.

[0020] Furthermore, when manually photographing the refrigerator compartment 21 using the camera unit 3, the shutter button 5, which is pressed by the user, may be installed on each of the left and right refrigerator compartment doors 211 and 212, or the shutter button 5 may be provided on the camera unit 3, and is not particularly limited (not shown).

[0021] Figure 4 is a plan view of the refrigerator 100 with the left and right refrigerator compartment doors 211 and 212 open. As shown in Figure 4, hinges 211a and 212a are installed on the top surface of the housing 1. The left hinge 211a has a pivot axis 211g and pivotally supports the refrigerator compartment door 211 (the right hinge 212a is similar). In the example in Figure 4, the camera unit 3 is positioned slightly to the left of the center of the housing 1 of the refrigerator 100 in the left-right direction. More specifically, the camera unit 3 is positioned directly above the joint of the closed refrigerator compartment doors 211 and 212 (see also Figure 1). This makes it easier for food items in the door pockets 211c and 212c to come into the field of view of the camera unit 3 when the refrigerator compartment doors 211 and 212 are opened.

[0022] Figure 5 is a perspective view of the camera unit 3 when viewed from diagonally below. As shown in Figure 5, the main body 31 of the camera unit 3 includes the lens 31a (see also Figure 2), a case 31b, and an LED cover 31c. The case 31b of the main body 31 is generally elongated in the front-to-back direction and has a rectangular parallelepiped shape. A circular hole (not shown) is provided on the lower surface near the front end of the case 31b, through which the lens 31a is exposed. On the lower surface of the case 31b, on the rear (back) side of the lens 31a, an elongated hole (not shown) is provided in the left-to-right direction, and the LED cover 31c is fitted into this hole. The LED cover 31c is a translucent resin member for protecting the camera LED 31d (see Figure 6B).

[0023] Figure 6A is a front view of the camera unit 3. As shown in Figure 6A, the main body 31 of the camera unit 3 is detachable and is installed on the upper side of the support 32. In the left-right direction, the main body 31 is installed near the center of the support 32.

[0024] Figure 6B is a cross-sectional view taken along the line II-II in Figure 6A. As shown in Figure 6B, the camera unit 3 includes the aforementioned lens 31a (see also Figure 5), LED cover 31c (see also Figure 5), and case 31b, as well as a camera LED 31d and a circuit board 31e.

[0025] The camera LED 31d (illumination unit) is a light source that illuminates storage compartments such as the refrigerator compartment 21. In other words, the camera LED 31d is installed on the outside of the housing 1 (in this embodiment, on the top side of the housing 1) so that the camera unit 3 can photograph the refrigerator compartment 21 (see Figure 3) and door pockets 211c, 212c (see Figure 3) under appropriate brightness. The camera LED 31d is illuminated when at least one of the refrigerator compartment doors 211, 212 (see Figure 3) is opened. In the example of Figure 6B, the camera LED 31d is positioned in front of the front end of the housing 1 of the refrigerator 100 (see Figure 2) (opening 10a of the housing 1). That is, in a plan view, the camera LED 31d is located in front of (outside) the opening 10a of the housing 1 (see Figure 2). This makes it easier for the light emitted from the camera LED 31d to enter the refrigerator compartment 21, etc., thus making it easier to obtain clear images. A translucent LED cover 31c (see also Figure 5) is installed on the underside of the camera LED 31d.

[0026] The circuit board 31e is a printed circuit board on which the image sensor 31f (see Figure 7) and the camera / communication control SoC 31g (System-on-a-Chip: see Figure 7) are mounted. By housing the lens 31a, camera LED 31d, and circuit board 31e in a single case 31b in this way, the camera unit 3 can be made smaller compared to when these components are provided separately.

[0027] Figure 7 is a system configuration diagram of the refrigerator 100 including the camera unit 3. As shown in Figure 7, in addition to the camera unit 3, the refrigerator 100 is equipped with a door sensor 4, a shutter button 5, an operation panel 6, an interior light 7, a control unit 8, and a wireless LAN unit 9. The wireless LAN unit 9 may be built into the camera unit 3.

[0028] The door sensor 4 outputs a predetermined signal to the control unit 8 indicating the open / closed state of the refrigerator doors 211 and 212 (see Figure 1). Although Figure 7 shows a single door sensor 4, in reality, multiple door sensors 4 are provided to correspond to each door of the refrigerator 100 (see Figure 1). For example, when the left refrigerator door 211 (see Figure 1) is opened, a predetermined open signal is output to the control unit 8 from the door sensor 4 corresponding to this refrigerator door 211.

[0029] The control panel 6 is a panel for inputting predetermined setting information based on user operations. The interior light 7 is a light source that illuminates the refrigerator compartment 21 (storage compartment) and is installed on the wall of the refrigerator compartment 21.

[0030] The control unit 8 is, for example, a microcontroller, and although not shown in the diagram, it is composed of electronic circuits including a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), and various interfaces. It reads the program stored in ROM, loads it into RAM, and the CPU executes various processes. The processes performed by the control unit 8 will be described later.

[0031] The camera unit 3 includes a lens 31a (see also Figure 6B), a camera LED 31d (see also Figure 6B), an image sensor 31f, a camera / communication control SoC 31g, and a buzzer 31h. The image sensor 31f is an element that converts light incident through the lens 31a into photoelectric data to generate captured image data. Examples of such image sensors 31f include CCD sensors (Charge Coupled Devices) and CMOS sensors (Complementary Metal Oxide Semiconductors).

[0032] The camera / communication control SoC31g (processing unit) is an integrated circuit designed to integrate microcontroller functions and other applied functions onto a single chip and to function in coordination. The camera / communication control SoC31g communicates with the control unit 8 as specified and outputs a capture command to the image sensor 31f. As a result, image data of the captured image is input from the image sensor 31f to the camera / communication control SoC31g.

[0033] The buzzer 31h emits a predetermined sound, for example, if an error occurs during shooting by the camera unit 3. Alternatively, the buzzer 31h may emit a predetermined sound when shooting begins with the camera unit 3.

[0034] The wireless LAN unit 9 transmits the captured image data output from the camera / communication control SoC 31g to the router 51. The router 51 is a communication relay device and transmits the captured image data received from the wireless LAN unit 9 to the server 53 via the network 52. The server 53 performs predetermined processing based on the captured image data and stores it in association with the identification information of the refrigerator 100. In addition, if the server 53 receives a request signal for captured images (interior images) of the refrigerator 100 from the user's mobile terminal 54, it transmits the captured image data to the mobile terminal 54. Such a mobile terminal 54 can be a mobile phone, smartphone, tablet, or wearable device.

[0035] Figure 8 shows an example of the results captured by camera unit 3, before any image processing has been performed. Note that in Figure 8, the top of the page is considered the front and the bottom of the page is considered the back, so the left and right are reversed compared to Figure 3, etc. (the same applies to Figure 9). In the example in Figure 8, both the left and right refrigerator doors 211 and 212 are open, and in addition to the refrigerator compartment 21, the door pocket 211c of the left refrigerator compartment door 211 and the door pocket 212c of the right refrigerator compartment door 212 are also captured.

[0036] In particular, when a fisheye lens is used as the lens 31a of the camera unit 3 (see Figure 5), imaging can be performed with a wide field of view.

[0037] Figure 9 shows an example of the image captured by the camera unit 3 after predetermined image processing has been performed. For example, based on the image captured in Figure 8, predetermined image processing is performed on the server 53 (see Figure 7) to convert it into an image (unfolded image) that looks as if the refrigerator compartment 21 and the left and right refrigerator compartment doors 211 and 212 are viewed from the front. When such an image is displayed on the mobile terminal 54 (see Figure 7), the user can see at a glance what kind of food is stored in the refrigerator compartment 21, etc. Note that the predetermined image processing may be performed not only on the server but also on the camera / communication control SoC 31g.

[0038] Furthermore, it is desirable that the image captured with the refrigerator doors 211 and 212 (see Figure 3) fully open be displayed on the mobile terminal 54 (see Figure 7). This is because if the opening angle of the refrigerator doors 211 and 212 is too small, parts of the door pockets 211c and 212c will be visible in the area of ​​the refrigerator compartment 21 in the unfolded image shown in Figure 10, resulting in poor visibility.

[0039] Therefore, in this embodiment, as will be described below, the camera / communication control SoC31g (hereinafter referred to as the processing unit) calculates the coordinates of the door pockets 211c and 212c in the image captured by the imaging unit in relation to an origin set at a predetermined location on the image, calculates the rotation angle of the refrigerator doors 211 and 212 from the coordinates, and processes the image according to the rotation angle.

[0040] Figure 10 is a flowchart of the image processing of the camera unit 3 (process S100). Figure 11 is a diagram showing the relationship between the origin and the door during the calculation of the door opening angle in Figure 10. In Figure 10, the processing unit has already incorporated the relationship formula between the X coordinates of the door pockets 211c and 212c (described later) and the door opening angle of each door (see Figures 13A and 13B). The processing unit determines whether there is an open signal input from the door sensor 4 (process S101). If there is no open signal input (process S101, No), it returns to process S101; if there is an open signal input (process S101, Yes), it proceeds to process S102.

[0041] In process S102, the processing unit determines whether a predetermined time has elapsed (process S102). If the predetermined time has not elapsed (process S102, No), it returns to process S102. If the predetermined time has elapsed (process S102, Yes), it issues an imaging command (process S103).

[0042] The processing unit then detects the door pocket using image recognition (processing S104). Note that the door pocket detection model in this embodiment is a general object detection model, but other methods utilizing deep learning, such as YOLO, SSD, and RCNN, may be selected, and the model is not particularly limited. Furthermore, the color, shape, and other features of the door pocket, which are training images, may be modeled using machine learning and stored on the server as information, and the door pocket may be recognized using this trained data. Now, the flow of this processing unit will be explained.

[0043] First, the door angle (rotation angle) is calculated from the coordinates of the door pocket (process S105). The processing unit determines whether the door angle (rotation angle) of any door is greater than or equal to a predetermined angle (for example, 80 degrees) (process S106). If it is less than the predetermined angle (process S106, No), the process returns to S102. If it is greater than or equal to the predetermined angle (process S106, Yes), the process proceeds to S107.

[0044] In process S107, the processing unit updates the candidate image and angle information to be sent to the server 53, and determines whether or not there is a closed signal input from the door sensor 4 (process S108). If there is no closed signal input (process S108, No), it returns to process S102; if there is a closed signal input (process S108, Yes), it proceeds to process S109.

[0045] In process 109, the processing unit determines whether or not the candidate for transmission to server 53 has been saved. If there is no candidate for transmission (process S109, No), the process ends. If there is a candidate for transmission (process S109, Yes), the saved image and information are sent to server 53 (process S110), and the process ends.

[0046] In other words, in process S110, the processing unit transmits the captured image and the calculated door rotation angle to the server, and the server 53 extracts the area showing the door pocket from the captured image based on the calculated door rotation angle and corrects it to a front view image (see Figure 9).

[0047] In the process shown in Figure 10, the door angle (rotation angle) and the image captured by the imaging unit are properly matched, simplifying image processing to make it easier for the user to view.

[0048] In process S109, No, the processing unit means that if the calculated rotation angle is less than a predetermined angle (for example, 80 degrees), it will not send the captured image to the server.

[0049] Figure 11 shows an example of door pocket detection in process S104 of Figure 10. The door pocket is detected by image recognition as thick solid bounding boxes B1 and B2. A bounding box is a rectangular frame that surrounds an object in an image or video, as determined by an object detection model.

[0050] In process S105 of Figure 10, assuming that an origin 0 (zero) indicated by reference numeral 210 is set in the upper left of the figure as shown in Figure 11, and that the left-right direction of the storage compartment is the X-axis direction and the front-to-back direction of the storage compartment is the Y-axis direction, the processing unit recognizes the area (bounding boxes B1, B2) in the image where the door pocket is visible, and calculates the X coordinate of that area away from the rotation axis of the door.

[0051] Furthermore, calculating the X-coordinate of the area away from the door's pivot axes 211g and 212g means, in the case of bounding box B1, calculating the X-coordinate of the central side of the four sides (four straight lines) that make up bounding box B1. Similarly, for bounding box B2, it means calculating the X-coordinate of the central side.

[0052] Generally, any coordinate of bounding boxes B1 and B2 can be used as the coordinate of the door pocket. However, the inventors' findings indicate that the X coordinate has a larger range of change with respect to door rotation (e.g., from 85 to 140 degrees) than the Y coordinate, and that this allows for smaller errors during calculation. Therefore, it is preferable to use the X coordinate to determine the door angle. In Figure 11, coordinates X1 and X2, which are the X coordinates of the left and right sides of the storage compartment near the center, are used. Note that in Figure 11, the pivot axis 212g is indicated by a dashed line at the base of the left refrigerator door 211 to the housing 1 (the right refrigerator door 212 also has a pivot axis 211g).

[0053] In other words, the processing unit includes a model that detects door pockets in response to image input. The training data used to train the model consists of images showing door pockets. The model is trained by inputting images in which the type of item placed in the door pocket and the door's rotation angle are varied.

[0054] Figure 12A shows an example of door pocket detection results when the door opening angle is 140 degrees. Figure 12B shows an example of door pocket detection results when the door opening angle is 90 degrees. Figure 12C shows an example of door pocket detection results when the door opening angle is 60 degrees. Figure 12D shows an example of door pocket detection results when the door opening angle is 10 degrees.

[0055] When the door opening angle is 140 degrees, 90 degrees, or 60 degrees, the door pocket is detected, but when the door opening angle is 10 degrees, the door pocket is not detected. Although no people are shown in Figures 12A to 12D, the door pocket will still be detected even if a person is captured near it.

[0056] Figure 13A is an explanatory diagram of the procedure for deriving the relational formula for calculating the door opening angle. Figure 13B shows the measurement results of the door pocket coordinates with respect to the door opening angle. As explained in Figure 10, in this embodiment, a relational formula between the X coordinates (coordinate values) of door pockets 211c and 212c and the door opening angle of each door is incorporated in advance. Below, an example of calculating this relational formula will be explained. Note that the relational formula described in this embodiment is just one example, and similar formulas can be considered depending on the object detection model, so it is not necessarily limited.

[0057] As shown in Figure 13B, the door opening angle is changed in 10-degree increments, and images are taken. The X coordinates (coordinates X1, X2) of the center of the door pocket are then measured from the captured images. The measurement results are shown in Figure 13B. Here, for example, the relationship between the door opening angle and the door pocket coordinates may be given by equation (1). θ = AX³ + BX² + CX + D (1) Note that θ is the door opening angle, X is the X-coordinate of the center of the captured image of the door pocket, and A, B, C, and D are constants, but there are no particular limitations.

[0058] The model is trained on a dataset of annotated real objects before object detection. Annotation, in machine learning, refers to adding metadata to data to give it meaning. By annotating a large amount of data and adding correct data (=training data), it becomes possible to determine what is correct about the machine learning model.

[0059] In the refrigerator shown in Figure 2, door pockets 211c and 212c consist of an upper door pocket, a middle door pocket, and a lower door pocket. Figures 12A to 12D show examples of detection results for the upper door pocket. The entire door pocket, including the upper, middle, and lower door pockets, may also be the target of detection.

[0060] Figure 14A shows an example of other detection results for the door pocket when the door opening angle is 140 degrees. Figure 14B shows an example of other detection results for the door pocket when the door opening angle is 90 degrees. Figure 14C shows an example of other detection results for the door pocket when the door opening angle is 60 degrees. Figure 14D shows an example of other detection results for the door pocket when the door opening angle is 10 degrees.

[0061] When the door opening angle is 140 degrees, 90 degrees, or 60 degrees, the entire door pocket is detected, while when the door opening angle is 10 degrees, the entire door pocket is not detected. Although no people are shown in Figures 14A to 14D, the entire door pocket is detected even if a person is captured near the entire door pocket. Note that the position of the X coordinate (coordinates X1, X2) is the same as in Figure 11 for Figures 12A-12C, 13A, and 14A-14C.

[0062] (In the case of a drawer storage room) Figure 15 is another flowchart (process S120) of the image processing of camera unit 3. Figure 16A is a diagram showing the relationship between the origin and the drawer storage chamber during the drawer amount calculation process in Figure 15. In Figure 15, the same reference numerals are used for the same processes as in Figure 10.

[0063] In Figure 15, the processing unit determines whether there is an open signal input from the door sensor 4 (process S121). If there is no open signal input (process S121, No), it returns to process S121. If there is an open signal input (process S121, Yes), it proceeds to process S102.

[0064] In process S102, the processing unit determines whether a predetermined time has elapsed (process S102). If the predetermined time has not elapsed (process S102, No), it returns to process S102. If the predetermined time has elapsed (process S102, Yes), it issues an imaging command (process S103).

[0065] The processing unit then uses image recognition to detect the drawer storage compartment (process S124). The detection model uses the drawer storage compartment detection model. The processing unit calculates the opening amount from the coordinates of the drawer storage compartment (process S125). The processing unit determines whether the opening amount is greater than or equal to a threshold (process S126). If the opening amount is less than the threshold (process S126, No), the process returns to process S102. If the opening amount is greater than or equal to the threshold (process S126, Yes), the process proceeds to process S127.

[0066] In process S127, the processing unit updates the image and opening amount information to the server 53 and determines whether or not there is a closing signal input from the door sensor 4 (process S128). If there is no closing signal input (process S128, No), the unit returns to process S102; if there is a closing signal input (process S128, Yes), the unit proceeds to process S109.

[0067] In process 109, the processing unit determines whether or not the candidate for transmission to server 53 has been saved. If there is no candidate for transmission (process S109, No), the process ends. If there is a candidate for transmission (process S109, Yes), the saved image and information are sent to server 53 (process S110), and the process ends.

[0068] Figure 16A shows an example of drawer storage chamber detection in process S124 of Figure 15. The drawer storage chamber is detected as a bounding box B3 with a thick solid line by image recognition.

[0069] Assuming that the origin 0 (zero), indicated by symbol 210, is set in the upper left of the diagram as shown in Figure 16A, and the left-right direction of the storage compartment is the X-axis direction, and the front-to-back direction of the storage compartment is the Y-axis direction, then the coordinate Y1, which is the Y-coordinate of bounding box B3, is used for the door pocket coordinates.

[0070] In other words, if the left-right direction of the storage unit is the X-axis direction and the front-to-back direction of the storage unit is the Y-axis direction, the processing unit should recognize the area (bounding box B3) in the image where the drawer storage compartment is visible, and calculate the Y coordinate (for example, coordinate Y1) within that area that is far from the main body of the storage unit.

[0071] The processing unit may include an object detection model that detects the drawer storage compartment and / or the trays inside the drawer storage compartment in response to image input. The training data used to train the object detection model is an image showing the drawer storage compartment and / or the trays. The object detection model can be trained by inputting images in which the type of items placed in the drawer storage compartment and the amount the door is extended are varied. Alternatively, the object detection model can be trained for each multi-tiered tray in the drawer storage compartment.

[0072] In the processing S110 shown in Figure 15, the processing unit transmits the captured image and the amount of the drawer storage chamber drawn out to the server 53. The server 53 then extracts the area containing the drawer storage chamber from the captured image based on the amount of the drawer storage chamber drawn out and corrects it to a front view image. Alternatively, the processing unit may use the recognized area (bounding box B3) as the extraction area.

[0073] Figure 16B shows an example of the image captured by the camera unit 3 after image processing. The server 53 (see Figure 7) performs predetermined image processing, converting the image into a top-down view of the vegetable compartment 25 (inside the vegetable compartment container 252), as shown in Figure 16B. This unfolded image is displayed on the mobile terminal 54 (see Figure 7), allowing the user to quickly understand what foods are stored in the vegetable compartment 25. Although not specifically shown in Figure 16B, predetermined characters or images indicating "vegetable compartment" may be associated with the captured image and displayed on the mobile terminal 54. This makes it easier for the user to recognize that the image is of the vegetable compartment 25. Incidentally, the control unit 8 can recognize which storage compartment the image was taken from, as each door is equipped with a door sensor 4.

[0074] In the processing shown in Figure 15, the correspondence between the amount of drawer pulled out of the drawer storage chamber and the image captured by the imaging unit is properly established, simplifying image processing to make it easier for the user to view.

[0075] In the processing of step S109 in Figure 15, if the extraction amount is less than a predetermined value, the processing unit will not send the captured image to the server.

[0076] The processing unit should use the coordinates of the drawer storage chamber in the image captured when the drawer storage chamber is fully extended as a reference, and compare the coordinates in the captured image with the coordinates when there is no mounting misalignment of the camera unit 3 to correct the amount the drawer storage chamber is extended.

[0077] (Initial setup of camera unit 3) It is desirable to perform an initial setup to ensure that the camera unit 3 is installed in the designated position. Specifically, when installing the camera unit 3, video is recorded, and a reference line is drawn on the recorded video and displayed on an information terminal such as a smartphone that has been authenticated and connected. The user then adjusts the installation position of the camera unit 3 according to the reference line.

[0078] Figure 17 is an explanatory diagram for adjusting the position of the camera unit 3. The refrigerator doors 211 and 212 are opened to 90 degrees, and the outer edges of the door pockets 211c and 212c on each door (shown as white oval marks in the figure) are adjusted to align with the reference line (dashed line in the figure). In Figure 17, before adjustment, the dashed line is misaligned with the white oval marks, but after adjustment, it is adjusted to align with them. At this time, the door pockets are detected by image recognition, and when the inner edge of the bounding box image overlaps with the reference line to some extent, the reference line may be turned green or other indicators may be displayed to notify that the adjustment is complete.

[0079] In other words, the processing unit should use the coordinates of the door pocket in the image captured when the door is opened 90 degrees as a reference, and correct the calculated rotation angle by comparing the coordinates in the captured image with the coordinates when there is no camera mounting misalignment.

[0080] Figure 18 is an explanatory diagram for other position adjustments of the camera unit 3. Close the refrigerator doors 211 and 212 and adjust them so that the gap between the doors aligns with the reference line. In Figure 18, before adjustment, the gap between the doors is misaligned with the reference line, but after adjustment, the gap between the doors aligns with the reference line. Note that even if the size of the doors is asymmetrical, it is best to adjust them so that the gap is in the center.

[0081] Furthermore, the processing unit should use the coordinates of the door pocket in the image captured when the door is opened 90 degrees as a reference, and correct the calculated rotation angle by comparing the coordinates when there is no camera mounting misalignment with the coordinates on the image captured by the imaging unit.

[0082] (Other methods for detecting door openness) As shown in Figure 11, door pocket detection is performed using image recognition. However, even with image recognition, door pocket detection is not 100% accurate. Therefore, this embodiment includes a backup function for determining whether the door is open or closed.

[0083] Figure 19 is an explanatory diagram of another example of door opening / closing detection. Figure 19 has a first detection area AR1, AR2 which is the detection area for the door pocket, and a second detection area BR1, BR2 which is the detection area for the door boundary, as the door opening / closing detection area. As a backup function for determining whether the door is open or closed, the processing unit determines whether the door is less than a predetermined angle (for example, 80 degrees) based on the first and second detection areas. Note that in Figure 19, the upper side of the paper is the front and the lower side is the rear, so the left and right are reversed compared to Figure 3, etc.

[0084] In door state 20B, the first detection area AR1 of the refrigerator door 211 shows the edge of the door pocket or an object. In this case, because there is an outline (edge) of the object, it is determined that a door pocket has been detected. Similarly, the first detection area AR2 of the refrigerator door 212 shows the edge of the door pocket. In this case, it is determined that a door pocket has been detected. In other words, in a case like door state 20B, the processing unit determines that the door is open but not fully open, and therefore does not send the captured image to the server 53.

[0085] In door state 20A, the edge of the door pocket is not visible in the first detection area AR1 of the refrigerator door 211. In this case, the processing unit determines that the door is at an angle greater than a predetermined angle if a boundary line is present in the second detection area BR1. Similarly, the edge of the door pocket is not visible in the first detection area A2 of the refrigerator door 212. In this case, the processing unit determines that the door is at an angle greater than a predetermined angle if a boundary line is present in the second detection area BR2.

[0086] In door state 20C, the edge of the door pocket is not visible in the first detection area AR1 of the refrigerator door 211. In this case, the processing unit determines that the door is closed if there is no boundary line in the second detection area BR1. Similarly, the edge of the door pocket is not visible in the first detection area AR2 of the refrigerator door 212. In this case, the processing unit determines that the door is closed if there is no boundary line in the second detection area BR2.

[0087] The following image processing device can be used to determine door state 20B, which is an intermediate state between door state 20A and door state 20C. In other words, it is an image processing device that processes images of the boundary between the storage room and the door that opens and closes the opening of the storage room (for example, first detection areas A1, A2). The image processing device is (1) An extraction step to extract multiple points from an image, (2) A derivation step of deriving a parameter for each of the multiple lines passing through the coordinates of each of the multiple points, (3) A detection step of detecting the peak of the frequency of occurrence of parameters distributed in the parameter coordinate space, (4) A determination step is performed to determine whether the door in the image is open or closed, depending on whether the peak size is above a threshold.

[0088] The parameters in (2) above are determined from the fact that, for a given point (X, Y), a line that may pass through the point can be expressed as Y = ax + b (parameters: a, b) or R = X cosθ + Y sinθ (parameters: R, θ).

[0089] In the determination step (4) above, the size of the peak indicates a high probability of a line being present. When the door is open, for example, the boundary between the door and the inner box is detected as a line, i.e., a large peak is detected. On the other hand, when the door is closed, no line is detected on the design surface of the door, i.e., the detected peak is small or absent.

[0090] To explain in more detail, The image processing device is (1A) A step of extracting n points from the boundary or a region near the boundary from the image, (2A) For each of the coordinates (k=1,···,n) that represent n points (xk,yk) in the coordinate system defined in the image, Representation using the angular parameter θ relative to the origin of the coordinate system Calculate ρθ=(xkcosθ+yksinθ) for multiple values ​​of θ, (3A) Repeat the process of increasing the number of votes for the point (θ,ρθ) obtained using the θ used and the calculated ρ in the parameter coordinate space (θ,ρ) for a number of times T (n × the number of θ values ​​set). (4A) It is recommended to determine whether the door in the image is open or closed based on whether the number of votes for the (θT,ρT) with the highest number of votes exceeds the threshold. Note that θT represents the number of occurrences for each angle parameter.

[0091] The image processing area that the image processing device processes is the region where edges are visible when the door is open and where edges are not visible when the door is closed (the second detection region BR1, BR2 in Figure 19).

[0092] The image processing device can determine the recognition of an intermediate state (door state 20B) between the open state (door state 20A) and the closed state (door state 20C) of the door by performing extraction, derivation, detection, and determination steps on the area (first detection area AR1, AR2) where the edge of the door pocket is captured in the image.

[0093] In this embodiment, the processing unit (camera / communication control Soc31g) transmits the captured image and the calculated door rotation angle to the server, and the server 53 extracts the area showing the door pocket from the captured image based on the calculated door rotation angle and corrects it to a front view image, but is not limited to this. The processing unit may also extract the area showing the door pocket from the captured image based on the calculated door rotation angle and correct it to a front view image.

[0094] Each embodiment is described in detail for the purpose of clearly illustrating this disclosure and is not necessarily limited to having all the configurations described. Furthermore, some of the configurations of the embodiments may be added, deleted, or replaced with other configurations. Furthermore, the mechanisms and configurations described above are those deemed necessary for explanatory purposes and do not necessarily represent all of the mechanisms and configurations shown in the actual product. [Explanation of symbols]

[0095] 1 cabinet 3 Camera Unit 4 Door Sensors 8 Control Unit 9 Wireless LAN Unit 10a opening 21. Refrigerated room (storage room) 31a lens 31f Image Sensor (Imaging Unit) 31g Camera / Communication Control SoC (Processing Unit) 53 Servers 54 Mobile devices 100 Refrigerator (storage unit) 210 sign (origin) 211,212 Refrigerator door (door) 211c, 212c Door pocket 211g, 212g pivot shaft 221 Ice maker door 231 Upper freezer door 241 Lower freezer door 251 Vegetable compartment door 252 Vegetable compartment container B1, B2, B3 bounding boxes AR1, AR2 First detection area BR1, BR2 Second detection area

Claims

1. An image processing apparatus is provided above a storage unit comprising a storage chamber, a door that rotates to open and close the opening of the storage chamber, and a door pocket located on the storage chamber side of the door, and having a processing unit for processing images captured by an imaging unit that includes the door pocket in its imaging range, The processing unit calculates the coordinates of the door pocket in the image captured by the imaging unit in relation to a predetermined origin set at a specific location on the image, calculates the rotation angle of the door from the coordinates, and processes the image according to the rotation angle. An image processing apparatus characterized by the following:

2. An image processing apparatus according to claim 1, If the left-right direction of the storage compartment is the X-axis direction and the front-to-back direction of the storage compartment is the Y-axis direction, The processing unit recognizes the area in the image that shows the door pocket and calculates the X coordinate of that area, which is away from the rotation axis of the door. An image processing apparatus characterized by the following:

3. An image processing apparatus according to claim 1, The processing unit transmits the captured image and the calculated door rotation angle to the server. The server extracts the area showing the door pocket from the captured image based on the calculated door rotation angle and corrects it to a front view image. An image processing apparatus characterized by the following:

4. An image processing apparatus according to claim 3, The processing unit does not send the captured image to the server if the calculated rotation angle is less than 80 degrees. An image processing apparatus characterized by the following:

5. An image processing apparatus according to claim 1, The processing unit uses the coordinates of the door pocket in the image captured when the door is opened 90 degrees as a reference, and corrects the calculated rotation angle by comparing the coordinates when there is no camera mounting misalignment with the coordinates on the image captured by the imaging unit. An image processing apparatus characterized by the following:

6. An image processing apparatus according to claim 1, If the left-right direction of the storage compartment is the X-axis direction and the front-to-back direction of the storage compartment is the Y-axis direction, The processing unit recognizes the area in the image that shows the drawer storage compartment and calculates the Y coordinate of the area that is far from the main body of the storage unit. An image processing apparatus characterized by the following:

Images