Foodstuff management system

Abstract

To manage foodstuff stored in a vegetable chamber where an upper stage and a lower stage are divided on the basis of a photographed image from an upper surface of the vegetable chamber of a refrigerator.SOLUTION: During a period when a door 11 is opened, an image of a vegetable chamber 10 divided into an upper stage 15 and a lower stage 16 by a partitioned tray 12 is photographed with a camera 21 provided for an upper part of a refrigerator 5. When the opening value of the partitioned tray 12 and existence information of a plurality of predetermined vegetable items in an image in the image are acquired by performing image processing of the photographed image by the camera 21, it is determined whether the image is that of the upper stage 15 or the lower stage 16 on the basis of the opening value. Respective stock data of the upper stage 15 and the lower stage 16 are updated by using the determination result of the upper stage 15 or the lower stage 16 and existence information of the respective vegetable items, and purchase dates of the respective vegetable items are stored on the basis of the photographing date of the image when the existence information is updated.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an ingredient management system, an ingredient management method, and a program, and more particularly to an ingredient management system, an ingredient management method, and a program that target ingredients (vegetables) stored in the crisper of a refrigerator. [Background technology]

[0002] A food ingredient management system that targets the amount of food ingredients held by a user, particularly food ingredients stored in a refrigerator, is described in JP 2021-128557 A (Patent Document 1). Patent Document 1 describes storing various data on food ingredients in a server system, and displaying information on a terminal device encouraging the purchase of food ingredients whose held quantity has decreased based on the stored data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-128557 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 assumes several ways of inputting food ingredient data, and mentions inputting images captured by a camera built into a refrigerator or a camera of a terminal device into a trained model that has been machine-learned for food ingredient recognition, in addition to manual input by a user. However, Patent Document 1 does not specifically disclose the specific processing content of data obtained by image recognition using a trained model based on images captured by a camera.

[0005] In addition, in the vegetable compartment of a refrigerator, a sliding partition tray is often provided on the upper level to separate the upper and lower levels. If an image of a vegetable compartment with such a structure is used to manage the vegetables in the vegetable compartment, it is necessary to recognize ingredients (vegetables) separately for the upper and lower levels. However, from the viewpoint of convenience, it is hesitant to ask the user to take special measures such as requesting specific images of the upper and lower levels when taking images.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and its purpose is to provide a technology for managing food ingredients stored in a vegetable compartment of a refrigerator, which is divided into an upper and lower compartment, based on an image captured from the top of the vegetable compartment. [Means for solving the problem]

[0007] In one aspect of the present disclosure, a food ingredient management system includes an imaging unit, an image processing unit, an image determination unit, and an update processing unit. The imaging unit is provided at the top of the refrigerator and is positioned so as to include a vegetable compartment within its field of view when the vegetable compartment has a sliding partition tray that divides an upper section and a lower section. The image processing unit processes an image of the vegetable compartment in an open state captured by the imaging unit to obtain information on whether or not each of a plurality of predetermined types of vegetables is recognized in the image, and an opening degree value indicating the position of the partition tray.

[0008] The image determination unit determines whether each image is an image of the upper or lower row based on the opening value obtained by the image processing unit. The update processing unit updates inventory data including presence / absence data of each of the multiple items of vegetables for each of the upper and lower rows based on the determination result of the image determination unit and the presence / absence information in each image. Furthermore, when the presence / absence data of a first item of the multiple items of vegetables changes from absent to present based on the presence / absence information in the first image, the update processing unit records the photographing date of the first image in the inventory data as data of the purchase date of the first item of vegetables.

[0009] A foodstuff management method according to another aspect of the present invention includes first to third steps. The first step is a step of acquiring information on whether each of a plurality of predetermined items of vegetables is recognized in the image and an opening value indicating the position of the partition tray by performing image processing on an image of the vegetable compartment taken from the upper side of the refrigerator when the door of the vegetable compartment having a sliding partition tray dividing the upper and lower tiers is opened. The second step is a step of judging whether each image is an image of the upper tier or the lower tier based on the opening value obtained in the first step. The third step is a step of updating inventory data including data on the presence or absence of each of the plurality of items of vegetables for each of the upper and lower tiers based on the judgment result in the second step and the presence or absence information in each image. Furthermore, in the third step, when the presence or absence data of the first item of vegetables among the plurality of items changes from absent to present based on the presence or absence information in the first image of the vegetable compartment, the photographing date of the first image is recorded in the inventory data as data on the purchase date of the first item of vegetables.

[0010] A program according to still another aspect of the present disclosure causes a processor of a computer to execute the above first to fifth steps. Effect of the Invention

[0011] According to the present invention, food ingredients stored in a vegetable compartment divided into an upper and lower level can be managed based on an image captured from the top surface of the vegetable compartment of a refrigerator. [Brief description of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram illustrating an example of the configuration of a foodstuff management system according to an embodiment of the present invention. [Diagram 2] 2 is a block diagram illustrating a schematic configuration of the imaging device shown in FIG. [Diagram 3] FIG. 2 is a conceptual side view of a refrigerator for explaining the function of an imaging device. [Figure 4] 4 is a plan view of the vegetable compartment as seen from the direction IV in FIG. 3. [Diagram 5]11 is a flowchart illustrating an example of a control process for acquiring the opening degree of a vegetable compartment door and a partition plate in a fully open state. [Figure 6] 6 is a conceptual diagram showing a display example of a user setting screen in the control process of FIG. 5. [Figure 7] 13 is a flowchart illustrating a control process for image capture after installation of the foodstuff management application. [Figure 8] 13 is a flowchart illustrating a control process for managing inventory data of a vegetable room. [Figure 9] 13 is a diagram illustrating an example of a structure of vegetable inventory data. [Figure 10] 13 is a table illustrating an example of a structure of data on the number of days until the expiration date of vegetables; [Figure 11] 9 is a flowchart illustrating details of an update process of the inventory data in FIG. 8. [Figure 12] 13 is a flowchart illustrating a control process in a stock data display process in response to a user request. [Figure 13] FIG. 13 is a conceptual diagram illustrating an example of a display screen in the inventory data display process. [Figure 14] 1 is a conceptual diagram showing a functional block diagram of a foodstuff management system according to an embodiment of the present invention; [Figure 15] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a server. [Figure 16] 11 is a conceptual diagram illustrating a measurement area of ​​a distance measuring sensor in the open / close detection system according to the second embodiment. FIG. [Figure 17] FIG. 17 is a conceptual diagram illustrating details of a planar region shown in FIG. 16. [Figure 18] 10 is a flowchart illustrating a control process performed by the open / close detection system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following, the same or corresponding parts in the drawings are given the same reference characters, and their description will not be repeated in principle.

[0014] [Embodiment 1] FIG. 1 is a conceptual diagram illustrating an example of the configuration of a foodstuff management system 2 according to the present embodiment.

[0015] Referring to FIG. 1, the food ingredient management system 2 includes an imaging device 20 provided in a refrigerator 5, an interface device 30 for communicating with an external communication network, a portable terminal device 40 configured as a smartphone or a tablet terminal, etc., and a server 60 connected to the Internet 50.

[0016] The refrigerator 5 has a freezer door 7 (drawer type), refrigerator compartment doors 8a, 8b (openable and closable from side to side by a rotating mechanism), an ice-making compartment door 9a (drawer type), an auxiliary freezer compartment door 9b (drawer type), and a vegetable compartment door 11 (drawer type). The imaging device 20 is attached to the upper front side of the housing of the refrigerator 5.

[0017] The interface device 30 can be typically configured by a wireless LAN (Local Area Network) router. As described later, the imaging device 20 has a communication function with the interface device 30. This allows the imaging device 20 to transmit and receive data to and from a server 60 connected to the Internet 50 via the interface device 30.

[0018] It is also possible for the terminal device 40 to communicate with the server 60 via the Internet 50. When the terminal device 40 is within a range where it can communicate with the interface device 30, the terminal device 40 can communicate with the server 60 by connecting to the interface device 30 via wireless communication. When the terminal device 40 is outside the home, for example, the terminal device 40 can communicate with the server 60 by connecting to the Internet 50 via a base station (not shown).

[0019] An application program related to the food ingredient management system 2 can be installed in the terminal device 40. For example, the application program can be downloaded from the server 60 to the terminal device 40. As will be described later, in the food ingredient management system 2, when initial settings of a food ingredient management app for the vegetable compartment are performed, the user can view the list of ingredients in the vegetable compartment at any time.

[0020] FIG. 2 is a block diagram illustrating a schematic configuration of the imaging device 20. As shown in FIG.

[0021] As shown in FIG. 2, the imaging device 20 includes a camera 21, a distance measuring sensor (hereinafter referred to as a “distance measuring sensor”) 22, a controller 25, and a communication unit .

[0022] The camera 21 is positioned to capture an image downward from above the refrigerator 5. The distance measuring sensor 22 is configured to measure the distance to an object by emitting a measurement wave (e.g., infrared light) and receiving a reflected wave from the object that reflects the measurement wave.

[0023] Controller 25 is configured to be able to transmit and receive data to and from each of camera 21, distance measurement sensor 22, and communication unit 26. Controller 25 controls image capture by camera 21 based on measurements by distance measurement sensor 22. Specifically, controller 25 detects whether vegetable compartment door 11 is open or closed based on measurements by distance measurement sensor 22, and can control camera 21 to capture images of vegetable compartment 10 from above at predetermined time intervals while door 11 is open.

[0024] Controller 25 receives data of images captured by camera 21 together with image capture date and time data (data on the image capture date and time). Controller 25 transmits data of at least a portion of the multiple images received from camera 21 to the outside of imaging device 20 via communication unit 26.

[0025] The controller 25 is configured by, for example, a microcomputer so as to execute the above-mentioned process of detecting whether the door 11 is open or closed, the process of controlling the camera 21, and the process of transmitting image data using the communication unit 26 by executing a program stored in advance.

[0026] In the configuration example of Figure 2, the controller 25 and the communication unit 26 are shown as separate blocks from the camera 21, but it is also possible to integrate the controller 25 and the communication unit 26 into an integrated unit by mounting them on the control board of the camera 21.

[0027] The communication unit 26 is connected to the interface device 30 (FIG. 1) by wireless communication or wired communication. The communication unit 26 can be typically configured by a communication module having a function of wireless LAN communication (for example, Wi-Fi (registered trademark) communication) with the interface device 30. This allows the controller 25 (imaging device 20) to transmit and receive data to and from a server 60 and a terminal device 40 (FIG. 1) connected to the Internet 50 via the interface device 30 (FIG. 1). This allows data of an image of the vegetable compartment 10 captured by the camera 21 to be transmitted to the server 60.

[0028] Fig. 3 is a conceptual side view of the refrigerator 5 for explaining the function of the imaging device 20. The left-right direction in Fig. 3 corresponds to the opening and closing direction of the door 11 of the vegetable compartment 10 from the front side of the refrigerator 5.

[0029] As shown in Fig. 3, by opening the drawer door 11, the inside of the vegetable compartment 10 moves into the field of view of the camera 21. The vegetable compartment 10 is provided with a sliding partition tray 12 for dividing the vegetable compartment 10 into an upper level 15 and a lower level 16. The partition tray 12 is configured to have a horizontal surface 12a and a vertical surface 12b. The partition tray 12 can be opened and closed by the user operating the vertical surface 12b in the left-right direction in Fig. 3.

[0030] In the following, the distance Xdr from the reference plane Prf, which corresponds to the position of the doors 7, 8a, 8b, 9a, 9b, and 11 in the closed state, to the door 11 is defined as the opening degree of the door 11, and the distance Xtr from the reference plane Prf to the vertical surface 12b of the partition tray 12 is defined as the opening degree of the partition tray 12.

[0031] Basically, the partition tray 12 is opened and closed integrally with the door 11. Therefore, when the door 11 is in a fully closed state (Xdr=0), the partition tray 12 is also stored integrally, resulting in a state of Xtr=0. When the door 11 is opened to the right in FIG. 3 (Xdr>0), the partition tray 12 is also opened integrally with the door 11. In this state, the upper level 15 of the vegetable compartment 10 is open to the user. Furthermore, when the user closes the partition tray 12 toward the refrigerator (to the left in FIG. 3), the lower level 16 of the vegetable compartment 10 is opened.

[0032] A reference plane Pcm (measurement distance lx=0) of the distance measuring sensor 22 is determined according to the placement position (vertical direction) of the distance measuring sensor 22 on the upper part of the housing of the refrigerator 5. The distance from the reference plane Pcm to the floor surface Pfl on which the refrigerator 5 is placed is lf. In contrast, the distance from the reference plane Pcm to the upper surface of the vegetable compartment 10 when the door 11 is open is ld.

[0033] When the door 11 is in a closed state, the measurement wave output from the distance measuring sensor 22 is reflected by the floor surface Pfl. By receiving the reflected wave from the floor surface Pfl, the measured distance by the distance measuring sensor 22 becomes Ix=lf.

[0034] On the other hand, when the door 11 is open, the measurement wave output from the distance measuring sensor 22 is reflected by the components of the crisper 10 or the stored food ingredients. Therefore, the measurement distance Ix by the distance measuring sensor 22 is close to ld, which is shorter than the distance lf to the floor. Therefore, it can be understood that it is possible to detect whether the door 11 is in the open state or the closed state based on the measurement distance lx by the distance measuring sensor 22.

[0035] As shown in Figure 3, when door 11 is open, an image can be obtained by capturing an image of the inside of vegetable compartment 10 from the direction "IV" in Figure 3 (i.e., from the top side) using camera 21 whose field of view includes the area indicated by the dotted line.

[0036] FIG. 4 shows a plan view of the vegetable compartment 10 as seen from the direction IV in FIG.

[0037] As can be seen from Figure 4, an image of the vegetable compartment 10 taken from direction IV will be either an image of the upper tier 15, an image of the lower tier 16, or an image including a portion each of the upper tier 15 and the lower tier 16, depending on the horizontal position of the partition tray 12 (vertical surface 12b).

[0038] In this embodiment, the opening degree of the door 11 (hereinafter referred to as the door opening degree) and the opening degree of the partition tray 12 (hereinafter referred to as the tray opening degree) are indicated as a percentage, with the fully open state being 100(%).

[0039] For example, when the above-mentioned foodstuff management application is initially set up, user guidance can be executed to acquire in advance an image of the door 11 and partition tray 12 in a fully open state.

[0040] Fig. 5 is a flow chart for explaining an example of a control process for acquiring in advance the opening degree of the fully open state of the vegetable compartment door 11 and the partition tray 12. Fig. 6 is an example of a user guidance screen displayed in the control process shown in Fig. 5.

[0041] 5 can be executed as part of initial settings when installing a food ingredient management application program from the terminal device 40, involving communication between the terminal device 40 and the imaging device 20 and the server 60. The user guidance screen can be output to a display of the terminal device 40.

[0042] 5, in step (hereinafter simply referred to as "S") 51, terminal device 40 executes shooting guidance to display user guidance screen 71a of Fig. 6(a). User guidance screen 71a is formed to include an image 72 of the field of view of camera 21, a guidance message 73a that urges the user to fully open door 11 and partition tray 12, i.e., to fully open the upper level of vegetable compartment 10, and a touch button 74 for inputting a shutter command to camera 21.

[0043] Returning to FIG. 5, when the user follows the guidance message 73a to fully open the upper level of the vegetable compartment 10 and then operates the touch button 74 (FIG. 6(a)), the imaging device 20 acquires a reference image of the upper level of the vegetable compartment 10 when the door 11 and the partition tray 12 are fully open (opening degree 100(%)) in S52.

[0044] When the terminal device 40 is notified by the imaging device 20 that the upper reference image has been acquired, it executes imaging guidance to display a user guidance screen 71b in Fig. 6(b) in S53. The user guidance screen 71b is formed to include an image 72 of the field of view of the camera 21, a guidance message 73b that urges the user to fully open the door 11 and fully close the partition tray 12 (opening degree 0(%)), i.e., to fully open the lower level of the vegetable compartment 10, and a touch button 74 for inputting a shutter command to the camera 21.

[0045] Returning to FIG. 5, when the user follows the guidance message 73b to fully open the lower level of the vegetable compartment 10 and then operates the touch button 74 (FIG. 6(b)), S54 causes the imaging device 20 to acquire a reference image of the lower level of the vegetable compartment 10 when the door 11 is fully open (opening degree 100(%)) and the partition tray 12 is fully closed (opening degree 0(%)).

[0046] In S55, the imaging device 20 transmits the upper and lower reference images acquired in S52 and S54 to the server 60 together with the identification information of the refrigerator 5 (refrigerator ID).

[0047] In S56, the server 60 receives the upper reference image and the lower reference image from the imaging device 20. In S57, the server 60 performs image processing on the received upper reference image and lower reference image to obtain reference opening data Xr equivalent to the distances Xdr, Xtr (FIG. 3) corresponding to an opening of 100(%). In S55, a machine-learned model for image recognition can be used to recognize at least the door 11 and the partition tray 12 (vertical surface 12b) from the image of the vegetable compartment 10 captured by the camera 21 and to determine the distances Xdr, Xtr in FIG. 3.

[0048] In the following description, a common reference opening degree data Xr is used for the door opening degree and the tray opening degree, but it is also possible to obtain reference opening degree data separately for each of the door opening degree and the tray opening degree.

[0049] The server 60 checks in S58 whether the reference opening data Xr has been normally acquired, and if normal (YES in S58), instructs the terminal device 40 to output a normal end message in S59. This ends the control process for acquiring the opening degree of the vegetable compartment door and partition tray in the fully open state. On the other hand, if the reference opening data Xr has not been acquired (NO in S58), the process returns to S51 (or S53) to reacquire the upper reference image and / or the lower reference image, thereby making it possible to retry acquiring the reference opening data Xr.

[0050] Next, a control process for capturing an image of the vegetable compartment 10 after the food management application is installed (after the initial settings are completed) will be described with reference to Fig. 7. The control process shown in Fig. 7 can be repeatedly executed by the controller 25 of the imaging device 20.

[0051] In the initial state (start) of FIG. 7, the door 11 of the vegetable compartment 10 is in a fully closed state, and the measured distance lx of the distance measuring sensor 22 is lf. In the initial state, the controller 25 repeatedly executes the opening of the door 11, i.e., the opening detection process of the vegetable compartment 10, based on the measured distance lx of the distance measuring sensor 22 in S110. Specifically, when the measured distance lx continues to be within a predetermined range including the distance ld from the reference plane Pcm to the top surface of the vegetable compartment 10 for a certain period of time (for example, about 2 to 3 seconds), the opening of the vegetable compartment 10 is detected, and a YES judgment is made in S110. While a NO judgment is made in S110, the judgment in S110 is repeatedly executed.

[0052] When controller 25 detects that vegetable room 10 (door 11) is open (YES in S110), in S120, camera 21 photographs vegetable room 10 at regular intervals (for example, one second). As a result, images of vegetable room 10 from direction IV in FIG. 3 are acquired at regular intervals.

[0053] After detecting the opening of the vegetable compartment 10, the controller 25 repeatedly executes the process of detecting the closing of the vegetable compartment 10 (door 11) by S130. Specifically, when the measured distance lx returns to the distance lf from the reference plane Pcm to the floor surface, the closing of the vegetable compartment 10 (door 11) is detected. When the closing of the vegetable compartment 10 is detected (YES determination in S130), the controller 25 stops the image capturing (S120) at regular intervals by the camera 21 by S140. As a result, during the period from when YES is determined in S110 to when YES is determined in S130, that is, from when the vegetable compartment 10 is opened once to when it is closed, images of the vegetable compartment 10 can be captured sequentially at regular intervals (for example, every second).

[0054] When the camera 21 stops capturing images (S140), the controller 250 extracts, at S150, a predetermined N images (N is a natural number equal to or greater than 2) captured last from the multiple images of the vegetable compartment 10 captured at regular intervals by S120. This extracts the N images captured immediately before the vegetable compartment 10 is closed. When closing the door 11 of the vegetable compartment 10, the user generally pushes the door 11 from the outside, so it is understood that the N images increase the possibility of capturing images of all of the ingredients in the vegetable compartment 10 without capturing the user's hand.

[0055] In S160, the controller 25 transmits the N images of the crisper 10 extracted in S150 to the server 60 together with the identification information (refrigerator ID) of the refrigerator 5 and the photographing date and time data. The N images are either an image of the upper tier 15, an image of the lower tier 16, or an image of a portion of each of the upper tier 15 and the lower tier 16, depending on the position of the partition tray 12 (vertical surface 12b) at the time of photographing.

[0056] When S160 ends, the controller 25 returns to the initial state (start) and waits for the next opening of the vegetable compartment 10. By repeating the control process in FIG. 7, every time the vegetable compartment 10 (door 11) is opened, N images immediately before the closing operation are associated with the refrigerator ID and the photographing date and time, transmitted from the imaging device 20 to the server 60, and stored in the server 60.

[0057] Fig. 8 shows a flowchart explaining a control process for managing inventory data of the vegetable compartment 10. The control process in Fig. 8 is started by the server 60 periodically (for example, once a day at a predetermined time) for each refrigerator ID.

[0058] At the start of the control process of Fig. 8, the server 60 stores a plurality of images of the vegetable room 10 transmitted from the imaging device 20 after the previous control process of Fig. 8 is completed. If there is no stored image, the start of the control process of Fig. 8 is skipped even if the above-mentioned predetermined time arrives.

[0059] When the control process of FIG. 8 is started, the server 60 sets the update completion flag Fu of the upper row 15 and the update completion flag Fl of the lower row 16 to the initial value "0" indicating no update (Fu=Fl=0) in S210.

[0060] In S220, the server 60 extracts one image with the most recent photographed date and time from the stored images, and processes the image in S230. In S230, a trained model for image recognition that has been machine-learned can be used to recognize the distances Xdr, Xtr (FIG. 3) between the door 11 and the partition tray 12 (vertical surface 12b) from the photographed image of the vegetable compartment 10, and the presence or absence of each of a predetermined M (M is a natural number of 2 or more) items of ingredients (typically vegetables). This allows the recognition result of presence or absence (i.e., "presence or absence information") for each ingredient (vegetable) to be obtained.

[0061] As a result, in S230, the door opening degree and tray opening degree, and presence / absence data indicating the recognition results (presence / absence) of each of the M items of ingredients (vegetables) are obtained for the captured image of the vegetable compartment 10 extracted in S220. As described above, the door opening degree and tray opening degree are indicated by the ratio (%) of the distances Xdr, Xtr obtained by image recognition to the reference opening degree data Xr.

[0062] In S240, the server 60 determines whether the door opening degree (%) exceeds a predetermined judgment value γ (for example, 90 (%)). In an image in which the door opening degree is equal to or less than the judgment value (NO in S240), it is highly likely that only a portion of the food in the vegetable compartment 10 is shown, so the subsequent steps S242 to S275 for updating the inventory data are skipped and the process proceeds to S280.

[0063] In an image in which the door opening degree is greater than the judgment value γ (YES judgment in S240), the tray opening degree (%) is compared with the judgment value α (e.g., α = 90 (%)) of the upper image and the judgment value β (e.g., β = 10 (%)) of the lower image in S242 and S245.

[0064] If the tray opening degree (%) is greater than the judgment value α (YES in S242), the image extracted in S220 is recognized as the "upper tier image" in S250. Furthermore, in S255, it is confirmed whether the update completion flag Fu of the upper tier 15 is 0 or not. When Fu=0, that is, when the inventory data of the upper tier 15 has not been updated (YES in S255), an update process of the inventory data of the upper tier 15 is executed in S300u. When the update process of S300u is executed, the update completion flag Fu of the upper tier 15 is changed from "0" to "1" in S270. On the other hand, when Fu=1 in S255 (NO in S255), that is, when the inventory data of the upper tier 15 has already been updated after the control process of FIG. 8 is started, S300u and S270 are skipped and the process proceeds to S280.

[0065] If the tray opening degree (%) is smaller than the judgment value β (YES in S245), the image extracted in S220 is recognized as the "lower image" in S260. Furthermore, in S265, it is confirmed whether the update completion flag Fl of the lower tier 16 is 0 or not. When Fl=0 (YES in S265), that is, when the inventory data of the lower tier 16 has not been updated, an update process of the inventory data of the lower tier 16 is executed in S300l. When the update process of S300l is executed, the update completion flag Fl of the lower tier 16 is changed from "0" to "1" in S275. On the other hand, when Fl=1 in S255 (NO in S265), that is, when the inventory data of the lower tier 16 has already been updated after the control process of FIG. 8 is started, S300l and S275 are skipped and the process proceeds to S280.

[0066] Also, if the tray opening degree (%) is between the judgment value α and the judgment value β (NO judgment in both S242 and S245), the image extracted in S220 is judged to be neither the "upper row image" nor the "lower row image". In this case, the process skips S300u, S331, S270, and S275 and proceeds to S280. That is, the image is not used to update the inventory management data of the upper row 15 and the lower row 16, and the update completion flags Fu and Fl are maintained at their current values. The judgment value α of the upper row image corresponds to one embodiment of the "first threshold value", and the judgment value β of the lower row image corresponds to one embodiment of the "second threshold value".

[0067] In S280, the server determines whether both update completion flags Fu and Fl have been changed to "1", i.e., whether the inventory data of the upper shelf 15 and the lower shelf 16 have been updated. When Fu=Fl=1, i.e., when the inventory data of both the upper shelf 15 and the lower shelf 16 have been updated (YES in S280), the server 60 ends the control process for managing the inventory data for the day in the vegetable compartment 10 of the refrigerator 5 identified by the refrigerator ID in S290.

[0068] On the other hand, if at least one of the update completion flag Fu of the upper row 15 and the update completion flag Fl of the lower row remains at "0" (NO in S280), the inventory data of at least one of the upper row 15 and the lower row 16 has not been updated. In this case, the process returns to S220 to extract the next image. In S220, an image with the most recent shooting date and time is extracted from the unextracted images among the multiple accumulated images, and the above-mentioned processes of S230 to S280 are performed on this newly extracted image.

[0069] Moreover, when there are no unextracted images remaining even when the process returns to S220, the process automatically proceeds to S290, and the control process for managing the inventory data for the day in the refrigerator 5 (vegetable compartment 10) is terminated.

[0070] Next, the structure of inventory data for ingredients (vegetables) and the update process will be described with reference to Figs.

[0071] FIG. 9 is a diagram showing an example of the structure of an ingredient inventory data table.

[0072] As shown in FIG. 9, an inventory data table 100U in the upper stage 15 and an inventory data table 100L in the lower stage 16 are provided separately, but the data structures of the inventory data tables 100U and 100L are the same.

[0073] Each of the inventory data tables 100U and 100L stores stock availability data (0 / 1), purchase date data, and expiration date data for each of the M vegetable items that are to be recognized in the image processing in S230 of FIG.

[0074] In the example of FIG. 9, spinach and broccoli are present among the M vegetable items, and purchase date data Db and expiration date data FDd are stored for spinach, which is recognized as being in stock (presence data="1") Similarly, purchase date data Dd and expiration date data FDd are stored for broccoli. On the other hand, no purchase date data or expiration date data is stored for vegetable items recognized as being out of stock (presence data="0")

[0075] The expiration date data is calculated by adding predetermined expiration date number of days data Dx to the purchase date stored in the inventory data tables 100U and 100L.

[0076] FIG. 10 shows an example of the structure of expiration date data for vegetables.

[0077] 10, the expiration date table 101 is configured to store expiration date data Dx for each of M vegetable items. The expiration date table 101 is created by writing in advance the values ​​of Dx=A1 (days) to AM (days) for M vegetable items.

[0078] The expiration date table 101 may be configured so that different values ​​are referenced for the values ​​A1 to AM of the expiration date data Dx depending on the season.

[0079] 9 and 10, the expiration date data FDb of spinach is set to indicate a date A2 (days) after the purchase date by adding the expiration date data Dx(A2) of spinach in the expiration date table 101. Similarly, the expiration date data FDd of broccoli is set to indicate a date A4 (days) after the purchase date by adding the expiration date data Dx(A4) of broccoli in the expiration date table 101 to the purchase date data Dd.

[0080] Fig. 11 shows a flowchart for explaining details of the inventory data update process in each of S300u and 300l in Fig. 8. In S300u, the inventory data table 100U is updated, while in S300l, the inventory data table 100L is updated, but the process for the inventory data table 100U or 100L is the same between S300u and S300l.

[0081] 11, when executing S300u and S300l, the server 60 initializes a parameter value i (i is an integer) to "0" in S310. Furthermore, the server 60 determines, in S320, the difference between the presence / absence data of the M vegetable items obtained in S230 (FIG. 8) and the presence / absence data of the M vegetable items in the data table to be updated, which is one of the inventory data tables 100U and 100L. As a result, in S320, K vegetable items (k is an integer between 0 and M) for which a difference has occurred in the presence / absence data are extracted from the M vegetable items.

[0082] The server 60 executes the processes of S330 to S380 for each of the K vegetable items. After the server 60 increments the parameter value i by 1 in S330, the server 60 branches the process for the i-th vegetable item among the K vegetable items extracted in S320 depending on whether the vegetable item has been purchased or consumed.

[0083] When the presence / absence data in the data table to be updated changes from "0" to "1" obtained in S230 (FIG. 8) (YES in S340), the server 60 determines that the i-th vegetable item has been purchased, and proceeds to S350 and S360. In S350, the presence / absence data of the vegetable item (i-th) in the data table to be updated is rewritten from "0" to "1," and the image capture date of the image processing target in S230 is stored as purchase date data in the data table to be updated. Furthermore, in S360, the expiration date data is calculated by adding the value (number of days) of the expiration date date data Dx stored in the expiration date date table 101 (FIG. 11) to the purchase date data stored in S350. The purchase date data and expiration date data obtained in S350 and S360 are stored in the item of the vegetable item in the inventory data table 100U or 100L, which is the table to be updated.

[0084] In response to this, if the presence / absence data in the data table to be updated changes from "1" to "0" obtained in S230 (FIG. 8) (NO in S340), the server 60 determines that the i-th vegetable item has been consumed and proceeds to S370. In S370, the presence / absence data for the vegetable item (i-th) is rewritten from "1" to "0" in the inventory data table 100U or 100L, which is the data table to be updated, and the purchase date data and expiration date data are cleared.

[0085] After the processing of S350, S360 (at the time of purchase) or S370 (at the time of consumption), the server 60 compares, by S380, the current value of the parameter value i with the number K of food item types extracted at S320. When i < K (when the determination at S380 is NO), the server 60 returns the processing to S330. As a result, the parameter value i is incremented by 1, and the processing of S340 to S380 is executed again. Finally, for each of the K food item types extracted at S320, the processing of S340 to S370 is executed.

[0086] When i ≥ K at S380 (when the determination at S380 is YES), since the processing of S340 to S370 has been completed for all of the K food item types extracted at S320, the processing proceeds to S390. As a result, the processing of S300u or S300l in FIG. 8 (the inventory table update processing) is completed. After this, the processing returns to FIG. 8, and S270 or S275 is executed.

[0087] Next, with reference to FIGS. 12 and 13, the inventory data display processing will be described.

[0088] FIG. 12 is a flowchart for explaining the control processing in the inventory data display processing. The control processing shown in FIG. 12 is activated when a browsing request for the list of foodstuffs (vegetables) in the refrigerator 5 (vegetable compartment 10) input to the terminal device 40 by a user's screen touch processing or the like for the terminal device 40 is transmitted to the server 60.

[0089] When the inventory data display processing is activated, the server 60 reads out, by S410, the upper inventory data table 100U in the upper stage 15 and the lower inventory data table 100L in the lower stage 16 corresponding to the refrigerator 5.

[0090] Furthermore, in S420, the server 60 extracts, based on the read inventory data table 100U, inventory vegetable data, which is data on vegetable items whose presence / absence data is "1" in the upper row 15. Similarly, in S430, based on the read inventory data table 100L, inventory vegetable data on vegetable items whose presence / absence data is "1" is extracted in the lower row 16.

[0091] In S440, the server 60 generates screen display data for the terminal device 40 using the stock vegetable data extracted in S420 and S430, and transmits the data to the terminal device 40. The stock vegetable data includes vegetable items, data for distinguishing between the upper row 15 and the lower row 16, purchase date data, and expiration date data.

[0092] FIG. 13 shows an example of a display screen based on the screen data transmitted in S440.

[0093] 13, a display screen 150 during the inventory data display process includes switching tags 152 for upper and lower levels 15 and 16, a photographed image 153 of crisper 10, and an inventory data list 160 of ingredients (vegetables).

[0094] As the captured image 153, the images used in the update process of the inventory data tables 100U, 100L in S300u, S300l (FIG. 8) are displayed together with the capture date and time of the images for each of the upper and lower rows 15, 16. At least the captured image 153 is switched in response to the operation of the switching tag 152.

[0095] The inventory data list 160 is configured to display the area (upper / lower), item, purchase date, and expiration date for the inventory vegetable data 165 extracted in S420 and S430. When each of the inventory vegetable data 165 is selected by tapping, an image of the vegetable item corresponding to the data is displayed in the photographed image 153 surrounded by a frame 155.

[0096] Furthermore, the display screen 150 is configured so that the purchase date data 170 included in each of the stock vegetable data 165 can receive manual correction input accompanied by selection by tapping. In this embodiment, since the stock data is managed by the control process shown in Fig. 8 and Fig. 11, the purchase date cannot be automatically updated when the same type of vegetable is purchased before it is completely consumed. By enabling manual correction of the purchase date data 170, it is possible to deal with the above-mentioned cases and improve usability for the user.

[0097] 13, the vegetable stock data 165 of the upper row 15 and the lower row 16 are displayed together on one screen. However, depending on the number of vegetable stock data 165, it is also possible to form the display screen 150 so that only the vegetable stock data 165 of the upper row 15 or the vegetable stock data 165 of the lower row 16 is displayed in response to the selection of the switching tag 152.

[0098] The display screen 150 of the terminal device 40 can be switched by operating a touch switch at the bottom of the screen. For example, when an ingredient management app for the refrigerator 5 is started on the terminal device 40 and the display screen 150 is output, the control process of Fig. 12 (processing for displaying inventory data of the vegetable compartment 10) may be started by operating the touch switch 151.

[0099] 12 again, server 60 checks in S450 whether or not the user has input a correction to purchase date data 170 on display screen 150 (FIG. 13) of terminal device 40. If the user has input a correction (YES in S450), server 60 updates and stores the purchase date data in inventory data table 100U or 100L according to the input date. On the other hand, if the user has not input a correction (NO in S450), the purchase date data correction process in S460 is skipped.

[0100] The server 60 determines in S470 whether or not a closing operation for the display screen 150 has been input to the terminal device 40. When information indicating that the closing operation has been input to the terminal device 40 is transmitted to the server 60, a YES determination is made in S470, and the processing related to the output of the display screen 150 by the server 60 is terminated. Meanwhile, the server 60 repeatedly executes the processing of S440 to S470 until the closing process is recognized (NO4 determination in S470) in order to be able to accept a user correction input in S450.

[0101] In this way, according to the food ingredient management system 2 of the present embodiment and the food ingredient management method executed by the food ingredient management system 2, it is possible to manage inventory vegetable data 165 that distinguishes between the upper level 15 and the lower level 16 of the vegetable compartment 10 divided by the partition tray 12, using an image that is automatically captured when the user uses the vegetable compartment 10. This realizes a technology that manages food ingredients (vegetables) stored in the vegetable compartment 10 of the refrigerator 5 by distinguishing between the upper level 15 and the lower level 16, based on an image that is automatically captured from above the vegetable compartment 10.

[0102] FIG. 14 shows a conceptual diagram in which the ingredient management system 2 according to this embodiment is expressed in functional blocks.

[0103] 14, the foodstuff management system 2 includes an imaging section 110, an image processing section 120, an image determination section 130, an update processing section 140 for updating inventory data 142, and a display processing section 145.

[0104] The imaging unit 110 corresponds to the imaging device 20 described above, and is provided on the upper part of the refrigerator 5 and positioned so as to include the inside of the vegetable compartment 10 within its field of view when the door 11 of the vegetable compartment 10, which has a sliding partition tray 12 that separates an upper level 15 and a lower level 16, is opened.

[0105] The image processing unit 120 performs image processing on the image of the vegetable compartment 10 in the open state captured by the imaging unit 110, and acquires information on whether a predetermined number of items (M items) of vegetables are recognized in the image, and an opening degree value indicating the position of the partition tray 12. The function of the image processing unit 120 is realized by the processing of S230 in Fig. 8. That is, S230 in Fig. 8 corresponds to one embodiment of the "acquiring step".

[0106] The image determination unit 130 determines whether each image is an image of the upper tier 15 or the lower tier 16 based on the opening degree value of the partitioned tray 12 obtained by the image processing unit 120. The inventory data 142 includes data on the presence or absence of each of the multiple items of vegetables for each of the upper tier 15 and the lower tier 16, and purchase date data for the vegetables that are recognized as being present. The function of the image determination unit 130 is realized by the processes of S242, S245, S250, and S260 in Fig. 8. These steps correspond to one embodiment of the "determining step".

[0107] The update processing unit 140 updates the inventory data 142 based on the determination result of the image determination unit 130 and information on whether multiple items (M items) of vegetables are recognized in each image. The inventory data 142 corresponds to a comprehensive representation of the data stored in the inventory data tables 100U and 100L (FIG. 9).

[0108] Specifically, when the inventory data of a first item of vegetables among the multiple items changes from absent to present based on information on whether multiple items (M items) of vegetables are recognized in the first image, update processing unit 140 records the photographing date of the first image as the purchase date of the first item of vegetables. The function of update processing unit 140 is realized by the control processes of S300u and S300l in Fig. 8 (i.e., the control process in Fig. 11). These steps correspond to one embodiment of the "updating step".

[0109] In response to a user request, the display processing unit 145 generates a display screen 150 (FIG. 13) for displaying each item that is recognized as having vegetables among a plurality of items in association with the purchase date of the vegetable, based on the inventory data 142 at the time the user request was generated. The function of the display processing unit 145 is realized by the processing of S410 to S470 shown in FIG. 12. These steps correspond to one embodiment of the "step of generating (display screen data)".

[0110] 1 to 13, an example has been described in which the functions of the image processing unit 120, the image determination unit 130, the update processing unit 140, and the display processing unit 145 are realized by software processing of the server 60. Therefore, an example of the hardware configuration of the server 60 will also be described with reference to FIG.

[0111] 15, the server 60 includes, as its main components, a processor 61 that executes a program, a ROM (Read Only Memory) 62 that stores data in a non-volatile manner, a RAM (Random Access Memory) 63 that volatilely stores data generated by the execution of the program by the processor 61 or input data, a HDD (Hard Disk Drive) 64 that stores data in a non-volatile manner, a communication IF (Interface) 65, an input device 66, a power supply circuit 67, and a display 68. The components are mutually connected by a data bus. The communication IF 65 is an interface for communicating with other devices.

[0112] The processing in the server 60 is realized by each piece of hardware and software (programs) executed by the processor 61. Such software may be pre-stored in the HDD 64. The software may also be stored in other storage media and distributed as a program product. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet. Such software is read from the storage medium by a reading device or downloaded via the communication IF 65 or the like, and then temporarily stored in the HDD 64. The software is read from the HDD 64 by the processor 61 and stored in the RAM 63 in the form of an executable program. The processor 61 executes the program.

[0113] Each component of the server 60 shown in Fig. 15 is a general one. Therefore, it can be said that the essential part of the present invention is the RAM 63, the HDD 64, the software stored in the storage medium, or the software downloadable via the network. Note that the operation of each piece of hardware of the server 60 is well known, so detailed description will not be repeated.

[0114] Furthermore, the food ingredient management system 2 according to this embodiment can be modified as appropriate so that some of the functions described as processing by the server 60 are executed by the terminal device 40 and / or the imaging device 20. Alternatively, the server 60 may be executed by multiple servers in a distributed arrangement. In other words, the entities that execute the functions constituting the food ingredient management system 2 are not limited to the examples given above, and any system design can be adopted.

[0115] [Embodiment 2] In the second embodiment, the system for detecting whether the door 11 of the vegetable compartment 10 is opened or closed using the distance measuring sensor 22 will be further described.

[0116] The open / close detection system according to the second embodiment is positioned as a preferred aspect of the process of detecting whether the door 11 is open or closed (S110, S130 in FIG. 7) among the functions of the imaging device 20 in the first embodiment.

[0117] FIG. 16 is a conceptual diagram illustrating a measurement area of ​​a distance measuring sensor in the open / close detection system according to the second embodiment.

[0118] 16, in the open / close detection system according to the second embodiment, distance measuring sensor 22 has a measurable area on the front side of refrigerator 5. The measurable area includes both an area where refrigerator compartment doors 8a and 8b are assumed to be located when opened by a rotating mechanism, and an area where door 11 of vegetable compartment 10 and doors 7, 9a and 9b (freezer and ice compartment) are assumed to be located when opened by a sliding mechanism. These areas correspond to areas where it is assumed that the measurement wave from distance measuring sensor 22 is received when each door is in an open state.

[0119] The distance measuring sensor 22 is configured to divide a planar area 200 within a measurable area into a plurality of mesh areas 201, and to be able to measure the distance to an object that reflects a measurement wave for each mesh area 201.

[0120] In Fig. 16, in order to illustrate the mesh division, the planar area 200 is shown as being inclined rather than perpendicular to the door surface (reference plane Prf: Fig. 3) of the refrigerator 5. However, in reality, the planar area 200 can be defined as being perpendicular to the door surface of the refrigerator 5. The distance measuring sensor 22 can measure the distance for each of the mesh areas 201 by emitting a plurality of measurement waves so that the measurement waves pass through each of the mesh areas 201.

[0121] FIG. 17 is a conceptual diagram illustrating the details of the planar area 200 shown in FIG.

[0122] Referring to FIG. 17, the plurality of mesh regions 201 obtained by dividing the planar region 200 are classified into a region 210 where the door 11 of the vegetable compartment 10 is assumed to be located when it is open, and regions 220 and 230 where the doors 8a and 8b by the rotation mechanism are assumed to be located when they are open. In the opening / closing detection system according to Embodiment 2, the measurement distances for each of the regions 210, 220, and 230 are classified, and the opening / closing detection of each door is executed. Note that the regions 210, 220, and 230 can be defined in any shape and size according to the positions and shapes of the doors of the refrigerator compartment and the vegetable compartment. The region 230 corresponds to an example of the "predetermined region".

[0123] FIG. 18 is a flowchart for explaining the control process by the opening / closing detection system according to Embodiment 2. FIG. 18 illustrates the control process for detecting the opening / closing of the door 11 of the vegetable compartment 10. The control process in FIG. 18 can be executed by the controller 25 of the imaging device 20.

[0124] Referring to FIG. 18, the controller 25 acquires, in S510, the measurement distance lx in each mesh region 201 included in the region 210 in FIG. 17. Further, in S520, it is determined whether the open state of the door 11 is detected from the acquired measurement distance lx in each mesh region 201.

[0125] For example, in S520, when the measurement distance lx is within the range of ld - < lx < ld + in a predetermined number or more of the mesh regions 201 within the region 210, S520 determines YES. At this time, it can be detected by S530 that the door 11 is in the open state. Here, the constants ld - and ld + can be set to ld - = ld - ε and ld + = ld + ε with respect to the distance ld (FIG. 3) from the reference plane Pcm to the upper surface of the vegetable compartment 10.

[0126] On the other hand, when the above condition is not satisfied in S520, S520 determines NO, and it can be detected by S540 that the door 11 is in the closed state.

[0127] It should be noted that the determination method in S520 is not limited to the above example. For example, by storing the measurement distance lx in each mesh region 201 when the reference image was captured in the initial setting described in Figures 5 and 6 as a reference pattern, it is possible to execute the determination in S520 by pattern matching processing between the distribution of the measurement distance lx of each mesh region 201 in the region 210 in the reference pattern and the distribution of the measurement distance lx of each mesh region 201 in the region 210 acquired in S510. In this case, when the degree of match quantitatively obtained by the pattern matching processing is higher than the determination value, S520 can be determined as YES, and otherwise, S520 can be determined as NO.

[0128] In this way, according to the open / close detection system of the second embodiment, the distance measuring sensor 22 measures the distance for each of the mesh areas 201, and open / close detection can be performed using only the measured distance in the mesh area 201 where the door to be detected for open / close (for example, the door 11 of the vegetable compartment 10) is located when it is in the open state. This can improve the accuracy of open / close detection.

[0129] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0130] 2 food management system, 5 refrigerator, 7, 8a, 8b, 9a, 9b door (refrigerator), 10 vegetable compartment, 11 door (vegetable compartment), 12 partition tray, 12a horizontal surface (tray), 12b vertical surface, 15 upper level, 16 lower level, 20 imaging device, 21 camera, 22 distance measurement sensor, 25 controller, 26 communication unit, 30 interface device, 40 terminal device, 50 Internet, 60 server, 61 processor, 62 ROM, 63 RAM, 64 HDD, 65 communication IF, 66 input device, 67 power supply circuit, 68 display, 71a, 71b user guidance screen, 72, 72a image, 73a, 73b guidance message, 74 touch button, 100L, 100U inventory data table, 101 expiration date table, 110 imaging unit, 120 Image processing unit, 130 image determination unit, 140 update processing unit, 142 inventory data, 145 display processing unit, 150 display screen, 151 touch switch, 152 switching tag, 153 captured image, 160 inventory data list, 165 inventory vegetable data, 170 purchase date data, 200 flat area, 201 mesh area, 210, 220, 230 area, 250 controller, Dx expiration date data, Fl, Fu update completion flag, Ix measured distance (distance measurement sensor), Pfl floor surface, Prf reference surface (door surface).

Claims

1. An imaging unit that is provided at the top of the refrigerator and is arranged so as to include the vegetable compartment in its field of view when a door of the vegetable compartment having a sliding partition tray that divides the vegetable compartment into an upper and lower compartments is opened; An image processing unit that performs image processing on an image of the vegetable compartment in an open state captured by the imaging unit to obtain information on whether or not each of a plurality of predetermined items of vegetables is recognized in the image and an opening degree value indicating the position of the partition tray; an image determination unit that determines whether each of the images is an image of the upper section or an image of the lower section based on the opening degree value obtained by the image processing unit; an update processing unit that updates inventory data including data on the presence or absence of each of the plurality of items of vegetables for each of the upper and lower stages based on a determination result of the image determination unit and the presence or absence information in each of the images, In the food ingredient management system, when the presence / absence data for a first item of the multiple items, a vegetable, changes from absent to present based on the presence / absence information in a first image, the update processing unit records the photographing date of the first image in the inventory data as data on the purchase date of the first item of vegetable.

2. The food ingredient management system of claim 1, further comprising a display processing unit that generates, in response to a user request, display screen data for displaying each item among the plurality of items for which vegetables are recognized to be available in association with the purchase date of the vegetable of that item, based on the inventory data at the time the user request was generated.

3. The food ingredient management system of claim 2, wherein the display processing unit generates the display screen data so as to further display, together with the purchase date, a date that is a predetermined number of days after the purchase date corresponding to each of the plurality of items as an expiration date.

4. the image determination unit determines that an image having an opening degree value larger than a first threshold value is an image of the upper section, while determining that an image having an opening degree value smaller than a second threshold value set smaller than the first threshold value is an image of the lower section, A food ingredient management system as described in any one of claims 1 to 3, wherein the update processing unit updates the inventory data for the upper tier using images determined by the image determination unit to have been taken of the upper tier, while updating the inventory data for the lower tier using images determined by the image determination unit to have been taken of the lower tier, and the presence / absence information of images whose opening value is greater than or equal to the second threshold and less than or equal to the first threshold is not used to update the inventory data.

5. The food management system of claim 4, wherein the opening degree value of the partitioned tray is indicated as a ratio to the opening degree values ​​of the fully open and fully closed states acquired by the image processing unit using reference images taken with the partitioned tray in a fully open and fully closed state.

6. The imaging unit includes: A camera for capturing the image; a controller for controlling the camera; The controller controls the camera to start taking the images at regular intervals when the opening of the door of the vegetable compartment is detected, and to end the taking of the images when the closing of the vegetable compartment is detected, A food management system as described in any one of claims 1 to 3, wherein the image processing unit obtains the presence / absence information and the opening degree value by subjecting the last predetermined number of images out of a plurality of images taken sequentially by the camera from when the door of the vegetable compartment is opened until it is closed, to the image processing.

7. The imaging unit includes: A camera for capturing the image; A distance measuring sensor that measures and compares the distance to an object that has reflected the measurement wave; a controller for controlling the camera using measurements by the distance measuring sensor; The distance measuring sensor is configured to measure the distance for each of a plurality of mesh areas obtained by dividing a measurable area on a front side of the refrigerator, The food management system of claim 6, wherein the controller detects whether the door is open or closed based on the measurement value of the distance measuring sensor in one or more predetermined specified areas among the plurality of mesh areas that are expected to receive the measurement wave when the vegetable compartment door is in an open state.

8. A step of acquiring information on whether or not each of a plurality of predetermined types of vegetables is recognized in an image by processing an image of the vegetable compartment taken from an upper side of the refrigerator when a door of the vegetable compartment having a sliding partition tray dividing an upper section and a lower section is opened, and an opening degree value indicating a position of the partition tray; determining whether each of the images belongs to the upper section or the lower section based on the opening degree value obtained in the obtaining step; and updating inventory data including data on the presence or absence of each of the plurality of items of vegetables for each of the upper and lower stages based on a result of the determination in the determining step and the presence or absence information in each of the images, In the updating step, when the presence / absence data for a first item of vegetables among the multiple items changes from absent to present based on the presence / absence information in a first image of the vegetable compartment, the photographing date of the first image is recorded in the inventory data as data on the purchase date of the first item of vegetables.

9. The food ingredient management method according to claim 8, further comprising a step of generating display screen data for displaying, in response to a user request, each of the plurality of items for which the vegetables are recognized to be available in association with the purchase date of the vegetable of the item, based on the inventory data at the time the user request was generated.

10. 10. The food ingredient management method of claim 9, wherein the generating step generates the display screen data so as to further display, together with the purchase date, a date that is a predetermined number of days after the purchase date corresponding to each of the plurality of items as an expiration date.

11. The determining step includes: determining that the image having the opening degree value larger than a first threshold value is an image captured of the upper section; determining that an image in which the opening degree value is smaller than a second threshold value set smaller than the first threshold value is an image of the lower section; The updating step includes: updating the inventory data of the upper level using the image determined to have been taken of the upper level by the determining step; updating the inventory data of the lower level using the image determined to have been taken of the lower level in the determining step; The food ingredient management method according to any one of claims 8 to 10, further comprising a step of not using the presence / absence information of an image whose opening value is greater than or equal to the second threshold and less than or equal to the first threshold in updating the inventory data.

12. The present invention further includes a step of outputting a user guidance screen for taking reference images taken in a fully open state and a fully closed state of the partition tray, The foodstuff management method according to claim 11 , wherein the opening degree value of the partition tray is indicated as a ratio to the opening degree values ​​in the fully open state and the fully closed state acquired by image processing the reference image.

13. A step of detecting opening and closing of the door of the vegetable compartment; A step of controlling a camera provided on an upper portion of the refrigerator so as to capture images of the vegetable compartment at regular intervals during a period from when the door is opened to when the door is closed by the detecting step; The foodstuff management method according to any one of claims 8 to 10, further comprising a step of selecting a last predetermined number of images from the plurality of images taken sequentially within the period as targets for the image processing.

14. The detecting step includes: The foodstuff management method according to claim 13, further comprising a step of detecting whether the door is opened or closed based on a change in a measurement value obtained by a distance measuring sensor which compares the distance to an object that has reflected a measurement wave.

15. The distance measuring sensor is configured to measure the distance for each of a plurality of mesh areas obtained by dividing a measurable area on a front side of the refrigerator, The step of detecting whether the door is open or closed includes: The food ingredient management method of claim 14, further comprising a step of detecting whether the door is in an open or closed state based on the measurement value of the distance measuring sensor in one or more predetermined specified areas among the plurality of mesh areas that are expected to receive the measurement wave when the vegetable compartment door is in an open state.

16. A program for causing a computer processor to execute the foodstuff management method according to any one of claims 8 to 10.

Images