Food information acquisition device
The food information acquisition device uses image capture and L*a*b* color system analysis to accurately determine food freshness and movement, addressing inaccuracies in existing methods by accounting for storage condition variations.
Patent Information
- Application Number
- JP2020202976
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-12-07
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-12-07
AI Technical Summary
Existing food freshness determination methods based on spectral reflectance are inaccurate due to variations in storage conditions, illumination direction, and light intensity, which affect spectral reflectance.
A food information acquisition device that includes a photographing means to capture images inside storage compartments, extracts color information using the L*a*b* color system, and determines freshness based on changes in a* and b* values over time, while also determining food movement and storage amount within the storage chamber.
Accurately determines food freshness and movement, regardless of storage conditions, by minimizing the influence of illumination changes and light intensity variations, thereby reducing food waste and improving inventory management.
Smart Images

Figure 0007581815000001 
Figure 0007581815000002 
Figure 0007581815000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a food information acquisition device. [Background technology]
[0002] There is known a refrigerator that includes a storage section (storage chamber) for storing vegetables, a light source that irradiates light into the storage section, a measurement section that measures the spectral reflectance of a specific wavelength of light irradiated from the light source into the storage section, and a judgment section that judges deterioration of the vegetables based on the difference between the spectral reflectance obtained at the time of storage and the spectral reflectance obtained at the time of deterioration confirmation (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2016-133296 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, the technology disclosed in Patent Document 1 determines the freshness of vegetables using the spectral reflectance of specific wavelengths. Depending on the storage conditions of the food in the storage room, the spectral reflectance changes when the direction of the lighting or the amount of light changes, so there is a risk that the accuracy of determining the freshness will decrease.
[0005] The present disclosure has been made to solve such problems. It is an object of the present invention to provide a food information acquisition device that can accurately determine the freshness of food regardless of the storage status of the food in a storage compartment. [Means for solving the problem]
[0006] The food information acquisition device according to the present disclosure includes an imaging means for capturing images of the inside of a storage compartment in which food is stored, a food image information extraction means for extracting one or both of the a* and b* values of the L*a*b* color system as food color information from the images captured by the imaging means, and a freshness determination means for determining the freshness of the food based on the food color information extracted by the food image information extraction means from each of the images captured at different times. a storage amount determination means for determining the amount of food stored in the storage compartment based on the color information of the food extracted by the food image information extraction means from each of the images taken at different times, and a movement determination means for determining the movement of food within the storage compartment based on the color information of the food extracted by the food image information extraction means from each of the images taken at different times; and a notification means for notifying a result of the freshness determination means. The food image information extraction unit obtains a boundary between the food area and the background from the difference in L* value of the L*a*b* color system in the image captured by the imaging means, specifies pixels inside the boundary as the food area, and creates a histogram of one or both of the a* value and the b* value and calculates the average value. The storage room is divided into a plurality of areas, the food movement determination means determines whether or not food has moved for each of the areas, and the notification means further notifies the areas where it is determined by the movement determination means that no food has moved. do. Effect of the Invention
[0007] The food information acquisition device according to the present disclosure has the advantage of being able to accurately determine the freshness of food regardless of the storage status of the food in the storage compartment. [Brief description of the drawings]
[0008] [Figure 1] 1 is a cross-sectional view showing a configuration of a refrigerator equipped with a food information acquisition device according to embodiment 1. FIG. [Diagram 2] 1 is a block diagram showing a functional configuration of a control system of a refrigerator equipped with a food information acquisition device according to embodiment 1. FIG. [Diagram 3] 1 is a block diagram showing a functional configuration of a control device of a refrigerator equipped with a food information acquisition device according to embodiment 1. FIG. [Figure 4] 3A to 3C are diagrams showing examples of images of ground beef before and after storage and binarization processing performed by the food information acquisition device in accordance with the first embodiment. [Diagram 5] 4 is a diagram showing an example of a histogram of a* values of red component of a ground beef image acquired by the food information acquisition device in accordance with the first embodiment; FIG. [Figure 6] 4 is a diagram showing an example of the amount of change in a* value before and after storage of ground beef by the food information acquisition device in accordance with embodiment 1. FIG. [Figure 7] 4A to 4C are diagrams showing example images of spinach before and after storage taken by the food information acquisition device in accordance with the first embodiment. [Figure 8]4 is a diagram showing an example of color change of spinach before and after storage by the food information acquisition device in accordance with embodiment 1. FIG. [Figure 9] 4 is a diagram showing an example of the amount of change in a* value and b* value of spinach before and after storage by the food information acquisition device in accordance with embodiment 1. FIG. [Figure 10] 3 is a flow diagram showing an example of a process flow in the food information acquisition device according to the first embodiment. [Figure 11] 1 is a diagram for explaining an overview of food storage amount and movement determination processing in the food information acquisition device in accordance with embodiment 1. FIG. [Figure 12] 5A to 5C are diagrams illustrating the noise removal process of an image in the food information acquisition device according to the first embodiment. [Figure 13] 4A to 4C are diagrams illustrating processing of an upper blank area of an image in the food information acquisition device according to the first embodiment. [Figure 14] 2 is a diagram illustrating an example of dividing a storage chamber into areas in the food information acquisition device according to the first embodiment. FIG. [Figure 15] 5A to 5C are diagrams showing examples of food storage amounts and movement determination results in the food information acquisition device in accordance with embodiment 1. [Figure 16] 3 is a flow diagram showing an example of a process flow in the food information acquisition device according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] The embodiment for implementing the food information acquisition device according to the present disclosure will be described with reference to the attached drawings. In each drawing, the same or corresponding parts are given the same reference numerals, and duplicated descriptions are appropriately simplified or omitted. In the following description, for convenience, the positional relationship of each structure may be expressed based on the illustrated state. Note that the present disclosure is not limited to the following embodiments, and the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted, within the scope of the gist of the present disclosure.
[0010] Embodiment 1 A first embodiment of the present disclosure will be described with reference to FIG. 1 to FIG. 16. FIG. 1 is a cross-sectional view showing a configuration of a refrigerator equipped with a food information acquisition device. FIG. 2 is a block diagram showing a functional configuration of a control system of a refrigerator equipped with a food information acquisition device. FIG. 3 is a block diagram showing a functional configuration of a control device of a refrigerator equipped with a food information acquisition device. FIG. 4 is a diagram showing an example of images of ground beef before and after storage by a food information acquisition device and an example of binarization processing. FIG. 5 is a diagram showing an example of a histogram of red component a* value of a ground beef image by a food information acquisition device. FIG. 6 is a diagram showing an example of a change in a* value before and after storage of ground beef by a food information acquisition device. FIG. 7 is a diagram showing an example of images of spinach before and after storage by a food information acquisition device. FIG. 8 is a diagram showing an example of color change before and after storage of spinach by a food information acquisition device. FIG. 9 is a diagram showing an example of a* value and b* value change before and after storage of spinach by a food information acquisition device. FIG. 10 is a flow diagram showing an example of a processing flow in a food information acquisition device. FIG. 11 is a diagram explaining an overview of a food storage amount and movement determination processing in a food information acquisition device. Fig. 12 is a diagram explaining noise removal processing of an image in a food information acquisition device. Fig. 13 is a diagram explaining upper blank area processing of an image in a food information acquisition device. Fig. 14 is a diagram explaining an example of area division in a storage compartment in a food information acquisition device. Fig. 15 is a diagram showing an example of food storage amount and movement determination result in a food information acquisition device. Fig. 16 is a flow diagram showing an example of a processing flow in a food information acquisition device.
[0011] In each drawing, the dimensional relationship and shape of each component may differ from the actual one. In addition, the positional relationship between each component (e.g., up-down relationship) is, in principle, when the refrigerator equipped with the food information acquisition device is installed in a usable state.
[0012] A refrigerator 1 equipped with a food information acquisition device according to this embodiment has an insulated box 40 as shown in FIG. 1. The insulated box 40 has an open front (front side) and a storage space formed inside. The insulated box 40 has an outer box, an inner box, and a thermal insulation material. The outer box is made of steel. The inner box is made of resin. The inner box is placed inside the outer box. The thermal insulation material is, for example, urethane foam, and is filled in the space between the outer box and the inner box. The storage space formed inside the insulated box 40 is divided by one or more partition members into a plurality of storage chambers for storing and preserving food.
[0013] 1, the refrigerator 1 here includes a plurality of storage compartments, for example, a refrigerator compartment 2, a switchable compartment 3, an ice-making compartment (not shown), a freezer compartment 4, and a vegetable compartment 5. These storage compartments are arranged in a heat-insulating box body 40 in four vertical levels.
[0014] The refrigerator compartment 2 is disposed on the topmost level of the insulated box 40. The switchable compartment 3 is disposed on one of the left and right sides below the refrigerator compartment 2. The cold storage temperature zone of the switchable compartment 3 can be selected and switched from one of a plurality of temperature zones. The plurality of temperature zones selectable as the cold storage temperature zone of the switchable compartment 3 include, for example, a freezing temperature zone (e.g., about -18°C), a refrigeration temperature zone (e.g., about 3°C), a chilled temperature zone (e.g., about 0°C), and a soft freezing temperature zone (e.g., about -7°C). The ice-making compartment is disposed adjacent to the side of the switchable compartment 3 and in parallel with it, that is, on the other left or right side below the refrigerator compartment 2.
[0015] The freezer compartment 4 is located below the switchable compartment 3 and the ice-making compartment. The freezer compartment 4 is mainly used for freezing and preserving storage objects for a relatively long period of time. The vegetable compartment 5 is located on the lowest level below the freezer compartment 4. The vegetable compartment 5 is mainly used for storing vegetables, fruits, etc.
[0016] An opening formed on the front of the refrigerator compartment 2 is provided with a rotating refrigerator compartment door 2a for opening and closing the opening. Here, the refrigerator compartment door 2a is a double door (double door) and is composed of a right door and a left door. An operation panel 30 is provided on the outer surface of the refrigerator compartment door 2a (for example, the left door) on the front of the refrigerator 1.
[0017] Each of the storage compartments other than the refrigerator compartment 2 (the switchable compartment 3, the ice-making compartment, the freezer compartment 4, and the vegetable compartment 5) is opened and closed by a pull-out door. These pull-out doors can be opened and closed in the depth direction (front-to-back direction) of the refrigerator 1 by sliding a frame fixed to the door along a rail formed horizontally on the inner wall surface on the left and right sides of each storage compartment. Inside the switchable compartment 3 and the freezer compartment 4, a switchable compartment storage case and a freezer storage case, which can store food and the like, are stored so that they can be pulled out freely. Inside the vegetable compartment 5, a vegetable storage case, which can store food such as vegetables and fruits, is stored so that they can be pulled out freely.
[0018] The refrigerator 1 is equipped with a refrigeration cycle circuit that cools the air supplied to each storage compartment. The refrigeration cycle circuit is composed of a compressor 8, a condenser (not shown), a throttling device (not shown), a cooler 9, etc. The compressor 8 compresses and discharges the refrigerant in the refrigeration cycle circuit. The condenser condenses the refrigerant discharged from the compressor 8. The throttling device expands the refrigerant flowing out from the condenser. The cooler 9 cools the air supplied to each storage compartment by the refrigerant expanded by the throttling device. The compressor 8 is disposed, for example, at the lower part on the rear side of the refrigerator 1.
[0019] In the refrigerator 1, an air passage 6 is formed for supplying air cooled by the refrigeration cycle circuit to each storage compartment. This air passage 6 is mainly disposed on the rear side inside the refrigerator 1. A cooler 9 of the refrigeration cycle circuit is disposed in this air passage 6. Also disposed in the air passage 6 is a blower fan 7 for sending the air cooled by the cooler 9 to each storage compartment.
[0020] When the blower fan 7 operates, the air (cold air) cooled by the cooler 9 is sent through the air passage 6 to the freezer compartment 4, the switchable compartment 3, the ice-making compartment, and the refrigerator compartment 2, cooling the insides of these storage compartments. The vegetable compartment 5 is cooled by introducing the return cold air from the refrigerator compartment 2 into the vegetable compartment 5 through the refrigerator compartment return air passage. The cold air that has cooled the vegetable compartment 5 is returned to the air passage 6 where the cooler 9 is located through the vegetable compartment return air passage (these return air passages are not shown). The cold air is then cooled again by the cooler 9, and circulated inside the refrigerator 1.
[0021] A damper (not shown) is provided at a location midway from air duct 6 to each storage chamber. Each damper opens and closes the location of air duct 6 leading to each storage chamber. By changing the open / close state of the damper, the amount of cold air supplied to each storage chamber can be adjusted. In addition, the temperature of the cold air can be adjusted by controlling the operation of compressor 8. The refrigeration cycle circuit consisting of compressor 8 and cooler 9, blower fan 7, air duct 6, and damper provided as described above constitute a cooling means for cooling the air sent into the inside of the storage chamber.
[0022] A refrigerator compartment camera 10 is installed inside refrigerator compartment door 2a. Refrigerator compartment camera 10 is capable of taking images of the inside of the storage compartment (refrigerator compartment 2 in this case) from the refrigerator compartment door 2a side. Refrigerator 1 according to this embodiment is equipped with a camera movement device 11. Camera movement device 11 is a device that moves refrigerator compartment camera 10 in the vertical direction. Camera movement device 11 is provided inside refrigerator compartment door 2a across the vertical direction.
[0023] The camera moving device 11 moves the refrigerator compartment camera 10 up and down within a movable range. The movable range is a range that is set in advance. Here, the upper end of the movable range is the upper end of the refrigerator compartment 2. Moreover, the lower end of the movable range is the lower end of the refrigerator compartment 2.
[0024] The camera moving device 11 moves the refrigerator camera 10 in the up-down direction over the aforementioned movable range, for example, by a stepping motor (not shown). In this case, the amount of movement of the refrigerator camera 10 is proportional to the amount of rotation of the stepping motor. The amount of rotation of the stepping motor can be controlled using the number of steps. Therefore, the camera moving device 11 can control the amount of movement of the refrigerator camera 10 using the number of steps of the stepping motor.
[0025] The refrigerator 1 according to this embodiment is also provided with a vegetable compartment camera 12. The vegetable compartment camera 12 is attached to the rear part inside the vegetable compartment 5. The vegetable compartment camera 12 can capture images inside the vegetable compartment 5, which is a storage compartment. The position of the vegetable compartment camera 12 is not limited to the rear part inside the vegetable compartment 5. If it can capture images of food inside the vegetable compartment 5, the vegetable compartment camera 12 may be installed on the ceiling surface or the like inside the vegetable compartment 5. The vegetable compartment camera 12 may also be installed at multiple locations inside the vegetable compartment 5. Similarly, the refrigerator compartment camera 10 may also be installed at multiple locations inside the refrigerator compartment 2. The refrigerator compartment camera 10 and the vegetable compartment camera 12 are imaging means for capturing images inside the storage compartments in which food is stored. The cameras, which are imaging means, may be installed not only in the refrigerator compartment 2 and the vegetable compartment 5, but also in other storage compartments (freezer compartment 4, switchable compartment 3, etc.).
[0026] The refrigerator 1 includes a control device 20. The control device 20 is provided, for example, at an upper part on the rear side of a thermally insulated box 40 of the refrigerator 1 as shown in Fig. 1. The control device 20 includes a control circuit for controlling the operation of the refrigerator 1, and the like. Each function of the control device 20 is realized by this control circuit.
[0027] 2 is a block diagram functionally illustrating the configuration of the main parts of the control system of the refrigerator 1 according to this embodiment. The control circuit of the control device 20 includes, for example, a processor 20a and a memory 20b. The control device 20 controls the refrigerator 1 by executing preset processes as a result of the processor 20a executing a program stored in the memory 20b.
[0028] The processor 20a is also called a CPU (Central Processing Unit), central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, or DSP. The memory 20b corresponds to, for example, a non-volatile or volatile semiconductor memory such as a RAM, a ROM, a flash memory, an EPROM, and an EEPROM, or a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, and a DVD.
[0029] The control circuit of the control device 20 may be formed as dedicated hardware, for example. A part of the control circuit of the control device 20 may be formed as dedicated hardware, and the control circuit may be provided with the processor 20a and the memory 20b. Examples of the control circuit partly formed as dedicated hardware include a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination of these.
[0030] The operation panel 30 includes an operation unit 31 and an alarm unit 32. The operation unit 31 is an operation switch or the like for setting the cold storage temperature of each storage compartment. The alarm unit 32 is a liquid crystal display unit that displays various information such as the temperature of each storage compartment, an indicator lamp, a speaker, or the like. The operation panel 30 may include a touch panel that serves as the liquid crystal display unit of both the operation unit 31 and the alarm unit 32. The operation unit 31 outputs a signal according to the operation of the operation unit 31 by the user to the control device 20. Then, a signal from the operation unit 31 of the operation panel 30 is input to the control device 20. The control device 20 also outputs an alarm control signal to the alarm unit 32 of the operation panel 30 to control the operation of the alarm unit 32.
[0031] The refrigerator 1 is equipped with a thermistor 13 and a door open / close detection switch 15. The thermistor 13 detects the temperature inside each storage compartment. The thermistor 13 is installed in each storage compartment. The door open / close detection switch 15 is an open / close detection means that detects the opening and closing of the door of the storage compartment. The door open / close detection switch 15 detects, for example, the opening and closing of the refrigerator compartment door 2a of the refrigerator compartment 2. The door open / close detection switch 15 is, for example, a general magnet type switch. That is, the door open / close detection switch 15 detects, for example, the proximity of a magnet embedded in the refrigerator compartment door 2a by a pair of reed switches installed on the refrigerator 1 main body side.
[0032] A detection signal of the temperature inside each storage compartment is input to the control device 20 from the thermistor 13. A detection signal from the door open / close detection switch 15 is also input to the control device 20. Based on the input signal, the control device 20 executes a process to control the operation of the compressor 8, the blower fan 7, the damper 14, etc. so that the inside of each storage compartment is maintained at a set temperature. That is, the control device 20 controls the cooling means and the like described above to control the operation of the refrigerator 1. The control device 20 also controls the shooting operation of the refrigerator compartment camera 10 and the vegetable compartment camera 12, and the movement of the refrigerator compartment camera 10 by the camera moving device 11. Image data captured by the refrigerator compartment camera 10 and the vegetable compartment camera 12 is input to the control device 20.
[0033] The control device 20 also includes a communication unit 20c. The communication unit 20c is a circuit that enables the control device 20 of the refrigerator 1 to communicate with external devices. Examples of external devices that communicate with the refrigerator 1 include a mobile terminal such as a smartphone, and an external server. For example, it is possible for a user to operate a mobile terminal to instruct the refrigerator 1 to change the set temperature or check the status inside the refrigerator. Also, some or all of the functions of the units of the control device 20 described below may be provided in an external server.
[0034] 3, the control device 20 includes a camera control unit 21, a food image information extraction unit 22, a freshness determination unit 23, a storage amount determination unit 24, a movement determination unit 25, a noise removal unit 26, a vacancy rate calculation unit 27, a notification control unit 28, and a memory unit 29. The functions of each of these units are realized, for example, by the processor 20a executing a program stored in the memory 20b to execute a preset process.
[0035] The camera control unit 21 controls the operations of the refrigerator compartment camera 10, the vegetable compartment camera 12, and the camera moving device 11. For example, the camera control unit 21 causes the refrigerator compartment camera 10 to capture an image of the inside of the refrigerator compartment 2 when the refrigerator compartment door 2a is opened or closed. This is because when the refrigerator compartment door 2a is opened or closed, food is taken in or out of the refrigerator compartment 2, and the storage state within the refrigerator compartment 2 may change. Similarly, the camera control unit 21 causes the vegetable compartment camera 12 to capture an image of the inside of the vegetable compartment 5 when the door of the vegetable compartment 5 is opened or closed.
[0036] The camera control unit 21 detects whether the refrigerator compartment door 2a is open or closed based on a signal output from the door open / close detection switch 15. The camera control unit 21 can detect when the closed refrigerator compartment door 2a has been opened, and when the open refrigerator compartment door 2a has been closed. For example, after detecting that the open refrigerator compartment door 2a has been closed, the camera control unit 21 causes the refrigerator compartment camera 10 to take an image while the inside of the refrigerator compartment 2 is illuminated. Similarly, for example, after detecting that the open door of the vegetable compartment 5 has been closed, the camera control unit 21 causes the vegetable compartment camera 12 to take an image while the inside of the vegetable compartment 5 is illuminated.
[0037] The camera control unit 21 causes the refrigerator compartment camera 10 to capture images of the inside of the refrigerator compartment 2 in coordination with the movement of the refrigerator compartment camera 10 by the camera movement device 11. Next, the photographing operation coordinated between the refrigerator compartment camera 10 and the camera movement device 11 will be described. The "photographing operation" here refers to the operation of the refrigerator compartment camera 10 capturing images of the inside of the refrigerator compartment 2 while moving from a start position to an end position. In this photographing operation, the camera movement device 11 moves the refrigerator compartment camera 10 from the start position to the end position. The start position is, for example, one end of the aforementioned movable range. The end position is, for example, the other end of the aforementioned movable range.
[0038] As a specific example, the start position is the upper end of the aforementioned movable range. And the end position is the lower end of the aforementioned movable range. In this example, in the shooting operation, the camera moving device 11 moves the refrigerator compartment camera 10 downward from the upper end to the lower end of the aforementioned movable range. Then, the refrigerator compartment camera 10 shoots when passing a specific position during this movement. Alternatively, the camera moving device 11 may stop the refrigerator compartment camera 10 at a specific position, and the refrigerator compartment camera 10 may shoot while this is stopped. The specific position where the refrigerator compartment camera 10 shoots is set so that the entire inside of the refrigerator compartment 2 can be shot by one shooting operation. Such a specific position is, for example, between adjacent shelves.
[0039] The movement direction of the refrigerator compartment camera 10 is not limited to the downward direction described above. That is, the camera control unit 21 may capture images with the refrigerator compartment camera 10 while moving the refrigerator compartment camera 10 upwardly by the camera movement device 11 from the lower end to the upper end of the aforementioned movable range. In this case, the start position is the lower end of the aforementioned movable range, and the end position is the upper end of the aforementioned movable range.
[0040] The amount of movement of the refrigerator camera 10 is proportional to the amount of rotation of the stepping motor of the camera movement device 11. The amount of rotation of the stepping motor can be controlled using the number of steps. In this embodiment, the camera control unit 21 controls the amount of movement of the refrigerator camera 10 using the number of steps of the stepping motor of the camera movement device 11. In other words, the camera control unit 21 can identify a specific position where shooting by the refrigerator camera 10 is performed by counting the number of steps of the stepping motor.
[0041] The images captured by the refrigerator compartment camera 10 and the vegetable compartment camera 12 are stored in chronological order in the memory unit 29. Therefore, images captured at different times are stored in the memory unit 29. The memory unit 29 also stores in advance images of the storage compartment when it is empty and nothing is stored therein.
[0042] The food image information extraction unit 22 is a food image information extraction unit that extracts one or both of the a* and b* values of the L*a*b* (CIELAB) color system as food color information from the images captured by the refrigerator compartment camera 10 and the vegetable compartment camera 12, which are the image capture unit. The food image information extraction unit 22 first captures the image captured by the image capture unit from which the food area is to be extracted. Next, the food image information extraction unit 22 extracts the luminance information L* value of the L*a*b* color system for each pixel of the captured image. There is a difference in the L* value, which represents luminance, between the food and the background in the image. Therefore, the food image information extraction unit 22 separates the food and the background for each pixel based on the difference in the L* value. In addition, since there is a difference in the L* value at the boundary between the food and the background, the food image information extraction unit 22 may utilize this feature to find the boundary between the food area and the background, and set the pixels inside the boundary as the food area and the pixels outside the boundary as the background. Furthermore, since the walls inside the storage compartment of the refrigerator 1 are mainly white, one or both of the color information a* and b* values may be used instead of the luminance information L* value. In this case, the food image information extraction unit 22 determines that the pixel area is a food area if one or both of the a* and b* values are equal to or greater than a preset reference value.
[0043] Then, the food image information extraction unit 22 generates a binarized image in which pixel values determined to be food regions are colored white and pixel values determined to be background are colored black. Figure 4 shows an example of a captured image and a binarized image when the target food is ground beef. By generating such a binarized image, it is easy to check whether the food region has been extracted.
[0044] Next, the food image information extraction unit 22 extracts pixel information of the original image, i.e., the image captured by the image capture unit, in the area where the pixel value of the generated binary image is white. At this time, the pixel information of the original image is in the RGB color system, but the food image information extraction unit 22 converts it to the L*a*b* color system. Then, the food image information extraction unit 22 extracts one or both of the a* value and the b* value of the L*a*b* color system as color information of the food. The a* value and the b* value extracted by the food image information extraction unit 22 can be histogrammed as shown in FIG. 5, for example, to confirm the color distribution of the target food. Since the target food in the example described here is ground beef, the a* value representing red is shown in the histogram in FIG. 5. In addition, the average value of the a* value and the b* value extracted by the food image information extraction unit 22 may be calculated.
[0045] The freshness determination unit 23 is a freshness determination means that determines the freshness of a food based on the color information of the food extracted by the food image information extraction unit 22 from each of the images taken at different times. The freshness determination of food by the freshness determination unit 23 will be explained using the example shown in Fig. 6. The figure shows an example of an experimental result of color change when ground beef is stored in the refrigerator 1 for three weeks. In the figure, (a) shows ground beef stored in the switchable compartment 3 with a set temperature of -8°C, (b) shows ground beef stored in the switchable compartment 3 with a set temperature of -7°C, and (c) shows ground beef stored in the freezer compartment 4 with a set temperature of -18°C.
[0046] The red color of meat is due to the color development of myoglobin, which becomes methemoglobin and turns brown as oxygen absorption progresses. As oxidation progresses further, it changes to cholestamyloglobin and turns green. Therefore, when the target food is ground beef, the freshness determination unit 23 can determine the freshness of the target food based on the change amount Δa* of the a* value. As shown in the figure, the change amount Δa* of the a* value of ground beef stored for three weeks is approximately -14 for (a), approximately -20 for (b), and approximately -3 for (c). Generally, since three weeks of storage in the switching compartment 3 is considered to be a guideline for maintaining the quality of ground beef, it can be considered that the limit of storage is when the a* value decreases by 20 compared to the start of storage. Therefore, the freshness determination unit 23 determines that the freshness of ground beef is such that consumption needs to be promoted when Δa* of the ground beef reaches the first reference value -15. Then, when Δa* for the ground beef reaches the second reference value −20, the freshness determining unit 23 determines that the freshness is such that disposal is required.
[0047] The notification control unit 28, for example, controls the operation of the notification unit 32 of the operation panel 30 to notify the user of the determination result of the freshness determination unit 23. The notification control unit 28 and the notification unit 32 are notification means for notifying the user of the determination result of the freshness determination unit 23. For example, when the freshness determination unit 23 determines that the freshness of the target food requires promotion of consumption, the notification control unit 28 controls the operation of the notification unit 32 to, for example, display a message on the liquid crystal display encouraging consumption of the food. Also, when the freshness determination unit 23 determines that the freshness of the target food requires promotion of disposal, the notification control unit 28 controls the operation of the notification unit 32 to, for example, display a message on the liquid crystal display encouraging disposal of the food.
[0048] Next, the freshness determination of food by the freshness determination unit 23 will be described with reference to the examples shown in Figs. 7 to 9. These figures show an example of the experimental results of the color change of spinach stored in the refrigerator 1 for one week. In Fig. 7(a), spinach is stored in the freezer compartment 4 with a set temperature of -18°C, in Fig. 7(b), in the switchable compartment 3 with a set temperature of -7°C, and in Fig. 7(c), in the vegetable compartment 5 with a set temperature of 6°C. When the quality of spinach deteriorates due to a long storage period, the green leaves first fade. Then, when the quality deteriorates further, the spinach turns yellow. After yellowing, the spinach becomes inedible, so it is desirable to draw attention to this before it turns yellow.
[0049] FIG. 8 plots the a* and b* values before and after storage in each storage compartment. FIG. 9 shows the change amount Δa* of the a* value and the change amount Δb* of the b* value before and after storage in each storage compartment. As can be seen from FIG. 8 and FIG. 9, when stored in the vegetable compartment 5, the a* value does not change significantly, and only the b* value increases significantly. In contrast, when stored in the freezer compartment 4 and the switchable compartment 3, the a* value increases slightly and the b* value decreases. Therefore, the freshness determination unit 23 first determines that the freshness of the spinach requires the consumption or disposal of the spinach when Δa* reaches the reference value 4 and Δb* reaches the reference value -6. In addition, the freshness determination unit 23 also determines that the freshness of the spinach requires the consumption or disposal of the spinach when Δb* exceeds the reference value 0.
[0050] According to the food information acquisition device configured as described above, one or both of the a* and b* values of the L*a*b* color system are extracted as food color information from the storage room image captured by the imaging means, and the freshness of the food is determined by evaluating the color change of the food while suppressing the influence of the luminance information (L* value). Therefore, the freshness of foods such as red meat and fish and green vegetables can be accurately determined regardless of the storage situation of the food in the storage room, that is, by suppressing the influence of changes in the orientation of the food, the direction of the lighting, the amount of light, etc. Also, by notifying the user of the deterioration of food freshness, it is possible to encourage consumption and reduce food waste, etc.
[0051] Next, an example of the process flow in the refrigerator 1 equipped with the food information acquisition device configured as above will be described with reference to the flow chart in Fig. 10. First, in step S11, the camera control unit 21 controls the refrigerator compartment camera 10 and vegetable compartment camera 12, which are imaging means, to capture images of the inside of the storage compartments of the refrigerator 1 (in this case, the refrigerator compartment 2 and vegetable compartment 5).
[0052] In the next step S12, the food image information extraction unit 22 extracts food areas from the image captured in step S11 and generates a binarized image. For past captured images stored in the storage unit 29, binarized images may be generated in advance and stored in the storage unit 29, or the binarization process may be performed whenever necessary. After step S12, the control device 20 performs the process of step S13.
[0053] In step S13, food image information extraction unit 22 converts pixel information of the original image, i.e., the image captured in step S11, in the food area extracted in step S12, into the L*a*b* color system. In the following step S14, food image information extraction unit 22 extracts one or both of the a* and b* values of the L*a*b* color system for the pixels in the food area as food color information. The a* and b* values extracted by food image information extraction unit 22 are then histogrammed, and their average values are calculated. After step S14, control device 20 performs the process of step S15.
[0054] In step S15, the freshness determination unit 23 calculates the amount of change in one or both of the a* and b* values extracted for images taken at different times. Then, in the following step S16, the freshness determination unit 23 determines whether the amount of change in one or both of the a* and b* values reaches a preset freshness determination criterion. If the amount of change in the a* and b* values does not reach the freshness determination criterion, the control device 20 returns to step S11 and continues processing.
[0055] On the other hand, if the amount of change in one or both of the a* and b* values reaches the preset freshness judgment criterion, the control device 20 performs the process of step S17. In step S17, the notification control unit 28 controls, for example, the operation of the notification unit 32 of the operation panel 30 to notify the user of the freshness judgment result. When step S17 is completed, the series of processes ends.
[0056] Storage amount determination unit 24 of control device 20 is a storage amount determination means that determines the amount of food stored in the storage compartment based on food color information extracted by food image information extraction unit 22 from each image taken at different times. Movement determination unit 25 of control device 20 is a movement determination means that determines the movement of food within the storage compartment based on food color information extracted by food image information extraction unit 22 from each image taken at different times.
[0057] 11 shows an overview of the process of determining the food storage amount by the storage amount determination unit 24 and the process of determining the movement of food by the movement determination unit 25. First, the determination of the food storage amount by the storage amount determination unit 24 will be described. The storage amount determination unit 24 determines the food storage amount using two images taken at different times: image A (hereinafter also referred to as "empty image A") showing an empty state with nothing stored in the storage compartment, and image C (hereinafter also referred to as "current image C") taken inside the storage compartment at the most recent time. As mentioned above, empty image A is stored in advance in the memory unit 29, for example.
[0058] Storage capacity determination unit 24 first calculates the difference between sky image A and current image C for each pixel of the food color information extracted by food image information extraction unit 22, i.e., one or both of the a* and b* values in the L*a*b* color system. Then, for each pixel, a binarized image is generated depending on whether the absolute value of the difference between sky image A and current image C is equal to or greater than a reference value. In Fig. 11, pixels whose absolute value of the difference is equal to or greater than the reference value are colored white, and pixels whose absolute value is less than the reference value are colored black.
[0059] However, the binary image generated in this way also contains many isolated points that become noise. The isolated points that become noise can occur, for example, when the color of the background and part of the food surface are very similar, or when the position or orientation of a food with an uneven outer shape changes slightly. Therefore, next, the noise due to the isolated points is removed by the noise removal unit 26 from the binary image generated by the storage amount determination unit 24.
[0060] The noise removal unit 26 compares the pixel value of the target pixel with the pixel values of the surrounding pixels while scanning each pixel of the binarized image. If the pixel value of the target pixel is significantly different from the pixel values of the surrounding pixels, it is determined to be an isolated point. On the other hand, if the pixel value of the target pixel is the same as or close to the pixel values of the surrounding pixels, it is determined not to be an isolated point. A specific example of the noise removal process by the noise removal unit 26 will be described with reference to FIG. 12. The noise removal unit 26 calculates the average value of pixel values for a pixel group within a pixel distance N=2 from the target pixel. If the calculated average value is equal to or greater than a preset reference value, the value of the target pixel is replaced with a white display value. On the other hand, if the calculated average value is less than the reference value, the value of the target pixel is replaced with a black display value.
[0061] Next, the storage amount determination unit 24 removes the blank area above the food from the binary image after noise removal by the noise removal unit 26, and determines the area where the food exists. This is because it is assumed that no more food can be stored in the area above the food between the food and the shelf or ceiling. Specifically, as shown in FIG. 13, while scanning each pixel of the binary image, if the pixel value below the target pixel is a white display value, the storage amount determination unit 24 replaces the value of the determination pixel above the target pixel to the ceiling or shelf with a white display value. In the figure, an example is shown in which pixels at a pixel distance N=2 and a pixel distance N=4 are used for the determination of the pixels below the target pixel.
[0062] If the number of determination pixels from the target pixel to the upper shelf or ceiling is greater than a preset number, it can be determined that there is sufficient space above the food between the food and the shelf or ceiling to store additional food. Therefore, if the number of determination pixels from the target pixel to the upper shelf or ceiling is greater than a preset number, the values of the determination pixels may be displayed as black up to the ceiling or shelf.
[0063] The storage amount determination unit 24 determines the amount of food stored in the storage compartment based on the binary image that has been subjected to the above processing. That is, it is determined that food is stored in the parts of the binary image whose pixel values are white, and that parts whose pixel values are black are empty. In this case, the storage compartment may be divided into a plurality of areas, and the storage amount determination unit 24 may determine the presence or absence of food in each of these areas. An example of the storage compartment divided into a plurality of areas is shown in FIG. 14. (a) of FIG. 14 shows the actual state of the refrigerator compartment 2, which is a storage compartment, and (b) of FIG. 14 is an example of the area division within the refrigerator compartment 2. In (b) of FIG. 14, different shaded areas indicate different divided areas.
[0064] Next, the determination of food movement by movement determination unit 25 will be described. Movement determination unit 25 determines food movement using two images taken at different times: image B taken inside the storage room just before the food is moved (hereinafter also referred to as "pre-movement image B"), and image C taken inside the storage room at the latest time, i.e., after the food is moved (hereinafter also referred to as "current image C"). Note that pre-movement image B is stored in memory unit 29 after being taken by the imaging means.
[0065] Movement determination unit 25 first calculates the difference between pre-movement image B and current image C for each pixel, for the color information of the food extracted by food image information extraction unit 22, i.e., one or both of the a* and b* values in the L*a*b* color system. Then, for each pixel, a binarized image is generated depending on whether the absolute value of the difference between pre-movement image B and current image C is equal to or greater than a reference value. In Fig. 11, pixels whose absolute value of the difference is equal to or greater than the reference value are colored white, and pixels whose absolute value is less than the reference value are colored black.
[0066] Next, noise removal unit 26 removes noise from the binary image generated by movement determination unit 25. Next, movement determination unit 25 removes blank areas above the food from the binary image after noise removal by noise removal unit 26, and determines the area where the food exists. This is the same as the process in storage capacity determination unit 24 described above.
[0067] Movement determination unit 25 determines the movement of food within the storage compartment based on the binary image that has been subjected to the above processing. That is, it determines that food has been moved in areas where pixel values in the binary image are white, and that food has not been moved in areas where pixel values are black. In this case, similar to the determination of the food storage amount by storage amount determination unit 24, the storage compartment may be divided into a plurality of areas, and the movement of food may be determined for each of these areas. In this case, movement determination unit 25 may determine the presence or absence of food movement within each area by determining whether the average pixel value within each area is equal to or greater than a preset reference value.
[0068] The movement of food determined by movement determination unit 25 indicates either that a new food item has been placed at the location or that a food item has been removed from the location. In a location where movement determination unit 25 determines that no food item has been moved, the location continues to have food or no food therein.
[0069] By combining the judgment results of storage amount judgment unit 24 and movement judgment unit 25, it is possible to judge whether the amount of food in the storage compartment has increased or decreased. That is, if storage amount judgment unit 24 judges that food is present in a location where movement judgment unit 25 judges that food has moved, it can be judged that new food has been placed in that location, i.e., the amount of food has increased. Also, if storage amount judgment unit 24 judges that no food is present in a location where movement judgment unit 25 judges that food has moved, it can be judged that food has been removed from that location, i.e., the amount of food has decreased.
[0070] The vacancy rate calculation unit 27 is a vacancy rate calculation means that calculates the vacancy rate of the storage chamber based on the storage amount of food in the storage chamber determined by the storage amount determination unit 24. In the configuration example described above, a binary image is obtained as the determination result by the storage amount determination unit 24. The vacancy rate calculation unit 27 calculates the vacancy rate of the storage chamber based on the binary image obtained as the determination result by the storage amount determination unit 24. As described above, the binary image obtained as the determination result by the storage amount determination unit 24 is obtained by removing noise from the binary image based on the difference between the sky image A and the current image C regarding the color information of the food extracted by the food image information extraction unit 22 by the noise removal unit 26, and further removing the blank area above the food to obtain the area where the food exists. The vacancy rate calculation unit 27 calculates the vacancy rate (%) of the storage chamber from the binary image obtained as the determination result by the storage amount determination unit 24 using the following formula (1).
[0071] (Vacancy rate of storage room) = 100 - {(number of white pixels) / (total number of pixels)} × 100 (1)
[0072] Also, the vacancy rate calculation unit 27 may calculate the rate of change of the storage state of the storage chamber based on the movement of food in the storage chamber determined by the movement determination unit 25. In the configuration example described above, a binary image is obtained as the determination result by the movement determination unit 25. Therefore, the vacancy rate calculation unit 27 calculates the rate of change of the storage state of the storage chamber based on the binary image obtained as the determination result by the movement determination unit 25. Note that, as described above, the binary image obtained as the determination result by the movement determination unit 25 is obtained by removing noise from the binary image based on the difference between the pre-movement image B and the current image C regarding the color information of the food extracted by the food image information extraction unit 22 by the noise removal unit 26, and further removing the blank area above the food to obtain the area where the food exists. The vacancy rate calculation unit 27 calculates the rate of change (%) of the storage state of the storage chamber from the binary image obtained as the determination result by the movement determination unit 25 using the following formula (2).
[0073] (Change rate of storage state in storage room) = {(Number of white pixels) / (Total number of pixels)} × 100 (2)
[0074] The notification control unit 28 may, for example, control the operation of the notification unit 32 of the operation panel 30 to notify the user of the judgment results of the storage amount judgment unit 24 and the movement judgment unit 25 and the calculation results of the vacancy rate calculation unit 27. For example, as the judgment results of the storage amount judgment unit 24 and the movement judgment unit 25, a storage room image in which an increase / decrease in food is represented in different colors for each area in the storage room is displayed on the liquid crystal display of the notification unit 32. As a specific example, in a captured image of the storage room, an area where no food has been moved is displayed in red, an area where food has increased is displayed in green, and an area where food has decreased is displayed in blue. In addition, for example, the vacancy rate of the storage room and the storage state change rate calculated by the vacancy rate calculation unit 27 are displayed on the liquid crystal display of the notification unit 32. FIG. 15 shows display examples of the before-movement image B, the current image C, the judgment results of the storage amount judgment unit 24 and the movement judgment unit 25, and the calculation results of the vacancy rate of the storage room and the storage state change rate for six patterns.
[0075] According to the food information acquisition device configured as described above, one or both of the a* and b* values of the L*a*b* color system are extracted as color information of food from the storage compartment image captured by the imaging means, and the presence or absence of food, storage state, movement, etc. of food can be determined by evaluating the shape of the food while suppressing the influence of luminance information (L* value). Therefore, the presence or absence of food can be determined accurately regardless of the storage state of the food in the storage compartment, that is, by suppressing the influence of changes in the orientation of the food, the direction of the lighting, the amount of light, etc. Also, by informing the user of the amount of food stored, the storage position, and the availability of the storage compartment, it is possible to use this as a reference for the amount to purchase when shopping. Also, for foods whose position has changed over a long period of time, caution can be drawn when using them.
[0076] Next, an example of the process flow in the refrigerator 1 equipped with the food information acquisition device configured as above will be described with reference to the flow chart in Fig. 16. First, in step S21, the camera control unit 21 controls the refrigerator compartment camera 10 and vegetable compartment camera 12, which are imaging means, to capture images of the inside of the storage compartments of the refrigerator 1 (in this case, the refrigerator compartment 2 and vegetable compartment 5).
[0077] In the next step S22, the food image information extraction unit 22 extracts a food region from the image captured in step S21 and generates a binarized image. For past captured images stored in the storage unit 29, a binarized image may be generated in advance and stored in the storage unit 29, or the binarization process may be performed whenever necessary. Next, the food image information extraction unit 22 converts the pixel information of the original image, i.e., the image captured in step S11, in the extracted food region into the L*a*b* color system. Then, the food image information extraction unit 22 extracts one or both of the a* value and the b* value of the L*a*b* color system for the pixels in the food region as food color information. After step S22, the control device 20 performs the process of step S23.
[0078] In step S23, the storage amount determination unit 24 and the movement determination unit 25 calculate the difference in the color information (a* value, b* value) of the food extracted in step S22 from each image taken at a different time point, and compare the absolute value of the difference with a reference value to generate a binary image. In the following step S24, the noise removal unit 26 removes noise from the binary image generated in step S23. In the further following step S25, the storage amount determination unit 24 and the movement determination unit 25 determine whether food can be inserted based on the size of the blank area above the food in each binary image from which noise has been removed in step S22, and if food cannot be inserted, remove the blank area and set the area as an area where food exists. After step S25, the control device 20 performs the process of step S26.
[0079] In step S26, storage amount determination unit 24 determines the amount of food stored in the storage compartment based on the binary image processed in step S25. That is, storage amount determination unit 24 determines the amount of food stored from the binary image obtained by comparing empty image A and current image C. Furthermore, movement determination unit 25 determines the movement of food in the storage compartment based on the binary image processed in step S25. Movement determination unit 25 determines the movement of food from the binary image obtained by comparing pre-movement image B and current image C. Then, control device 20 determines the increase or decrease in food in the storage compartment based on the determination results of storage amount determination unit 24 and movement determination unit 25. After step S26, control device 20 performs the process of step S27.
[0080] In step S27, the notification control unit 28 controls, for example, the operation of the notification unit 32 of the operation panel 30 to notify the user of the determination result in step S26. When step S27 is completed, the series of processes ends.
[0081] In the above, a configuration example in which the food information acquisition device is provided in the refrigerator 1 has been described. However, this is not limiting, and the food information acquisition device may be provided separately from the refrigerator 1. Specifically, as described above, an image captured inside a storage compartment may be sent to an external server capable of communicating with the control device 20 of the refrigerator 1, and the food image information extraction unit 22, freshness determination unit 23, storage capacity determination unit 24, movement determination unit 25, noise removal unit 26, vacancy rate calculation unit 27, memory unit 29, etc. may be provided in the external server to configure the food information acquisition device. In addition, the storage compartment targeted by the food information acquisition device does not have to be that of the refrigerator 1. [Explanation of symbols]
[0082] 1. Refrigerator 2 Refrigerator 2a Refrigerator door 3. Switch Room 4 Freezer 5 Vegetable compartment 6 Wind path 7 Blower fan 8. Compressor 9 Cooler 10 Refrigerator Camera 11 Camera movement device 12 Vegetable Compartment Camera 13 Thermistor 14 Damper 15 Door opening / closing detection switch 20 Control device 20a processor 20b Memory 20c Communication Department 21 Camera control unit 22 Food image information extraction unit 23 Freshness Determination Section 24 Storage volume determination unit 25 Movement determination section 26 Noise removal section 27 Vacancy Rate Calculation Section 28 Notification control section 29 Memory section 30 Operation Panel 31 Operation section 32 Notification Department 40 Insulated box
Claims
1. An image capturing means for capturing an image of the inside of a storage compartment in which food is stored; a food image information extraction means for extracting one or both of the a* value and the b* value of the L*a*b* color system as color information of the food from the image captured by the imaging means; a freshness determination means for determining the freshness of the food based on the color information of the food extracted by the food image information extraction means from each of the images taken at different times; a storage amount determination means for determining the amount of food stored in the storage compartment based on the color information of the food extracted by the food image information extraction means from each of the images taken at different times; a movement determination means for determining movement of food within the storage compartment based on color information of the food extracted by the food image information extraction means from each of the images taken at different times; A notification means for notifying a result of the freshness determination means, The food image information extraction unit obtains a boundary between the food area and the background from the difference in L* value of the L*a*b* color system in the image captured by the imaging means, identifies pixels inside the boundary as the food area, and creates a histogram of one or both of the a* value and the b* value and calculates the average value. The storage chamber is divided into a plurality of areas, The food movement determination means determines whether or not food has moved for each area, The notification means of the food information acquisition device further notifies the area in which it has been determined by the movement determination means that no movement of food has occurred.
2. The food information acquisition device of claim 1, wherein the storage amount determination means and the movement determination means determine whether or not food can be added based on the size of the blank area above the food, and if food cannot be added, the blank area is determined to be an area where food exists.
3. The food information acquisition device according to claim 1 or 2, further comprising a vacancy rate calculation means for calculating a vacancy rate of the storage compartment based on the amount of food stored in the storage compartment determined by the storage amount determination means.
4. The storage compartment is provided in a refrigerator, The food information acquisition device according to claim 1 , wherein the image capturing means is provided in the storage compartment of the refrigerator.
5. The food information acquisition device according to claim 4 , further comprising a moving means for moving the photographing means vertically within the storage chamber.
Citation Information
Patent Citations
Refrigerator and refrigerator internal display system
JP2014209048A
Refrigerator
JP2016133296A
Refrigerator and network system including the same
JP2017089913A
Refrigerator and refrigerator system
WO2019102616A1
