Food monitoring device, refrigerator containing the same, and method of operation thereof

Abstract

To provide a food monitoring apparatus, a refrigerator including the same, and an operating method thereof.SOLUTION: Provided is a food monitoring apparatus including at least one light source configured to selectively radiate light of a first wavelength band and light of a second wavelength band, different from the first wavelength band, to food, at least one image sensor configured to obtain a visible image of the food and a hyperspectral image of the food based on the light scattered, emitted, or reflected from the food, and at least one processor configured to obtain first information on the food based on the visible image, and to obtain second information on the food based on the obtained first information and the hyperspectral image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a food monitoring device, a refrigerator including the same, and an operation method thereof.

Background Art

[0002] Techniques for obtaining information related to food and analyzing the characteristics or state of food based on the obtained information are not only for research purposes but also widely used for the purpose of providing convenience to users.

[0003] Among the conventional methods for analyzing the state of food, the method of crushing food to prepare a sample and measuring the pH, volatile basic nitrogen (VBN) content, number of microorganisms, etc. of the sample is suitable for measuring the freshness of food, etc., but the measured food is destroyed and it is not suitable for general consumers to use. Therefore, there is a need for a technique that can accurately provide information related to the state of food without destroying the measured food.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to provide a monitoring device, a refrigerator including the same, and an operation method thereof. The technical problems to be solved by the present disclosure are not limited to the technical problems as described above, and other technical problems can be inferred from the following embodiments.

Means for Solving the Problems

[0005] As a means to solve the aforementioned technical problems, a one-sided food monitoring device may include: at least one light source that selectively irradiates food with light in a first wavelength band and light in a second wavelength band different from the first wavelength band; at least one image sensor that acquires a visible image or a superspectral image related to the food from light scattered, emitted, or reflected from the food; and at least one processor that acquires first information related to the food based on the visible image and acquires second information related to the food based on the acquired first information and the superspectral image.

[0006] According to some embodiments, the at least one processor can use the acquired first information to correct at least one of the superspectral image, the parameters used to analyze the superspectral image, and the correlation between the parameters and the second information.

[0007] For example, the first information includes at least one of the type and part of the food, and the at least one processor can obtain the second information by selectively applying weighted values ​​to the parameters based on the first information, and by analyzing the superspectral image using the parameters to which the weighted values ​​have been applied.

[0008] Furthermore, the first information includes the packaging conditions of the food, and the at least one processor can correct the parameters based on the packaging conditions and obtain the second information based on the corrected parameters.

[0009] The aforementioned parameters may include at least one of the following in the food: NADH (1,4-dihydronicotinamide adenine dinucleotide) content, porphyrin content, FAD (flavin adenine dinucleotide) content, and collagen content.

[0010] Furthermore, the first information includes the location of the food in the food monitoring device, and the at least one processor controls the at least one light source to irradiate a region corresponding to the location of the food with light in the second wavelength band, and can acquire the superspectral image from the light received from the region corresponding to the location of the food.

[0011] On the other hand, the first wavelength band is in the range of 400 nm to 700 nm, the second wavelength band is in the range of 250 nm to 400 nm, and at least one image sensor can measure light in the wavelength band range of 300 nm to 700 nm.

[0012] Furthermore, a refrigerator including a food monitoring device in other aspects may include a main body having a storage space, a door for opening and closing the storage space, and the food monitoring device disposed in the storage space.

[0013] According to some embodiments, the second information includes at least one of the freshness, maturity, and spoilage of the food, and the at least one processor can determine at least one of the edible state, edible period, optimal time of consumption, and cooking method of the food based on at least one of the freshness, maturity, and spoilage.

[0014] The refrigerator may further include a control device that controls at least one of the temperature, humidity, and gas composition ratio within the storage space.

[0015] For example, the second information includes the degree of maturity of the food, and the at least one processor can determine, based on the degree of maturity of the food, whether it is necessary to accelerate or decelerate the maturation of the food in order to reach a target degree of maturity on the planned date of consumption, and can use the control device to control at least one of the temperature, humidity, and gas composition ratio in the storage space.

[0016] Furthermore, if the at least one processor determines that it is necessary to accelerate or slow down the maturation of the food, it may raise or lower the temperature in the storage space above the basic set temperature for at least a portion of the period between the time the maturation level of the food is determined and the planned date of consumption.

[0017] The refrigerator may further include a display device located in the door that outputs the second information received from the food monitoring device.

[0018] The refrigerator may further include a communication interface for communicating with an external device in order to transmit the second information to the external device.

[0019] Furthermore, a method of operating a food monitoring device from another perspective may include the steps of: using at least one light source to selectively irradiate food with light in a first wavelength band and light in a second wavelength band different from the first wavelength band; using at least one image sensor to acquire a visible image or a superspectral image relating to the food from light scattered, emitted, or reflected from the food; acquiring first information relating to the food based on the visible image; and acquiring second information relating to the food based on the acquired first information and the superspectral image.

[0020] According to some embodiments, the step of acquiring the second information may include a step of using the acquired first information to correct at least one of the superspectral image, the parameters used to analyze the superspectral image, and the correlation between the parameters and the second information.

[0021] For example, the first information includes at least one of the type and part of the food, and the step of obtaining the second information is to selectively apply weighted values ​​to the parameters based on the first information, and then use the parameters to which the weighted values ​​have been applied to analyze the superspectral image to obtain the second information.

[0022] In addition, the first information includes the packaging conditions of the food, and in the step of obtaining the second information, the parameters can be corrected based on the packaging conditions, and the second information can be obtained based on the corrected parameters.

[0023] The parameters may include at least one of the NADH content, porphyrin content, FAD content, and collagen content in the food.

[0024] In addition, the first information includes the position of the food in the food monitoring device, and in the step of obtaining the visible image or hyperspectral image related to the food, at least one light source is controlled to irradiate the region corresponding to the position of the food with light in the second wavelength band, and the hyperspectral image can be obtained from the light received from the region corresponding to the position of the food.

Brief Description of the Drawings

[0025] [Figure 1] It is a block diagram showing an example of a food monitoring device according to some embodiments. [Figure 2] It is a conceptual diagram of a food monitoring device according to some embodiments. [Figure 3] It is a drawing showing an example of parameters obtained from a hyperspectral image with different parts of the food according to some embodiments. [Figure 4] It is a drawing showing an example of parameters obtained from a hyperspectral image with different parts of the food according to some embodiments. [Figure 5] It is a drawing showing an example of data obtained from a hyperspectral image with different packaging states of the food according to some embodiments. [Figure 6] It is a flowchart for explaining the operation method of a food monitoring device according to some embodiments. [Figure 7] It is a perspective view of a refrigerator according to some embodiments. [Figure 8]This is a conceptual diagram of a refrigerator, which further includes a communication interface for communicating with an external device, according to one embodiment. [Figure 9] This is a flowchart illustrating a method, according to some embodiments, for a refrigerator to control at least one of the temperature, humidity, and gas composition ratio within the storage space. [Modes for carrying out the invention]

[0026] The terminology used in this embodiment was selected, as far as possible, to be commonly used terms, taking into account the functions of this embodiment. However, this may vary depending on the intentions of engineers in the art, precedents, or the emergence of new technologies. In some cases, terms have been arbitrarily selected, and in such cases, their meanings will be described in detail in the description of the embodiment. Therefore, the terminology used in this embodiment must not be simply names of terms, but must be defined based on the meaning of the term and the overall content of this embodiment.

[0027] In the description relating to this embodiment, singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, when a part is said to "include" a certain component, it does not mean to exclude other components, but rather that it may include other components, unless otherwise stated.

[0028] The terms "composed of" or "including" used in this embodiment should not be interpreted as necessarily including all of the various components or stages described in the specification, but rather as meaning that some of the components or stages may not be included, or that further components or stages may be included.

[0029] Furthermore, terms including ordinal numbers, such as "first" or "second" as used herein, may be used to describe a variety of components, but such components are not limited by such terms. These terms are also used to distinguish one component from another.

[0030] The description relating to the embodiments below is not intended to be interpreted as limiting the scope of rights, and anything that a person skilled in the art can easily infer should be interpreted as falling within the scope of rights of the embodiments. Hereinafter, illustrative embodiments will be described in detail with reference to the attached drawings.

[0031] Figure 1 is a block diagram showing an example of a food monitoring device according to some embodiments.

[0032] The food monitoring device 10 can be any device that determines the state of food, without limitation. For example, the food monitoring device 10 may be installed in food storage equipment such as refrigerators, kimchi refrigerators, warming cabinets, storage containers, and airtight containers, and may be a device that determines the state of food, but is not necessarily limited to these. The food monitoring device 10 also falls under the category of a device used to analyze the state of food for research purposes. On the other hand, "food" refers to food and beverages consumed on a daily basis, and may include, but is not necessarily limited to, meat, fish, eggs, grains, vegetables, fruits, dairy products, etc.

[0033] Referring to Figure 1, the food monitoring device 10 may include at least one light source 110, at least one image sensor 120, and at least one processor 130. However, only the components related to this embodiment are shown in the food monitoring device 10 illustrated in Figure 1. Therefore, it will be obvious to those skilled in the art that the food monitoring device 10 may further include other general-purpose components in addition to those shown in Figure 1. For example, the food monitoring device 10 may further include memory (not shown).

[0034] Furthermore, if the objectives of this disclosure can be achieved by including only some of the components shown in Figure 1, then a device including only some of the components shown in Figure 1 may also be considered a food monitoring device 10. For example, the food monitoring device 10 includes only at least one image sensor 120 and at least one processor 130, and at least one light source 110 is also located outside the food monitoring device 10.

[0035] The memory is hardware that stores various types of data processed within the food monitoring device 10. For example, the memory can store data processed by the food monitoring device 10 and data that is being processed. The memory can also store applications, drivers, and the like that are driven by the food monitoring device 10.

[0036] The memory includes RAM (random access memory) such as DRAM (dynamic random access memory) and SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), CD-ROM (compact disc read-only memory), Blu-ray or other optical disc storage, HDD (hard disk drive), SSD (solid static drive), or flash memory, and may also include other external storage devices that can be accessed by the food monitoring device 10.

[0037] At least one light source 110 may mean a device for irradiating food with light. At least one light source 110 can irradiate food with light in multiple different wavelength bands. For example, at least one light source 110 can selectively irradiate food with light in a first wavelength band and light in a second wavelength band different from the first wavelength band. The first wavelength band may be in the range of 400 nm to 700 nm, and the second wavelength band may be in the range of 250 nm to 400 nm.

[0038] For example, at least one light source 110 may be an LED (light-emitting diode) or fluorescent lamp that emits broad light in the visible light band, or a laser diode that emits high-intensity light of a short wavelength. However, it is not necessarily limited to these. At least one light source 110 may emit light in a wavelength band suitable for acquiring first or second information related to food.

[0039] Furthermore, at least one light source 110 is a single light source capable of selectively irradiating food with light from multiple different wavelength bands. However, it is not limited to this, and at least one light source 110 may include multiple light sources, each irradiating light from one wavelength band. Also, at least one light source 110 may include multiple light sources capable of selectively irradiating food with light from multiple different wavelength bands.

[0040] At least one image sensor 120 may mean a device for acquiring a visible image or hyperspectral image related to food from light scattered, emitted, or reflected from food. In one example, at least one image sensor 120 can acquire a visible image or hyperspectral image related to food by measuring light in the wavelength range of 300 nm to 700 nm. However, it is not limited to these, and at least one image sensor 120 can measure light in any wavelength range suitable for acquiring information related to food.

[0041] For example, at least one image sensor 120 may be, but is not limited to, a photodiode array, a CCD (charge coupled device) sensor, or a CMOS (complementary metal oxide semiconductor) sensor capable of acquiring a visible image. At least one image sensor 120 may acquire a visible image that includes information relating to the appearance of the food, such as its hue or shape. For example, at least one image sensor 120 may acquire an RGB image relating to the food.

[0042] Furthermore, at least one image sensor 120 can acquire a superspectroscopic image containing information related to fluorescence emitted from the food. Indicator substances in the food can emit fluorescence when irradiated with light from at least one light source 110. For example, the indicator substance may include at least one of NADH (1,4-dihydronicotinamide adenine dinucleotide), porphyrin, FAD (flavin adenine dinucleotide), and collagen. The NADH may refer to the reduced form of NAD (nicotinamide adenine dinucleotide). Parameters indicating the content of the indicator substance are also acquired from the superspectroscopic image and are used to acquire secondary information. As an example, since NADH is a substance produced as a result of cellular respiration and is produced only by microorganisms after the death of an animal, information such as the freshness, maturity, and spoilage of the food can be acquired from the NADH content.

[0043] Furthermore, at least one image sensor 120 is a single image sensor capable of acquiring a visible image or a superspectral image related to food from light scattered, emitted, or reflected from food. However, it is not limited to this, and at least one image sensor 120 may include multiple image sensors, each acquiring one visible image or a superspectral image related to food from light scattered, emitted, or reflected from food. Alternatively, at least one image sensor 120 may include multiple image sensors, each acquiring multiple visible images or superspectral images related to food from light scattered, emitted, or reflected from food.

[0044] At least one processor 130 is responsible for performing general functions for controlling the food monitoring device 10. For example, at least one processor 130 can control the operation of at least one light source 110 and at least one image sensor 120. Alternatively, at least one processor 130 can also be represented by an array of numerous logic gates, or by a combination of a general-purpose microprocessor and a memory in which a program that can be executed by the microprocessor is stored.

[0045] At least one processor 130 can acquire first information related to food based on a visible image acquired from at least one image sensor 120, and acquire second information related to food based on the acquired first information and a superspectral image acquired from at least one image sensor 120. In this way, at least one processor 130 does not acquire second information by using only the information acquired through the analysis of the superspectral image, but rather acquires second information by comprehensively considering the information acquired through the analysis of the visible image, and the accuracy of the second information related to food may be increased.

[0046] For example, the first information may include at least one of the following: type of food, part of the food, packaging conditions, and location of the food in the food monitoring device, and the second information may include at least one of the following: freshness, maturity, and spoilage of the food, but is not necessarily limited to these.

[0047] At least one processor 130 can use a recognition algorithm to identify the presence of food and acquire information relating to the type of food, part, packaging conditions, and location of the food within the food monitoring device. This recognition algorithm may include classification algorithms, clustering algorithms, ensemble learning algorithms, general algorithms for predicting arbitrary structured labels, and regression algorithms.

[0048] The classification algorithm may include SVC, Naive Bayes Classifier, K-nearest Neighbors Classifier, Ensemble Classifiers, SGD Classifier, kernel approximation, neural network, SVM, decision trees, logistic regression, etc., and the clustering algorithm may include Spectral Clustering, Kmeans Clustering, MiniBatch Kmeans Clustering, etc. Furthermore, the ensemble learning algorithm may include Boosting (meta-algorithm), Bootstrap aggregating ("bagging"), Ensemble averaging, etc., and general algorithms for predicting arbitrary structured labels may include Bayesian networks and Markov random fields, and the regression algorithm may include Gaussian process regression (kriging), Linear / Nonlinear regression and extensions, Neural networks and Deep learning methods, independent component analysis (ICA), principal component analysis (PCA), etc.

[0049] On the other hand, the neural network may also be a deep neural network (DNN) or an n-layer neural network containing one or more hidden layers. The DNN may include, but is not limited to, CNNs (convolutional neural networks), RNNs (recurrent neural networks), Deep Belief Networks, Restricted Boltzman Machines, etc. If the pattern recognition algorithm is a neural network, the estimation results of the pattern recognition algorithm also correspond to the inference results of the neural network.

[0050] On the other hand, at least one processor 130 can determine the freshness, maturity, and spoilage of food based on at least one of the following: NADH content, porphyrin content, FAD content, and collagen content, or the ratio of their contents. When food is irradiated with light in a specific wavelength range, at least one processor 130 can measure the NADH content based on a spectral distribution related to the wavelength range of approximately 430 nm to approximately 550 nm, and can measure the porphyrin content based on a spectral distribution related to the wavelength range of approximately 570 nm to approximately 630 nm. Furthermore, at least one processor 130 can measure the FAD content based on a spectral distribution related to the wavelength range of approximately 500 nm to approximately 550 nm, and can measure the collagen content based on a spectral distribution related to the wavelength range of approximately 360 nm to approximately 420 nm.

[0051] For example, at least one processor 130 can use the information acquired through visible image analysis to correct at least one of the following: a superspectral image, parameters used to analyze the superspectral image, and the correlation between the parameters and information related to the state of the food. As will be described later, if the information acquired through visible image analysis includes the location of the food in the food monitoring device, at least one processor 130 can acquire a clearer superspectral image related to the object being measured by having at least one light source 110 irradiate the region corresponding to the food location with light in the second wavelength band. If the information acquired through visible image analysis includes the packaging conditions of the food, at least one processor 130 can amplify the parameter content according to a predetermined ratio for each packaging condition. Furthermore, if the information acquired through visible image analysis includes at least one of the type and part of the food, at least one processor 130 can assign weighted values ​​to the parameters. However, it is not necessarily limited to these.

[0052] Thus, at least one processor 130 does not determine the state of the food by using only the information obtained through the analysis of the superspectral image, but rather by comprehensively considering the information obtained through the analysis of the visible image to determine the state of the food, although the accuracy of determining the state of the food may be increased.

[0053] The process by which the food monitoring device 10 operates will be explained in more detail below, with reference to Figure 2.

[0054] Figure 2 is a conceptual diagram of a food monitoring device according to one embodiment.

[0055] Referring to Figure 2, a conceptual diagram illustrating the process by which the food monitoring device 10 of Figure 1 determines the state of the food 140 is shown. At least one light source 110 can selectively irradiate the food 140 with light in a first wavelength band and light in a second wavelength band different from the first wavelength band, causing the light to be scattered, emitted, or reflected from the food.

[0056] According to some embodiments, at least one processor 130 can acquire information relating to the location of the food 140 based on a visible image acquired from at least one image sensor 120, and precisely irradiate the location of the food 140 with light in the second wavelength band. In such a case, the light can be concentrated on the food 140 being measured, compared to when the light is irradiated over the entire measurement space of the food monitoring device 10. As a result, signals from other foods or articles other than the food 140 can be filtered out, and the maximum output of at least one light source 110 can be used to determine the state of the food 140, while the accuracy of the information relating to the state of the food acquired from the superspectral image can be increased.

[0057] For example, if the information obtained through visible image analysis includes the location of food in the food monitoring device, at least one processor 130 can control at least one light source 110 to irradiate the region corresponding to the food location with light in the second wavelength band, and a superspectral image can be obtained from the light received from the region corresponding to the food location.

[0058] Figures 3 and 4 are diagrams illustrating examples of parameters obtained from superspectroscopic images of different parts of food according to some embodiments.

[0059] Figure 3 shows graphs 310 and 320, which represent the intensity of light scattered, emitted, and reflected from the adipose tissue of meat in the wavelength range of 350 nm to 750 nm. Graph 310 corresponds to the measurement results for the adipose tissue of fresh meat, and graph 320 corresponds to the measurement results for the adipose tissue of spoiled meat. Figure 4 shows graphs 410 and 420, which represent the intensity of light scattered, emitted, and reflected from the muscle tissue of meat in the wavelength range of 350 nm to 750 nm. Graph 410 corresponds to the measurement results for the muscle tissue of fresh meat, and graph 420 corresponds to the measurement results for the muscle tissue of spoiled meat.

[0060] In the illustrated graph, the intensity in the wavelength range of approximately 430 nm to 550 nm corresponds to the NADH content, and the intensity in the wavelength range of approximately 570 nm to 630 nm corresponds to the porphyrin content. Based on such graph patterns and the intensity in specific wavelength ranges, information related to the freshness, spoilage, and maturity of food can be obtained. For example, information related to the freshness, spoilage, and maturity of food can be obtained using the recognition algorithm explained with reference to Figure 1.

[0061] However, a comparison of Figures 3 and 4 reveals that the patterns of adipose tissue measurement in graphs 310 and 320, and the patterns of muscle tissue measurement in graphs 410 and 420, differ depending on the freshness level. Therefore, the criteria for determining the freshness of food vary depending on the part of the food.

[0062] Furthermore, the way parameters change can differ depending on the type of food, for example, whether the food being measured is meat, fish, eggs, grains, vegetables, fruits, or dairy products. Therefore, similar to the case of food parts, the criteria for determining the freshness of food will also differ depending on the type of food. In one example, food types can be classified by their porphyrin content, and the porphyrin content can be analyzed based on the hue of the food's appearance obtained from a visible image.

[0063] For example, if the information obtained through visible image analysis includes at least one of the food type and part, at least one processor (e.g., at least one processor 130 in Figure 1 or Figure 2) can selectively apply weighted values ​​to parameters based on such information, and then use the weighted parameters to analyze the superspectral image to obtain information related to the state of the food. When weighted values ​​are applied to parameters based on the food type or part, the state of the food can be determined by using parameters that are more suitable for analyzing the state of the food being measured. Therefore, the accuracy of information related to the state of the food can be increased.

[0064] Figure 5 is a diagram illustrating examples of data obtained from superspectroscopic images of food in different packaging states according to some embodiments. Graph 510 shows data obtained for food wrapped in plastic wrap, and Graph 520 shows data obtained for food vacuum-packed in plastic.

[0065] In the graphs 510 and 520 shown, the intensity in the wavelength range of approximately 430 nm to 550 nm corresponds to the NADH content, and the intensity in the wavelength range of approximately 570 nm to 630 nm corresponds to the porphyrin content.

[0066] Referring to Figure 5, in the case of Graph 520, compared to Graph 510, a prominent peak is shown in the wavelength band of approximately 430 nm to approximately 550 nm, which corresponds to the NADH content. If signals from items other than the food being measured are not removed, the accuracy of the method for determining the state of the food will decrease, so the parameters must be corrected according to the packaging conditions of the food.

[0067] For example, if the information obtained through visible image analysis includes the packaging conditions of the food, at least one processor (e.g., at least one processor 130 in Figure 1 or Figure 2) can correct the parameters based on the packaging conditions and, based on the corrected parameters, obtain information related to the state of the food.

[0068] For example, if signals from packaging are removed, the patterns or parameter changes in the graph become clearer, which can increase the accuracy of information related to the state of the food.

[0069] Furthermore, if some packaging material presents on the food that interferes with the irradiation of light in the second wavelength band, the light in the second wavelength band can be irradiated only to the portion of the food that does not have such packaging material. In such cases, the signal from the packaged portion of the food being measured will decrease. Therefore, the accuracy of information related to the state of the food may be increased.

[0070] Figure 6 is a flowchart illustrating the operation method of a food monitoring device according to one embodiment.

[0071] Referring to Figure 6, the operation method of the food monitoring device consists of stages processed chronologically in the food monitoring device 10 shown in Figures 1 and 2. Therefore, even if some details are omitted below, it can be seen that the content described above for Figures 1 to 5 also applies to the operation method of the food monitoring device in Figure 6.

[0072] In step 610, the food monitoring device can selectively irradiate the food with light in a first wavelength band and light in a second wavelength band different from the first wavelength band, using at least one light source.

[0073] In one example, the food monitoring device may utilize at least one light source to selectively irradiate with light in a first wavelength band ranging from 400 nm to 700 nm and light in a second wavelength band ranging from 250 nm to 400 nm. However, it is not necessarily limited to these.

[0074] In step 620, the food monitoring device can use at least one image sensor to acquire a visible image or a superspectral image of the food from light scattered, emitted, or reflected from the food. At least one image sensor can acquire a visible image that includes information relating to the appearance of the food, such as its hue or shape. At least one image sensor can also acquire a superspectral image that includes information relating to fluorescence emitted from the food.

[0075] In one example, the food monitoring device may utilize at least one image sensor to measure light in the wavelength range of 300 nm to 700 nm, but is not necessarily limited to this.

[0076] In step 630, the food monitoring device can acquire first information relating to the food based on a visible image. This first information may include any information necessary to accurately measure the condition of the food. For example, the first information may include, but is not limited to, the type of food, part of the food, packaging conditions, and the location of the food within the food monitoring device.

[0077] In step 640, the food monitoring device can acquire second information relating to the food based on the acquired first information and hyperspectral image. The second information relating to the food may include all information indicating the state of the food. For example, the second information may include at least one of the freshness, maturity, and spoilage of the food, but is not necessarily limited to these.

[0078] The food monitoring device may include a step of using the acquired first information to correct at least one of the following: a superspectral image, parameters used to analyze the superspectral image, and the correlation between the parameters and the second information. For example, the parameters may include, but are not necessarily limited to, at least one of the following: NADH content, porphyrin content, FAD content, and collagen content in the food.

[0079] On the other hand, if the first information includes at least one of the food type and part, the food monitoring device can also obtain second information by selectively applying weighted values ​​to parameters based on the first information and analyzing a superspectral image using the parameters to which the weighted values ​​have been applied.

[0080] Furthermore, if the first information includes the packaging conditions of the food, the food monitoring device can also correct the parameters based on the packaging conditions and acquire the second information based on the corrected parameters.

[0081] On the other hand, if the first information includes the location of food within the food monitoring device, the food monitoring device can control at least one light source to irradiate a region corresponding to the location of food with light in a second wavelength band, and acquire a superspectral image from the light received from the region corresponding to the location of food.

[0082] On the other hand, the operating method of the aforementioned food monitoring device is also recorded on a computer-readable recording medium on which one or more programs containing the instruction words to execute that method are recorded. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs (compact disc read-only memory) and DVDs (digital versatile discs); magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instruction words include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

[0083] Figure 7 is a perspective view of a refrigerator according to one embodiment.

[0084] Referring to Figure 7, a refrigerator 70 including a main body 710, a door 720, a display device 730, and a food monitoring device (not shown) may be provided. The food monitoring device corresponds to the food monitoring device 10 in Figures 1 and 2, so a redundant explanation will be omitted.

[0085] The main unit 710 has a storage space in which not only one food item but multiple food items can be placed. The door 720 can open and close the storage space, thereby creating an environment within the storage space that is independent of the external environment. The display device 730 is positioned in front of the door 720 and can output second information received from the food monitoring device. The display device 730 provides the user with the second information received from the food monitoring device by outputting video or audio.

[0086] According to some embodiments, if the second information includes at least one of the freshness, maturity, and spoilage of the food, at least one processor included in the food monitoring device (for example, at least one processor 110 in Figure 1 or Figure 2) can determine at least one of the following based on at least one of the freshness, maturity, and spoilage: whether the food is edible, the period during which it can be eaten, the optimal time for consumption, and the cooking method. The at least one processor can also provide the user with the determined edibleness, period during which it can be eaten, the optimal time for consumption, or the cooking method via the display device 730.

[0087] According to some embodiments, the refrigerator 70 may further include a control device that controls at least one of the temperature, humidity, and gas composition ratio within the storage space. The control device controls the temperature within the storage space to a range of -25°C to 10°C, but is not necessarily limited to that range. Since the storage space forms an environment independent of the outside, it is possible to control the gas composition ratio within the storage space, and preferably, the oxygen ratio in the gas within the storage space is controlled.

[0088] For example, if the second piece of information includes the degree of maturity of the food, at least one processor can determine, based on the degree of maturity of the food, whether it is necessary to accelerate or slow down the maturation process related to the food in order to reach the target degree of maturity on the day the food is to be consumed. Based on such a determination, at least one processor can control the temperature, humidity, and gas composition ratio within the storage space using a control device that controls at least one of these three.

[0089] At least one processor calculates the estimated time required from the time the food's maturity is determined until the planned date of consumption. If the temperature at the time the food's maturity is determined is not high enough to reach the target maturity level by the planned consumption date, it decides to accelerate the maturation process. If the temperature is high enough to exceed the target maturity level by the planned consumption date, it decides to slow down the maturation process.

[0090] Furthermore, if at least one processor determines that it is necessary to accelerate or slow down the maturation of the food, it may raise or lower the temperature in the storage space above the default temperature for at least a portion of the period between the time the maturation level of the food is determined and the planned date of consumption.

[0091] The maturation rate is related to the ambient temperature, humidity, and gas composition ratio. Increasing the temperature, decreasing the humidity, or increasing the oxygen ratio in the gas increases the maturation rate, while decreasing the temperature, increasing the humidity, or decreasing the oxygen ratio in the gas reduces the maturation rate.

[0092] For example, if accelerating food maturation is required, at least one processor can accelerate the chemical reactions necessary for maturation by increasing the temperature within the storage space. Thus, food maturation can be accelerated. If slowing down food maturation is required, at least one processor can slow down the chemical reactions necessary for maturation by decreasing the temperature within the storage space. This, in turn, can slow down food maturation.

[0093] If it is necessary to accelerate or slow down food maturation, at least one processor can calculate the temperature difference between the temperature in the storage space at the time the food's maturation level is determined and the temperature at which the target maturation level can be reached on the planned consumption date, and raise or lower the temperature in the storage space by that temperature difference. In such a case, the user can obtain food at the desired maturation level on the consumption date, without the need for further maturation time due to insufficient maturation, or a deterioration in the taste or texture of the food due to over-maturation.

[0094] Furthermore, if the temperature in the storage space is raised or lowered from the default setting temperature to accelerate or slow down food maturation, at least one processor may change the temperature in the storage space to the default setting temperature or a temperature preferred by the user for consumption or cooking before the user consumes or cooks the food. In such cases, the user can obtain food at a suitable temperature for consumption or cooking, thus saving time that would otherwise be required for heating or cooling the food, such as thawing frozen food, before consumption or cooking.

[0095] On the other hand, the processing performed by the aforementioned at least one processor is also carried out by a separate processor provided in the refrigerator, in addition to the at least one processor included in the food monitoring device.

[0096] Figure 8 is a conceptual diagram of a refrigerator that further includes a communication interface for communicating with an external device, according to one embodiment.

[0097] Referring to Figure 8, the refrigerator 70 may include a communication interface (not shown) for communicating with an external device in order to transmit second information to the external device. The refrigerator 70 can communicate with the server device 810 and / or the mobile terminal 820 via the communication interface. The communication interface may include a short-range wireless communication interface, a mobile communication unit, etc. The short-range communication unit may include, but is not limited to, a Bluetooth communication unit, a BLE (Bluetooth low energy) communication unit, a near-field communication interface, a WLAN (wireless local area network) (Wi-Fi (wireless fidelity)) communication unit, a ZigBee communication unit, an infrared (IrDA: infrared data association) communication unit, a WFD (Wi-Fi direct) communication unit, a UWB (ultra-wideband) communication unit, an Ant+ communication unit, etc.

[0098] The refrigerator 70, which includes a communication interface for communicating with external devices, transmits information related to the condition of food to the user's mobile terminal 820, enabling the user to monitor information related to the condition of food in the storage space without restrictions on location or time. Furthermore, since information related to food is provided to the server device 810 from multiple refrigerators 70, making it easy for the user to identify and manage the condition of a large number of arbitrary products, the refrigerator 70, which includes a communication interface for communicating with external devices, can also be applied to distribution management.

[0099] Figure 9 is a flowchart illustrating a method, according to some embodiments, for a refrigerator (for example, refrigerator 70 (Figure 7)) to control at least one of the temperature, humidity, and gas composition ratio within the storage space.

[0100] Referring to Figure 9, the method by which the refrigerator controls at least one of the temperature, humidity, and gas composition ratio within the storage space consists of steps processed sequentially in the refrigerator 70 shown in Figures 7 and 8. Therefore, even if the details are omitted below, it can be seen that the contents described above for Figures 7 and 8 also apply to the method by which the refrigerator in Figure 9 controls at least one of the temperature, humidity, and gas composition ratio within the storage space.

[0101] Furthermore, the method shown in Figure 9 can also be performed by a food monitoring device (e.g., food monitoring device 10 (Figure 1 or Figure 2)) included in a refrigerator (e.g., refrigerator 70 (Figure 7 or Figure 8)), or by a separate processor outside the food monitoring device. On the other hand, when the method shown in Figure 9 is performed by the food monitoring device, the method shown in Figure 9 can also be performed by any component included in the food monitoring device (e.g., at least one processor 130 (Figure 1 or Figure 2)).

[0102] In step 910, the refrigerator can selectively irradiate food with light in a first wavelength band and light in a second wavelength band different from the first wavelength band, using at least one light source. Since step 910 in Figure 9 corresponds to step 610 in Figure 6, a redundant explanation is omitted.

[0103] In step 920, the refrigerator can use at least one image sensor to acquire a visible image or a superspectral image of the food from the light scattered, emitted, or reflected from the food. Since step 920 in Figure 9 corresponds to step 620 in Figure 6, a redundant explanation will be omitted.

[0104] In step 930, the refrigerator can acquire first information related to the food based on the visible image. Since step 930 in Figure 9 corresponds to step 630 in Figure 6, a redundant explanation is omitted.

[0105] In step 940, the refrigerator can acquire second information related to the food based on the acquired first information and the superspectral image. Since step 940 in Figure 9 corresponds to step 640 in Figure 6, a redundant explanation is omitted.

[0106] In step 950, if the second piece of information includes the degree of maturity of the food, the refrigerator can determine, based on the degree of maturity of the food, whether it is necessary to accelerate or slow down the maturation of the food in order to reach the target degree of maturity on the planned date of consumption. For example, the refrigerator can calculate the estimated time required from the time the degree of maturity of the food is determined until the planned date of consumption, and if the temperature at the time the degree of maturity of the food is determined is not high enough to reach the target degree of maturity on the planned date of consumption, it can decide to accelerate the maturation; if it is high enough to exceed the target degree of maturity on the planned date of consumption, it can decide to slow down the maturation.

[0107] In step 960, if it is determined that it is necessary to accelerate or slow down the maturation of the food, the refrigerator may use a control device that controls at least one of the temperature, humidity, and gas composition ratio to control at least one of these in the storage space. For example, the temperature in the storage space may be raised or lowered above the basic set temperature for at least a portion of the period between when the degree of maturation of the food is determined and the planned date of consumption.

[0108] The refrigerator calculates the temperature difference between the temperature inside the storage space at the time the food's maturity is determined and the temperature at which the target maturity can be reached by the planned consumption date, and can raise or lower the temperature inside the storage space by that temperature difference. Furthermore, if the temperature inside the storage space is raised or lowered from the basic setting temperature to accelerate or slow down the maturation of the food, the refrigerator can change the temperature inside the storage space to the basic setting temperature or a temperature preferred by the user for consumption or cooking before the user consumes or cooks the food.

[0109] If, in step 970, it is determined that there is no need to accelerate or slow down the maturation of the food, the refrigerator can maintain the temperature, humidity, and gas composition ratio within the storage space.

[0110] Although these embodiments have been described in detail above, the scope of the present invention is not limited thereto. Many modifications and improvements made by those skilled in the art, utilizing the basic concepts of the present invention as defined in the claims, also fall within the scope of the present invention. [Explanation of Symbols]

[0111] 10 Food monitoring devices 70 Refrigerator 110 Light source 120 Image Sensors 130 processors 140 Food 710 Main Unit 720 doors 730 Display Devices 810 Server Device 820 Mobile devices

Claims

1. A food monitoring device, A food is irradiated with at least one light source that selectively emits light in a first wavelength band and light in a second wavelength band different from the first wavelength band, The system includes at least one image sensor that acquires a visible image and a superspectral image related to the food from light scattered, emitted, or reflected from the food, The system includes at least one processor that acquires first information relating to the food based on the visible image, and acquires second information relating to the food based on the acquired first information and the superspectral image, The processor obtains from the superspectroscopic image at least one parameter indicating the content of an indicator substance in the food that emits fluorescence when irradiated with light from the light source, A food monitoring device that corrects at least one of the parameters obtained from the superspectral image based on the first information obtained based on the visible image, and obtains second information based on the corrected parameter.

2. The food monitoring apparatus according to claim 1, wherein the at least one processor uses the acquired first information to correct at least one of the superspectral image, the parameters used to analyze the superspectral image, and the correlation between the parameters and the second information.

3. The first information includes at least one of the types and parts of the food, The food monitoring apparatus according to claim 2, wherein the at least one processor selectively applies weighted values ​​to the parameters based on the first information, and obtains the second information by analyzing the superspectral image using the parameters to which the weighted values ​​have been applied.

4. The first information includes the packaging conditions of the food, The food monitoring apparatus according to claim 2, wherein the at least one processor corrects the parameters based on the packaging conditions and obtains the second information based on the corrected parameters.

5. The food monitoring device according to claim 2, wherein the parameter indicating the content of the indicator substance includes at least one of the following in the food: NADH (1,4-dihydronicotinamide adenine dinucleotide) content, porphyrin content, FAD (flavin adenine dinucleotide) content, and collagen content.

6. The first information includes the location of the food within the food monitoring device, The food monitoring apparatus according to any one of claims 1 to 5, wherein the at least one processor controls the at least one light source to irradiate a region corresponding to the location of the food with light in the second wavelength band, and acquires the superspectral image from the light received from the region corresponding to the location of the food.

7. The first wavelength band is in the range of 400 nm to 700 nm. The second wavelength band is in the range of 250 nm to 400 nm. The food monitoring apparatus according to any one of claims 1 to 6, wherein the at least one image sensor measures light in a wavelength band in the range of 300 nm to 700 nm.

8. A main body having a storage space, A door for opening and closing the aforementioned storage space, A refrigerator comprising a food monitoring device according to any one of claims 1 to 7, which is disposed in the aforementioned storage space.

9. The second piece of information includes at least one of the freshness, maturity, and spoilage of the food. The refrigerator according to claim 8, wherein the at least one processor determines at least one of the following based on freshness, maturity, and spoilage: whether the food is edible, the period during which it can be consumed, the optimal time for consumption, and the cooking method.

10. The refrigerator according to claim 8, further comprising a control device for controlling at least one of the temperature, humidity, and gas composition ratio within the storage space.

11. The second information includes the degree of maturity of the food, The refrigerator according to claim 10, wherein the at least one processor determines, based on the degree of maturation of the food, whether or not it is necessary to accelerate or decelerate the maturation of the food in order to reach a target degree of maturation on the day the food is to be consumed, and controls at least one of the temperature, humidity and gas composition ratio in the storage space using the adjustment device.

12. The aforementioned at least one processor is The refrigerator according to claim 11, wherein, if it is determined that it is necessary to accelerate or slow down the maturation of the food, the temperature inside the storage space is raised or lowered above the basic set temperature for at least a portion of the period between the time when the degree of maturation of the food is determined and the planned date of consumption.

13. The refrigerator according to claim 8, further comprising a display device disposed on the door and outputting the second information received from the food monitoring device.

14. The refrigerator according to claim 8, further comprising a communication interface for communicating with an external device in order to transmit the second information to the external device.

15. In the operation method of a food monitoring device, A step of selectively irradiating food with light in a first wavelength band and light in a second wavelength band different from the first wavelength band, using at least one light source. A step of using at least one image sensor to obtain a visible image and a superspectral image related to the food from light scattered, emitted, or reflected from the food, Based on the aforementioned visible image, a step is taken to obtain first information relating to the food, The process includes the step of acquiring second information relating to the food based on the first information acquired and the superspectroscopic image, The stage of obtaining the aforementioned second information is, A step of obtaining from the superspectroscopic image at least one parameter indicating the content of an indicator substance in the food that emits fluorescence when irradiated from the light source, A method comprising the steps of: correcting at least one parameter obtained from the superspectroscopic image based on the first information obtained based on the visible image, and obtaining second information based on the corrected parameter.

16. The method according to claim 15, wherein the step of acquiring the second information includes a step of using the acquired first information to correct at least one of the superspectroscopic image, the parameters used to analyze the superspectroscopic image, and the correlation between the parameters and the second information.

17. The first information includes at least one of the types and parts of the food, The method according to claim 16, wherein the step of acquiring the second information is to selectively apply weighted values ​​to the parameters based on the first information, and to acquire the second information by analyzing the superspectroscopic image using the parameters to which the weighted values ​​have been applied.

18. The first information includes the packaging conditions of the food, The method according to claim 16, wherein the step of obtaining the second information involves correcting the parameters based on the packaging conditions, and obtaining the second information based on the corrected parameters.

19. The method according to claim 16, wherein the parameter indicating the content of the indicator substance includes at least one of the following in the food: NADH (1,4-dihydronicotinamide adenine dinucleotide) content, porphyrin content, FAD (flavin adenine dinucleotide) content, and collagen content.

20. The first information includes the location of the food within the food monitoring device, The method according to any one of claims 15 to 19, wherein the step of acquiring a visible image or superspectral image relating to the food is to control the at least one light source so as to irradiate a region corresponding to the location of the food with light in the second wavelength band, and acquire the superspectral image from the light received from the region corresponding to the location of the food.

Images