Food measuring method, device and program

Abstract

The present invention relates to a method for measuring food, and the present invention calculates the amount of food based on information regarding the size, depth, and capacity of a plurality of spaces formed in a dish in which food is served and an image captured by a photographing means.

Description

[Technical field]

[0001] The present invention relates to a method for measuring food, and in particular to a method for measuring the amount of food using volume. [Background technology]

[0002] In recent years, the demand for food measuring technology has increased as more people try to maintain healthy diets, focusing on well-being and dieting.

[0003] This system can be used in places that provide meals to large numbers of people, such as schools, companies, the military, government offices, and hospitals, to measure the amount of food served and distributed to people.By measuring the amount of food distributed and the amount of food left over, this system has many advantages, such as being able to predict demand and supply, allowing for efficient food distribution management and the calorie content of people being served.

[0004] However, most of the technologies currently available are limited to simply searching images taken by a camera, which results in significantly reduced accuracy. In addition, the reduced accuracy in image searching leads to even larger errors in the next step, such as calculating calories.

[0005] The inventor then came up with an invention that goes beyond simply searching for food images, but can also accurately analyze and calculate them. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a food measuring method that calculates the volume of food using height information for each pixel of an image captured by a stereo camera or a depth measurement device.

[0007] Another object of the present invention is to provide a food measuring method that corrects the calculated volume of food by using information about serving dishes.

[0008] The problems that the present invention is intended to solve are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0009] A food measuring method according to one embodiment of the present invention for solving the above-mentioned problems includes a step of an analysis unit analyzing an image of a serving dish (a dish in which a user is served a meal) and recognizing one or more foods contained in the dish, a step of a calculation unit calculating a volume of each food using dish information and an image of the food in the dish image, and a step of the calculation unit calculating a weight of each food using the food information and the calculated volume of each food, and the server includes a database storing an image of an empty dish, dish information including size, depth and capacity of a plurality of spaces formed in the dish, and food information including weight per volume of each food.

[0010] In addition, a food measuring server according to one embodiment of the present invention for solving the above-mentioned problems includes a database storing an image of an empty tableware, tableware information including the size, depth and capacity of multiple spaces formed in the tableware, and food information including the weight per volume of each food, an analysis unit that analyzes an image of a serving tableware (tableware in which a user is served food) and recognizes one or more foods contained in the tableware, and a calculation unit that calculates the volume of each food using the tableware information and the image of the food in the tableware image, and calculates the weight of each food using the food information and the calculated volume of each food.

[0011] Furthermore, a food measuring device according to one embodiment of the present invention for solving the above-mentioned problems includes a database storing an image of an empty tableware, tableware information including the size, depth and capacity of multiple spaces formed in the tableware, and food information including the weight per volume of each food, an analysis unit that analyzes an image of a serving tableware (tableware in which a user is served food) and recognizes one or more foods contained in the tableware, and a calculation unit that calculates the volume of each food using the tableware information and the image of the food in the tableware image, and calculates the weight of each food using the food information and the calculated volume of each food.

[0012] In addition, other methods and systems for implementing the present invention, and a computer-readable recording medium having a computer program for executing the methods may also be provided. Effect of the Invention

[0013] According to the present invention as described above, the volume of food can be calculated using height information for each pixel of an image captured by a stereo camera or a depth measuring device, thereby providing an advantage of being able to calculate an accurate food volume.

[0014] In addition, the present invention has the effect of correcting the calculated volume of food based on information regarding the size, depth and capacity of the serving dish space, thereby enabling a more accurate calculation of the volume of food.

[0015] The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. [Brief description of the drawings]

[0016] [Figure 1] 2 is a flowchart of a food measuring method according to an embodiment of the present invention. [Figure 2A] 1 is a perspective view of a tableware according to an embodiment of the present invention. [Figure 2B] 1 is a perspective view of tableware having an identification code attached to the back or side thereof; [Diagram 3] 1 is a perspective view of a food container according to an embodiment of the present invention; [Figure 4] FIG. 4 is a perspective view of the tableware in FIG. 3 in which two foods are stacked in one space. [Diagram 5] FIG. 2 is a perspective view of a dish in which the space is not divided. [Figure 6] 1 is a flow diagram of a method for providing an enhanced local model to improve accuracy of food type determination. [Figure 7] 4 is a flowchart of a food measuring method according to another embodiment of the present invention. [Figure 8] FIG. 13 is a perspective view of a table illustrating a background target. [Figure 9] 2 is a flowchart illustrating a method for providing restaurant operation information and dining subject management information according to an embodiment of the present invention; [Figure 10] FIG. 2 is a block diagram of a food measurement server according to an embodiment of the present invention. [Figure 11] FIG. 11 is a detailed block diagram of the management unit of FIG. 10. [Figure 12] 1 is a block diagram of a food measuring device according to an embodiment of the present invention. [Figure 13] 1 is a perspective view of a food measuring device according to an embodiment of the present invention; [Figure 14] 1 is a perspective view of a food measuring device according to an embodiment of the present invention; [Figure 15] 1 is a perspective view of a food measuring device according to an embodiment of the present invention; [Figure 16] 1 is a perspective view of a food measuring device according to an embodiment of the present invention; [Figure 17] 2 is a screen of a food measuring device according to an embodiment of the present invention. [Figure 18] FIG. 2 is a block diagram of a food measurement server according to an embodiment of the present invention. [Figure 19] FIG. 2 is a block diagram of a user terminal according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The advantages and features of the present invention, and the methods for achieving them, will become clear from the following detailed description of the embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be embodied in various different forms. However, the embodiments are provided to complete the disclosure of the present invention and to allow those skilled in the art to fully understand the scope of the present invention, and the present invention is only defined by the scope of the claims.

[0018] The terms used in this specification are for the purpose of describing the embodiments and are not intended to limit the present invention. In this specification, the singular includes the plural unless otherwise specified. The terms "comprises" and / or "comprising" used in this specification do not exclude the presence or addition of one or more other components other than the components mentioned. The same reference numerals throughout this specification refer to the same components, and "and / or" includes each and every combination of one or more of the components mentioned. Even if "first", "second", etc. are used to describe various components, these components are not limited by these terms. These terms are used simply to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may also be the second component within the technical spirit of the present invention.

[0019] Unless otherwise defined, all terms (including technical and scientific terms) used in this specification are used as meanings commonly understood by ordinary skilled artisans in the technical field to which the present invention belongs. Furthermore, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly and specifically defined.

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0021] Prior to the description, the meanings of the terms used in this specification will be briefly explained. However, since the explanations of the terms are intended to facilitate understanding of this specification, it should be noted that unless explicitly stated as matters limiting the present invention, they are not used to limit the technical ideas of the present invention.

[0022] A food measurement method according to an embodiment of the present invention will be described with reference to Figures 1 to 5. The food measurement method according to an embodiment of the present invention is performed by a computer, and the computer here means a food measurement server 100 or a food measurement device 200. That is, the photographing is performed by the food measurement device 200, but the other steps can be performed by the food measurement server 100 or the food measurement device 200.

[0023] FIG. 1 is a flowchart of a food measuring method according to one embodiment of the present invention, FIG. 2A is a perspective view of a tableware according to one embodiment of the present invention, FIG. 2B is a perspective view of a tableware having an identification code attached to the back or side, FIG. 3 is a perspective view of a tableware containing food according to an embodiment of the present invention, FIG. 4 is a perspective view of the tableware in which two foods are stacked in one space in FIG. 3, FIG. 5 is a perspective view of a tableware in which the space is not divided, and FIG. 6 is a flow chart of a method of providing an advanced local model to improve the accuracy of determining the type of food.

[0024] The food measuring method according to the embodiment of the present invention can be performed at a serving or return door of a restaurant, and the restaurant can be a self-service restaurant such as a cafeteria or a buffet where a user moves the tableware 500 to receive food, or a place that provides meals to an organization such as a school, a company, or an army. Here, the restaurant can have a serving door for receiving food or tableware 500, and a return door for disposing of leftover food and returning the tableware 500 after a meal, but is not limited to this.

[0025] However, the food measuring method according to an embodiment of the present invention is not limited to being performed in a restaurant, particularly a self-service restaurant, but may also be performed in other types of restaurants or at home, or at each seat.

[0026] For ease of explanation, the following description will be given using a plate-shaped tableware 500 with a space 510 defined therein, but the shape of the tableware 500 is not limited to this.

[0027] Referring to FIG. 1, the receiving unit 110 receives an image of a plate containing food photographed by the photographing unit 250 of the food measuring device 200 (step S20).

[0028] The receiving unit 110 normalizes the received image of the food serving dish. Normalization refers to adjusting or transforming the received image of the food serving dish according to a predetermined criterion (e.g., size, brightness, rotation, gradient, etc.).

[0029] There is no limitation on the configuration of the photographing unit 250 as long as it can acquire image data and depth data. For example, the photographing unit 250 may include at least one of an RGB camera, a 2D camera, a 3D camera, a ToF (Time of Flight) camera, a light field camera, a stereo camera, an event camera, and an infrared camera, and there is no limitation on the configuration as long as it can measure image data and depth data.

[0030] Meanwhile, the photographing unit 250 can photograph the food or the tableware on which the food is served, as well as photograph the biometric information of the eater to identify the eater. Here, the biometric information of the eater may be, but is not limited to, the face, iris, or fingerprint of the eater. One photographing unit 250 can photograph the food or the tableware on which the food is served and the biometric information of the eater simultaneously or separately, but the photographing unit 250a for photographing the food or the tableware on which the food is served and the photographing unit 250b for photographing the biometric information of the eater can be separated.

[0031] In various embodiments, when photographing the tableware 500 moving along the conveyor belt 270, the photographing unit 250 of the food measuring device 200 may include a single camera and may obtain depth data in addition to image data. This is because if a single photographing unit 250 is used to photograph the tableware 500 when it is at a first position on the conveyor belt 270 and when it is moving on the conveyor belt 270 to a second position, the same effect can be obtained as when a stationary tableware 500 is photographed using a photographing unit 250 including multiple cameras.

[0032] Since a camera is a device that recognizes light, it is subject to a lot of environmental influences such as light reflection. Therefore, it is necessary to minimize the environmental influences by taking pictures using the image capturing unit 250 including two or more cameras. However, since there is a possibility that the image capturing unit 250 including two or more cameras may still be subject to the environmental influences when taking pictures, it is necessary to correct for this.

[0033] Therefore, the computer can learn the on-site conditions through images captured by the learning module in order to optimize according to the characteristics and environment of the restaurant, and recognize the environmental conditions based on filtering, fixed objects, and reference objects (background targets described below).

[0034] Meanwhile, when photographing the tableware 500 using the food measuring device 200 according to an embodiment of the present invention, the photographing unit 250 photographs the tableware 500 from above, and the food measuring device 200 can sense the identification information 530 related to the tableware 500 or the identification information related to the subject of the meal. That is, the subject of the meal can be identified (S21) using the food measuring device 200 simultaneously with or separately from the dish image receiving step (S20).

[0035] Specifically, referring to Fig. 2B, the identification information regarding the tableware 500 may be included in the tableware 500, for example, in the form of a QR code (530, Quick Response Code) or a barcode, and various other code-type identification information, such as a combination of numbers or letters, a combination of figures and symbols, may also be included in the tableware 500. In addition, the identification information regarding the tableware 500 may be recorded in various semiconductor devices such as an RFID chip and included in the tableware 500. For this purpose, the food measuring device 200 includes an identification information sensing unit 251, which may be disposed in the food measuring device 200 so that when the photographing unit 250 photographs the tableware 500 from above, the identification information sensing unit 251 can also simultaneously sense the identification information regarding the tableware 500.

[0036] For example, as shown in FIG. 2B, when the QR code (registered trademark) 530 is located on the back or side of the tableware 500, the identification information sensing unit 251 may be located below facing the photographing unit 250, or may be located horizontally so as to recognize the QR code (registered trademark) 530. However, the position of the identification information sensing unit 251 is not limited thereto, and it may be located in various positions as long as the photographing unit 250 can sense the identification information of the tableware 500 at the same time as photographing the top surface of the tableware 500.

[0037] In some embodiments, when the identification information of the dish 500 is located on the top of the dish 500, the food measuring device 200 may not include a separate identification information sensing unit 251, and may analyze the image of the dish being measured using the photographing unit 250 to confirm the identification information of the dish 500.

[0038] In addition, identification information regarding the subject of the meal can be sensed by the authentication unit 160, and for example, methods such as facial recognition, iris recognition, and fingerprint recognition can be applied to confirm biometric information by capturing an image of the subject of the meal.

[0039] In addition, the authentication unit 160 can sense identification information regarding the subject of the meal by recognizing a tag containing identification information in various code forms, such as a tag containing an RFID chip held by the subject of the meal, a QR code (registered trademark), a barcode, a combination of numbers or letters, a combination of figures, and a combination of symbols.

[0040] When using the food measurement method according to an embodiment of the present invention, various information can be calculated using the measurement results, but the calculated information may differ depending on whether the food measurement device 200 can recognize the identification information 530 related to the tableware 500 or the identification information related to the subject of the meal.

[0041] First, when neither the food measuring device 200 installed at the serving opening nor the food measuring device 200 installed at the return opening can recognize identification information 530 related to the tableware 500, but can recognize only identification information related to the subjects of the meal, the food measuring method of this embodiment can be used to calculate information related to each subject of the meal and information related to the group to which the subjects of the meal belong.

[0042] Conversely, if neither the food measuring device 200 installed at the serving opening nor the food measuring device 200 installed at the return opening can recognize identification information related to the subjects of a meal, and only the identification information 530 related to the tableware 500 can be recognized, then the food measuring method of this embodiment cannot calculate information related to each subject of a meal, but can only calculate information related to the group to which the subject of the meal belongs.

[0043] In addition, if the food measuring device 200 installed at the serving opening can recognize both the identification information 530 related to the tableware 500 and the identification information related to the subject of the meal, and the food measuring device 200 installed at the return opening cannot recognize the identification information 530 related to the tableware 500, but can only recognize the identification information related to the subject of the meal, even if the food measuring device 200 installed at the return opening cannot recognize the identification information related to the tableware 500, the identification information related to the tableware 500 held by a specific subject of the meal can be identified using the information at the serving opening, and therefore information related to each subject of the meal and information related to the group to which the subject of the meal belongs can be calculated using the food measuring method of this embodiment.

[0044] Meanwhile, the food identification device 30 may not include a separate authentication unit 160 and may receive identification information from a subject of a meal via an input device (not shown) of the food measurement device 200 .

[0045] Therefore, in several embodiments, in the step (S520) in which the receiving unit 110 receives the image of the tableware photographed by the photographing unit 250, the receiving unit 110 can receive at least one of identification information regarding the tableware 500 and identification information regarding the subject of the meal in addition to the image of the tableware.

[0046] Next, referring to FIG. 1, the correction unit 150 can correct the image of the tableware containing the food (step S25).

[0047] More specifically, the correction unit 150 can remove noise components using the image data of the dish with food on it, recognize the gradient of the image of the dish, and correct it to a horizontal image. The image corrected in this way can be used in the volume calculation step (S50) described below.

[0048] When photographing tableware in a restaurant, the tableware can be photographed while placed in a specific position and not shaking, or the photographing unit 250 may photograph the tableware 500 as it moves on the conveyor belt 270, or the tableware 500 may be photographed by the food measuring device 200 while the subject of the meal is holding the tableware 500, and in some embodiments the photographing unit 250 may photograph a user walking away carrying the tableware 500.

[0049] Therefore, since the image of the dish may be captured unclearly or at a tilted angle, the correction unit 150 recognizes the sway or gradient of the dish image, selects the image of the dish with the least sway or gradient, and corrects the image of the dish to an image in a horizontal state, thereby enabling the volume of food to be accurately calculated.

[0050] First, the correction unit 150 can analyze a plurality of standardized dish images to select the dish image with the least shaking from among the standardized dish images. For example, the correction unit 150 can check the degree of blurring of the image on a pixel-by-pixel basis, thereby selecting the dish image with the least shaking and the clearest image.

[0051] Next, the correction unit 150 can calculate the distance from the photographing unit 250 to a plurality of exposed regions of the tableware 500 that are not obstructed by food from the image of the tableware, and can calculate the gradient of the tableware 50 from the image of the tableware using the calculated distance to the plurality of exposed regions. Next, the correction unit 150 can correct the photographed image of the tableware to be horizontal using the calculated gradient.

[0052] Specifically, the correction unit 150 can extract the shape of the tableware 500 using the depth information from the image of the tableware. Since there is a difference in depth information between the tableware 500 and the background other than the tableware 500 in the image of the tableware, the correction unit 150 can extract the tableware 500 from the image of the tableware using this.

[0053] The correction unit 150 checks flat areas among the exposed areas of the tableware 500 that are not obstructed by food, determines at least three or more points among the flat areas, and checks depth information of the three or more determined points.

[0054] Here, checking the depth information of three or more points is intended to generate a virtual plane using three or more points and check the gradient of the plane, and the correction unit 150 can calculate the gradient of the tableware 500 using the depth information of the three or more determined points and correct the image of the tableware photographed using the calculated gradient.

[0055] In some embodiments, after determining at least three or more points in a flat region, instead of checking the depth information of the determined three or more points, the average depth information of the points surrounding each of the determined three or more points is used as the depth information of each point, thereby utilizing relatively more information for calculating the gradient, thereby improving the accuracy of the calculated gradient.

[0056] In various embodiments, in the process of calculating the gradient of the tableware 500 using depth information of three or more predetermined points, a gradient value can be derived based on a line connecting each point, thereby calculating the gradient of the tableware 500, and the image of the photographed tableware can be corrected using the calculated gradient.

[0057] In some embodiments, the correction unit 150 may determine the contour of the dish 50 from the image of the dish and calculate the gradient of the dish 50 in the photographed image of the dish by comparing the contour of the dish 50 in the photographed image of the dish with the contour of the dish 50 in an image of the dish that is not tilted to serve as a reference. The correction unit 150 may then correct the photographed image of the dish using the calculated gradient.

[0058] In addition, since noise may occur in the image data of the two dishes containing food received through the receiving unit 110 due to on-site conditions (shaking, light reflection, etc.), the two dish images can be compared to remove the noise.

[0059] Next, referring to FIG. 1, the extracting unit 120 extracts image data of food contained in each space 510 of the tableware 500 from the image data of the tableware on which food is served (S530).

[0060] More specifically, since the tableware image includes both the tableware 500 and food images, this means that food image data is extracted from the tableware image. For example, the extraction unit 120 can extract food image data using a model trained using an artificial intelligence algorithm to distinguish between food and non-food. Here, if the tableware 500 includes multiple spaces 510, the extraction unit 120 can extract food image data for each space 510.

[0061] 5, multiple foods are served on one undivided tableware 500, but since there is no space within the tableware 500, the extraction unit 120 may recognize multiple foods as one food image 550. In this case, the extraction unit 120 must additionally perform a process of separating and extracting each food image data from the entire food image data.

[0062] Next, referring to FIG. 1, the determination unit 130 determines the type of each food based on the extracted food image data (step S40).

[0063] The determination unit 130 may determine the type of food using the extracted food image data. For example, the determination unit 130 may include a model learned based on an image to recognize the type of food, but the method of determining the type of food is not limited thereto.

[0064] In order to improve the food type recognition rate in the food measurement method according to the present embodiment, the model trained using an artificial intelligence algorithm can be optimized for an individual subject using the food measurement method in the following manner: The individual subject may be, for example, but is not limited to, an individual restaurant or an individual diners.

[0065] First, referring to FIG. 6, the receiving unit 110 receives optimization information for optimizing a model learned from an individual object such as a restaurant or a person who dines (step S31).

[0066] The trained model to be optimized is a master model, which is a model trained to classify food types using image data captured by the food measuring device 200 or the user terminal 300 or processed results as a data set.

[0067] The master model is constructed from a data set of all captured image data or processed results used in the food measurement method, so in order to determine the type of specific food image data, the master model will determine what type of food the specific food image data is from tens, hundreds, or thousands of food type classes.

[0068] Optimizing the master model reduces the number of food type class cases that need to be considered to identify the food type of a particular food image data, and thus allows obtaining optimization information for individual objects such as restaurants or dining subjects.

[0069] That is, the receiving unit 110 may receive optimization information for each individual object. The optimization information may be information that is likely to be consumed by each individual object or by a diner using the individual object, such as menu information sold or provided at an individual restaurant or types of food frequently consumed by individual diners.

[0070] Next, referring to FIG. 6, the determination unit 130 optimizes the classes in the master model based on the optimization information, and calculates a local model (step S32).

[0071] The determination unit 130 selects a food type class that is highly related to the corresponding individual object based on the optimization information, optimizes the class of the master model by reducing the food type class considered in the master model to the food type class that is highly related to the corresponding individual object, and calculates a local model for the corresponding individual object.

[0072] Next, referring to FIG. 6, the determination unit 130 performs learning using image data related to an individual object to improve the local model (step S33).

[0073] The determination unit 130 may improve the local model by learning the calculated local model using image data related to the individual object or a result of processing related to the individual object.

[0074] Here, the image data related to the individual object may be image data taken in an individual restaurant for the individual object or the result of processing image data taken in an individual restaurant, or image data taken by an individual dining subject for the individual object or the result of processing image data taken by an individual dining subject.

[0075] Meanwhile, in the process of upgrading the local model by learning the calculated local model using image data related to an individual object or a result processed in relation to an individual object, the judgment unit 130 may advance the learning while changing various variables such as a learning parameter and a learning cycle in the process of learning using an artificial intelligence algorithm, and may advance the upgrading process using a learning method with the highest accuracy among the learning methods that have been advanced.

[0076] Next, referring to FIG. 6, the judgment unit 130 can provide an advanced local model, and the judgment unit 130 can improve accuracy by using this to judge the type of each food based on the extracted food image data (step S34).

[0077] In some embodiments, the method may further include a preceding step of the receiving unit 110 receiving information on a list of foods to be served at the restaurant (S10).

[0078] In this case, in the step of determining the type of food (step S40), the determination unit 130 may determine the type of each food by matching the extracted food image data with the food list information.

[0079] In this way, the determination unit 130 can analyze the food image data to determine the type of food independently, or can match the data with stored food list information to determine the type of food, but the method of determining the type of food is not limited to these.

[0080] Next, referring to FIG. 1, the calculation unit 140 calculates the volume of each food by using height information (ie, three-dimensional distance data) for each pixel of the extracted food image data (step S50).

[0081] As described above, the image of the tableware is captured by the photographing unit 250, which can acquire depth data, so that the food image data includes pixel-by-pixel height information (3D distance data) as depth data, which can be used to calculate the volume of each food item.

[0082] The photographing unit 350 includes a computer including a database 170 in which tableware information including tableware images and the size, depth and capacity of a plurality of spaces 510 formed in the tableware 500 is stored.

[0083] The volume calculation step (step S50) may further include a step in which the correction unit 150 corrects the calculated volume of each food using dish information including the size, depth and capacity of the space 510 in which each food is contained.

[0084] Since the depth of each space 510 varies depending on the type of tableware 500, the degree to which food is stacked can be determined more accurately by the correction unit 150 using the tableware information. Since data on tableware 500 used in restaurants is stored in the computer database 170 according to this embodiment, the volume of each food item extracted can be calculated more accurately by using the food measuring method according to this embodiment using the tableware information.

[0085] For example, when the judgment unit 130 determines that the type of food contained in a specific space 510 in the dish 500 is liquid, the calculation unit 140 can calculate the volume of the liquid using dish information including the position where the liquid meets the corresponding space 510 and the size, depth, and capacity of the corresponding space 510.

[0086] When liquid food such as "soup" is placed in a specific space 510, there will inevitably be a portion where the "soup" meets the corresponding space 510 at a certain height. The calculation unit 140 uses this to recognize the position where the liquid food meets the corresponding space 510 from the dish image, and calculates the volume of the food using dish information (size, depth, capacity) of the corresponding space 510.

[0087] For example, the area "A" in Fig. 3 shows the area where the "soup" meets the space 510, and the calculation unit 140 can confirm the position where the liquid meets the space 510 that contains the liquid by a part such as area A in the image data. Then, the calculation unit 140 can calculate the volume of the "soup" by calculating the volume up to the height of area "A" using the dish information (e.g., capacity).

[0088] In some embodiments, as shown in Fig. 5, additional correction can be performed on a single dish 500 in which the space is not divided. If the calculation unit 140 determines from the food image data in the volume calculation step that multiple foods are contained in the undivided space in a single dish 500, it requests the extraction unit 120 to provide image data of each of the multiple foods and calculates the volume of each food.

[0089] Then, the calculation unit 140 calculates the volume of the entire food using the entire extracted food image data, and corrects the volume of each food by comparing the sum of the volumes of each food calculated in the volume calculation step with the volume calculated based on the entire food image data.

[0090] In some embodiments, the volume calculation step (step S50) may further include a step of calculating a volume of the stacked food (step S51).

[0091] More specifically, multiple foods can be stacked in a specific space 510 in the tableware 500. Therefore, when the calculation unit 140 recognizes that different foods are stacked in the specific space 510 in the tableware 500, it calculates the volume of the food located below using the image data of the food located above, the calculated volume information, and the size, depth, and capacity information of the corresponding space 510.

[0092] 4 is a diagram illustrating an example in which a fried egg is stacked on top of rice in a space 510 in a dish containing rice. The calculation unit 140 calculates the size and volume of the fried egg based on the dish information and image data of the fried egg.

[0093] In addition, the calculation unit 140 can predict the height of the rice contained in the corresponding space 510 based on the size and volume information of the fried egg, and can also calculate the volume of the rice contained in the corresponding space 510 based on the dish information.

[0094] Specifically, the area of ​​the food located below (e.g., rice) that is blocked by the food located above (e.g., fried egg) is calculated, and the height of the area of ​​the food located below (e.g., rice) that is not blocked by the food located above (e.g., fried egg) is used to calculate the height of the area blocked by the food located above (e.g., fried egg), thereby making it possible to calculate the volume of the area of ​​the food located below (e.g., rice) that is blocked by the food located above (e.g., fried egg).

[0095] Therefore, the height of the area where the food located below (e.g., rice) is blocked by the food located above (e.g., fried egg) may be set as the average height of the area where the food located below (e.g., rice) is not blocked by the food located above (e.g., fried egg), but is not limited to this.

[0096] In some embodiments, data on foods that are stacked on top of other foods (e.g., fried eggs, fish, seaweed, etc.) may be pre-stored in the computer database 170. For example, data on foods that are primarily stacked on top of other foods may include volume data or weight data, which allows the calculation unit 140 to quickly calculate the volume of the foods stacked on top.

[0097] Meanwhile, the volume calculation step (step S50) may further include a step (step S52) in which the correction unit 150 corrects noise components photographed together with the tableware in the image of the tableware. That is, the correction unit 150 may further include a step of detecting noise components that are not food but have volume from the tableware image data on which food is served, and correcting the correction to exclude the volume of the detected noise components from the food volume calculated in the step of calculating the volume of each food.

[0098] Here, the noise components are not food but have a volume, and the volume of the food may be calculated to be larger than the actual volume due to the volume of the noise components. Therefore, in order to accurately calculate the volume of the food, the correction unit 150 recognizes the noise components in the captured image and performs correction to remove the volume of the noise components from the volume of the food calculated by the calculation unit 140.

[0099] For example, hands and tableware (e.g., spoons, forks, knives, etc.) are noise components that have volume even though they are not food, and other examples include, but are not limited to, dairy product lids, etc. For example, noise information on objects that are noise components can be stored in advance in the computer database 170, and the correction unit 150 can recognize the noise components in the captured image and perform correction to remove the volume of the corresponding noise components from the volume of the food calculated by the calculation unit 140.

[0100] As an example, in the step of correcting noise components, the correction unit 150 may recognize an image of the user's hand from the image of the dish, and perform a correction to exclude the volume value of the hand from the volume value of the food calculated by the calculation unit 140. Specifically, the food measuring device 200 may take an image of the dish 500 while the user is holding it, but since the image data of the hand holding the dish 500 may cause an error in the calculation of the volume of the food, the correction unit 150 may perform a correction to exclude the volume value of a body part such as the hand from the volume value of the food calculated from the image data of the dish with food in it, thereby improving the accuracy in the food volume calculation step (S50).

[0101] As an example, the correction unit 150 is pre-trained on image data for an empty dish 500 and a spoon, and when a spoon is included in a dish image, the correction unit 150 can recognize this and perform correction to exclude the volume value of the spoon.

[0102] More specifically, the correction unit 150 detects whether a user's hand or tableware is present in the image data of the tableware on which food is served, and if the image of the tableware contains the user's hand or tableware, the correction unit 150 excludes the volume value of the user's hand or tableware from the volume value of the food calculated based on the image data of the tableware on which food is served.

[0103] Due to the above-mentioned configuration and operation of the compensation unit 150, the food measuring method according to the embodiment of the present invention has the effect of increasing accuracy when the subsequent steps are performed.

[0104] Next, referring to FIG. 1, the calculation unit 140 calculates meal information including the weight of each food by using the volume information of each food calculated in step S50 and the food information (step S60).

[0105] The computer database 170 stores food information including the weight per volume of each food, calorie information per weight for each food, and nutritional information per weight for each food.

[0106] Foods have different weights per volume and different calories depending on the type. Therefore, in an embodiment of the present invention, food information including the weight per volume of each food is stored in the database 170, and the calculation unit 140 uses this to calculate the weight of the food from the food volume information.

[0107] Using this, the calculation unit 140 can calculate the calories of the food contained in the user's serving dish using the calculated weight information of each food after the weight calculation step (step S60), and can also calculate information on the nutritional components ingested by the user and calculate additional specific information such as allergy warnings based on the food or the nutritional components contained in the food.

[0108] In addition, the calculation unit 140 can calculate price information about the food by using the type of food and the volume or weight of each food.

[0109] The calculation unit 140 can provide the information included in the above-mentioned diet information to the display unit 210 of the food measuring device 200 or the display unit 310 of the user terminal 300. The UI of the screen provided to the display units 210, 310 will be described later.

[0110] For example, the calculation unit 140 may display different information on the display unit 210 / 310 at the serving entrance from information displayed on the display unit 210 / 310 at the return entrance. For example, the display unit 210 / 310 at the serving entrance may display information on food type, calorie information, nutritional information, allergy information, price information, etc., and the display unit 210 / 310 at the return entrance may additionally display information on the amount actually consumed in addition to the above information. However, the meal information calculated by the calculation unit 140 is not limited thereto, and various information may be displayed without restriction using photographed information or information stored in the database 170.

[0111] This allows the diner to make payment using the payment unit 240 of the food measuring device 200 or the user terminal 300 by cash, card, account transfer, QR code (registered trademark) tagging, RFID tagging, facial recognition, or other methods based on the provided price information.

[0112] As an example, the restaurant may be provided with a weight measuring device (not shown) capable of measuring the weight of the tableware 500, but is not limited thereto, and the method may further include a step in which the receiving unit 110 receives weight information of the serving tableware 500 measured from a weight measuring device (not shown) provided in the restaurant, and the correction unit 150 matches the sum of the weight of each food item and the weight of the empty tableware calculated in step S560 with the received weight information to correct the weight of each food item.

[0113] The above-mentioned correction step can be selectively adopted and used. During the weight measurement process, the weight may be measured differently from the actual weight due to the user's hand or various other factors. Therefore, if the weight information is measured differently enough to exceed a threshold value, the correction step may not be performed.

[0114] Hereinafter, a food measurement method according to another embodiment using a user terminal 300 as the food measurement device 200 will be described with reference to Figures 7 and 8. However, the following will focus on the differences from the food measurement method according to the one embodiment of the present invention, and a description of the same contents will be omitted. Figure 7 is a flowchart of the food measurement method according to the other embodiment of the present invention, and Figure 8 is a perspective view of a table explaining a background target.

[0115] A food measurement method according to another embodiment of the present invention is performed by a computer, and the computer here means the food measurement server 100 or the user terminal 300. That is, the photographing is performed by the user terminal 300, but the other steps can be performed by the food measurement server 100 or the user terminal 300.

[0116] Here, the user terminal 300 is a mobile terminal on which a service application related to a food measurement method is installed. The user terminal 300 can acquire image data and depth data. To this end, the user terminal 300 can include at least one of an RGB camera, a 2D camera, a 3D camera, a Time of Flight (ToF) camera, a light field camera, a stereo camera, an event camera, and an infrared camera.

[0117] For example, the user terminal 300 may be any electronic device such as a computer, an Ultra Mobile PC (UMPC), a workstation, a netbook, a personal digital assistant (PDA), a portable computer, a web tablet, a wireless phone, a mobile phone, a smart phone, an e-book, a portable multimedia player (PMP), a portable game console, a navigation device, a black box, or a digital camera, and may include any user device capable of installing and executing a food measurement application associated with the food measurement server 100 or the food measurement device 200.

[0118] 7, the receiving unit 110 receives an image of a dish photographed by the photographing unit 350 of the user terminal 300 (step S20).

[0119] Here, food can be photographed in response to a photographing request from a service application installed on the user terminal 300, and guidance on how to photograph food, angles, etc. can be provided to the user as images and sounds via the service application.

[0120] As an example, the service application may request that the food be photographed two or more times from different angles. If a 3D camera or a ToF camera is not used, photographing the food two or more times from different angles allows the depth data to be obtained by comparing image data of two or more plates on which the food is served. In addition, even if the purpose is not to obtain depth data, photographing the food two or more times increases the accuracy of the calculation and correction steps.

[0121] Next, referring to FIG. 7, the correction unit 150 corrects the tableware image (step S25).

[0122] As an example, the dish image receiving step (step S10) may receive gradient information of the user terminal 300 measured by a gyro sensor (not shown) of the user terminal 300 when food is photographed by the photographing unit 350 of the user terminal 300. The correction unit 150 may correct the gradient of the photographed dish image data containing food by using the gradient information of the user terminal 300.

[0123] According to an embodiment of the present invention, when a user holds the user terminal 300 and photographs food using the photographing unit 350, a certain angle of tilt may occur, so the above-mentioned steps can be performed to correct the image data of the dish on which the food is served to a horizontal state.

[0124] Next, referring to FIG. 7, the extracting unit 120 extracts image data of food contained in the space 510 of the dish 500 from the image data of the dish on which food is served (step S30).

[0125] Since the image of the tableware captured by the user terminal 300 does not only include food, in order to calculate an accurate volume, only food image data is extracted from the image at this stage.

[0126] The extraction unit 120 accurately extracts food from the background using a function of recognizing and extracting food from the background, and when two or more foods are included, it separately extracts image data of the two or more foods, and in this process, a model trained using an artificial intelligence algorithm may be used.

[0127] Next, referring to FIG. 7, the determination unit 130 determines the type of each food according to the extracted food image data (step S40).

[0128] The determination unit 130 uses a model trained using an artificial intelligence algorithm to determine the type of food of each food image data extracted by the extraction unit 120. In various embodiments, when multiple determination results with similar accuracy are obtained, the determination unit 130 may output the multiple determination results to the user terminal 300, and allow the user to input or select a correct result through a service application of the user terminal 300.

[0129] Next, referring to FIG. 7, the calculation unit 140 calculates the volume of each food by using height information (ie, three-dimensional distance data) for each pixel of the extracted food image data (step S50).

[0130] The volume calculation step (step S50) may include a volume correction step (step S53). As an example, a background target may be used for volume correction in the volume correction step (step S53).

[0131] In the volume correction step (step S53), the determination unit 130 recognizes background targets contained in the received dish image, and the correction unit 150 can correct the volume of each food calculated in step S50 based on size information and position information of the recognized background targets.

[0132] Referring to FIG. 8, the correction unit 150 can compare the height information of each food with the background target to correct the volume of each food, and can also compare the size information of each food with the background target to correct the volume of each food.

[0133] Here, the background target is photographed by the photographing unit 350 together with the food or separately, and may be a tableware 500 containing food, a table 540, a spoon, a user's finger, etc. However, the background target is not limited thereto, and any object having a normal size may be used. For example, the background target may be any object with a fixed size, such as a coin, a bill, or a smartphone (product type).

[0134] In some embodiments, the background target may be separately photographed by the photographing unit 350 of the user terminal 300, and size information may be stored in advance in the database 170. This allows the correction unit 150 to accurately predict the volume of food from the extracted food image data based on the size or height of the background target.

[0135] As a first example, the service application may prompt the user to photograph the user's hand at a certain distance using the photographing unit 350. Alternatively, the service application may prompt the user to measure and input the actual size of the user's hand / fingers.

[0136] The receiving unit 110 then analyzes the captured image of the user's hand and stores information regarding the actual size of the user's hand in the database 170.

[0137] Then, when the determination unit 130 recognizes that a user's hand is included in the received dish image, it sets the user's hand as a background target, and the correction unit 150 can correct the volume of each food item calculated by the calculation unit 140 based on the position information and size information of the background target.

[0138] As a second example, the service application may allow the user to take pictures of tableware 500, a table 540, spoons, etc. used at home at a certain distance using the photographing unit 350. In this case, the table 540 may refer to a dining table or a table on which the user places the tableware 500 and eats.

[0139] Specifically, the computer can analyze the image of the tableware 500 used by the user at home captured by the photographing unit 350 of the user terminal 300, and store tableware 500 information including the size, depth, and capacity information of the tableware 500 in the database 170.

[0140] More specifically, since pixel-by-pixel height information is stored in the image captured by the photographing unit 350 of the user terminal 300 in accordance with an embodiment of the present invention, when the calculation unit 140 receives image data of the empty tableware 500, it can calculate the size, depth and capacity information of the tableware using the pixel-by-pixel height information of the tableware image data, and store this in the database 170 as tableware information.

[0141] If the calculation unit 140 finds that food is contained in the dish 500 as a result of analyzing the food image, it performs volume correction for each food calculated using the size, depth and capacity information of the dish 500 in which the food is contained.

[0142] As described above, if the size, depth and volume information of the corresponding dish 500 is known, a correction step can be performed to obtain more accurate results.

[0143] In some embodiments, the service application may request the user to measure the actual size of the tableware 500, table 540, spoon, etc., and input it into the user terminal 300 (for example, the width, length, or circumference information of the table 540).

[0144] As a third example, a case where a background target not related to food is used will be described. For example, a user can take a picture of a 500 won coin together with an image of food, and input the fact that the 500 won coin was taken together as a background target through a service application.

[0145] Therefore, the judgment unit 130 recognizes the 500 won coin (background target) contained in the received image of the dish containing the food, and the correction unit 150 corrects the volume of each food calculated based on the size information of the 500 won coin (background target).

[0146] In this way, since an object that is generally used by most people and has a fixed size can be applied as a background target, a user can simply photograph the surrounding object together with food and have the correction unit correct the volume of each food.

[0147] In addition, the accuracy can be improved by correcting the volume using a variety of information, including location information (restaurants, food service facilities, etc.), weather information (UV rays, reduced variety of food), user condition information (dieting, exercising, fighting an illness, taking medicine, etc.), personal preference information, and surrounding device information.

[0148] As a fourth example, the computer can store image data taken by the user terminals 300 of multiple users, information calculated from the image data, and information about restaurants that match the locations where the images were taken, to construct big data.

[0149] For example, when user A orders menu C at restaurant B and photographs menu C using user terminal 300, the types of food included in menu C are determined and the volume of each food item is calculated. The computer then stores this information in database 170 and utilizes it as big data.

[0150] Then, food image data is received via the user terminal 300 of a specific user, and the location information where the food image was taken and the type of food determined are matched with data in the database 170 in the computer to detect previous calculation results of the matching details, and the calculated volume information of each food can be corrected using this.

[0151] For example, if user D takes a picture of food at a location that is matched with restaurant B and transmits the image data of the food, the location information can be matched with big data to determine that user D is located at restaurant B, and the result data before it was matched with the ordered menu can be referenced to make corrections to the calculated volume information.

[0152] In relation to this, the opposite case can also be applied. More specifically, data on food is stored first, and when the user calculates what food he or she ordered at a restaurant, the stored data can be read out and volume and nutritional information can be recorded.

[0153] In summary, the service application can request the user to capture the background target by video or audio before or together with the food when taking a picture using the user terminal 300. Then, the determination unit 130 recognizes the received background target, and the correction unit 150 can correct the volume of each food calculated by the calculation unit 140 based on the position information and size information of the background target.

[0154] Next, referring to FIG. 7, the calculation unit 140 calculates meal information including the weight of each food by using the volume information of each food calculated in step S50 and the food information (step S60).

[0155] Using the food measuring method according to one embodiment of the present invention described above, a computer can measure the portion size (meaning the amount of food eaten before the meal) and the amount of leftover food for each person eating at a restaurant or home, and use this to calculate the amount of food eaten.

[0156] The computer can then use the serving size, amount of leftover food, and amount of food eaten by diners eating at a dining hall or at home to generate dining hall operation information and management information for the diners, and the management information for the diners can include, for example, the user's nutritional status, eating habits, whether or not they have picky eating habits, etc.

[0157] Hereinafter, a method for providing restaurant operation information and dining subject management information according to an embodiment of the present invention will be described with reference to Fig. 9. Fig. 9 is a flow chart of a method for providing restaurant operation information and dining subject management information according to an embodiment of the present invention.

[0158] First, referring to Fig. 9, the receiving unit 110 acquires pre-meal tableware data (pre-meal tableware image data) photographed by the serving tray photographing unit 250 installed at the serving tray or the photographing unit 350 of the user terminal 300, and the calculation unit 140 uses this to calculate the serving amount of the meal subject (step S110). The detailed explanation is as described with reference to Figs. 1 to 8, and the identification step for the meal subject (step S120) can be performed at the same time. Here, the pre-meal tableware refers to the tableware 500 of the meal subject in a state in which food has been served on it before the meal.

[0159] Next, referring to Fig. 9, the post-meal dishware data (post-meal dishware image data) photographed by the return port photographing unit 250 provided at the return port or the photographing unit 350 of the user terminal 300 is acquired, and the calculation unit 140 uses this to calculate the amount of leftover food of the meal subject (step S120). The detailed explanation is as described with reference to Figs. 1 to 8, and the identification step for the meal subject (step S120) can be performed at the same time. Here, the post-meal dishware refers to the meal subject dishware 500 after the meal.

[0160] Next, referring to FIG. 9, the calculation unit 140 calculates the meal amount of the corresponding meal subject using the calculated served meal amount and the amount of leftover food (step S130).

[0161] Through steps S110 to S120, the computer calculates the amount of each food item served in the dining room that was served to the subject of the meal, the amount of food left over, and the amount consumed (meal amount), thereby obtaining information regarding how much rice and side dishes the subject of the meal consumed and how much was left over.

[0162] In addition, the computer according to the embodiment of the present invention can perform the following operations to improve the accuracy of calculating the amount of food and drink consumed.

[0163] As one embodiment, the calculation unit 140 verifies whether the sum of the meal amount and leftover food amount of the meal subject calculated in the above steps matches the amount of food served, and if the verification results in a mismatch, the calculation unit 140 can correct at least one of the meal amount, leftover food amount, and served amount calculated in the above steps so that the sum of the meal amount or leftover food amount matches the amount of food served, and records the result in the database 170.

[0164] In addition, the calculation unit 140 verifies whether the sum of the amount of food eaten by the dining subjects in the cafeteria and the sum of the amount of leftover food calculated in the above steps matches the total amount of food served. If the verification results in a mismatch, the calculation unit 140 can correct at least one of the total amount of food eaten, the total amount of leftover food, and the total amount of food served calculated in the above steps so that the recorded total amount of food eaten and the total amount of leftover food matches the total amount of food served, and records the result of this in the database 170.

[0165] Furthermore, the calculation unit 140 verifies whether the sum of the meal amounts of the dining subjects and the sum of the amounts of leftover food calculated in the above steps match the sum of the amounts of food served, and records the result in the database 170. More specifically, the calculation unit 140 verifies whether the difference between the sum of the meal amounts of the dining subjects eating in the dining room and the sum of the amounts of leftover food matches the sum of the meal amounts.

[0166] In addition, the calculation unit 140 verifies the total amount of food and drink prepared in the cafeteria by comparing it with the total amount of food waste generated in the cafeteria, the total amount of food waste generated in the cafeteria, and the total amount of food eaten by the dining subjects of the cafeteria. If a difference occurs as a result of the verification, the calculation unit 140 can correct at least one of the total amount of food waste, the total amount of food waste, and the total amount of food eaten, for example, the amount of food eaten, so that the total amount of food and drink prepared in the cafeteria matches the total amount of food waste generated in the cafeteria, the total amount of food waste generated in the cafeteria, and the total amount of food eaten by the dining subjects of the cafeteria, and can record the result in the database 170.

[0167] Here, the amount of leftover food is the amount of food served to the main participants of the meal and left over by them, and the amount of leftover food can be the amount of food that is not served to the main participants of the meal and left over, and the amount of leftover food can be calculated by the above steps or by another method.

[0168] In addition, the database 170 may store the measurement results of the total weight of the amount of food and drink prepared for serving in the cafeteria each time, the total volume (weight) of the amount of leftover food, and the total weight of the amount of leftover food, and the calculation unit 140 verifies the total volume (weight) of the amount of food and drink prepared by comparing it with the total volume (weight) of the amount of leftover food, the total volume (weight) of the amount of leftover food, and the total sum of the amounts of food eaten by the dining subjects in the cafeteria calculated in the above steps. If a difference occurs as a result of the verification, the calculation unit 140 may correct the total sum of the amounts of food eaten by the dining subjects calculated in the above steps so that the total sum of the amount of food and drink prepared in the cafeteria matches the total sum of the amount of leftover food generated in the cafeteria, the total amount of leftover food generated in the cafeteria, and the total sum of the amounts of food eaten by the dining subjects in the cafeteria, and the result thereof may be recorded in the database 170.

[0169] In this way, the calculation unit 140 performs verification using various information and records the accuracy of the calculation / correction, making it possible to identify and improve problems.

[0170] For example, the database 170 can store the measurement results of the total volume (weight) of food and drink prepared for serving each time in the dining hall and the total volume (weight) of leftover food, and the calculation unit 140 can compare and verify the difference between the total volume (weight) of food and drink prepared and the total volume (weight) of leftover food with the total amount of food eaten by the dining subjects in the dining hall calculated in the above steps, and record the result in the database 170. (For example, if the total volume of menu A prepared in the dining hall is 200 and all of it is served with no leftovers, the total volume of menu A served to the dining subjects should be close to 200.)

[0171] Meanwhile, to accurately calculate the amount of food eaten, images must be taken before and after a meal, but there may be cases where a user mistakenly does not take a photo after eating.

[0172] Therefore, if a preset time has elapsed after receiving image data of dishes with food served before a meal, and image data of dishes after a meal is not captured and received, the calculation unit 140 determines that image data of dishes after a meal was not captured, and calculates the user's expected amount of food to be eaten.

[0173] For example, calculation unit 140 can calculate the user's expected meal amount assuming that the user did not leave any food uneaten, or can calculate the user's expected meal amount based on the user's average meal amount stored in database 170, status information (dieting, gaining weight, exercising, etc.), level of hunger, the user's preferences for menu items included in the pre-meal food image, information related to the taste of the menu items, and the like, and store the calculated amount in database 170.

[0174] Next, referring to FIG. 9, the management unit 180 of the computer generates restaurant operation information based on the serving amount, leftover amount, and meal amount of the meal subjects registered in the restaurant (step S140).

[0175] Next, referring to FIG. 9, the management unit 180 of the computer generates management information for each dining subject based on the serving amount, leftover amount, and meal amount of the dining subject registered in the restaurant (step S150).

[0176] Step S140 is to generate operation information useful for operating the restaurant.

[0177] More specifically, the serving amount adjustment module 181 of the management unit 180 uses the total serving amount prepared for serving in the cafeteria, the sum of the serving amounts for the dining subjects whose serving has been completed, and information on the remaining number of dining subjects to calculate the target serving amount per remaining dining subject, and provides it to the cafeteria manager device (not shown).

[0178] In this case, the cafeteria manager refers to the person in charge of planning the menu in the cafeteria and directly serving the food to the dining subjects, and a cafeteria manager device such as a display device 30 can be placed in front of the cafeteria manager in the cafeteria together with the food to be served.

[0179] Thus, the dining manager can adjust serving sizes for the remaining dining entities using the serving size adjustment module 181 that provides information about the target serving sizes per person for the remaining dining entities on the dining manager display.

[0180] In an embodiment of the present invention, the cafeteria manager may refer to a person who directly serves food to diners in the cafeteria as described above, or may refer to a manager who manages the cafeteria or a nutritionist of the cafeteria.

[0181] With this configuration, when there is a large amount of food preparation remaining compared to the number of remaining meal subjects, the target food serving amount per person for the remaining meal subjects is increased, thereby achieving the effect of reducing food waste.

[0182] In addition, if the amount of food prepared for distribution is insufficient compared to the number of remaining meal recipients, the target amount of food to be distributed per person to the remaining meal recipients can be reduced, preventing people (meal recipients) from being unable to eat due to a lack of food.

[0183] As one embodiment, the operation information generating step (step S140) may include a step in which the catering group management module 183 checks the affiliation of each catering staff member, stores the amount of food served, the amount of leftovers, and the amount of food eaten by each affiliation, and generates information by affiliation using the stored information, including at least one of favorite foods, disliked foods, amount of leftovers, and nutritional status for each affiliation, and a step in which the weather, day of the week, and schedule information for each affiliation are matched with the information by affiliation to extract related information thereto.

[0184] In this case, the affiliation of the caterer may mean the company, group, team, etc. in the case of a company, the department, class, etc. in the case of a school, and the corps, division, regiment, battalion, company, platoon, squad, etc. in the case of the military.

[0185] And, schedule information by affiliation may mean a quarter, an ongoing project, or a business in the case of a company, a semester, an exam period, etc. in the case of a school, or various training periods in the case of the military.

[0186] Therefore, according to an embodiment of the present invention, the food preferences of each affiliation can be understood using information on the likes and dislikes of foods according to the affiliation of the diner, and an accurate guide for future meal delivery menus can be created.

[0187] Managers will also be able to keep track of the amount of leftover food and nutritional status by department.

[0188] By matching the weather, day of the week, and schedule information for each department with the department-specific information and extracting related information, administrators can check changes in appetite and food preferences depending on the weather and day of the week.In the case of the military, this makes it possible to understand how the amount of food eaten by soldiers changes according to various training schedules.

[0189] Using this information, the administrator can appropriately adjust the menu and amount of food served depending on the military affiliation and training schedule, and provide meals suitable for soldiers who are completing their training schedule.

[0190] As an example, in the operation information generating step (step S140), if the serving amounts of food are similar between restaurants but there is a difference of a certain amount or more in the amount of leftover food, the management unit determines that the restaurant with the less amount of leftover food has a better quality of ingredients and cooking method, and can provide the ingredient supply and demand information and food cooking method of the corresponding restaurant to the restaurant manager devices of other restaurants.

[0191] For example, if two or more restaurants have similar food menus and portions, but a large difference in the amount of food left over, it may mean that the restaurant with the least amount of food left over uses higher quality ingredients and better cooking methods, while the restaurant with the most amount of food left over uses lower quality ingredients and inferior cooking methods.

[0192] Therefore, the management unit provides food supply and demand information and food cooking methods of a restaurant with a small amount of leftover food to the restaurant manager devices of other restaurants, thereby improving the quality of the restaurants.

[0193] As one embodiment, the management information generation step (step S150) includes a step in which the eating subject management module 185 generates subject-specific information including at least one of each eating subject's favorite foods, disliked foods, average amount of leftover food, nutritional status, and average amount of food eaten, using the physical information, serving size, amount of leftover food, and amount of food eaten of each eating subject.

[0194] In addition, the food subject management module 185 calculates physical strength management information and obesity risk of each food subject using the target-specific information, and if the amount of food eaten by a food subject not set to a diet state decreases to a certain amount or less within a certain period of time, or the amount of leftover food increases to a certain amount or more within a certain period of time, the food subject is set as a target for attention.

[0195] For example, most soldiers must be constantly provided with nutrition to maintain a regular lifestyle and schedule, and in most cases, they maintain a certain amount of food unless they have a personal problem. However, if a personal problem occurs, causing them to lose motivation and become ill, this will be reflected in the amount of food they eat, and this can be used to detect individual problems early on.

[0196] Therefore, when the amount of food eaten by a food subject not set to a diet state decreases below a certain amount within a certain period of time, the food subject management module 185 determines that a problem has occurred in the food subject and appetite has decreased, and sets the food subject as an object of attention.

[0197] Furthermore, when the amount of leftover food of a meal subject not set to a diet state increases to a certain amount or more within a certain period of time, the meal subject management module 185 determines that the amount of food and drink served by the meal subject in question will be about the same as before, but that appetite has decreased, the amount of food eaten has decreased, and the amount of leftover food has increased, and sets the meal subject in question as a target for attention.

[0198] In this way, by having the meal subject management module 185 manage each individual meal subject and set meal subjects suspected of having problems as targets for attention, it has the effect of detecting signs of danger early on before an accident occurs to a soldier who requires care in a military setting, and allowing the administrator to take measures such as consulting with the soldier.

[0199] In addition, the eating subject management module 185 generates information for each eating subject, including information such as the nutritional status and average food intake of each eating subject, so that the administrator can use such information to check the characteristics and status of each individual as well as the characteristics and status of each affiliation.

[0200] As one embodiment, the operation information creation step (step S140) includes a step of calculating the average favorite foods, least favorite foods, and meal amounts of the dining subjects, and using this, the serving menu recommendation module 187 derives one or more recommended menus and serving preparation amounts for the restaurant, and provides them to the restaurant manager device.

[0201] In a cafeteria where meals are served at least once or twice a day, it is a very difficult task for the cafeteria manager to decide the menu to be served each day, taking into account the preferences and nutritional status of the diners.

[0202] Furthermore, when nutritional status is taken into consideration, the food may not match the preferences (favourite foods) of the person taking the meal, resulting in a large amount of leftover food (food waste); therefore, if the food is tailored to the preferences of the person taking the meal, it becomes difficult to maintain a balanced nutritional status.

[0203] Therefore, the serving menu recommendation module 187 calculates the average favorite foods, least favorite foods, and the amount of food eaten by each food menu of the dining subject, and uses this to derive and provide the serving preparation amount together with the recommended menu, thereby having the effect of helping the cafeteria manager and administrator manage the cafeteria.

[0204] To realize such an algorithm, the relevance of each food is stored in the computer database 170. For example, if foods A, B, and C have high relevance and the diners at the restaurant like food B, it can be determined that they are also likely to like foods A and C.

[0205] At this time, the serving menu recommendation module 187 can derive recommended menus using food menu preference information and taste evaluation scores, which will be described later.

[0206] As one embodiment, the food serving menu evaluation module 189 of the management unit 180 generates operation information and management information based on the serving amount, the amount of leftover food, and the amount of meal, and evaluates the preference of each food menu and the evaluation score of the taste of the served food itself using the change in the serving amount of each served food and the change in the amount of leftover food.

[0207] A reference value for the serving amount of food served in a restaurant is stored in the computer database 170. If the food looks delicious based on its appearance or smell, the dining subject will increase the serving amount, and if not, will decrease the serving amount. If the food does not taste good while eating, the subject will leave a lot of the food.

[0208] The above-mentioned configuration evaluates the score using such features, and more specifically, the score is determined according to the following criteria.

[0209] First, when the amount of a particular food served is greater than a reference value and the amount of leftover food is less than a reference value, the food serving menu evaluation module 189 increases the preference for the food menu and increases the evaluation score for the taste of the served food itself.

[0210] In this case, the server (the person eating the meal) sees the food with their own eyes as it is being served and, since it is a food they like, increases the amount of that food served, and while eating, judges it to be delicious and either eats all of the food or leaves a little.

[0211] Therefore, the food delivery menu evaluation module 189 will give a high rating to the food menu and also give a high rating to the taste of the delivered food itself, thereby determining that the ingredients were selected and the food was cooked well.

[0212] Secondly, if the amount of food served of a particular food is greater than a reference value and the amount of leftover food is greater than a reference value, the food serving menu evaluation module 189 increases the preference for the food menu and decreases the evaluation score for the taste of the served food itself.

[0213] In this case, the server sees the food with the naked eye while serving it and sees that it is a food the person likes, so increases the amount of that food served. However, while eating, the server determines that the taste is different from what the person expected or that it is not tasty, so leaves a lot of the food and determines that the amount of leftovers is higher than the standard.

[0214] Therefore, the menu evaluation module 189 will increase the preference score for the menu and will evaluate the taste of the delivered food itself as low, thereby assessing that the selection of ingredients and cooking of the food were not done well.

[0215] Third, when the amount of a particular food served is less than a reference value and the amount of leftover food is less than a reference value, the menu evaluation module 189 reduces the preference for the food menu and increases the evaluation score for the taste of the served food itself.

[0216] In this case, the server sees the food with the naked eye while serving it and determines that it is a food the person does not like, so reduces the amount of that food served, but while eating, the person finds that it is unexpectedly delicious and either eats all of the food or leaves a small amount.

[0217] Therefore, the food delivery menu evaluation module 189 will evaluate the preference score for the menu item low and the evaluation score for the taste of the delivered food itself high, and will evaluate that although the food item is not the server's favorite, the ingredients were well selected and the food was well prepared, so the server is satisfied.

[0218] Fourth, if the amount of food served of a particular food is less than a reference value and the amount of leftover food is more than a reference value, the food serving menu evaluation module 189 reduces the preference of the food menu and reduces the evaluation score of the taste of the served food itself.

[0219] In this case, the server may see the food with the naked eye while serving and determine that it is not to his or her liking, so the server reduces the amount of that food served, and while eating, may determine that the food is not tasty, so the server leaves a lot of the food and determines that the amount of leftovers is greater than the standard.

[0220] Therefore, the menu evaluation module 189 will give a low preference score to the menu and a low evaluation score to the taste of the delivered food itself, concluding that the food is not to the server's liking and that the selection of ingredients and cooking of the food were not done well.

[0221] With the configuration of the menu evaluation module 189, the restaurant manager can prepare menus and ingredients for the diners based on the diners' preferences (preference scores) for each food menu and the evaluation scores of the food's taste, and can also check whether there are any problems with the food cooking process.

[0222] As one embodiment, the eating subject management module 185 of the management unit 180 generates nutritional information ingested by each eating subject based on the amount of food eaten by the eating subject, and derives and provides current nutrition intake information based on the nutritional information generated at a certain time period.

[0223] At that time, the meal subject management module 185 can provide the derived information to the meal subject's terminal, that is, the user terminal 300, the meal subject's guardian's terminal, the meal subject's manager's terminal, the cafeteria manager's terminal, and the like.

[0224] As an example, the management unit 180 can execute a program for improving nutritional status and reducing food waste using information on the amount of food eaten and eating habits of the dietary subject.

[0225] The management unit 180 grasps nutritional information and eating habit information of each eating subject based on the amount of food eaten and the amount of leftover food of the eating subject. Then, using this, rewards or penalties are provided according to changes in the nutritional state and amount of leftover food of each eating subject.

[0226] In more detail, the management unit 180 grasps whether each dining subject has a balanced nutritional intake and correct eating habits based on the amount of leftovers and meals eaten by each dining subject, and if it is determined that the nutritional status has improved compared to the previous record at the time of serving the food, it provides the dining subject with a reward such as a reward or points, and if it is determined that the nutritional status has not improved or has deteriorated, it provides a penalty such as a punishment.

[0227] In addition, the management unit 180 stores and monitors the amount of leftover food from each meal subject when serving food. If it is determined that the amount of leftover food from a particular meal subject is decreasing, it provides a reward to the meal subject, and if it is determined that the amount of leftover food has not decreased or has increased, it provides a penalty such as a punishment to the meal subject.

[0228] In this way, the system grasps the nutritional information and eating habits of each meal participant and provides rewards or penalties according to changes in behavior, which naturally encourages meal participants to avoid picky eating and to eat a nutritionally balanced diet, thereby reducing the generation of food waste.

[0229] For example, when the subject of the meal requires parental supervision, such as an elementary or junior high school student or a patient, the meal subject management module 185 of the management unit 180 can provide images of the meal subject's pre-meal tableware and post-meal tableware obtained by the photographing unit 250 to the terminal of the meal subject's guardian.

[0230] The parent or guardian in charge of feeding can then view the photo via the application to see whether the person in charge of feeding is eating properly.

[0231] In the case of elementary and junior high school students and patients, they need to eat a nutritionally balanced diet to stay healthy, so continuous management is necessary and it is necessary to provide parents with information about the nutritional status of their children.

[0232] Therefore, the food-based management module 185 periodically inputs and manages physical information or health information of each elementary and junior high school student or patient, and derives information regarding the basal metabolic rate and necessary nutrients for each elementary and junior high school student, taking into account the physical information and school schedules, etc.

[0233] The dietary subject management module 185 determines the nutritional state and eating habits of the elementary and junior high school student or patient based on the meal menu, food amount data, and basal metabolic rate.

[0234] The food subject management module 185 provides the information thus secured and determined to the user terminal 300 and the food subject guardian's terminal, and derives a recommended menu that the food subject should take at home based on the nutritional status and eating habits of each food subject, and can recommend nutritional supplements according to the situation.

[0235] For example, if an evaluation of the eating habits and nutritional status of a primary and junior high school student A determines that he or she is at risk of being deficient in vitamins A and D, a menu that ensures sufficient intake of vitamins A and D will be recommended. If the deficiency is determined to be serious, nutritional supplements will be recommended.

[0236] Here, even if the subject of the meal is not an elementary or junior high school student or a patient, the computer can determine what nutrients the user needs and what menu is suitable for the user's nutritional condition based on the analyzed nutritional condition, eating habits, whether or not the user has a limited appetite, if the user's nutritional balance is unbalanced, and provide this information to the user terminal 300. It can also have an advertising effect by providing information on various nutritional supplements.

[0237] In addition, a description will be given of services that can be provided when using the user terminal 300. Since the computer can grasp the amount of food consumed by the user for each type of food, it can analyze the user's eating habits and grasp the user's nutritional intake using information on the nutrients contained in each food.

[0238] Furthermore, the user's physical information can be input through a service application installed in the user terminal 300, and the user's basal metabolic rate can be calculated using such physical information.

[0239] Then, using the nutritional status and eating habits information of the user derived in this manner, one or more food menus matching the user can be derived and provided to the service application, and the information can also be provided in conjunction with other food ordering applications.

[0240] At this time, the service application allows the user to directly input the foods they like and dislike, and receives information about the user's allergies, illnesses, etc., so that it can remove foods that the user should not eat from the recommended menu.

[0241] In addition, the judgment unit 130 can use such information to provide a warning (notification) to the service application if a food image captured by the photographing unit 55 of the portable device 50 includes food that the user should not eat, thereby preventing the user from mistakenly consuming the food.

[0242] In one embodiment, the service application may receive various measurement data in conjunction with a wearable device used and worn by a user. When such a configuration is included, the computer may obtain the user's basal metabolic rate, health information, and the like in more detail and use the information as various health care information.

[0243] The computer collects data on the amount of food and drink eaten before each meal, the amount of food eaten, and the amount of food left over at each meal via the user terminal 300 and stores the data in the database 170 to form information on the user's nutritional status, eating habits, etc., and collects data on the user's physical information and the user's activity index, activity information, exercise information, etc. collected via various health and exercise applications to form health care information for the user.

[0244] In addition, by linking with health management applications and medical information applications, it is possible to collect medical information about the user and provide the user with information on their nutritional status, eating habits, health care, and medical information, as well as information on customized meal plans and dietary supplements based on the medical information. It is also possible to sell related products directly through the service application.

[0245] For example, when subscribing to a service application provided by an external server (not shown), various types of services can be provided, such as a health check, customized dietary supplements for the user, and a custom-made menu delivered once a week.

[0246] Hereinafter, an apparatus and a server for implementing a food measuring method according to an embodiment of the present invention will be described.

[0247] A food measurement server 100 according to an embodiment of the present invention will be described with reference to Fig. 10. Fig. 10 is a block diagram of a food measurement server according to an embodiment of the present invention, and Fig. 11 is a detailed block diagram of the management unit of Fig. 10.

[0248] Referring to FIG. 5, the food measurement server 100 according to the embodiment of the present invention includes a receiving unit 110, an extracting unit 120, a determining unit 130, a calculating unit 140, a correcting unit 150, an authenticating unit 160, a database 170, and a communicating unit 190.

[0249] However, in various embodiments, server 10 may include fewer or more components than those shown in FIG.

[0250] The receiving unit 110 receives images of the tableware captured by one or more photographing units 250.

[0251] The extraction unit 120 extracts image data of food contained in each space 510 of the tableware 500 from the image data of the tableware on which food is served.

[0252] The determination unit 130 determines the type of each food according to the extracted food image data.

[0253] The calculation unit 140 calculates the volume of each food by using height information for each pixel of the extracted food image data.

[0254] The database 170 stores an image of an empty dish, dish information including the size, depth and capacity of a plurality of spaces 510 formed in the dish 500, and food information including the weight per volume of each food item.

[0255] The management unit 180 includes a serving amount adjustment module 181, a serving group management module 183, a serving target management module 185, a serving menu recommendation module 187, and a serving menu evaluation module 189, and the functions of each module are as described above.

[0256] The communication unit 190 communicates to receive images of tableware captured by the photographing unit 250 installed in the dining room, and serves to transmit various information calculated and generated by the server 10 to a person in charge or a user terminal.

[0257] The authentication unit 160 authenticates user information received from devices such as the food measuring device 200 and the kiosk device 30 to identify the user, and loads various information related to the user.

[0258] In addition, information about the weight, calories, and the like of the food in the serving dish calculated by the calculation unit 140 of the server 10 is provided to the user terminal authenticated by the authentication unit 160. At that time, the information can be provided via a service application provided by the server 10 itself.

[0259] Furthermore, the computer (server, device) according to an embodiment of the present invention can perform the following operations to improve the accuracy of food calculations.

[0260] As one embodiment, the calculation unit 140 verifies whether the sum of the meal amount and leftover food amount of the meal subject calculated in the above steps matches the amount of food served, and if the verification result shows that they do not match, the calculation unit 140 can correct at least one of the meal amount, leftover food amount, and served amount calculated in the above steps so that the sum of the meal amount or leftover food amount matches the amount of food served, and records the result in the database 170.

[0261] In addition, the calculation unit 140 verifies whether the sum of the total amount of food eaten by the dining subjects in the cafeteria and the total amount of leftover food calculated in the above steps matches the total amount of food served, and if the verification shows that they do not match, the calculation unit 140 can correct at least one of the total amount of food eaten, the total amount of leftover food, and the total amount of food served calculated in the above steps so that the sum of the total amount of food eaten and the total amount of leftover food matches the total amount of food served, and records the result of this in the database 170.

[0262] More specifically, the calculation unit 140 verifies whether or not the difference between the total amount of food served to the dining subjects eating in the dining room and the total amount of leftover food matches the total amount of food eaten.

[0263] In addition, the calculation unit 140 verifies the total amount of food prepared in the cafeteria by comparing it with the total amount of leftover food generated in the cafeteria, the total amount of leftover food generated in the cafeteria, and the total amount of food eaten by the dining subjects in the cafeteria. If a difference occurs as a result of the verification, the calculation unit 140 can correct at least one of the total amount of leftover food, the total amount of leftover food, and the total amount of food eaten, for example, the amount of food eaten, so that the total amount of food prepared in the cafeteria matches the total amount of leftover food generated in the cafeteria, the total amount of leftover food generated in the cafeteria, and the total amount of food eaten by the dining subjects in the cafeteria, and can record the result of this in the database 170.

[0264] Here, the amount of leftover food can be the amount of food served to the meal subject and left over by the meal subject, or the amount of leftover food can be the amount of food not served to the meal subject and left over, and the amount of leftover food can be calculated by the above steps or by another method.

[0265] More specifically, the database 170 may store the measurement results of the total volume (weight) of food prepared for serving each time in the restaurant, the total volume (weight) of leftover food, and the total volume (weight) of leftover food, and the calculation unit 140 verifies the total volume (weight) of prepared food by comparing it with the total volume (weight) of leftover food, the total volume (weight) of leftover food, and the total sum of the meal amounts of the meal subjects of the restaurant calculated in the above step, and if a difference occurs as a result of the verification, the calculation unit 140 may correct the sum of the meal amounts of the meal subjects calculated in the above step so that the total amount of food prepared in the restaurant matches the total amount of leftover food generated in the restaurant, the total amount of leftover food generated in the restaurant, and the total sum of the meal amounts of the meal subjects of the restaurant, and may record the result in the database 170. (For example, if the total volume of menu A prepared in the restaurant is 200 and all of it is served without any leftovers, the sum of the volumes of menu A served to the meal subjects should be close to 200.)

[0266] In this way, the calculation unit 140 performs verification using various information and records the accuracy of the calculation / correction, so that problems can be identified and improved.

[0267] As an example, the AI ​​algorithm of the calculation unit 140 according to the embodiment of the present invention can compare and complement the accuracy of volume recognition using image learning results as a measure to verify volume recognition and increase reliability. For example, images of different amounts of bread are input, and a judgment is made on the volume. If an error or noise occurs, re-learning can be performed to increase the accuracy.

[0268] The food measurement server 100 according to the embodiment of the present invention described above has the same content as the food measurement method described in Figures 1 to 9, with the only difference being the category of the invention, so duplicated explanations and examples will be omitted.

[0269] Below, FIG. 12 is a block diagram of a food measuring device according to an embodiment of the present invention, FIGS. 13 to 16 are perspective views of a food measuring device according to an embodiment of the present invention, and FIG. 17 is a screen of a food measuring device according to an embodiment of the present invention.

[0270] The food measuring device 200 can be installed at a location where food is served, and includes a receiving unit 110, an extracting unit 120, a determining unit 130, a calculating unit 140, a correcting unit 150, an authenticating unit 160, a database 170, a communicating unit 190, a display unit 210, a payment unit 240, and an image capturing unit 250. However, in various embodiments, the food measuring device 200 can include fewer or more components than those shown in FIG.

[0271] 13 and 14, the food measuring device 200 is provided with a measurement area 220 or a plate 230 on which a dish 500 is placed. Referring to Fig. 13 and 14, the photographing unit 250 photographs the dish 500 placed in the measurement area 220 or the plate 230.

[0272] In addition, the measurement area 220 or the plate 230 has a weight measuring device (not shown) formed therein, so that the weight of a serving dish placed on the upper surface can be measured.

[0273] The receiving unit 110 receives an image of the tableware photographed by one or more photographing units 250. Here, the photographing unit 250 is located above the tableware 500 so as to photograph the top surface of the tableware 500, and if an identification information sensing unit 251 is present, the identification information sensing unit 251 may be located so as to sense the identification information of the tableware 500 from the top, bottom, or side at the same time as the photographing unit 250 photographs the tableware 500.

[0274] The extraction unit 120 extracts image data of food contained in each space 510 of the tableware 500 from the image data of the tableware on which food is served.

[0275] The determination unit 130 determines the type of each food according to the extracted food image data.

[0276] The calculation unit 140 calculates the volume of each food by using height information for each pixel of the extracted food image data.

[0277] The database 170 stores images of empty dishes, dish information including the size, depth and capacity of the spaces 510 formed in the dish 500, and food information including the weight per volume of each food item.

[0278] The communication unit 190 communicates to receive images of tableware captured by the photographing unit 250 installed in the dining room, and serves to transmit various information calculated and generated by the server 10 to a person in charge or a user terminal.

[0279] The authentication unit 160 authenticates user information received from a device such as the food measuring device 200 or a kiosk device to identify the user and loads various information related to the user.

[0280] Information regarding the weight, calories, etc. of the food in the serving dish calculated by the calculation unit 140 of the server 10 is provided to the user terminal authenticated by the authentication unit 160. At that time, the information can be provided via a service application provided by the server 10 itself.

[0281] The food measuring device 200 includes a display 240, and can receive user authentication information for authenticating a user and can display a captured image through the display.

[0282] 13, when a user places a dish 500 in the measurement area 220, the photographing unit 250 installed in the measurement area 220 photographs the dish to obtain an image of the dish and an image of the dish after eating. The food measuring device 200 itself is equipped with a payment unit 240, so that a user can measure food and make a payment at the same time.

[0283] In some embodiments, referring to FIG. 14(C), the food measuring device 200 may not include a plate 230, and photographing by the photographing unit 250 and sensing by the identification information sensing unit 251 may be performed while the subject of the meal is holding the dish 500.

[0284] 15, in some embodiments, the food measuring device 200 may be provided at a return port where a conveyor belt 270 is installed. As the tableware 500 moves along the conveyor belt 270, the photographing unit 250 may photograph the tableware 500 from above the tableware 500.

[0285] In various embodiments, the food measuring device 200 may include a photographing unit 250, an identification information sensing unit 251, and an authentication unit 160, with the photographing unit 250 located in a relatively highest area, and the authentication unit 160 located between the photographing unit 250 and the identification information sensing unit 251, with the identification information sensing unit 251 located in a relatively lowest area. Because the food measuring device 200 has this structure, the photographing unit 250, the identification information sensing unit 251, and the authentication unit 160 can obtain necessary information from the subject of the meal and the tableware 500 simply by the subject of the meal standing in front of the food measuring device 200, thereby reducing inconvenience to the subject of the meal.

[0286] The reason why the photographing unit 250 is located in the highest area is to photograph the dishware 500 from above. If the authentication unit 160 is to recognize the face of the eating subject, when the eating subject holds the dishware 500, the dishware 500 is located below the eating subject's face, and therefore the authentication unit 160 may be located relatively higher than the identification information sensing unit 251. Here, the identification information 530 of the dishware 500 is located on the side of the dishware 500 and can be recognized by the identification information sensing unit 251 when the eating subject is standing and holding the dishware 500.

[0287] On the other hand, the identification information sensing unit 251 and the authentication unit 160 do not operate all the time, but can only operate when the tableware 500 is photographed and recognized by the photographing unit 250. Thus, the recognition of the tableware 500 by the photographing unit 250 acts as a kind of enabler, allowing the identification information sensing unit 251 and the authentication unit 160 to operate effectively.

[0288] 16, food measurement device 200 can be placed on a table or hung on a wall. A display unit 210 is located on the front of food measurement device 200, and an image capturing unit 250a for capturing an image of food or a dish containing food, and an image capturing unit 250b for capturing biometric information of a person eating are located on the front of food measurement device 200.

[0289] The food measuring device 200 can be tilted with the top inclined forward, and the photographing unit 250a for photographing food or tableware with food in it is located in the tilted area of ​​the food measuring device 200 and can photograph the tableware 500 below. The photographing unit 250b for photographing the biometric information of the eater is shown not to be in the tilted area, but may be located in the tilted area.

[0290] When the person eating takes a picture of the tableware 500 on which food is served using the food measuring device 200 or the user terminal 300, the screen of FIG. 17 can be displayed on the display unit 210 / 310.

[0291] For example, an image of a dish with food photographed by the photographing unit 250 / 350 can be displayed, and information about the type of food and the associated calories can be displayed. If a drinking water or the like is in a cup, the level of the drinking water can be displayed in the cup. In addition, information about the exercise and exercise time required to burn calories when ingesting the food can also be displayed. In addition, nutritional information and price information about the food to be ingested can also be displayed. In addition, the only difference between FIGS. 18 and 19 and FIGS. 10 and 12, respectively, is that the user terminal 300 plays the role of the food measuring device 200, and therefore a detailed description will be omitted.

[0292] The detailed configuration of the food measuring device 200 according to one embodiment of the present invention described above is the same as that of the food measuring method described in Figures 1 to 9, except for the category of the invention, so duplicated explanations and examples will be omitted.

[0293] The food measuring method according to an embodiment of the present invention may be embodied as a program (or application) and stored on a medium in order to be executed in combination with a server, which is hardware.

[0294] The above-mentioned program may include code coded in a computer language such as C, C++, JAVA, machine language, etc., which is read by the processor (CPU) of the computer through the device interface of the computer so that the computer can read the program and execute the method implemented in the program. Such code may include functional code related to functions defining functions necessary to execute the method, and may include control code related to execution procedures necessary for the processor of the computer to execute the function according to a predetermined procedure. In addition, such code may further include memory reference-related code for referring to a location (address) of the internal or external memory of the computer where additional information or media necessary for the processor of the computer to execute the function should be referenced. Furthermore, if the processor of the computer needs to communicate with any other remote computer or server to execute the function, the code may further include communication-related code for how to communicate with any other remote computer or server using a communication module of the computer, and what information or media should be sent or received during communication.

[0295] The storage medium means a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short time such as a register, cache, memory, etc. Specifically, examples of the storage medium include, but are not limited to, a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. That is, the program can be stored in various recording media on various servers to which the computer can be connected, or various recording media on the user's computer. Also, the medium can be distributed among computer systems connected via a network, and computer-readable code can be stored in a distributed manner.

[0296] In connection with the embodiments of the present invention, the steps of the methods or algorithms described may be implemented directly in hardware, or by software modules executed by hardware, or by a combination thereof. The software modules may be resident in a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Flash Memory, a hard disk, a removable disk, or any other form of computer readable storage medium known in the art to which the present invention pertains.

[0297] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, it should be understood by those skilled in the art to which the present invention pertains that the present invention may be embodied in other specific forms without changing the technical spirit or essential features of the present invention. Therefore, the above-described embodiments should be understood as illustrative in all respects and not restrictive.

Claims

1. 1. A computer-implemented food measuring method comprising: receiving image data of the dish with food on it and depth data of the dish with food on it after photographing and measuring the dish with food on it; a correction unit calculating a gradient of the dish on which the food is served, and correcting image data of the dish on which the food is served and depth data of the dish on which the food is served according to the calculated gradient; an extracting unit extracting image data of food contained in a dish from the corrected image data of the dish; a determining unit determining a type of food based on the extracted food image data; A calculation unit calculates a volume of food using the corrected depth data of the dish in which the food is served; Including, In the step of calculating the volume of the food, the correction unit recognizes an image of a noise component having a volume other than food in the image data of the tableware on which the food is served, and performs correction to remove the volume value of the recognized noise component from the calculated volume value of the food; The food measuring method according to claim 1, wherein the noise components include at least one of a hand and a dishware.

2. The correction unit calculates the gradient of the tableware on which the food is served, 2. The food measuring method of claim 1, wherein the correction unit calculates the gradient of the dish with food on it by analyzing image data of the dish with food on it, calculates the gradient of the dish with food on it using depth data for a plurality of points on the dish with food on it, calculates the gradient of the dish with food on it based on the gradient of a reference object photographed and measured such as the dish with food on it, or calculates the gradient of the dish with food on it based on the gradient of an imaging and measuring device.

3. In the step of correcting, The food measuring method according to claim 1, characterized in that the correction unit corrects the image data of the dish with food on it according to the calculated gradient, by correcting the image data of the dish with food on it to image data of a horizontal state according to the gradient calculated so that the dish in the image data of the dish with food on it is horizontal.

4. In the step of calculating the volume, The food measuring method according to claim 1, wherein the correction unit corrects the depth data of the dish on which food is served according to the calculated gradient, the correction unit correcting the depth data of the dish on which food is served according to the calculated gradient so that the dish calculated based on the depth data of the dish on which food is served is in a horizontal state.

5. The computer includes: A database is included in which image data of an empty dish and dish data including at least one of a size, a depth, and a capacity of at least one space formed in the dish are stored; The step of calculating the volume comprises: The food measuring method of claim 1, further comprising: correcting the volume of the food calculated using the tableware data including at least one of a size, depth, and capacity of a space in which the food is contained.

6. The database includes: Food data including the weight per volume of each food is stored, The method of claim 5, further comprising, after the step of calculating the volume, calculating a weight of the food by the calculation unit using the calculated food volume data and the food data.

7. The computer includes: a database having food-related information stored therein, the food-related information including at least one of food menu data, restaurant location data, date data, and time data; The determining step includes:

2. The food measuring method according to claim 1, wherein the determining unit determines the type of each food after deriving a food list candidate group by matching the extracted food image data with the food-related information.

8. The step of calculating the volume of each food includes: The food measuring method according to claim 1, further comprising: when different foods are stacked in the dish, the calculation unit calculates a volume of an area in which an upper food blocks a lower food.

9. The calculation unit calculates a volume of an area in which food located below is blocked by food located above, The calculation unit calculates an area of ​​a region in which a food located below is blocked by a food located above, the calculation unit calculates a height of an area of ​​the food located below that is not blocked by the food located above, The food measuring method according to claim 7 , wherein the calculation unit calculates a volume of an area in which an upper food block an underlying food by using the calculated area and height.

10. The step of calculating the volume comprises:

2. The method of claim 1, further comprising the step of calculating a volume of the liquid based on a position where the liquid meets the space and information on the size, depth and volume of the space when the food contained in the space is determined to be liquid.

11. a receiving unit for receiving image data of the dish on which food is served and depth data of the dish on which food is served after photographing and measuring the dish on which food is served; a correction unit that calculates a gradient of the dish on which the food is served, and corrects image data of the dish on which the food is served and depth data of the dish on which the food is served according to the calculated gradient; an extracting unit for extracting image data of food contained in the dish from the corrected image data of the dish on which food is served; a determination unit for determining a type of food based on the extracted food image data; a calculation unit for calculating a volume of food using the corrected depth data of the dish on which the food is served; Including, the correction unit recognizes an image of a noise component having a volume other than food in the image data of the tableware on which food is served, and performs correction to remove the volume value of the recognized noise component from the calculated volume value of the food; The food measurement server, wherein the noise components include at least one of a hand and tableware.

12. a receiving unit for receiving image data of the dish on which food is served and depth data of the dish on which food is served after photographing and measuring the dish on which food is served; a correction unit that calculates a gradient of the dish on which the food is served, and corrects image data of the dish on which the food is served and depth data of the dish on which the food is served according to the calculated gradient; an extracting unit for extracting image data of food contained in the dish from the corrected image data of the dish on which food is served; a determination unit for determining a type of food based on the extracted food image data; a calculation unit for calculating a volume of food using the corrected depth data of the dish on which the food is served; Including, the correction unit recognizes an image of a noise component having a volume other than food in the image data of the tableware on which food is served, and performs correction to remove the volume value of the recognized noise component from the calculated volume value of the food; The food measuring device, wherein the noise components include at least one of a hand and a dishware.

13. A program stored on a medium for causing a computer to execute the method according to any one of claims 1 to 10.

Images