Food recognition method and apparatus using object recognition model

Abstract

A food recognition method and apparatus using an object recognition model are disclosed. According to an aspect of the present invention, provided is the food recognition method using an object recognition model, the food recognition method comprising the steps of: acquiring a recognition target image; recognizing, by an object recognition unit, an object in the recognition target image by using a first object recognition model that has learned an object-related data set; recognizing, by a food recognition unit, food in the recognition target image by using a second object recognition model that has learned a food-related data set through a pre-input food image; and comparing, by an information analysis unit, object data recognized by the object recognition unit and food data recognized by the food recognition unit, and when location information of data regarding a non-food object from among the object data corresponds to the food data, recognizing, as food, food data remaining after excluding the corresponding food data.

Description

Food recognition method and device using an object recognition model

[0001] The present invention relates to a method and device for recognizing food using an object recognition model.

[0002] The technology to recognize objects from images is used in many industrial fields, such as autonomous driving, facial recognition, defect detection, and lesion detection, depending on the type of object being targeted.

[0003] Recently, the performance of image recognition technology using artificial intelligence has been greatly improved, but there is a problem that a large amount of training data is required to train the object recognition model used to recognize objects contained in image data, and training using this data also requires a lot of computation.

[0004] Furthermore, object recognition typically involves determining the types of objects that can be predicted during the learning phase, and predictions are only possible for objects of the learned type. Among various recognition fields, food recognition in particular shows a trend of increasing the number of types of food that must be recognized. In line with this trend, continuously expanding the range of foods that can be recognized will inevitably require continuous investment of time and money.

[0005] Various research efforts are underway to overcome the limitation of being able to predict only the types of objects learned. One such approach, open set recognition, is inapplicable in situations where a trained object recognition model already exists. Therefore, it may not be an appropriate choice for industries that require rapid response and adaptation.

[0006] [Prior Art Literature]

[0007] [Patent Document]

[0008] Republic of Korea Patent Publication No. 10-2023-0040480 (March 23, 2023)

[0009] The present invention provides a method and device for recognizing food using an object recognition model that analyzes object recognition results using multiple object recognition models.

[0010] According to one aspect of the present invention, a method for recognizing food using an object recognition model is provided, including the steps of: obtaining a recognition target image; recognizing an object in the recognition target image by an object recognition unit using a first object recognition model trained with an object-related data set; recognizing food in the recognition target image by a food recognition unit using a second object recognition model trained with a food-related data set through a previously input food image; and comparing object data recognized by the object recognition unit and food data recognized by the food recognition unit, and, if positional information of data for an object other than food corresponds among the food data and the object data, excluding the corresponding food data and recognizing the remaining food data as food.

[0011] The step of recognizing an object by an object recognition unit is performed by creating a bounding box in an area where an object is placed and by mathematical expressions 1 and 2 below, and the step of recognizing food by a food recognition unit is performed by creating a bounding box in an area where food is placed in an image to be recognized and by mathematical expressions 3 and 4 below.

[0012] (Equation 1)

[0013]

[0014] (Equation 2)

[0015]

[0016] (Equation 3)

[0017]

[0018] (Equation 4)

[0019]

[0020] The step of recognizing food can be determined by determining that the food data and the data for non-food objects among the object data correspond if the value calculated by the mathematical expression 5 below for the bounding box of the food data and the bounding box of the data for non-food objects among the object data is greater than the reference value.

[0021] (Equation 5)

[0022]

[0023] An object-related dataset might contain images of objects containing food.

[0024] In the step of recognizing food, if data for objects other than food corresponds to food data and object data, and the class index of the object data is not included in the food index of the food data, the corresponding food data can be excluded.

[0025] In the step of recognizing food, if the data for objects other than food corresponds to the food data and the object data, and the Class index of the object data is the Background index, the Class index of the object data can be determined to be included in the Food index of the food data.

[0026] The food recognition step can recognize food data with a confidence score higher than the first confidence criterion score calculated by evaluating the second object recognition model with the test data set as food.

[0027] The food recognition step can recognize food data with a confidence score higher than the second confidence criterion score calculated using a second object recognition model among the object-related data sets among the food data with a confidence score lower than the first confidence criterion score as food.

[0028] The second reliability criterion score can be calculated according to the mathematical formula 6 below.

[0029] (Equation 6)

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] According to another aspect of the present invention, a food recognition device using an object recognition model is provided, including an image acquisition unit for acquiring a recognition target image, an object recognition unit for recognizing an object in the recognition target image using a first object recognition model learned from an object-related data set, a food recognition unit for recognizing food in the recognition target image using a second object recognition model learned from a food-related data set through previously input food photos, and an information analysis unit for comparing object data recognized by the object recognition unit and food data recognized by the food recognition unit, and when location information of data for an object other than food corresponds among the food data and the object data, excluding the corresponding food data and recognizing the remaining food data as food.

[0036] According to the present invention, food can be recognized by analyzing the object recognition results using an object recognition model learned from an object-related data set and an object recognition model learned through an existing input food image.

[0037] FIG. 1 is a block diagram illustrating a food recognition device using an object recognition model according to one embodiment of the present invention.

[0038] FIG. 2 is a flowchart illustrating a food recognition method using an object recognition model according to one embodiment of the present invention.

[0039] FIG. 3 is a diagram showing the conceptual sequence of a food recognition method using an object recognition model according to one embodiment of the present invention.

[0040] FIG. 4 is a diagram illustrating a step of comparing object recognition and food recognition results of a food recognition method using an object recognition model according to one embodiment of the present invention and excluding objects that are not food.

[0041] [Explanation of symbols]

[0042] 10: Recognition target image

[0043] 100: Food recognition device using an object recognition model

[0044] 110: Image acquisition unit

[0045] 120: Object recognition unit

[0046] 130: Food Recognition Department

[0047] 140: Information Analysis Department

[0048] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention. In describing the present invention, detailed descriptions of related known technologies will be omitted if they are deemed to obscure the gist of the present invention.

[0049] Terms such as first, second, etc. may be used to describe various components, but these components should not be limited by these terms. These terms are used solely to distinguish one component from another.

[0050] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0051] In addition, the term "coupling" is used as a concept that encompasses not only cases where each component is physically in direct contact with the other components in the contact relationship between each component, but also cases where another component is interposed between each component and each component is in contact with the other component.

[0052] Hereinafter, an embodiment of a food recognition method and device using an object recognition model according to the present invention will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical or corresponding components are assigned the same drawing numbers, and redundant descriptions thereof will be omitted.

[0053]

[0054] According to one embodiment of the present invention, a method for recognizing food using an object recognition model is provided, including a step (S110) of obtaining a recognition target image (10), a step (S120) of recognizing an object in the recognition target image (10) by using a first object recognition model in which an object-related data set is learned by an object recognition unit (120), a step (S130) of recognizing food in the recognition target image (10) by using a second object recognition model in which a food-related data set is learned through a previously input food image, and a step (S140) of comparing object data recognized by the object recognition unit (120) and food data recognized by the food recognition unit (130), and if positional information of data for an object other than food corresponds among the food data and the object data, excluding the corresponding food data and recognizing the remaining food data as food.

[0055] The food recognition method and device using the object recognition model of the present embodiment can recognize food by analyzing the object recognition results using a first object recognition model learned from an object-related data set and a second object recognition model learned through an existing input food image.

[0056] This embodiment can improve recognition accuracy by recognizing non-food objects through a first object recognition model, thereby reducing the misrecognition rate of a second object recognition model learned through food images.

[0057] Furthermore, the present embodiment can recognize a type of food that has not been learned without changing the model network and without additionally learning a data set for recognizing non-food objects separately during the process of performing food recognition.

[0058] Hereinafter, with reference to FIGS. 1 to 4, each step of the food recognition method using the object recognition model according to the present embodiment and each component of the food recognition device (100) using the object recognition model will be described.

[0059]

[0060] First, as illustrated in FIGS. 2 to 4, the food recognition method using the object recognition model of the present embodiment can recognize food in the recognition target image (10) by performing the steps of: acquiring a recognition target image (10); recognizing an object in the recognition target image (10) by using a first object recognition model in which an object-related data set has been learned; recognizing food in the recognition target image (10) by using a second object recognition model in which a food-related data set has been learned through a previously input food image; and recognizing food by comparing object data recognized by the object recognition unit (120) and food data recognized by the food recognition unit (130). Each step can be performed by the image acquisition unit (110), the object recognition unit (120), the food recognition unit (130), and the information analysis unit (140) illustrated in FIG. 1 to perform each step and perform each process.

[0061] As illustrated in FIGS. 1 to 3, the image acquisition unit (110) performs a step of acquiring a recognition target image (10), and may capture a recognition target image (10) using a camera and various devices including a camera, or may receive a recognition target image (10) that has already been captured.

[0062] As illustrated in FIGS. 1 to 3, the object recognition unit (120) may perform a step of recognizing an object in a recognition target image (10) using a first object recognition model trained on an object-related data set. Here, the object-related data set may include COCO, VOC, and ImageNet, and may include images of objects including food.

[0063] As illustrated in FIGS. 1 to 3, the food recognition unit (130) can perform a step of recognizing food in a recognition target image (10) using a second object recognition model that has learned a food-related data set through previously input food images.

[0064] The first object recognition model and the second object recognition model described above may include a YOLO (You only Look Once) series model and an SSD (Single Shot Detector) series model, and may include various models that perform object recognition by creating a bounding box in an area where a recognition target object is placed.

[0065] More specifically, the object recognition unit (120) can perform the step of recognizing an object by creating a bounding box in the area where the object is placed and using the following mathematical expressions 1 and 2.

[0066] (Equation 1)

[0067]

[0068] (Equation 2)

[0069]

[0070] In addition, the food recognition unit (130) can perform the food recognition step by creating a bounding box in the area where the food is placed in the recognition target image (10) and using the following mathematical expressions 3 and 4.

[0071] (Equation 3)

[0072]

[0073] (Equation 4)

[0074]

[0075] As illustrated in FIGS. 1 to 4, the information analysis unit (140) can recognize food by comparing object data recognized by the object recognition unit (120) and food data recognized by the food recognition unit (130), and more specifically, if the location information of data for an object other than food among the food data and the object data corresponds, the information analysis unit (140) can perform a step of excluding the corresponding food data and recognizing the remaining food data as food.

[0076] Here, in the step of recognizing food (S140), the information analysis unit (140) can determine that the data for non-food objects among the food data and the data for non-food objects among the object data correspond to each other when the value calculated by the mathematical expression 5 below is greater than the reference value for the bounding box of the food data and the bounding box of the data for non-food objects among the object data.

[0077] (Equation 5)

[0078]

[0079] In addition, as illustrated in FIG. 4, in the step of recognizing food (S140), when data for an object other than food corresponds to the food data and the object data, if the Class index of the object data is not included in the Food index of the food data, the information analysis unit (140) can exclude the corresponding food data (S142).

[0080] For example, if the Class index of the object data is recognized in the range of {0,1,2,쪋,79} and the Food index of the food data recognized through the recognition target image (10) is {41,45,46,47,48,49,50,51,52,53,54,55,56,60}, if the Class index of the object data is not included in the Food index of the food data, the corresponding food data can be excluded.

[0081] In addition, as illustrated in FIG. 4, in the step of recognizing food, the information analysis unit (140) can determine that, in the case where data for an object other than food corresponds to the food data and the object data, the Class index of the object data is the Background index, and the Class index of the object data is included in the Food index of the food data.

[0082] Here, the Background index is a Class index for an object that holds food or is placed together with food, such as a tableware. For example, in the example described above, Class index 41 may be designated as a cup, 45 as a bowl, and 60 as a table. The bounding boxes predicted as cups, bowls, and tables are objects that overlap greatly with food, and the objects are considered as food, thereby preventing food data recognized by the food recognition unit (130) from being excluded.

[0083] As illustrated in FIGS. 1 to 3, the information analysis unit (140) can recognize food data having a reliability score higher than the first reliability criterion score calculated by evaluating the second object recognition model with a test data set as food in the food recognition step, and can be calculated using the following mathematical expression 7.

[0084] In addition, as illustrated in FIGS. 1 to 3, in the step of recognizing food (S140), the information analysis unit (140) can recognize food data having a reliability score higher than the second reliability criterion score calculated using the second object recognition model as an object-related data set other than food among the object-related data sets among the food data having a reliability score lower than the first reliability criterion score as food (S144).

[0085] The food recognition step using this second reliability criterion score can enable the second object recognition model of the food recognition unit (130) to recognize even foods that it has not learned.

[0086] In addition, the second confidence criterion score can be calculated according to the mathematical formula 6 below, and can be calculated through a set of confidence scores among the inference values ​​of the Non-Food data set.

[0087] (Equation 6)

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] (Equation 7)

[0094]

[0095] The second reliability criterion score described above is obtained by calculating the distribution of reliability score values ​​from the inferred result values ​​using data composed of a non-food object data set as shown in the above mathematical expression 6, and selecting the median value from this distribution.

[0096] In addition, the information analysis unit (140) can obtain a threshold for food recognition through the first reliability criterion score and the second reliability criterion score described above through test data. Here, when the first reliability criterion score is regarded as conf1th and the second reliability criterion score is regarded as conf2th, the fixed values ​​of conf1th and conf2th can be set, and an ordered pair of conf0th and IoUth corresponding to the threshold can be obtained.

[0097] The information analysis unit (140) can obtain an optimal threshold value by inputting all ordered pairs of conf0th and IoUth corresponding to threshold values ​​while performing the entire process shown in FIG. 3 and comparing the F1 scores derived accordingly. In other words, it can obtain an ordered pair of conf0th and IoUth with the highest F1 score, and this threshold value can be used as a threshold value in all evaluation stages.

[0098] For example, if the ordered pairs of conf0th and IoUth are composed of values ​​from 0.1 to 0.99 with an interval of 0.01, and conf0th_IoUth = {(0.1, 0.1), (0.1, 0.2), (0.1, 0.3), ... (0.99, 0.99)}, then when checking the F1 score,

[0099] F1 score = 0.9 when conf0th = 0.1, conf1th = 0.5, conf2th = 0.2, IoUth = 0.1,

[0100] F1 score = 0.8 when conf0th = 0.1, conf1th = 0.5, conf2th = 0.2, and IoUth = 0.2.

[0101] (syncopation)

[0102] When conf0th = 0.99, conf1th = 0.5, conf2th = 0.2, IoUth = 0.99, the F1 score can be 0.5, and if the highest F1 score is 0.9, conf0th = 0.1, IoUth = 0.1 can be selected as the threshold.

[0103]

[0104] Meanwhile, the components of the aforementioned embodiments can be easily understood from a process perspective. That is, each component can be understood as a separate process. Furthermore, the processes of the aforementioned embodiments can be easily understood from the perspective of the device components.

[0105] In addition, the technical contents described above may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiments or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only programming language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0106] Above, one embodiment of the present invention has been described, but a person having ordinary skill in the art will be able to modify and change the present invention in various ways by adding, changing, deleting or adding components, etc., within the scope that does not depart from the spirit of the present invention described in the claims, and this will also be considered to be included within the scope of the rights of the present invention.

Claims

1. Step of acquiring a recognition target image; A step in which an object recognition unit recognizes an object in the recognition target image using a first object recognition model learned from an object-related data set; A step of recognizing food in the recognition target image using a second object recognition model that has learned a food-related data set through a previously input food image; and A method for recognizing food using an object recognition model, comprising the step of: comparing, by an information analysis unit, object data recognized by the object recognition unit and food data recognized by the food recognition unit, and, if location information of data for objects other than food among the food data and the object data corresponds, excluding the corresponding food data and recognizing the remaining food data as food.

2. In paragraph 1, The step of recognizing an object by the above object recognition unit is performed by creating a bounding box in the area where the object is placed and by the mathematical expressions 1 and 2 below. A food recognition method using an object recognition model, wherein the step of recognizing food by the above food recognition unit is performed by generating a bounding box in an area where food is placed in the recognition target image and by the following mathematical expressions 3 and 4. (Mathematical formula 1) (Mathematical formula 2) (Mathematical formula 3) (Mathematical formula 4) 3. In paragraph 2, The steps to recognize the above food are: A food recognition method using an object recognition model, wherein a case where a value calculated by the mathematical expression 5 below for the bounding box of the food data and the bounding box of the data for non-food objects among the object data is greater than a reference value is determined as a case where the food data and the data for non-food objects among the object data correspond. (Mathematical formula 5) 4. In paragraph 3, A method for recognizing food using an object recognition model, wherein the above object-related data set includes images of objects containing food.

5. In paragraph 4, The steps to recognize the above food are: A food recognition method using an object recognition model, wherein when data for an object other than food corresponds to the food data and the object data, if the Class index of the object data is not included in the Food index of the food data, the corresponding food data is excluded.

6. In paragraph 5, The steps to recognize the above food are: A food recognition method using an object recognition model, wherein when the Class index of the object data is a Background index in a case where data for an object other than food corresponds to the food data, the Class index of the object data is determined to be included in the Food index of the food data.

7. In paragraph 3, The steps to recognize the above food are: A food recognition method using an object recognition model, which recognizes the food data having a reliability score higher than a first reliability criterion score calculated by evaluating the second object recognition model with a test data set as food.

8. In paragraph 7, The steps to recognize the above food are: A food recognition method using an object recognition model, wherein among the food data having a reliability score lower than the first reliability criterion score, the food data having a reliability score higher than the second reliability criterion score calculated using a second object recognition model among the object-related data sets that are not food are also recognized as food.

9. In paragraph 8, A food recognition method using an object recognition model, wherein the second reliability criterion score is calculated according to the mathematical formula 6 below. (Mathematical formula 6) 10. Image acquisition unit for acquiring a recognition target image; An object recognition unit that recognizes an object in the recognition target image using a first object recognition model learned from an object-related data set; A food recognition unit that recognizes food in the recognition target image using a second object recognition model that has learned a food-related data set through previously input food images; and A food recognition device using an object recognition model, comprising an information analysis unit for comparing object data recognized by the object recognition unit and food data recognized by the food recognition unit, and, if positional information of data for objects other than food among the food data and the object data corresponds, excluding the corresponding food data and recognizing the remaining food data as food.

Images