Method and apparatus for object identification using hierarchical models

Abstract

The present invention relates to an object identification method and device using a hierarchical model, and an object identification method using a hierarchical model according to one embodiment of the present invention includes a step of an object identification device detecting an object area in which an object is located from an object image based on a learned object area extraction model and extracting the detected object area, and a step of an object management server identifying object type information located in the object area extracted from the object identification device based on an object type inference model.

Description

[Technical Field]

[0001] The present invention relates to a method and apparatus for identifying objects. [Background technology]

[0002] In recent years, interest in health has increased, but at the same time, the number of people suffering from overweight and obesity has also been steadily increasing. Such overweight and obesity are serious problems that can lead to various diseases such as diabetes and high blood pressure.

[0003] Therefore, in order to solve such overweight and obesity, it is necessary to analyze one's eating habits first. Generally, a person knows what he or she likes and dislikes, but does not remember the types and frequency of the objects that he or she actually consumes. Therefore, in order to analyze one's eating habits, it is necessary to identify the objects that are actually consumed and analyze the individual's eating habits according to the information about the identified objects.

[0004] However, most of the currently available technologies are limited to simple image search of object images taken with a camera, resulting in significantly lower accuracy. Furthermore, if the accuracy of object type identification during image search is low, there is a problem that larger errors will occur in subsequent steps such as calorie calculation.

[0005] For example, in order to identify an object to be ingested, a user must input object information using a mobile device or an input device. Here, the user photographs the object by executing an object-related application or a general photography mode executed on the mobile device. In this case, the user is inconveniently required to search for the photographed object image using the object-related application and then enter the object information for the object again. Alternatively, when photographing in the general photography mode, the user is inconveniently required to search for the photographed object image in an album, upload the found object image to the object-related application, and then search for the uploaded object image again to enter the object information. Therefore, since photographing an object and identifying the object are performed as separate operations, which is inconvenient, the frequency with which the user uploads object information rapidly decreases after photographing a number of object images.

[0006] For example, a user may photograph an object only once and then input the object information later. If a long time has passed since the object image was photographed, the user may not remember the object information they intended to input when photographing the object, and may be unable to record the object information, or may enter incorrect object information, resulting in an error in the eating habit analysis. Furthermore, even if a user wishes to input various object information for a photographed object image, they may be forced to input only a fixed number of pieces of object information set for each application. In such cases, the number of object information pieces is limited, and the user's eating habit analysis may be inaccurate. This added inconvenience to the user leads to a decline in satisfaction with services that analyze personal eating habits, for example, through object identification. Summary of the Invention [Problem to be solved by the invention]

[0007] The embodiments of the present invention aim to provide an object identification method and device using a hierarchical model to increase the object identification speed by identifying object regions in an object identification device and identifying object type information in an object management server.

[0008] The present invention provides an object identification method and apparatus using a hierarchical model for quickly determining an object located in an object image by classifying object categories in stages using an object region extraction model and an object type inference model and identifying object type information.

[0009] However, the problem to be solved by the present invention is not limited thereto, and may be expanded in various environments without departing from the spirit and scope of the present invention. [Means for solving the problem]

[0010] According to one embodiment of the present invention, there may be provided an object identification method using a hierarchical model, which is performed by an object identification system, including a step in which an object identification device detects an object area where an object is located from an object image based on a learned object area extraction model and cuts out the detected object area, and a step in which an object management server identifies object type information located in the object area cut out from the object identification device based on an object type inference model.

[0011] The step of extracting the detected object region can detect the object region from the object image using a box surrounding the object or pixels occupied by the object based on the learned object region extraction model.

[0012] The step of extracting the detected object region can extract first feature points through a first image encoder of the learned object region extraction model, and extract a feature point region from the extracted first feature points to infer the object region.

[0013] The method further includes a step in which the object identification device classifies object type categories located in the extracted object area in a stepwise manner, and the step of identifying the object type information can involve the object management server identifying the object type information using an object type inference model corresponding to the classified category.

[0014] The step of classifying the object type categories stepwise may include classifying the categories of the objects located in the extracted object region stepwise from the highest major category to the lowest minor category.

[0015] In the step of identifying the object type information, the object management server can identify the object type information by using an object type inference model corresponding to the lowest level subcategory, which is the classified category.

[0016] In the step of identifying the object type information, the object management server can determine the object type information as any one of a plurality of classes included in the lowest classification.

[0017] The step of identifying the object type information may involve the object management server extracting second feature points through a second image encoder of an object type inference model corresponding to the classified category, and inputting the extracted second feature points into a feature point decoder to infer the object type information.

[0018] Meanwhile, according to another embodiment of the present invention, there may be provided an object identification method using a hierarchical model, which is performed by an object identification device, including the steps of detecting an object region where an object is located from an object image based on a learned object region extraction model and cutting out the detected object region, and identifying object type information located in the cut-out object region based on an object type inference model.

[0019] Meanwhile, according to another embodiment of the present invention, there may be provided an object identification method using a hierarchical model, which includes the steps of: detecting an object area where an object is located from an object image based on a learned object area extraction model, and segmenting the detected object area; and identifying object type information located in the segmented object area based on an object type inference model.

[0020] Meanwhile, according to another embodiment of the present invention, there may be provided an object identification system using a hierarchical model, comprising: an object identification device that detects an object region where an object is located from an object image based on a learned object region extraction model and cuts out the detected object region; and an object management server that identifies object type information located in the object region cut out from the object identification device based on an object type inference model.

[0021] The object identification device can detect an object region from the object image based on the learned object region extraction model using a box surrounding the object or pixels occupied by the object.

[0022] The object identification device can extract first feature points through a first image encoder of the learned object region extraction model, and extract a feature point region from the extracted first feature points to infer an object region.

[0023] The object identification device classifies object type categories located in the extracted object area in stages, and the object management server can identify object type information using an object type inference model corresponding to the classified categories.

[0024] The object identification device may classify the categories of the objects located in the extracted object region stepwise from the highest major classification to the lowest minor classification.

[0025] The object management server can identify object type information using an object type inference model corresponding to the lowest level subclassification, which is the classified category.

[0026] The object management server can determine the object type information as one of a plurality of classes included in the lowest classification.

[0027] The object management server extracts second feature points through a second image encoder of an object type inference model corresponding to the classified category, and inputs the extracted second feature points to a feature point decoder to infer object type information.

[0028] Meanwhile, according to another embodiment of the present invention, there may be provided an object identification device using a hierarchical model, which includes a memory for storing one or more programs and a processor for executing the one or more stored programs, wherein the processor detects an object region in which an object is located from an object image based on a learned object region extraction model, cuts out the detected object region, and identifies object type information located in the cut-out object region using an object type inference model.

[0029] Meanwhile, according to another embodiment of the present invention, an object management server using a hierarchical model may be provided, which includes a memory for storing one or more programs in the object management server, and a processor for executing the one or more stored programs, wherein the processor detects an object region in which an object is located from an object image based on a learned object region extraction model, cuts out the detected object region, and identifies object type information located in the cut-out object region using an object type inference model. [Effects of the Invention]

[0030] The disclosed technology may have the following effects. However, this does not mean that a particular embodiment should include all or only the following effects, and therefore the scope of the disclosed technology should not be understood to be limited thereby.

[0031] The embodiments of the present invention can increase the object identification speed by identifying the object area in the object identification device and identifying the object type information in the object management server.

[0032] The embodiments of the present invention can quickly determine an object located in an object image by gradually classifying object categories and identifying object type information using an object region extraction model and an object type inference model.

[0033] The embodiments of the present invention can increase the efficiency of object identification operations by adjusting the categories to be classified, taking into consideration the real-time nature of the classification, the resource status and computational load of the object identification device, etc.

[0034] The embodiments of the present invention can improve the object identification time and accuracy by quickly performing the object category classification operation and object information determination operation, which are difficult for the object identification device to perform, in cooperation with the object management server. [Brief explanation of the drawings]

[0035] [Figure 1] FIG. 1 is a diagram showing the configuration of an object identification system using a hierarchical model according to an embodiment of the present invention.

[0036] [Figure 2] 1 is a flowchart illustrating a method for identifying objects using a hierarchical model according to an embodiment of the present invention.

[0037] [Figure 3] FIG. 1 illustrates an object domain reasoning model used in one embodiment of the present invention.

[0038] [Figure 4] FIG. 1 illustrates an object type inference model used in one embodiment of the present invention.

[0039] [Figure 5] 1 is a flowchart illustrating a method for identifying objects using a hierarchical model according to an embodiment of the present invention.

[0040] [Figure 6] FIG. 10 illustrates an example of an object identification operation using a hierarchical model according to an embodiment of the present invention.

[0041] [Figure 7] FIG. 10 illustrates an example of an object identification operation using a hierarchical model according to an embodiment of the present invention.

[0042] [Figure 8] FIG. 10 illustrates an example of an object identification operation using a hierarchical model according to an embodiment of the present invention.

[0043] [Figure 9] FIG. 10 illustrates an example of an object identification operation using a hierarchical model according to an embodiment of the present invention.

[0044] [Figure 10] 10 is a diagram illustrating an operation of an object identification method using a hierarchical model in conjunction with an object management server according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0045] The present invention may be modified in various ways and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to the specific embodiments, and it should be understood that the present invention includes all modifications, equivalents, and alternatives within the technical spirit and scope of the present invention. In describing the present invention, if it is determined that detailed descriptions of related publicly known technologies may obscure the gist of the present invention, such detailed descriptions will be omitted.

[0046] Terms such as "first" and "second" may be used to describe various components, but the components are not limited by the terms. The terms are used only to distinguish one component from another.

[0047] The terms used in the present invention are merely used to describe specific embodiments and are not intended to limit the present invention. The terms used in the present invention are generally used and currently widely, taking into consideration the functions of the present invention. However, these terms may change depending on the intentions of engineers in the field, precedents, or the emergence of new technologies. In addition, in certain cases, the applicant may arbitrarily select terms, and in such cases, the meanings of these terms will be described in detail in the relevant description of the invention. Therefore, the terms used in the present invention should be defined based on the meanings of the terms and the overall content of the present invention, rather than simply by their names.

[0048] The singular expression includes the plural expression unless the context clearly dictates otherwise. As used herein, the terms "comprise" or "have" and the like are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0049] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components will be given the same drawing numbers and redundant description thereof will be omitted.

[0050] FIG. 1 is a diagram showing the configuration of an object identification system using a hierarchical model according to an embodiment of the present invention.

[0051] First, as shown in FIG. 1, an object identification system 10 according to the present invention can include an object identification device 100 and an object management server 200 that communicates with the object identification device 100.

[0052] The object identification device 100 can detect an object region where an object is located from an object image based on the learned object region extraction model, and can cut out the detected object region. Then, the object identification device 100 can transmit the cut-out object region to the object management server 200.

[0053] The object management server 200 can identify the object type information located in the object area cut out from the object identification device 100 based on the object type inference model.

[0054] In this way, the operation of identifying the object type information can be performed in the object management server 200, rather than in the object identification device 100. Here, the object management server 200 can be realized as a cloud server. Alternatively, the operation of detecting the object area and the operation of identifying the object type information can both be performed in the object identification device 100.

[0055] This is to increase the speed of the object region detection and object type identification operations. For example, when a user takes a photo of an object (food or drink) using a mobile terminal including the object identification device 100, the original image file taken is 4 MB. If the 4 MB original image file is sent to the object management server 200 in the cloud, the file size will be too large. Therefore, the object identification device 100 can extract only the object (food or drink) region from the original image and send only 200 KB of the image to the object management server 200 in the cloud. In this way, the mobile terminal can perform only the operation of detecting the object region, and the cloud can perform the operation of identifying object type information. Alternatively, the mobile terminal can perform both the operation of detecting the object region and the operation of identifying object type information.

[0056] Here, which of the operations to detect the object region and identify the object type information is performed may be determined based on at least one of the data volume to be transmitted to the object management server 200, the hardware performance of the object identification device 100, the communication speed between the object management server 200 and the object identification device 100, the size of the object region extraction model or the object type inference model, and the identification accuracy of the object type information. For example, when the data volume is smaller than a preset threshold, the hardware performance is high, the communication speed is slow, the model size is small, and the accuracy is high, it may be advantageous for the object identification device 100 to perform the operation to detect the object region and the operation to identify the object type information. Conversely, when the data volume is larger than a preset threshold, the hardware performance is low, the communication speed is fast, the model size is large, and the accuracy is low, it may be advantageous for the object identification device 100 to perform the operation to detect the object region and for the object management server 200 to perform the operation to identify the object type information. In this case, the object identification device 100 may convert the detected object region and transmit it to the object management server 200 instead of extracting the detected object region. For example, the object identification device 100 can lower the resolution of the detected object area, convert it into a thumbnail, and send it quickly to the object management server 200, or when sending a file containing the object area, can convert the extension or file format and send it to the object management server 200.

[0057] 1, the object identification device 100 may include a communication module 110, a camera 120, a display 130, a memory 140, and a processor 150. The object management server 200 may include a communication module 210, a memory 220, and a processor 230.

[0058] However, not all of the components shown are required components, and the object identification system 10 may be realized with more components than those shown, or may be realized with fewer components.

[0059] The specific configuration and operation of each component of the object identification system 10 in FIG. 1 will now be described.

[0060] First, the specific configuration and operation of each component of the object identification device 100 will be described.

[0061] The communication module 110 may include one or more modules that enable communication between the object identification device 100 and a wireless communication device or between the object identification device 100 and the object management server 200. The communication module 110 may also include one or more modules that couple the object identification device 100 to one or more networks.

[0062] The camera 120 can capture images or videos in response to user operation. The camera 120 can be configured with a single camera, multiple cameras, a single image sensor, or multiple image sensors. The camera 120 can be configured with at least one of at least one 2D camera, at least one 3D camera, at least one stereo camera, and at least one image sensor. Here, the camera 120 can capture images or videos of objects before and after eating in response to user operation.

[0063] The display 130 may be layered with or integrated with a touch sensor to provide a touch screen, which can provide an input interface between the object identification device 100 and a user, as well as an output interface between the object identification device 100 and a user.

[0064] The memory 140 may store data supporting various functions of the object identification device 100. The memory 140 may store one or more programs, a plurality of application programs, or applications run by the object identification device 100, and data and commands for the operation of the object identification device 100. At least some of these application programs may be downloaded from the object management server 200 via wireless communication. Also, at least some of these application programs may exist for the basic functions of the object identification device 100. Meanwhile, the application programs may be stored in the memory 140, installed on the object identification device 100, and driven by the processor 150 to perform the operations (or functions) of the object identification device 100.

[0065] In addition to operations related to the application programs, the processor 150 may generally control the overall operation of the object identification device 100. The processor 150 may process signals, data, information, etc. input or output via the components described above, or may run application programs stored in the memory 140 to provide or process appropriate information or functions to a user.

[0066] In one embodiment, the processor 150 executes one or more programs stored in the memory 140 to detect an object region in which an object is located from an object image based on the learned object region extraction model, cut out the detected object region, and identify object type information located in the cut-out object region based on the object type inference model.

[0067] According to an embodiment, the processor 150 can detect an object region from an object image based on a learned object region extraction model using a box surrounding the object or pixels occupied by the object.

[0068] According to the embodiment, the processor 150 can extract first feature points via a first image encoder of the learned object region extraction model, extract feature point regions from the extracted first feature points, and infer the object region.

[0069] According to the embodiment, the processor 150 detects an object region in which an object is located from an object image based on the learned object region extraction model, cuts out the detected object region, gradually classifies the object type categories located in the cut-out object region, and identifies object type information using the object type inference model corresponding to the classified category.

[0070] According to the embodiment, the processor 150 can classify the categories of the objects located in the cut-out object region stepwise from the highest major classification to the lowest minor classification.

[0071] According to some embodiments, the processor 150 can identify object type information using an object type inference model corresponding to the lowest level subclassification, which is the classified category.

[0072] According to the embodiment, the processor 150 can determine that the object type information is one of a plurality of classes included in the lowest-level classification.

[0073] According to the embodiment, the processor 150 extracts second feature points through a second image encoder of an object type inference model corresponding to the classified category, and inputs the extracted second feature points to a feature point decoder to infer object type information.

[0074] According to an embodiment, the object identification device 100 further includes a communication module 210 that communicates with the object management server 200, and the processor 150 transmits the cut-out object area and the classified object type category to the object management server 200 via the communication module 210 to request object type identification, and can receive object type information determined using the classified category and the corresponding object type inference model from the object management server 200.

[0075] 1 to run an application program stored in memory 140. Furthermore, processor 150 can operate at least two or more components included in object identification device 100 in combination with each other to run the application program. Here, processor 150 will be described in detail later.

[0076] Next, the specific configuration and operation of each component of the object management server 200 will be described.

[0077] The communications module 210 may include one or more modules that enable communications between the object management server 200 and a wireless communication device or between the object management server 200 and the object identification device 100. The communications module 210 may also include one or more modules that couple the object management server 200 to one or more networks.

[0078] The memory 220 may store data supporting various functions of the object management server 200. The memory 220 may store one or more programs, a plurality of application programs, or applications run by the object management server 200, and data and commands for the operation of the object management server 200. At least some of these application programs may be downloaded to the object management server 200 via an external communication network by wireless communication. At least some of these application programs may exist for the basic functions of the object management server 200. Meanwhile, the application programs may be stored in the memory 220, installed on the object management server 200, and driven by the processor 230 to perform the operations (or functions) of the object management server 200.

[0079] In addition to operations related to the application programs, the processor 230 may generally control the overall operation of the object management server 200. The processor 230 may process signals, data, information, etc. input or output via the components described above, or may run application programs stored in the memory 220 to provide or process appropriate information or functions to a user.

[0080] In one embodiment, the processor 230 may execute one or more programs associated with an operation of determining object information from an object image in conjunction with the object identification device 100 stored in the memory 220. As an example, the processor 230 may stepwise classify the remaining categories not classified by the object identification device 100 among the categories of objects located in the segmented object region, and determine object type information using an object type inference model corresponding to the classified category.

[0081] The configuration of the object identification system 10 according to the present invention has been described in detail above with reference to Figure 1. The object identification system 10 may include fewer or more components than those shown in Figure 1.

[0082] Hereinafter, object identification methods according to various embodiments of the present invention will be described with reference to FIGS.

[0083] FIG. 2 is a flowchart illustrating a method for identifying objects using a hierarchical model according to an embodiment of the present invention.

[0084] 2, in step S101, an object identification device 100 using a hierarchical model according to an embodiment of the present invention detects an object region where an object is located from an object image based on a trained object region extraction model, and cuts out the detected object region. Then, the object identification device 100 transmits the detected object region to the object management server 200.

[0085] In step S102, the object management server 200 identifies the object type information located in the cut-out object region based on the object type inference model.

[0086] As described above, the object identification device 100 according to an embodiment of the present invention detects an object region where an object is located from an object image based on an object region extraction model, and the object management server 200 hierarchically classifies objects using a hierarchical model including an object type inference model. The advantages of such hierarchical object classification are as follows: The advantages of configuring a system using an artificial intelligence (AI) model that can find a higher-level category without immediately grasping the detailed class of an object in one stage, and a model that extracts the object found by the higher-level model and classifies the object class, are as follows:

[0087] For example, in the case of food, food images are collected to train a model. In this case, food images such as rice, cabbage kimchi, and kkakdugi appear relatively frequently, while food images such as bellflower root salad, bracken namul, and stir-fried dried sardines appear less frequently. Therefore, when food images include rice, cabbage kimchi, kkakdugi, bellflower root salad, bracken namul, and stir-fried dried sardines, the training data to create a one-stage model must label all of the food images.

[0088] However, as in the embodiment of the present invention, it is possible to create an AI system that combines two models: model (A) that searches for food and drink, and model (B) that extracts the food and drink and classifies it. In the embodiment of the present invention, once all food and drink has been extracted from the food and drink image, model (A) does not need to further learn from this food and drink image. Therefore, in the embodiment of the present invention, model (B) for food classification can label and extract only the data it needs (e.g., bracken namul, fried dried sardines, and salad with bellflower root) and proceed with learning.

[0089] An additional advantage is that in the case of a one-stage model, there may be cases where the food or drink has never been seen before and the model is completely unable to recognize it.

[0090] However, the object identification device 100 according to an embodiment of the present invention operates through a model that has learned various types of food and drink as a single class called "food and drink," and therefore has the advantage of being able to predict that a new food and drink is being eaten by using a model that has learned the overall characteristics of food and drink.

[0091] Furthermore, embodiments of the present invention can be applied not only to the food and beverage domain but also to the autonomous driving domain. By categorizing the transportation objects that a vehicle should recognize (e.g., automobiles, bicycles, tillers, etc.) and then classifying the detailed categories, if there is a model that captures automobiles well enough, the cost of labeling automobiles can be reduced and more resources can be used to label tillers. This allows for efficient labeling and rapid model development. Furthermore, embodiments of the present invention can increase the likelihood that a vehicle will recognize a tractor as a "transportation device" even when it is first seen.

[0092] FIG. 3 illustrates an object domain reasoning model used in one embodiment of the present invention.

[0093] As shown in FIG. 3, the object identification apparatus 100 can detect an object region where an object is located from an object image based on an object region inference model 300 .

[0094] The object region inference model 300 executed in the object identification apparatus 100 performs a first encoding operation by inputting an input object image into a first encoder 310. The object region inference model 300 extracts feature points from the object image through the first encoding operation. Here, the object region inference model 300 may have a structure in which a plurality of first encoders 310 and a plurality of feature point extraction blocks 320 are connected in order.

[0095] The object region inference model 300 then extracts a feature point region using the extracted feature points (330), and infers an object region from the object image using the extracted feature point region (340).

[0096] Then, the object region inference model 300 segments the inferred object region and outputs the segmented object region.

[0097] FIG. 4 illustrates an object type inference model used in one embodiment of the present invention.

[0098] As shown in FIG. 4, the object identification device 100 can identify object type information using an object type inference model 400 corresponding to the classified category.

[0099] The object type inference model 400 executed in the object identification device 100 performs a second encoding operation by inputting the input cut-out object area image into a second encoder 410. The object type inference model 400 extracts feature points from the cut-out object area image through the second encoding operation (420). Unlike the object area inference model 300, the object type inference model 400 may include one second encoder 410 and one feature point decoder 430.

[0100] The object type inference model 400 then inputs the feature points for the extracted segmented object region to the feature point decoder 430 to perform a feature point decoding operation. The object type inference model 400 then infers the type of the object located in the segmented object region using the feature point decoding result (440).

[0101] The object type inference model 400 then outputs the inferred object type.

[0102] Meanwhile, when analyzing an object using general artificial intelligence (AI) techniques, the object location determination (e.g., box or pixel) and object type determination occur simultaneously in one model.

[0103] The object identification device 100 according to an embodiment of the present invention can detect an object region where an object is located from an object image based on the object region inference model 300, and can identify object type information based on the object type inference model 400. That is, the object identification device 100 includes the object region inference model 300 that searches for an object from an object image, and the object type inference model 400 that analyzes the type of the object searched for in the object image.

[0104] Here, the object identification device 100 may analyze the object type by classifying the object type category located in the extracted object region, or may analyze the object type without classifying the object type category located in the extracted object region.

[0105] In this way, when the object region inference model 300 and the object type inference model 400 are separated, the following advantages can be obtained.

[0106] First, since the object region inference model 300 and the object type inference model 400 are separated, the training data for each model can be separated. When the object model is used as a single model, object images are collected to create training data. When the object image contains both new objects and previously known objects, the known objects must be labeled. If the objects are not labeled, the object model may forget about them. Conversely, when the object models are separated as in one embodiment of the present invention, only new objects can be labeled. Since only the area where the object region is located is extracted from the object image for training, the object identification device 100 does not need to label known objects.

[0107] Second, because the object region inference model 300 and the object type inference model 400 are separated, recognition performance for new objects that may appear in object images can be improved. If the object model is used as a single model, it will only search for familiar objects, making it highly likely that new objects will not be recognized. Conversely, when the object model is separated as in one embodiment of the present invention, it is an object model that has learned the characteristics of objects from object images by observing various objects. Therefore, even when an object that has never been seen before appears in an object image, the object region inference model 300 can recognize and sense that it is an object, even if it cannot accurately recognize the object type.

[0108] FIG. 5 is a flowchart illustrating a method for identifying objects using a hierarchical model according to an embodiment of the present invention.

[0109] As shown in FIG. 5, in step S201, an object identification device 100 using a hierarchical model according to an embodiment of the present invention detects an object region in which at least one object is located from an object image based on an object region extraction model.

[0110] In step S202, the object identification device 100 extracts a detected object region and classifies the category of the object located in the extracted object region.

[0111] In step S203, the object identification device 100 determines the object by using the object type inference model corresponding to the classified category.

[0112] 6 to 9 are diagrams illustrating an example of an object identification operation using a hierarchical model according to an embodiment of the present invention.

[0113] The object identification device 100 can determine object information through the operations shown in FIGS. 6 to 9 in order to distinguish which objects are present in the object image.

[0114] 6, the object identification device 100 may receive an object image 510. The object image 510 may be an original image including an object 511. The object image 510 may be an object image captured by a camera provided in the object identification device 100, or an object image currently captured by a camera and displayed on a screen. The object image 510 may include at least one object 511. The object image 510 may include an object 511 placed in a dish, tray, or the like.

[0115] 7, the object identification device 100 can determine in which region an object 511 is located from an object image 510 based on the learned object region extraction model 300. The object identification device 100 can detect an object region 512 where an object is located from the object image 510 and a non-object region where something other than the object is located.

[0116] As an example, the object identification device 100 can search for a box enclosing an object 511 from an object image 510 based on the learned object region extraction model 300. The object identification device 100 can detect an object region 512 using the box enclosing the object. The object identification device 100 can detect the remaining region other than the box enclosing the object 511 as a region other than the object. Alternatively, the object identification device 100 can detect edges enclosing the object and detect a region connecting the detected edges as the object region.

[0117] As another example, the object identification device 100 can search for pixels occupied by an object 511 from an object image 510 based on the learned object region extraction model 300. The object identification device 100 can detect an object region 512 using the pixels occupied by the object. The object identification device 100 can detect the remaining region other than the pixels occupied by the object as a region other than the object.

[0118] The operation of detecting an object region for a box or pixel in the object image 510 will be described as follows. The object identification device 100 first receives the object image. Then, the object identification device 100 can extract a feature map for the object image through a convolution neural network. Then, the object identification device 100 can extract an object region having object features from the extracted feature map.

[0119] 8, the object identification apparatus 100 can cut out the object region detected in FIG. 6 and classify the category of the object 511 located in the cut-out object region 513. Here, the object identification apparatus 100 can cut out the object region 512 by cropping a box including the object region 512 detected in FIG. 6.

[0120] In this case, the object identification device 100 may classify the object 511 included in the cropped object region 513 into object categories in a stepwise downward order from the highest category. For example, the object identification device 100 may classify the object into a specific major category among a plurality of major categories corresponding to the highest category. For example, if an object included in the cropped object region is Korean food, the object identification device 100 may classify the object into a category corresponding to a comprehensive range of Korean food. The object identification device 100 may search for the category of the object within a comprehensive range such as rice, soup, rice bowl, noodles, kimchi, desserts, etc. In FIG. 8, the object identification device 100 may identify objects included in each cropped object region as types of rice and proceed with classification into the rice category. If the top category including the object is Korean food, Japanese food, Chinese food, or Western food, the object category may be searched for within the top category.

[0121] The object category is not limited to a specific category, and the object identification device 100 may classify the object in descending order from the highest category that can include the object with the highest priority. Here, when the highest category of the object is a major category, the next highest category is a medium category, and the lowest category is a minor category, the object identification device 100 may primarily classify the major category for the object region, secondarily classify the medium category for the object region among the classified major categories, and tertiarily classify the minor category for the object region among the classified medium categories.

[0122] 9, the object identification device 100 may first classify the category of the object 511 for all objects included in the object region 512 of the object image 510, and then finally determine the object information 514 using the object type inference model 400 corresponding to the classified category. If the object identification device 100 classifies the category of the object region 512 into rice types, it may determine the object information 514 as millet rice using the rice object inference model corresponding to the rice type. For example, the object identification device 100 may determine a specific rice type using a rice object inference model corresponding to the rice type, i.e., an object model that can distinguish between rice, black rice, and black bean rice, which belong to subcategories within the rice type. In FIG. 8, the object identification device 100 may finally determine the object included in the object region as millet rice using the rice object inference model corresponding to the rice type.

[0123] Thereafter, the object identification device 100 performs an object information determination operation using the same object identification method for each object included in the object region detected from the object image 510, thereby accurately determining which objects are present in the object image. For example, the object identification device 100 may finally determine object information for each object included in the object tray as millet rice, spinach miso soup, beef and quail's egg stewed in soy sauce, kimchi pancake, kkakdugi, mandarin orange, etc. In this case, the object identification device 100 determines object categories down to the lowest category, and may determine object information corresponding to the lowest object category using the types of object type inference models 400, for example, a rice object inference model, a soup object inference model, a side dish object inference model, etc., within the determined object category.

[0124] FIG. 10 is a diagram illustrating an operation of an object identification method using a hierarchical model in cooperation with an object management server according to an embodiment of the present invention.

[0125] 7 shows an operation of identifying an object through the cooperation of the object identification device 100 and the object management server 200 when an object included in an object region cut out from an object image is classified into major, middle, and minor categories from the highest to the lowest category. Here, the object categories are not limited to a specific number of hierarchical levels. For example, the object categories may be classified into only two levels, i.e., major and middle categories, or three levels, i.e., major, middle, and minor categories. The object categories may be classified into categories including a natural number n levels, and are not limited to a specific level.

[0126] In this case, the object identification device 100 may preset a category hierarchy to be classified in advance during the overall category classification operation. Alternatively, the object identification device 100 may adjust the category to be classified in consideration of real-time characteristics at the time of classification, the device resource status, the calculation load, etc. Correspondingly, the object management server 200 has higher classification performance and more available resources than the object identification device 100, and therefore may perform an operation that is less efficient in terms of time or accuracy in the object identification device 100 and transmit the result to the object identification device 100.

[0127] For example, the operation of cutting out an object area from an object image and classifying the cut-out object area into large, medium, and small classifications takes little time or load, whereas the operation of determining specific object information requires a large amount of calculation and results in low accuracy. In this case, the object management server 200 can perform such an operation on behalf of the object identification device 100. In this way, the object management server 200 can perform an operation that is difficult for the object identification device 100 to perform by itself, thereby improving the identification time and accuracy.

[0128] As shown in FIG. 10, in step S301, the object identification apparatus 100 detects an object region in which at least one object is located from an object image based on an object region extraction model.

[0129] In step S302, the object identification device 100 extracts a detected object region and primarily classifies the categories of objects located in the extracted object region into specific major categories.

[0130] In step S303, the object identification device 100 secondarily classifies the object category of the object region image into a specific intermediate category using the specific major category information.

[0131] In step S304, the object identification device 100 classifies the object category of the object region image into specific minor categories three-dimensionally using the specific middle classification information.

[0132] In step S305, the object identification device 100 transmits the cut-out object region image and the classified major classification, medium classification, and minor classification information.

[0133] In step S306, the object management server 200 determines the object type information of the object area image using the object type inference model corresponding to the specific minor classification information.

[0134] In step S307, the object management server 200 transmits the determined object information to the object identification device 100.

[0135] In step S308, the object identification apparatus 100 checks the object type information for each object included in the object image using the received object type information.

[0136] 10 illustrates an example in which the categorization operation is performed by the object identification device 100 and the object information determination operation is performed by the object management server 200. Here, the entity that performs each step may not be fixed. For example, the entity that performs the stage-by-stage categorization operation and the object information identification operation may change depending on the number of objects included in the object image, the size or capacity of the object image, the time required for the object information included in the object image, the communication status between the object identification device 100 and the object management server 200, the status of the object model stored in the object identification device 100, etc.

[0137] Meanwhile, a non-transitory computer-readable storage medium for storing instructions that, when executed by a processor, cause the processor to perform a method may be provided, the non-transitory computer-readable storage medium including the steps of: an object identification device detecting an object region in which an object is located from an object image based on a learned object region extraction model, and segmenting the detected object region; and an object management server identifying object type information located in the segmented object region from the object identification device based on an object type inference model.

[0138] Meanwhile, according to one embodiment of the present invention, the various embodiments described above may be implemented by software including instructions stored in a machine-readable storage medium. The machine is a device that can access instructions stored in the storage medium and operate according to the accessed instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or using other components under the control of the processor. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not include a signal and is tangible, and does not distinguish between data being stored semi-permanently or temporarily on the storage medium.

[0139] Furthermore, according to an embodiment of the present invention, the methods according to the various embodiments described above may be provided as being included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or online via an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored in a storage medium such as a memory of a manufacturer's server, an application store server, or an intermediary server, or may be temporarily generated.

[0140] Furthermore, according to one embodiment of the present invention, the various embodiments described above may be implemented in a recording medium readable by a computer or similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented in a processor itself. In a software implementation, the procedures and functions described herein may be implemented in separate software modules. Each of the software modules may perform one or more of the functions and operations described herein.

[0141] Meanwhile, computer instructions for performing the processing operations of the devices according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When executed by a processor of a particular device, the computer instructions stored in such a non-transitory computer-readable medium cause the particular device to perform the processing operations of the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0142] Furthermore, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of one or more entities, and some of the subcomponents described above may be omitted, or other subcomponents may be further included in various embodiments. Generally or additionally, some components (e.g., modules or programs) may be integrated into a single entity to perform the same or similar functions as those performed by the respective components before integration. Operations performed by modules, programs, or other components according to various embodiments may be performed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be performed in a different order, omitted, or other operations may be added.

[0143] While the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the disclosure pertains without departing from the gist of the present invention as claimed in the claims. However, such modifications should not be understood individually from the technical ideas and perspectives of the present invention. [Explanation of symbols]

[0144] 10 Object Identification System 100 Object Identification Device 110 Communication Module 120 Camera 130 Display 140 memory 150 processors 200 Object Management Server 210 Communication Module 220 memory 230 processors

Claims

1. 1. An object identification method performed by an object identification system, comprising: an object identification device detecting an object region where the object is located from the object image based on the learned object region extraction model, and extracting the detected object region; an object management server identifying object type information located in the object area extracted from the object identification device based on an object type inference model; Including, The object identification device may further classify the object types located in the extracted object region into categories in stages, In the step of identifying the object type information, the object management server identifies the object type information by using an object type inference model corresponding to the classified category.

2. The step of extracting the detected object region includes:

2. The object identification method using a hierarchical model according to claim 1, wherein an object region is detected from the object image using a box surrounding the object or pixels occupied by the object based on the learned object region extraction model.

3. The step of extracting the detected object region includes:

2. The object identification method according to claim 1, further comprising: extracting first feature points through a first image encoder of the learned object region extraction model; extracting a feature point region from the extracted first feature points; and inferring an object region.

4. The step of classifying the object types into categories in stages comprises:

2. The object identification method using a hierarchical model according to claim 1, wherein the object identification device classifies the categories of the objects located in the extracted object region in stages from the highest major classification to the lowest minor classification.

5. The step of identifying object type information includes:

5. The object identification method using a hierarchical model according to claim 4, wherein the object management server identifies object type information using an object type inference model corresponding to the lowest level classification that is the classified category.

6. The step of identifying object type information includes:

5. The object identification method using a hierarchical model according to claim 4, wherein the object management server determines the object type information as one of a plurality of classes included in the lowest classification.

7. The step of identifying object type information includes:

2. The object identification method using a hierarchical model according to claim 1, wherein the object management server extracts second feature points through a second image encoder of an object type inference model corresponding to the classified category, and inputs the extracted second feature points to a feature point decoder to infer object type information.

8. In an object identification system, an object identification device that detects an object region where an object is located from an object image based on the learned object region extraction model and cuts out the detected object region; an object management server that identifies object type information located in the object area extracted from the object identification device based on an object type inference model; Equipped with the object identification device classifies the object type located in the extracted object region into categories in stages, The object management server identifies object type information by using an object type inference model corresponding to the classified categories, in an object identification system using a hierarchical model.

9. The object identification device 9. The object identification system using a hierarchical model according to claim 8, wherein an object region is detected from the object image using a box surrounding the object or pixels occupied by the object based on the learned object region extraction model.

10. The object identification device 9. The object identification system using a hierarchical model according to claim 8, further comprising: extracting first feature points through a first image encoder of the learned object region extraction model; extracting a feature point region from the extracted first feature points; and inferring an object region.

11. The object identification device 9. The object identification system using a hierarchical model according to claim 8, wherein the categories of the objects located in the extracted object regions are classified stepwise from the highest major classification to the lowest minor classification.

12. The object management server 12. The object identification system using a hierarchical model according to claim 11, wherein object type information is identified using an object type inference model corresponding to the lowest level classification that is the classified category.

13. The object management server The object identification system using a hierarchical model according to claim 11, wherein the object type information is determined to be one of a plurality of classes included in the lowest classification.

14. The object management server 9. The object identification system using a hierarchical model according to claim 8, wherein second feature points are extracted through a second image encoder of an object type inference model corresponding to the classified category, and the extracted second feature points are input to a feature point decoder to infer object type information.

Images