Method and apparatus for identifying objects using hierarchical models

The hierarchical model enhances object identification by using a device and server to detect regions and classify types, addressing low accuracy and inconvenience in existing technologies, thereby improving the efficiency and accuracy of analyzing eating habits.

JP2024545378AActive Publication Date: 2024-12-06NUVI LABS CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2023571484
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-11-09
Filing Date
2023-09-07
Publication Date
2024-12-06
Estimated Expiration
2043-09-07

AI Technical Summary

Technical Problem

Existing object identification technologies suffer from low accuracy and inconvenience in identifying ingested objects, leading to errors in calorie calculation and incomplete analysis of eating habits due to separate operations for object photography and information input, which decreases user engagement and accuracy.

Method used

A hierarchical model is used to identify object regions and types through an object identification device and management server, involving object region extraction and type inference models to classify objects step-by-step, allowing for efficient and accurate identification of objects in images.

Benefits of technology

The hierarchical model increases object identification speed and accuracy by separating object region detection and type classification, improving the efficiency and completeness of analyzing eating habits.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024545378000001_ABST
    Figure 2024545378000001_ABST
Patent Text Reader

Abstract

The present invention relates to an object identification method and device using a hierarchical model, and an object identification method using a hierarchical model according to one embodiment of the present invention includes a step of an object identification device detecting an object area in which an object is located from an object image based on a learned object area extraction model and extracting the detected object area, and a step of an object management server identifying object type information located in the object area extracted from the object identification device based on an object type inference model.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for identifying an object. [Background technology]

[0002] In recent years, interest in health has increased, but at the same time, the number of people suffering from overweight and obesity is also gradually increasing. Such overweight and obesity are serious problems that can lead to various diseases such as diabetes and high blood pressure.

[0003] Therefore, in order to solve such overweight and obesity, it is necessary to analyze one's eating habits. Generally, a person knows what he likes and dislikes, but does not remember the type and frequency of the objects he actually ingests. Therefore, in order to analyze one's eating habits, it is necessary to grasp the objects that are actually ingested, and analyze the individual's eating habits according to the information about the grasped objects.

[0004] However, most of the currently available technologies are limited to simple image search of object images taken through a camera, which results in a significant drop in accuracy. Also, if the accuracy of object type identification in image search is low, there is a problem that larger errors will occur in the next step, such as calorie calculation.

[0005] For example, in order to know an object to be actually ingested, a user must input object information using a mobile terminal or an input device. Here, the user executes an object-related application or a general photography mode executed on the mobile terminal to photograph an object. At this time, the user is inconvenient in that the user must search for the photographed object image in the object-related application and input the object information for the object again. Alternatively, when photographing in the general photography mode, the user is inconvenient in that the user must search for the photographed object image from an album, upload the searched object image to the object-related application, and search the uploaded object image again to input the object information. Therefore, since it is inconvenient that the object photographing and the object identification are performed as separate operations, the frequency with which the user uploads object information decreases rapidly after photographing several object images.

[0006] For example, a user may take pictures of an object only to input object information later. If a long time has passed since the object image was taken, the user may not remember the object information that he or she intended to input when taking the picture, and may not be able to record the object information, or may enter incorrect object information, causing an error in the eating habit analysis. Even if the user wishes to input various object information for the taken object image, he or she must input only a fixed number of pieces of object information set for each application. In such a case, the number of object information is limited, and the user's eating habit analysis may not be performed accurately. As a result of this increased inconvenience to the user, the satisfaction level of the service that analyzes an individual's eating habit, for example, through object identification, is decreasing. Summary of the Invention [Problem to be solved by the invention]

[0007] The embodiments of the present invention aim to provide an object identification method and apparatus using a hierarchical model to increase the object identification speed by identifying object areas in an object identification device and identifying object type information in an object management server.

[0008] The present invention provides an object identification method and apparatus using a hierarchical model to quickly determine an object located in an object image by classifying object categories in a hierarchical manner using an object region extraction model and an object type inference model and identifying object type information.

[0009] However, the problem to be solved by the present invention is not limited thereto, and may be expanded in various ways in environments without departing from the spirit and scope of the present invention. [Means for solving the problem]

[0010] According to one embodiment of the present invention, in an object identification method performed by an object identification system, an object identification method using a hierarchical model may be provided, the method including: an object identification device detecting an object area in which an object is located from an object image based on a learned object area extraction model and cutting out the detected object area; and an object management server identifying object type information located in the object area cut out from the object identification device based on an object type inference model.

[0011] The step of extracting the detected object region may detect the object region from the object image using a box surrounding the object or pixels occupied by the object based on the learned object region extraction model.

[0012] The step of extracting the detected object region can include extracting first feature points through a first image encoder of the learned object region extraction model, and extracting a feature point region from the extracted first feature points to infer the object region.

[0013] The method further includes a step of the object identification device classifying categories of object types located in the extracted object area in a hierarchical manner, and the step of identifying the object type information may include the object management server identifying the object type information using an object type inference model corresponding to the classified categories.

[0014] The step of classifying the object type categories stepwise may include classifying the categories of the objects located in the extracted object region by the object identification device stepwise from a top major category to a bottom minor category.

[0015] In the step of identifying the object type information, the object management server may identify the object type information by using an object type inference model corresponding to a lowest level classification, which is the classified category.

[0016] The step of identifying the object type information may include the object management server determining the object type information as any one of a plurality of classes included in the lowest classification.

[0017] The step of identifying the object type information may involve the object management server extracting second feature points through a second image encoder of an object type inference model corresponding to the classified category, and inputting the extracted second feature points to a feature point decoder to infer the object type information.

[0018] Meanwhile, according to another embodiment of the present invention, in an object identification method performed by an object identification device, an object identification method using a hierarchical model may be provided, the object identification method including the steps of: detecting an object area in which an object is located from an object image based on a learned object area extraction model, and cutting out the detected object area; and identifying object type information located in the cut-out object area based on an object type inference model.

[0019] Meanwhile, according to another embodiment of the present invention, there may be provided an object identification method using a hierarchical model, which includes the steps of: detecting an object area in which an object is located from an object image based on a learned object area extraction model and cutting out the detected object area; and identifying object type information located in the cut-out object area based on an object type inference model, in an object identification method performed by an object management server.

[0020] Meanwhile, according to another embodiment of the present invention, an object identification system using a hierarchical model may be provided, which includes an object identification device that detects an object area in which an object is located from an object image based on a learned object area extraction model and cuts out the detected object area, and an object management server that identifies object type information located in the object area cut out from the object identification device based on an object type inference model.

[0021] The object identification device can detect an object region from the object image using a box surrounding the object or pixels occupied by the object based on the learned object region extraction model.

[0022] The object identification device can extract first feature points through a first image encoder of the learned object area extraction model, and extract a feature point area from the extracted first feature points to infer an object area.

[0023] The object identification device classifies object type categories located in the extracted object area in a stepwise manner, and the object management server can identify object type information using an object type inference model corresponding to the classified categories.

[0024] The object identification device may classify the categories of the objects located in the extracted object regions stepwise from a top major classification to a bottom minor classification.

[0025] The object management server can identify object type information by using an object type inference model corresponding to the lowest classification, which is the classified category.

[0026] The object management server can determine that the object type information is one of a plurality of classes included in the lowest classification.

[0027] The object management server can extract second feature points through a second image encoder of an object type inference model corresponding to the classified category, and input the extracted second feature points to a feature point decoder to infer object type information.

[0028] Meanwhile, according to another embodiment of the present invention, there may be provided an object identification apparatus using a hierarchical model, comprising: a memory for storing one or more programs; and a processor for executing the one or more stored programs, wherein the processor detects an object area in which an object is located from an object image based on a learned object area extraction model, cuts out the detected object area, and identifies object type information located in the cut-out object area using an object type inference model.

[0029] Meanwhile, according to another embodiment of the present invention, an object management server using a hierarchical model may be provided, which includes a memory for storing one or more programs, and a processor for executing the one or more stored programs, in which the processor detects an object area in which an object is located from an object image based on a learned object area extraction model, cuts out the detected object area, and identifies object type information located in the cut-out object area using an object type inference model. Effect of the Invention

[0030] The disclosed technology may have the following effects. However, this does not mean that a particular embodiment should include all or only the following effects, and therefore the scope of the disclosed technology should not be understood to be limited thereby.

[0031] The embodiments of the present invention can increase the object identification speed by identifying object regions in an object identification device and identifying object type information in an object management server.

[0032] The embodiments of the present invention can quickly determine an object located in an object image by classifying object categories in a stepwise manner and identifying object type information using an object region extraction model and an object type inference model.

[0033] The embodiments of the present invention can increase the efficiency of object identification operations by adjusting the category to be classified in consideration of the real-time nature of classification, the resource status and computational load of the object identification device, and the like.

[0034] According to the embodiments of the present invention, the object identification device can quickly perform object category classification and object information determination operations, which are difficult for the object identification device to perform, in cooperation with the object management server, thereby improving the object identification time and accuracy. [Brief description of the drawings]

[0035] [Figure 1] FIG. 1 is a diagram showing a configuration of an object identification system using a hierarchical model according to an embodiment of the present invention.

[0036] [Diagram 2] 1 is a flowchart illustrating a method for identifying an object using a hierarchical model according to an embodiment of the present invention.

[0037] [Diagram 3] FIG. 2 illustrates an object domain reasoning model used in one embodiment of the present invention.

[0038] [Figure 4] FIG. 2 illustrates an object type inference model used in one embodiment of the present invention.

[0039] [Diagram 5] 1 is a flowchart illustrating a method for identifying an object using a hierarchical model according to an embodiment of the present invention.

[0040] [Figure 6] FIG. 2 illustrates an example of an object identification operation using a hierarchical model according to an embodiment of the present invention.

[0041] [Figure 7] FIG. 2 illustrates an example of an object identification operation using a hierarchical model according to an embodiment of the present invention.

[0042] [Figure 8] FIG. 2 illustrates an example of an object identification operation using a hierarchical model according to an embodiment of the present invention.

[0043] [Figure 9] FIG. 2 illustrates an example of an object identification operation using a hierarchical model according to an embodiment of the present invention.

[0044] [Figure 10] 13 is a diagram illustrating an operation of an object identification method using a hierarchical model in conjunction with an object management server according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] The present invention can be modified in various ways and can have various embodiments, and a specific embodiment will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to the specific embodiment, and it can be understood that the present invention includes any modifications, equivalents, or alternatives included in the technical idea and technical scope of the present invention. In describing the present invention, if it is determined that a detailed description of related publicly known technology may obscure the gist of the present invention, the detailed description will be omitted.

[0046] Terms such as first, second, etc. may be used to describe various components, but the components are not limited by the terms, etc. The terms, etc. are used only for the purpose of distinguishing one component from another component.

[0047] The terms used in the present invention are merely used to describe certain embodiments and are not intended to limit the present invention. The terms used in the present invention are selected from currently widely used general terms, taking into consideration the functions of the present invention, but these may change depending on the intentions of engineers in the field, precedents, or the emergence of new technologies. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meanings will be described in detail in the description of the relevant invention. Therefore, the terms used in the present invention must be defined based on the meanings that the terms have and the overall content of the present invention, not simply the names of the terms.

[0048] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, the terms "comprise" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preclude the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0049] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components will be given the same drawing numbers and duplicate descriptions thereof will be omitted.

[0050] FIG. 1 is a diagram showing the configuration of an object identification system using a hierarchical model according to an embodiment of the present invention.

[0051] First, as shown in FIG. 1, an object identification system 10 according to the present invention can include an object identification device 100 and an object management server 200 that communicates with the object identification device 100.

[0052] The object identification device 100 can detect an object region where an object is located from an object image based on the learned object region extraction model, and can extract the detected object region. Then, the object identification device 100 can transmit the extracted object region to the object management server 200.

[0053] The object management server 200 can identify the object type information located in the object area extracted from the object identification device 100 based on the object type inference model.

[0054] In this manner, the operation of identifying the object type information may be performed in the object management server 200, not in the object identification device 100. Here, the object management server 200 may be realized as a cloud server. Alternatively, the operation of detecting the object area and the operation of identifying the object type information may be performed together in the object identification device 100.

[0055] This is to increase the speed of the object area detection and object type identification operations. For example, when a user takes a picture of an object (food and drink) with a mobile terminal including the object identification device 100, the original image file taken is 4 MB. If the 4 MB original image file is sent to the object management server 200 in the cloud, the capacity will be too large. Therefore, the object identification device 100 can extract only the object (food and drink) area from the original image and send only 200 KB, for example, to the object management server 200 in the cloud. In this way, the mobile terminal can only perform the operation of detecting the object area, and the cloud can perform the operation of identifying the object type information. Alternatively, the mobile terminal can perform both the operation of detecting the object area and the operation of identifying the object type information.

[0056] Here, the subject of the operation of detecting the object area and the operation of identifying the object type information can be determined based on at least one of the data capacity to be transmitted to the object management server 200, the hardware performance of the object identification device 100, the communication speed between the object management server 200 and the object identification device 100, the size of the object area extraction model or the object type inference model, and the identification accuracy of the object type information. For example, when the data capacity is smaller than a preset threshold, the hardware performance is high, the communication speed is slow, the model size is small, and the accuracy is high, it may be advantageous for the object identification device 100 to perform the operation of detecting the object area and the operation of identifying the object type information. Conversely, when the data capacity is larger than a preset threshold, the hardware performance is low, the communication speed is fast, the model size is large, and the accuracy is low, it may be advantageous for the object identification device 100 to perform the operation of detecting the object area and the object management server 200 to perform the operation of identifying the object type information. At this time, the object identification device 100 may convert the detected object area and transmit it to the object management server 200 instead of cutting out the detected object area. For example, the object identification device 100 can lower the resolution of the detected object area and convert it into a thumbnail to quickly send it to the object management server 200, or when sending a file containing the object area, it can convert the extension or file format and send it to the object management server 200.

[0057] 1, the object identification device 100 may include a communication module 110, a camera 120, a display 130, a memory 140, and a processor 150. The object management server 200 may include a communication module 210, a memory 220, and a processor 230.

[0058] However, not all of the components shown in the figure are required components, and the object identification system 10 may be realized with more components than those shown in the figure, or may be realized with fewer components.

[0059] The specific configuration and operation of each component of the object identification system 10 in FIG. 1 will now be described.

[0060] First, the specific configuration and operation of each component of the object identification device 100 will be described.

[0061] The communication module 110 may include one or more modules that enable communication between the object identification device 100 and a wireless communication device or between the object identification device 100 and the object management server 200. The communication module 110 may also include one or more modules that couple the object identification device 100 to one or more networks.

[0062] The camera 120 can capture images or videos according to a user's operation. The camera 120 can be composed of a single camera, multiple cameras, a single image sensor, or multiple image sensors. The camera 120 can be composed of at least one of at least one 2D camera, at least one 3D camera, at least one stereo camera, and at least one image sensor. Here, the camera 120 can capture images or videos of objects before and after a meal according to a user's operation.

[0063] The display 130 may be layered with a touch sensor or may be integrally formed with the display 130 to provide a touch screen, which may provide an input interface between the object identification device 100 and a user, and an output interface between the object identification device 100 and a user.

[0064] The memory 140 may store data supporting various functions of the object identification device 100. The memory 140 may store one or more programs, a plurality of application programs, or applications, data and commands for the operation of the object identification device 100, which are run by the object identification device 100. At least some of the application programs may be downloaded from the object management server 200 via wireless communication. Also, at least some of the application programs may exist for the basic functions of the object identification device 100. Meanwhile, the application programs may be stored in the memory 140, provided on the object identification device 100, and driven by the processor 150 to perform the operations (or functions) of the object identification device 100.

[0065] In addition to operations related to the application programs, the processor 150 may generally control the overall operation of the object identification device 100. The processor 150 may process signals, data, information, etc. input or output via the components described above, or may run application programs stored in the memory 140 to provide or process appropriate information or functions to a user.

[0066] In one embodiment, the processor 150 executes one or more programs stored in the memory 140 to detect an object region in which an object is located from an object image based on the learned object region extraction model, cut out the detected object region, and identify object type information located in the cut-out object region based on the object type inference model.

[0067] According to an embodiment, the processor 150 can detect an object region from an object image using a box surrounding the object or pixels occupied by the object based on a learned object region extraction model.

[0068] According to an embodiment, the processor 150 can extract first feature points via a first image encoder of a trained object area extraction model, and extract feature point areas from the extracted first feature points to infer the object area.

[0069] According to an embodiment, the processor 150 detects an object region in which an object is located from an object image based on the learned object region extraction model, cuts out the detected object region, gradually classifies categories of object types located in the cut-out object region, and identifies object type information using an object type inference model corresponding to the classified category.

[0070] According to the embodiment, the processor 150 can classify the category of the object located in the cut-out object region stepwise from the highest major classification to the lowest minor classification.

[0071] According to some embodiments, the processor 150 can identify object type information using an object type inference model that corresponds to the lowest level subgroup, which is the classified category.

[0072] According to the embodiment, the processor 150 can determine that the object type information is one of a plurality of classes included in the lowest classification.

[0073] According to an embodiment, the processor 150 can extract second feature points through a second image encoder of an object type inference model corresponding to the classified category, and input the extracted second feature points to a feature point decoder to infer object type information.

[0074] According to an embodiment, the object identification device 100 further includes a communication module 210 that communicates with the object management server 200, and the processor 150 transmits the cut-out object area and the classified object type category to the object management server 200 via the communication module 210 to request object type identification, and can receive object type information determined using the classified category and the corresponding object type inference model from the object management server 200.

[0075] 1 to operate the application program stored in the memory 140. Furthermore, the processor 150 can operate at least two or more of the components included in the object identification device 100 in combination with each other to operate the application program. Here, the processor 150 will be described in detail later.

[0076] Next, the specific configuration and operation of each component of the object management server 200 will be described.

[0077] The communications module 210 may include one or more modules that enable communications between the object management server 200 and a wireless communication device or between the object management server 200 and the object identification device 100. The communications module 210 may also include one or more modules that couple the object management server 200 to one or more networks.

[0078] The memory 220 may store data supporting various functions of the object management server 200. The memory 220 may store one or more programs, a plurality of application programs or applications, and data and commands for the operation of the object management server 200, which are run by the object management server 200. At least some of the application programs may be downloaded to the object management server 200 via an external communication network by wireless communication. Also, at least some of the application programs may exist for the basic functions of the object management server 200. Meanwhile, the application programs may be stored in the memory 220 and installed on the object management server 200, and driven by the processor 230 to perform the operations (or functions) of the object management server 200.

[0079] In addition to operations related to the application programs, the processor 230 may generally control the overall operation of the object management server 200. The processor 230 may process signals, data, information, etc. input or output via the components described above, or may run application programs stored in the memory 220 to provide or process appropriate information or functions to a user.

[0080] In one embodiment, the processor 230 may execute one or more programs associated with an operation of determining object information in an object image in conjunction with the object identification device 100 stored in the memory 220. As an example, the processor 230 may stepwise classify the remaining categories not classified by the object identification device 100 among the categories of objects located in the cut-out object region, and determine object type information using an object type inference model corresponding to the classified category.

[0081] The configuration of the object identification system 10 according to the present invention has been described in detail above with reference to Fig. 1. The object identification system 10 may include fewer or more components than those shown in Fig. 1.

[0082] Hereinafter, object identification methods according to various embodiments of the present invention will be described with reference to FIGS.

[0083] FIG. 2 is a flowchart illustrating a method for identifying objects using a hierarchical model according to an embodiment of the present invention.

[0084] 2, in step S101, the object identification device 100 using a hierarchical model according to an embodiment of the present invention detects an object region where an object is located from an object image based on a trained object region extraction model, and extracts the detected object region. Then, the object identification device 100 transmits the detected object region to the object management server 200.

[0085] In step S102, the object management server 200 identifies object type information located in the cut-out object region based on the object type inference model.

[0086] In this manner, the object identification device 100 according to an embodiment of the present invention detects an object region where an object is located from an object image based on an object region extraction model, and the object management server 200 classifies objects hierarchically using a hierarchical model including an object type inference model. The advantages of such hierarchical object classification will be described. That is, the advantages of configuring a system using an artificial intelligence (AI) model that can search for a higher category without immediately grasping the detailed class of an object in one stage and a model that extracts an object searched by the higher model and classifies the object class are as follows.

[0087] For example, in the case of eating and drinking, food images are collected to train a model. In this case, when it comes to eating and drinking, dishes such as rice, cabbage kimchi, and kkakdugi appear relatively frequently, while dishes such as bellflower root salad, bracken namul, and stir-fried anchovies appear less frequently. Therefore, when there are food images including rice, cabbage kimchi, kkakdugi, bellflower root salad, bracken namul, and stir-fried anchovies, the training data to create a one-stage model must label all the dishes.

[0088] However, as in the embodiment of the present invention, an AI system can be created that combines model (A) that searches for food and drink and model (B) that extracts the searched food and drink and classifies it as such. In the embodiment of the present invention, once all food and drink has been searched for from the food and drink image, model (A) does not need to learn further about this food and drink image. Therefore, in the embodiment of the present invention, model (B) for food classification may label and extract only necessary data (e.g. bracken namul, fried dried sardine, bellflower root salad) and proceed with learning.

[0089] An additional advantage is that in the case of a one-stage model, there may be cases where the food or drink has never been seen before and the model is completely unable to recognize it.

[0090] However, the object identification device 100 according to an embodiment of the present invention operates through a model that has learned various types of eating and drinking as a single class called "eating and drinking", and therefore has the advantage of being able to predict that an eating or drinking event is occurring even when a new eating or drinking event is observed, by using a model that has learned the overall characteristics of eating and drinking.

[0091] Furthermore, the embodiment of the present invention can be applied not only to the food and beverage domain but also to the autonomous driving domain. If the objects of transportation that a car should recognize are grouped into categories (e.g., cars, bicycles, tillers, etc.) and then classified into detailed categories, when there is a model that captures cars well enough, the cost of labeling cars can be reduced and more resources can be used to label tillers. This allows for efficient labeling and rapid model development. Also, the embodiment of the present invention can increase the possibility of recognizing a tractor as a "transportation means" even when seeing it for the first time.

[0092] FIG. 3 illustrates an object domain reasoning model used in one embodiment of the present invention.

[0093] As shown in FIG. 3, the object identification apparatus 100 can detect an object region in which an object is located from an object image based on an object region inference model 300 .

[0094] The object region inference model 300 executed in the object identification device 100 performs a first encoding operation by inputting an input object image into a first encoder 310. The object region inference model 300 extracts feature points from the object image through the first encoding operation. Here, the object region inference model 300 may have a structure in which a plurality of first encoders 310 and a plurality of feature point extraction blocks 320 are connected in sequence.

[0095] The object region inference model 300 then extracts a feature point region using the extracted feature points (330), and infers an object region from the object image using the extracted feature point region (340).

[0096] Then, the object region inference model 300 segments the inferred object region and outputs the segmented object region.

[0097] FIG. 4 illustrates an object type inference model used in one embodiment of the present invention.

[0098] As shown in FIG. 4, the object identification device 100 can identify object type information by using an object type inference model 400 corresponding to a classified category.

[0099] The object type inference model 400 executed in the object identification device 100 performs a second encoding operation by inputting the input cut-out object area image into a second encoder 410. The object type inference model 400 extracts feature points from the cut-out object area image through the second encoding operation (420). Here, unlike the object area inference model 300, the object type inference model 400 may include one second encoder 410 and a feature point decoder 430.

[0100] Then, the object type inference model 400 performs a feature point decoding operation by inputting the feature points for the extracted cut-out object region to the feature point decoder 430. Next, the object type inference model 400 infers the type of the object located in the cut-out object region using the feature point decoding result (440).

[0101] The object type inference model 400 then outputs the inferred object type.

[0102] Meanwhile, when analyzing objects using general artificial intelligence (AI) techniques, the operations of determining the object location (e.g., box or pixel) and determining the object type occur simultaneously in one model.

[0103] The object identification device 100 according to an embodiment of the present invention can detect an object region where an object is located from an object image based on the object region inference model 300, and can identify object type information based on the object type inference model 400. That is, the object identification device 100 includes the object region inference model 300 that searches for an object from an object image, and the object type inference model 400 that analyzes the type of the object searched for from the object image.

[0104] Here, the object identification device 100 may analyze the object type by classifying the categories of the object types located in the extracted object region, or may analyze the object type without classifying the categories of the object types located in the extracted object region.

[0105] In this way, when the object domain inference model 300 and the object type inference model 400 are separated, the following advantages can be obtained.

[0106] First, since the object region inference model 300 and the object type inference model 400 are separated, the learning data for each model can be separated. When the object model is used as one model, object images are collected to create learning data. When there are new objects and previously known objects in the object image, the known objects must be labeled. If the objects are not labeled, the object model may forget about the objects. Conversely, when the object models are separated as in one embodiment of the present invention, only new objects can be labeled. Since only the area where the object region is located is cut out from the object image and learned, the object identification device 100 does not need to label known objects.

[0107] Secondly, since the object region inference model 300 and the object type inference model 400 are separated, the recognition performance for new objects that may appear in an object image can be improved. When the object model is used as a single model, it is highly likely that new objects cannot be recognized because only objects known to the model are searched for. On the other hand, when the object model is separated as in one embodiment of the present invention, it is an object model that has learned the feature points of objects in object images by looking at various objects. Therefore, even when an object seen for the first time appears in an object image, the object region inference model 300 can recognize and sense that it is an object even if it cannot accurately recognize the object type.

[0108] FIG. 5 is a flowchart illustrating a method for identifying objects using a hierarchical model according to an embodiment of the present invention.

[0109] As shown in FIG. 5, in step S201, an object identification device 100 using a hierarchical model according to an embodiment of the present invention detects an object area in which at least one object is located from an object image based on an object area extraction model.

[0110] In step S202, the object identification apparatus 100 extracts a detected object region and classifies a category of an object located in the extracted object region.

[0111] In step S203, the object identification device 100 determines an object by using an object type inference model corresponding to the classified category.

[0112] 6 to 9 are diagrams illustrating an example of an object identification operation using a hierarchical model according to an embodiment of the present invention.

[0113] The object identification device 100 can determine object information through the operations shown in FIGS. 6 to 9 in order to distinguish which objects are present in the object image.

[0114] 6, the object identification device 100 may receive an object image 510. The object image 510 may be an original image including an object 511. The object image 510 may be an object image captured by a camera provided in the object identification device 100, or an object image currently captured by a camera and displayed on a screen. The object image 510 may include at least one object 511. The object image 510 may include an object 511 placed in a dish, a tray, or the like.

[0115] 7, the object identification device 100 can determine in which region an object 511 is located from an object image 510 based on the learned object region extraction model 300. The object identification device 100 can detect an object region 512 where an object is located from the object image 510 and a non-object region where something other than the object is located.

[0116] As an example, the object identification device 100 can search for a box surrounding an object 511 from an object image 510 based on the learned object region extraction model 300. The object identification device 100 can detect an object region 512 using the box surrounding the object. The object identification device 100 can detect the remaining region other than the box surrounding the object 511 as a region other than the object. Alternatively, the object identification device 100 can detect edges surrounding the object and detect a region connecting the detected edges as the object region.

[0117] As another example, the object identification device 100 can search for pixels occupied by an object 511 from an object image 510 based on the learned object region extraction model 300. The object identification device 100 can detect an object region 512 using the pixels occupied by the object. The object identification device 100 can detect the remaining region other than the pixels occupied by the object as a region other than the object.

[0118] The operation of detecting an object region for a box or pixel in an object image 510 will be described as follows. The object identification device 100 first receives an object image. Then, the object identification device 100 can extract a feature map for the object image through a convolution neural network in the object image. Then, the object identification device 100 can extract an object region having object features from the extracted feature map.

[0119] 8, the object identification apparatus 100 may extract the object region detected in FIG. 6 and classify the category of the object 511 located in the extracted object region 513. Here, the object identification apparatus 100 may extract the object region 512 by cropping a box including the object region 512 detected in FIG.

[0120] At this time, the object identification device 100 may classify the object categories of the object 511 included in the cropped object region 513 from the highest category downward in stages. As an example, the object identification device 100 may classify the object category in a specific major category among a plurality of major categories corresponding to the highest category. For example, if an object included in the cropped object region is Korean food, the object identification device 100 may classify the object into a category corresponding to a comprehensive range of Korean food. The object identification device 100 may search for the category of the object in a comprehensive range such as rice, soup, bowl, noodles, kimchi, desserts, etc. In FIG. 8, the object identification device 100 may classify the objects included in each cropped object region into the rice category by determining that the objects are types of rice. If the top category in which the object is included is Korean food, Japanese food, Chinese food, or Western food, the object category may be searched for in the top category.

[0121] The object category is not limited to a specific category, and the object identification device 100 may classify the objects in descending order from the highest category that can include the objects with the highest priority. Here, when the highest category of the object is a major category, the next highest category is a medium category, and the lowest category is a minor category, the object identification device 100 may primarily classify the major category for the object region, secondarily classify the medium category for the object region among the classified major categories, and tertiarily classify the minor category for the object region among the classified medium categories.

[0122] As shown in FIG. 9, the object identification device 100 may first classify the category of the object 511 for all objects included in the object region 512 in the object image 510, and may finally determine the object information 514 using the object type inference model 400 corresponding to the classified category. When the object identification device 100 classifies the category of the object region 512 into rice types, the object identification device 100 may determine the object information 514 as millet rice using the rice object inference model corresponding to the rice type. For example, the object identification device 100 may determine a specific rice by using a rice object inference model corresponding to the rice type, that is, an object model that can distinguish whether the rice type is boiled rice, black rice, or black bean rice, which are subclassifications of the rice type. In FIG. 8, the object identification device 100 may finally determine the object included in the object region as millet rice by using the rice object inference model corresponding to the rice type.

[0123] Thereafter, the object identification device 100 performs an object information determination operation for each object included in the object region detected from the object image 510 using the same object identification method, and can accurately determine which object is present in the object image. For example, the object identification device 100 can finally determine object information for each object included in the object tray as millet rice, spinach miso soup, beef and quail eggs stewed in soy sauce, kimchi pancake, kakdugi, mandarin oranges, etc. At this time, the object identification device 100 can determine the object category up to the lowest category, and determine object information corresponding to the lowest object category using the types of object type inference models 400, for example, a rice object inference model, a soup object inference model, a side dish object inference model, etc., in the determined object category.

[0124] FIG. 10 is a diagram illustrating an operation of an object identification method using a hierarchical model in conjunction with an object management server according to an embodiment of the present invention.

[0125] 7 shows an operation of identifying an object through the interlocking operation of the object identification device 100 and the object management server 200 when an object included in an object region cut out from an object image is classified into major, middle and minor categories from the highest to the lowest category. Here, the object categories are not limited to a specific number of hierarchical levels. For example, the object categories may be classified into only two levels, i.e., major and middle categories, or may be classified into three levels, i.e., major, middle and minor categories. The object categories may be classified into categories including a natural number n of levels, and are not limited to a specific level.

[0126] In this case, the object identification device 100 may preset a category hierarchy in which the objects should be classified in the overall category classification operation. Alternatively, the object identification device 100 may adjust the categories to be classified in consideration of real-time characteristics at the time of classification, device resource status, calculation load, etc. Correspondingly, the object management server 200 has higher classification performance and more available resources than the object identification device 100, and therefore may perform an operation that is less efficient in terms of time or accuracy in the object identification device 100 and transmit the result to the object identification device 100.

[0127] For example, an operation of cutting out an object area from an object image and classifying the cut-out object area into major, middle and minor classifications takes little time or load, whereas an operation of determining specific object information requires a large amount of calculation and is less accurate, so that the object management server 200 can perform such an operation on behalf of the object identification device 100. In this way, the object management server 200 can perform an operation that is difficult for the object identification device 100 to perform by itself, thereby improving the identification time and accuracy.

[0128] As shown in FIG. 10, in step S301, the object identification apparatus 100 detects an object region in which at least one object is located from an object image based on an object region extraction model.

[0129] In step S302, the object identification device 100 extracts a detected object region, and primarily classifies the category of an object located in the extracted object region into a specific major classification.

[0130] In step S303, the object identification device 100 secondarily classifies the object category of the object region image into a specific intermediate category using the specific major category information.

[0131] In step S304, the object identification device 100 classifies the object category of the object region image into specific minor categories three-dimensionally using the specific intermediate category information.

[0132] In step S305, the object identification device 100 transmits the cut-out object region image and the classified major classification, medium classification, and minor classification information.

[0133] In step S306, the object management server 200 determines the object type information of the object area image by using the object type inference model corresponding to the specific subcategory information.

[0134] In step S 307 , the object management server 200 transmits the determined object information to the object identification device 100 .

[0135] In step S308, the object identification apparatus 100 checks the object type information for each object included in the object image using the received object type information.

[0136] 10, the category classification operation is performed by the object identification device 100, and the object information determination operation is performed by the object management server 200. Here, the entity performing each step may not be fixed. For example, the entity performing the stage-by-stage category classification operation and the object information identification operation may change depending on the number of objects included in the object image, the size or capacity of the object image, the time required for the object information included in the object image, the communication state between the object identification device 100 and the object management server 200, the state of the object model stored in the object identification device 100, etc.

[0137] Meanwhile, a non-transitory computer-readable storage medium for storing instructions which, when executed by a processor, cause the processor to perform a method may be provided, the non-transitory computer-readable storage medium including the steps of: an object identification device detecting an object region in which an object is located from an object image based on a learned object region extraction model, and cutting out the detected object region; and an object management server identifying object type information located in the object region cut out from the object identification device based on an object type inference model.

[0138] Meanwhile, according to an embodiment of the present invention, the various embodiments described above may be realized by software including instructions stored in a machine-readable storage medium. The machine is a device capable of calling up instructions stored in the storage medium and operating according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or using other components under the control of the processor. The instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not include a signal and is tangible, and does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0139] Also, according to an embodiment of the present invention, the methods according to the various embodiments described above may be provided in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read only memory (CD-ROM)) or online via an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a storage medium such as a memory of a manufacturer's server, an application store server, or an intermediary server.

[0140] Also, according to an embodiment of the present invention, the various embodiments described above can be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination of these. In some cases, the embodiments described herein can be implemented in the processor itself. According to a software implementation, the embodiments, such as the procedures and functions described herein, can be implemented in separate software modules. Each of the software modules can perform one or more functions and operations described herein.

[0141] Meanwhile, computer instructions for performing the processing operations of the device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When the computer instructions stored in such a non-transitory computer-readable medium are executed by a processor of a particular device, the particular device performs the processing operations of the device according to the various embodiments described above. The non-transitory computer-readable medium means a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc. Specific examples of the non-transitory computer-readable medium may include a CD, a DVD, a hard disk, a Blu-ray disk, a USB, a memory card, a ROM, etc.

[0142] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of one or more entities, and some of the subcomponents described above may be omitted, or other subcomponents may be further included in various embodiments. Generally or additionally, some components (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions as those performed by the respective components before integration. Operations performed by modules, programs, or other components according to various embodiments may be performed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be performed in a different order, omitted, or other operations may be added.

[0143] Although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the above-mentioned specific embodiments, and various modifications may be made by a person having ordinary skill in the art to which the disclosure pertains without departing from the gist of the present invention as claimed in the claims. However, such modifications should not be understood individually from the technical ideas and perspectives of the present invention. [Explanation of symbols]

[0144] 10 Object Identification System 100 Object Identification Device 110 Communication Module 120 Camera 130 Display 140 Memory 150 processors 200 Object Management Server 210 Communication Module 220 Memory 230 Processor

Claims

1. 1. An object identification method performed by an object identification system, comprising: The object identification device detects an object region in which the object is located from the object image based on the learned object region extraction model, and extracts the detected object region; an object management server identifying object type information located in the object area extracted from the object identification device based on an object type inference model; The object identification method includes:

2. The step of extracting the detected object region includes:

2. The method of claim 1, further comprising: detecting an object region from the object image using a box surrounding the object or pixels occupied by the object based on the learned object region extraction model.

3. The step of extracting the detected object region includes:

2. The method of claim 1, further comprising: extracting first feature points through a first image encoder of the learned object region extraction model; extracting a feature point region from the extracted first feature points; and inferring an object region.

4. The object identification device further includes a step of classifying the object type located in the extracted object region into categories in a stepwise manner, 2. The method for identifying objects using a hierarchical model according to claim 1, wherein the step of identifying the object type information comprises the step of identifying the object type information by the object management server using an object type inference model corresponding to the classified category.

5. The step of classifying the object types into categories in stages comprises:

5. The object identification method using a hierarchical model according to claim 4, wherein said object identification device classifies the categories of the objects located in said cut-out object region stepwise from a top major classification to a bottom minor classification.

6. The step of identifying object type information comprises:

6. The method for identifying objects using a hierarchical model according to claim 5, wherein the object management server identifies object type information by using an object type inference model corresponding to the lowest classification, which is the classified category.

7. The step of identifying object type information comprises:

6. The method for identifying objects using a hierarchical model according to claim 5, wherein the object management server determines the object type information as one of a plurality of classes included in the lowest classification.

8. The step of identifying object type information comprises:

5. The object identification method using a hierarchical model according to claim 4, wherein the object management server extracts second feature points through a second image encoder of an object type inference model corresponding to the classified category, and inputs the extracted second feature points to a feature point decoder to infer object type information.

9. 1. An object identification method performed by an object identification device, detecting an object region in which an object is located from the object image based on the learned object region extraction model, and extracting the detected object region; identifying object type information located in the segmented object region based on an object type inference model; The object identification method includes:

10. 1. An object identification method performed by an object management server, comprising: detecting an object region in which an object is located from the object image based on the learned object region extraction model, and extracting the detected object region; identifying object type information located in the segmented object region based on an object type inference model; The object identification method includes:

11. In an object identification system, an object identification device for detecting an object region in which an object is located from an object image based on the learned object region extraction model and cutting out the detected object region; an object management server for identifying object type information located in the object area extracted from the object identification device based on an object type inference model; The object identification system uses a hierarchical model.

12. The object identification device includes: The object identification system using a hierarchical model according to claim 11, wherein an object region is detected from the object image using a box surrounding the object or pixels occupied by the object based on the learned object region extraction model.

13. The object identification device includes:

12. The object identification system using a hierarchical model according to claim 11, further comprising: extracting first feature points through a first image encoder of the learned object region extraction model; extracting a feature point region from the extracted first feature points; and inferring an object region.

14. The object identification device classifies categories of object types located in the extracted object region in stages, The object identification system using a hierarchical model according to claim 11, wherein the object management server identifies object type information by using an object type inference model corresponding to the classified categories.

15. The object identification device includes:

15. The object identification system using a hierarchical model according to claim 14, wherein the categories of the objects located in the extracted object regions are classified stepwise from a top major classification to a bottom minor classification.

16. The object management server includes:

16. The object identification system using a hierarchical model according to claim 15, wherein object type information is identified using an object type inference model corresponding to the lowest classification, which is the classified category.

17. The object management server includes:

16. The system for identifying objects using a hierarchical model according to claim 15, wherein the object type information is determined to be one of a plurality of classes included in the lowest classification.

18. The object management server includes:

15. The object identification system using a hierarchical model according to claim 14, further comprising: extracting second feature points through a second image encoder of an object type inference model corresponding to the classified category; and inputting the extracted second feature points to a feature decoder to infer object type information.

19. In the object identification device, a memory for storing one or more programs; a processor for executing said stored one or more programs; Equipped with The processor, Detecting an object region in which an object is located from the object image based on the learned object region extraction model, and extracting the detected object region; An object identification device using a hierarchical model, which identifies object type information located in the extracted object region by using an object type inference model.

20. In the object management server, a memory for storing one or more programs; a processor for executing said stored one or more programs; Equipped with The processor, Detecting an object region in which an object is located from the object image based on the learned object region extraction model, and extracting the detected object region; An object management server using a hierarchical model, which uses an object type inference model to identify object type information located in the segmented object region.

Citation Information

Patent Citations

  • Method and system for performing non-invasive genetic testing using artificial intelligence (AI) models

    JP2022551683A

  • Efficient object detection method using convolutional neural network-based hierarchical feature modeling

    KR101882743B1

  • Method for providing food information based on food suitability and apparatus using the method

    KR102110766B1

  • Apparatus and Method for Food Search Service

    KR102433391B1

  • Apparatus for removing degraded concrete of pier using water jet

    KR102614919B1