Data labeling system and method using lifelong learning device

Abstract

The present invention provides a system and method that can improve the data labeling process through lifelong learning techniques. A data labeling system using a lifelong learning device according to one embodiment of the present invention comprises: a terminal of a plurality of annotators; and a lifelong learning device. The lifelong learning device comprises: a receiving unit that receives data labeling results from the terminal of the annotator; a labeling model generating unit that generates a labeling model based on the data labeling results; a labeling model learning unit that trains the labeling model through a lifelong learning technique; and an annotator labeling ability evaluation unit that evaluates the data labeling ability of the annotator by calculating the performance of the labeling model. A data labeling system and method using a lifelong learning device according to the present invention can improve the quality of data labeling.

Description

Technology Field

[0001] The present invention relates to a data labeling system and method, and more specifically, to a system and method capable of improving the data labeling process using a lifelong learning device and / or lifelong learning technique. Background Technology

[0002] In the case of deep learning techniques, deep neural networks are trained through supervised learning, but due to the infinite number of parameters, a large amount of training data is required.

[0003] The process of collecting and processing training data necessary for learning is called labeling.

[0004] In general, the role of the annotator performing the labeling is crucial for acquiring high-quality training data. While methods such as active learning can be utilized for this purpose, the training process takes a very long time, making it difficult for the annotator to check the training results of the data completed so far in real time.

[0005] In addition, there is a lack of mechanisms to provide feedback to data labeling workers regarding their work results to date, thereby enabling them to produce improved results. In other words, generally, to evaluate a worker based on the dataset they have worked on, the evaluation is conducted either by evaluating the quantity of the dataset or through third-party verification.

[0006] However, this not only takes a long time for the evaluation process but also creates a subjective problem where the standard for qualitative evaluation must be based on human eyes. Prior art literature

[0007] Published Patent Application 10-2023-0047531 (2023.04.10) The problem to be solved

[0008] The present invention provides a system and method for improving a data labeling process for artificial intelligence by utilizing a lifelong learning device and / or a lifelong learning technique.

[0009] The problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0010] A data labeling system using a lifelong learning device according to one embodiment of the present invention is provided, and the data labeling system using the lifelong learning device is,

[0011] Terminals of multiple annotators; and

[0012] A lifelong learning device, wherein the lifelong learning device comprises,

[0013] A receiving unit that receives data labeling results from the terminal of the above-mentioned worker;

[0014] A labeling model generation unit that generates a labeling model based on the above data labeling results;

[0015] A labeling model learning unit that trains the above-mentioned labeling model through lifelong learning techniques;

[0016] It includes a worker labeling ability evaluation unit that evaluates the data labeling ability of the worker by calculating the performance of the above labeling model.

[0018] Preferably,

[0019] The lifelong learning technique performed by the above-mentioned lifelong learning device is,

[0020] It is configured to perform supervised learning on a labeled dataset and unsupervised learning on an unlabeled dataset, and then selectively perform supervised or unsupervised learning on an additional labeled dataset or an additional unlabeled dataset.

[0022] Preferably,

[0023] The lifelong learning device for performing the above-mentioned lifelong learning technique is,

[0024] processor; and

[0025] It further includes memory connected to the above processor,

[0026] The above memory is,

[0027] A deep learning model updated by the above labeling model is input into a first dataset with labels to perform supervised learning of the deep learning model, and

[0028] The first parameter determined by the above supervised learning is applied to the deep learning model, and

[0029] A second dataset without labels is input to a deep learning model to which the first parameter is applied to perform unsupervised learning of the deep learning model, and

[0030] The second parameter determined by the above unsupervised learning is applied to the deep learning model, and

[0031] To enable the deep learning model to which the above-mentioned second parameter is applied to selectively input a labeled third dataset or an unlabeled fourth dataset to train the deep learning model indefinitely.

[0032] It stores program instructions executed by the above processor.

[0034] Preferably,

[0035] The first to fourth datasets above include data related to at least one of object detection, behavior analysis, autonomous driving, and smart factory.

[0037] Preferably,

[0038] The first parameter and the second parameter include a weight, a bias, and an activation function.

[0040] Preferably,

[0041] The above unsupervised learning is performed using the SimCLR (A simple framework for contrastive learning of visual representations) algorithm.

[0043] Preferably,

[0044] The above labeling model is characterized by being generated separately for each worker.

[0046] A data labeling method using a lifelong learning device according to another embodiment of the present invention is provided, and the data labeling method using the lifelong learning device is,

[0047] The above-mentioned lifelong learning device is configured to include a receiving unit, a labeling model generation unit, a labeling model learning unit, and a worker labeling ability evaluation unit, and

[0048] A receiving step of receiving a data labeling result from the operator's terminal by the receiving unit;

[0049] A labeling model generation step in which a labeling model is generated based on the data labeling result by the labeling model generation unit;

[0050] A labeling model learning step in which the labeling model is trained through a lifelong learning technique by the labeling model learning unit;

[0051] It includes a worker labeling ability evaluation step that evaluates the data labeling ability of the worker by calculating the performance of the labeling model by the worker labeling ability evaluation unit.

[0053] Preferably,

[0054] The lifelong learning technique performed by the above-mentioned lifelong learning device is,

[0055] It is configured to perform supervised learning on a labeled dataset and unsupervised learning on an unlabeled dataset, and then selectively perform supervised or unsupervised learning on an additional labeled dataset or an additional unlabeled dataset.

[0057] Preferably,

[0058] The lifelong learning device for performing the above-mentioned lifelong learning technique is,

[0059] processor; and

[0060] It further includes memory connected to the above processor,

[0061] The above memory is,

[0062] A deep learning model updated by the above labeling model is input into a first dataset with labels to perform supervised learning of the deep learning model, and

[0063] The first parameter determined by the above supervised learning is applied to the deep learning model, and

[0064] A second dataset without labels is input to a deep learning model to which the first parameter is applied to perform unsupervised learning of the deep learning model, and

[0065] The second parameter determined by the above unsupervised learning is applied to the deep learning model, and

[0066] To enable the deep learning model to which the above-mentioned second parameter is applied to selectively input a labeled third dataset or an unlabeled fourth dataset to train the deep learning model indefinitely.

[0067] It stores program instructions executed by the above processor.

[0069] Preferably,

[0070] The first to fourth datasets above include data related to at least one of object detection, behavior analysis, autonomous driving, and smart factory.

[0072] Preferably,

[0073] The first parameter and the second parameter include a weight, a bias, and an activation function.

[0075] Preferably,

[0076] The above labeling model is characterized by being generated separately for each annotator.

[0078] Specific details of other embodiments are included in the detailed description and drawings. Effects of the invention

[0079] A data labeling system and method using a lifelong learning device according to the present invention can improve and enhance the quality of data labeling.

[0080] According to the data labeling system and method using a lifelong learning device according to the present invention, workers can be evaluated based on the performance of a learned model and used as a basis for performance-based bonuses to motivate workers.

[0081] According to the data labeling system and method using a lifelong learning device of the present invention, workers can learn how to improve model performance through the data labeling process, thereby enabling the production of higher quality data.

[0082] According to the data labeling system and method using a lifelong learning device according to the present invention, workers can easily identify errors in the labels they have worked on while confirming in real time that performance is improving.

[0083] However, the effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0084] FIG. 1 is a schematic diagram of a data labeling system using a lifelong learning device according to one embodiment of the present invention. Figure 2 is a diagram illustrating a tree representing a series of various lifelong learning methods and various branches within each series. FIG. 3 is a diagram illustrating a lifelong learning process according to one embodiment of the present invention. Figure 4 is a diagram illustrating the SimCLR framework. FIG. 5 is a diagram illustrating the configuration of a lifelong learning device for implementing a lifelong learning technique according to one embodiment of the present invention. FIG. 6 is a flowchart illustrating a data labeling method using a lifelong learning device according to one embodiment of the present invention. FIG. 7 is a drawing illustrating an exemplary computing device capable of implementing a device and / or system according to various embodiments of the present invention. Specific details for implementing the invention

[0085] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Throughout the specification, the same reference numerals refer to the same components.

[0086] The embodiments described herein will be described with reference to cross-sectional and / or plan views, which are exemplary illustrations of the invention. In the drawings, the thickness of the components is exaggerated for effective description of the technical content. Accordingly, the components illustrated in the drawings are schematic in nature, and the shapes of the components illustrated in the drawings are intended to illustrate specific forms of the components and are not intended to limit the scope of the invention. Although terms such as first, second, third, etc., have been used to describe various components in the various embodiments of this specification, these components should not be limited by such terms. These terms are used merely to distinguish one component from another. The embodiments described and illustrated herein also include their complementary embodiments.

[0087] The terms used herein are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used herein, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0088] Unless otherwise defined, all terms used in this specification (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0089] Hereinafter, the concept of the present invention and embodiments according thereto will be described in detail with reference to the drawings.

[0091] FIG. 1 is a schematic diagram of a data labeling system using a lifelong learning device according to one embodiment of the present invention.

[0092] A data labeling system (100) using a lifelong learning device includes terminals (110 to 130) of a plurality of annotators and a lifelong learning device (150).

[0093] Worker terminals (110 to 130) and lifelong learning devices (150) are interconnected through a network (140). In FIG. 1, three worker terminals are shown, but the number of worker terminals can be more or fewer.

[0094] In FIG. 1, the worker terminal is depicted as a portable terminal such as a smartphone, but is not limited thereto. The worker terminal may include a personal computer, a laptop, a tablet, or other server device capable of performing the lifelong learning device of the present invention.

[0095] The lifelong learning device (150) includes a receiving unit (151), a labeling model generating unit (152), a labeling model learning unit (153), and a worker labeling ability evaluation unit (154).

[0096] The receiving unit (151) receives data labeling results from the worker's terminal (110 to 130).

[0097] The labeling model generation unit (152) generates a labeling model based on the data labeling results. The labeling model is generated separately for each worker.

[0098] The labeling model learning unit (153) trains the labeling model through lifelong learning techniques.

[0099] The worker labeling ability evaluation unit (154) evaluates the data labeling ability of the worker by calculating the performance of the labeling model.

[0100] The labeling model is characterized by being generated separately for each annotator.

[0101] The data labeling system (100) using the lifelong learning device of the present invention is characterized by improving data labeling performance by utilizing the lifelong learning device and / or lifelong learning technique.

[0102] In this invention, data labeling refers to the process of inputting various information into data such as images, videos, and text according to a specific purpose, using data processing tools by a person (manual) or a machine (automatic), so that artificial intelligence can learn.

[0103] As a specific example, data labeling for object recognition of objects in an image refers to the process of drawing a box on a data image to indicate the location of an object, such as a person or a car, and adding annotations to classify whether the box is a person or a car.

[0104] Simply put, drawing a box around a car in an image and labeling it 'car' is data labeling.

[0105] Although data labeling may appear simple and easy, it requires concentration and attention to detail because it must be performed in accordance with data input standards.

[0106] Accuracy is crucial for the data used to train artificial intelligence. This is because incorrect training data can lead to the failure of the AI.

[0107] Data formats for artificial intelligence learning are diverse, such as images, videos, sound, and documents.

[0108] To perform data labeling effectively, utilize tools with labeling capabilities tailored to the training data.

[0109] As used in the present invention, the term 'labeling model' refers to a tool having a labeling function and may also be called a labeler or labeler.

[0110] The quality of data labeling depends on whether the tool features and guides suitable for each data format have been selected.

[0111] By utilizing tools that are easy and fast to use, you can quickly generate data in various formats in the file format requested by the customer.

[0112] The basic functions of a labeling model, which is a data processing tool, include bounding boxes, polygons, etc.

[0113] A bounding box is a labeling method that draws an object so that it is contained within a rectangular box, and it is the most commonly used method in data labeling tasks.

[0114] A polygon is a labeling method that draws points along the outline of an object's visible area in a polygonal shape. It is a feature that can handle errors that may occur due to empty spaces included outside the object.

[0115] Additionally, polylines are used to distinguish sidewalks, lanes, etc. by using a line with multiple points to label specific areas.

[0116] Point is the task of labeling specific points and is a technology that requires precise and delicate work, such as emotion analysis through facial recognition.

[0117] A Cuboid is a labeling form that generates 3D objects that cannot be worked with in 2D as cubes.

[0118] Body is a method of creating objects on the body when it is necessary to detect human movement, such as in overall motion capture or abnormal behavior.

[0119] Face is a method of creating objects on a face when it is necessary to detect facial feature points.

[0120] Hands is used when it is necessary to detect hand joints to identify hand movements, such as sign language.

[0121] The labeling model in the present invention is configured to automatically perform labeling that was previously performed solely manually by an operator.

[0122] The automatic labeling model is configured to recognize objects within an image through a pre-trained recognition model.

[0123] By configuring an automatic labeling model, the repetitive and laborious manual labeling process can be automated, dramatically shortening the 'verification and correction' steps.

[0124] Back-end online recognition methods are an example of automatic labeling models.

[0125] In the back-end online recognition method, while the software itself used for labeling can operate either online or offline, the object detection unit based on pre-recognition models—which is the core of automatic labeling—is processed online on a server, thereby reducing the burden on the operator environment. In particular, multiple operators can share the server's recognition functions, reducing costs and improving usability. Furthermore, recognition models can be easily updated and replaced as needed; as training data accumulates, pre-recognition performance improves, and operators can significantly reduce their workload by gradually shifting from modifications to verification tasks.

[0126] Essential considerations for performing automatic labeling through back-end online object recognition are the pre-recognized class name conversion table and the activation of a multiple recognition model pool.

[0127] The Pre-recognition Class Name Conversion Table is used for the post-processing task of assigning class names, which is required alongside the pre-region recognition task. Since individual operators may target different class groups, this is performed by the client-side software. For example, objects recognized during pre-recognition are returned with an object region and an object class name, but the operator can specify the class name separately. If an operator wishes to assign a class name such as 'Truck' to all objects even if pre-recognition returns it, they may need to manually change it every time. To address this, the 'Pre-recognition Class Name Conversion Table' can be configured to reduce the repetitive work of changing returned class names.

[0128] Generally, deep learning models require loading trained model data into CPU or GPU memory during inference, so repeating this loading process for every inference is inefficient. To address this issue, a multi-recognition model pool and activation are necessary. By utilizing a multi-recognition model pool and activation, models can be loaded into memory prior to the launch of a web service to enable continuous inference, thereby significantly improving overall response speed. This is considerably faster than manually processing box and polygon areas during the labeling process, and the automatic labeling function, which guarantees a consistent time, becomes even more effective as the workload and processing time increase.

[0129] As an embodiment of the present invention, the labeling model is a manual labeling model and is characterized by being generated separately according to each annotator.

[0130] If the labeling model is a manual labeling model, the data labeling ability of each worker can be evaluated by calculating the performance of the manual labeling model generated for each worker.

[0131] In addition, labeling models are generated individually using data collected by each worker.

[0132] For a clear explanation, one labeling model M (s) Let's assume that the training dataset processed by the worker is X. (s) , Y (s) When saying that, M (s) is X (s) , Y (s) It refers to the dataset trained on. s is an identification number that can identify each worker.

[0133] If there are N workers, N labeling models can be generated. The form of the loss function used to train these labeling models will vary depending on the task, but if it is an image classification task, it will be a cross-entropy loss function.

[0134] If the validation set is X (v) , Y (v) If given, the score for evaluating each worker is defined as the accuracy on the validation set, which is equal to Equation (1).

[0135] (1)

[0136] N (v) represents the total number of validation sets, and is the i-th data of the validation set, represents the i-th label for the validation set. If, the labeling model M (s)If these were models with the same original weights and performance, their performance would be determined by the training dataset acquired by the operator. In other words, the structure is such that the operator who collected the training dataset that resulted in the best-performing model receives the highest score.

[0137] The evaluation of workers' labeling ability is the S defined above. (s) It is defined based on, and if you want to select the best worker, you can use formula (2).

[0138] (2)

[0139] If the task is object detection, Average Precision can be used for the score function, and if the task is segmentation, the score function can take the form of IoU (Intersection over Union).

[0141] Figure 2 is a diagram illustrating a tree representing a series of various lifelong learning methods and various branches within each series.

[0142] When a new class appears that does not correspond to the classes of the dataset trained to perform a deep learning algorithm, it is necessary to train a new model that includes this new class.

[0143] When deep learning models learn a new task, a fatal forgetting phenomenon occurs in which they lose a large amount of information from previously learned tasks; the technology that can solve this is lifelong learning.

[0144] Lifelong learning is a skill that enables learning to remember information from previously learned tasks while learning the current task.

[0145] Lifelong learning stores important information about tasks through a continuous and sequential learning method, and is classified into three methods as shown in Figure 2 depending on how the stored information is utilized.

[0146] Referring to Figure 2, Replay methods save samples from previous tasks in raw form or create pseudo samples using a generative model, and solve the catastrophic forgetting problem by replaying samples from previous tasks when training a new task.

[0147] Regularization-based methods utilize a regularization-based loss function that imposes constraints on the weights of the previous task when training a new task.

[0148] Parameter isolation methods are a method of extending a network to learn new tasks, specifying different model parameters for each task.

[0149] Research in the field of such lifelong learning focuses primarily on supervised learning models.

[0150] However, supervised learning requires large-scale, high-quality labeled datasets, which has the disadvantage of being very inefficient in terms of time and cost.

[0151] To address this, it is possible to utilize lifelong learning algorithms based on unsupervised learning techniques, but this requires that all data used must be free from human intervention.

[0152] In lifelong learning, sequentially input tasks are distinguished by the presence or absence of labels; therefore, when tasks of the two structures are mixed, constraints arise in applying the optimal algorithmic methodology.

[0153] While unlabeled data is accumulating in real time, the number of labeled data is limited.

[0154] Therefore, the demand for deep learning technology that utilizes both labeled and unlabeled data continues to rise.

[0156] FIG. 3 is a diagram illustrating a lifelong learning process according to one embodiment of the present invention.

[0157] Referring to FIG. 3, in this embodiment, supervised learning based on a labeled dataset and unsupervised learning based on an unlabeled dataset are sequentially performed on a deep learning model, and then, either a supervised learning-based or unsupervised learning-based algorithm is sequentially applied to perform lifelong learning.

[0158] More specifically, in Task 1, a labeled dataset is input into a deep learning model to perform supervised learning.

[0159] Here, the deep learning model may be a Convolutional Neural Network (CNN) that includes multiple convolutional layers and pooling layers for extracting feature maps from input data included in the dataset, and a fully connected layer for classification.

[0160] The dataset according to the present embodiment may include data related to object detection, behavior analysis, autonomous driving, and smart factories.

[0161] In supervised learning of the first task, a loss function such as contrastive loss can be used.

[0162] After the training of the first task is completed, the first parameter of the deep learning model obtained is set as the initial parameter of the second task (Task 2), and unsupervised learning is performed by inputting the second dataset (Unlabeled Dataset) into the deep learning model.

[0163] Here, the parameters can be weights, biases, and activation functions.

[0164] According to the present embodiment, after supervised learning of the first task is completed, unsupervised learning can be performed using an algorithm such as SimCLR (A simple framework for contrastive learning of visual representations), which is a type of contrastive learning.

[0165] SimCLR is a type of contrastive learning, which is the most representative methodology of self-supervised learning (unsupervised learning). It is a method that learns the representation distances of patches obtained from the same image to be close to each other, while other distances are far apart.

[0167] Figure 4 is a diagram illustrating the SimCLR framework.

[0168] Referring to Figure 4, SimCLR applies two different data augmentations to each image, defining results from the same image as positive pairs and results from different images as negative pairs, thereby applying a contrastive learning method.

[0169] A single image (x) is divided into two images (xi, xj) by undergoing two different augmentation transformations, and these two transformed images are defined as a positive pair because they were obtained from the same image. If transformed images yi and yj are obtained from another image y, then xi and yi (or yj) are defined as a negative pair because they were obtained from different images.

[0170] Each transformed image (xi, xj) passes through a CNN-based network (f) to be converted into a visual representation embedding vector (hi, hj). The contrastive loss is calculated using this transformed output (zi, zj).

[0171] According to the present embodiment, optimized parameters within the learning space of features are explored and shared using parameters of labeled and unlabeled tasks, and self-supervised lifelong learning is performed to minimize the loss function.

[0172] As described above, after supervised and unsupervised learning for the deep learning model is completed once each, in the third task, a labeled third dataset or an unlabeled fourth dataset is input to selectively perform supervised or unsupervised learning.

[0173] Subsequently, the lifelong learning device according to the present embodiment can sequentially perform supervised learning or unsupervised learning of a deep learning model by selectively inputting a labeled n-th dataset or an unlabeled n+1-th dataset.

[0174] According to the present embodiment, after the learning of the second task is completed, it is determined whether there exists an n-th dataset or an n+1-th dataset of a size greater than a preset size, and among the n-th dataset or the n+1-th dataset, the dataset of a size greater than the preset size is selectively input to perform supervised learning or unsupervised learning of the deep learning model.

[0176] FIG. 5 is a diagram illustrating the configuration of a lifelong learning device for implementing a lifelong learning technique according to one embodiment of the present invention.

[0177] Referring to FIG. 5, a lifelong learning device (500) for implementing a lifelong learning technique according to the present embodiment includes a processor (510) and a memory (520).

[0178] The lifelong learning device (500) for performing the lifelong learning technique illustrated in FIG. 5 may be implemented as an independent computing device or server separate from the lifelong learning device (150) for implementing data labeling, including a processor (510) and memory (520).

[0179] At this time, a lifelong learning device (500) for performing lifelong learning techniques and a lifelong learning device (150) for implementing data labeling can be connected to each other through a network (140).

[0180] In another embodiment, a lifelong learning device (500) for implementing a lifelong learning technique and a lifelong learning device (150) for implementing data labeling may be integrated into a single device.

[0181] That is, the processor (510) and memory (520) can be configured to be integrated within the lifelong learning device (150) with the receiving unit (151), the labeling model generation unit (152), the labeling model learning unit (153), and the worker labeling ability evaluation unit (154).

[0182] The processor (510) may include a CPU (central processing unit), a GPU (graphics processing unit), or other virtual machines capable of executing computer programs.

[0183] The memory (520) may include a non-volatile storage device such as a fixed hard drive or a removable storage device. The removable storage device may include a compact flash unit, a USB memory stick, etc. The memory (520) may also include volatile memory such as various random access memory.

[0184] In the memory (520) according to the present embodiment, program instructions for performing lifelong learning using both labeled and unlabeled datasets are stored.

[0185] Such program instructions can be stored on a computer-readable recording medium.

[0186] The program instructions according to the present embodiment input a first dataset with labels into a deep learning model updated by a labeling model to perform supervised learning of the deep learning model, apply a first parameter determined by the supervised learning to the deep learning model, input a second dataset without labels into the deep learning model to which the first parameter is applied to perform unsupervised learning of the deep learning model, apply a second parameter determined by the unsupervised learning to the deep learning model, and selectively input a third dataset with labels or a fourth dataset without labels into the deep learning model to which the second parameter is applied to perform continuous learning of the deep learning model.

[0188] FIG. 6 is a flowchart illustrating a data labeling method using a lifelong learning device according to one embodiment of the present invention.

[0189] The present invention provides a data labeling method using a lifelong learning device.

[0190] A data labeling method using a lifelong learning device includes a receiving step (S610), a labeling model generation step (S620), a labeling model training step (S630), and a worker labeling ability evaluation step (S640).

[0191] A lifelong learning device (150) for performing the data labeling method of the present invention includes, as described with reference to FIG. 1, a receiving unit (151), a labeling model generating unit (152), a labeling model learning unit (153), and a worker labeling ability evaluation unit (153).

[0192] The present invention includes a receiving step (S610) of receiving data labeling results from a worker's terminal (110 to 130) by a receiving unit (151).

[0193] The present invention includes a labeling model generation step (S620) that generates a labeling model based on data labeling results by a labeling model generation unit (152).

[0194] The present invention includes a labeling model learning step (S630) in which a labeling model is learned through a lifelong learning technique by a labeling model learning unit (153).

[0195] The present invention includes a worker labeling ability evaluation step (S640) that evaluates the worker's data labeling ability by calculating the performance of the labeling model by the worker labeling ability evaluation unit (154).

[0196] The labeling model is characterized by being generated separately for each annotator.

[0197] The lifelong learning technique performed by the lifelong learning device of the present invention is configured to perform supervised learning on a labeled dataset and unsupervised learning on an unlabeled dataset, and then selectively perform supervised learning or unsupervised learning on an additional labeled dataset or an additional unlabeled dataset.

[0198] In a lifelong learning device for performing a lifelong learning technique, the memory (520) stores program instructions executed by the processor (510) to perform supervised learning of a deep learning model by inputting a first dataset with labels into the deep learning model updated by the labeling model, applying a first parameter determined by supervised learning to the deep learning model, applying a second dataset without labels into the deep learning model with the first parameter applied to the deep learning model to perform unsupervised learning of the deep learning model, applying a second parameter determined by unsupervised learning to the deep learning model, and optionally inputting a third dataset with labels or a fourth dataset without labels into the deep learning model with the second parameter applied to perform lifelong learning of the deep learning model.

[0199] The first to fourth datasets include data related to at least one of object detection, behavior analysis, autonomous driving, and smart factory.

[0200] The first parameter and the second parameter include a weight, a bias, and an activation function.

[0202] FIG. 7 is a drawing illustrating an exemplary computing device (700) capable of implementing a device and / or system according to various embodiments of the present invention.

[0203] Referring to FIG. 7, an exemplary computing device (700) capable of implementing devices according to some embodiments of the present disclosure will be described in more detail.

[0204] A computing device (700) may include one or more processors (710), a bus (750), a communication interface (770), a memory (730) for loading a computer program (791) executed by the processor (710), and a storage (790) for storing the computer program (791). However, only components related to embodiments of the present disclosure are illustrated in FIG. 7.

[0205] Therefore, a person skilled in the art to which this disclosure belongs will understand that other general-purpose components may be included in addition to the components shown in FIG. 7.

[0206] The processor (710) controls the overall operation of each component of the computing device (700). The processor (710) may be configured to include a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphic Processing Unit), or any form of processor (710) well known in the art of the present disclosure. Additionally, the processor (710) may perform operations for at least one application or program for executing the method according to the embodiments of the present disclosure. The computing device (700) may have one or more processors (710). The computing device (700) may refer to artificial intelligence (AI).

[0207] The memory (730) stores various data, commands and / or information. The memory (730) may load one or more programs (791) from storage (790) to execute a method according to embodiments of the present disclosure. The memory (730) may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

[0208] The bus (750) provides communication functions between components of the computing device (700). The bus (750) can be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

[0209] The communication interface (770) supports wired and wireless internet communication of the computing device (700). Additionally, the communication interface (770) may support various communication methods other than internet communication. To this end, the communication interface (770) may be configured to include a communication module well known in the art of the present disclosure.

[0210] According to some embodiments, the communication interface (770) may be omitted.

[0211] Storage (790) can store one or more programs (791) and various data non-temporarily.

[0212] Storage (790) may be configured to include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which this disclosure belongs.

[0213] The computer program (791) may include one or more instructions that cause the processor (710) to perform a method / operation according to various embodiments of the present disclosure when loaded into memory (730). That is, the processor (710) may perform a method / operation according to various embodiments of the present disclosure by executing the one or more instructions.

[0215] Although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the patent claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

Claims

Claim 1 Terminals of multiple annotators; The lifelong learning device includes: a receiving unit that receives data labeling results from a terminal of the worker; a labeling model generating unit that generates a labeling model based on the data labeling results, but generates the labeling model separately for each worker; a labeling model training unit that trains the labeling model through a lifelong learning technique; and a worker labeling ability evaluation unit that evaluates the data labeling ability of the worker by calculating the performance of the labeling model. The labeling model training unit inputs a first dataset with labels into a deep learning model updated by the labeling model to perform supervised learning of the deep learning model, applies a first parameter determined by the supervised learning to the deep learning model, inputs a second dataset without labels into the deep learning model to which the first parameter is applied to perform unsupervised learning of the deep learning model, applies a second parameter determined by the unsupervised learning to the deep learning model, and applies a third dataset with labels or a fourth dataset without labels to the deep learning model to which the second parameter is applied. A data labeling system using a lifelong learning device, wherein a deep learning model is trained by selectively inputting a dataset, and the evaluation unit calculates accuracy using a validation set for the labeling model trained with the training dataset worked on by the operator, and evaluates the operator's data labeling ability based on the calculated accuracy by utilizing the fact that the performance of the labeling model is determined according to the training dataset obtained by the operator while the labeling models have the same initial weights and performance. Claim 2 delete Claim 3 delete Claim 4 A data labeling system using a lifelong learning device according to claim 1, wherein the first to fourth datasets include data related to at least one of object detection, behavior analysis, autonomous driving, and smart factory. Claim 5 A data labeling system using a lifelong learning device according to claim 1, wherein the first parameter and the second parameter include a weight bias and an activation function. Claim 6 A data labeling system using a lifelong learning device according to claim 1, wherein the unsupervised learning is performed using the SimCLR (A simple framework for contrastive learning of visual representations) algorithm. Claim 7 delete Claim 8 A data labeling method using a lifelong learning device comprises: a receiving unit, a labeling model generating unit, a labeling model learning unit, and a worker labeling ability evaluation unit; a receiving step of receiving a data labeling result from a worker's terminal by the receiving unit; a labeling model generating step of generating a labeling model based on the data labeling result by the labeling model generating unit, wherein the labeling model is generated separately for each worker; a labeling model learning step of training the labeling model through a lifelong learning technique by the labeling model learning unit; and a worker labeling ability evaluation step of evaluating the data labeling ability of the worker by calculating the performance of the labeling model by the worker labeling ability evaluation unit; wherein the labeling model learning step involves supervising the deep learning model by inputting a first dataset containing labels into a deep learning model updated by the labeling model, applying a first parameter determined by the supervised learning to the deep learning model, and inputting a second dataset without labels into the deep learning model to which the first parameter is applied to the deep learning model A data labeling method using a lifelong learning device, wherein unsupervised learning is performed, a second parameter determined by the unsupervised learning is applied to the deep learning model, and a third dataset with labels or a fourth dataset without labels is selectively input to the deep learning model to which the second parameter is applied to learn the deep learning model, thereby learning the deep learning model in a lifelong manner; and the evaluation step calculates accuracy using a validation set for the labeling model learned with the training dataset worked on by the operator, and evaluates the operator's data labeling ability based on the calculated accuracy by utilizing the fact that the performance of the labeling model is determined according to the training dataset obtained by the operator while the labeling models have the same initial weights and performance. Claim 9 delete Claim 10 delete Claim 11 A data labeling method using a lifelong learning device according to claim 8, wherein the first to fourth datasets include data related to at least one of object detection, behavior analysis, autonomous driving, and smart factory. Claim 12 A data labeling method using a lifelong learning device according to claim 8, wherein the first parameter and the second parameter include a weight bias and an activation function. Claim 13 delete

Images