Data generation support device, data generation support method and program

Abstract

To provide an automatic annotation technology for reducing annotation costs.SOLUTION: A data generation support device comprises: a data acquisition unit which acquires annotation target data; a model selection unit which selects an annotation model from a first machine learning model and a second machine learning model, on the basis of a predetermined index; an annotation processing unit which executes annotation processing, by using the selected annotation model; and a user interface unit which provides the annotated data created by the annotation processing.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a data generation support device, a data generation support method, and a program. [Background technology]

[0002] In recent years, the use of machine learning models has progressed in various technical fields. When machine learning models are used in operational fields, data collected at the operational sites may be used to update the machine learning models in operation and improve their accuracy. For example, such a mechanism is known as MLOps (Machine Learning Operations).

[0003] After being annotated, data collected at the operation site may be used as training data for updating or retraining an operational machine learning model. In order to rapidly generate a training dataset, automatic annotation may be performed on data using a machine learning model. For example, data annotated by automatic annotation is confirmed by a user such as an annotator. If it is determined that the annotated data is appropriate and does not need to be corrected, the data is added to the training dataset. On the other hand, if it is determined that the annotated data is inappropriate and needs to be corrected, the automatically assigned annotation may be corrected by the annotator, and the corrected data may be added to the training dataset.

[0004] Several techniques have been proposed for automatic annotation using such machine learning models. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-2338 [Patent Document 2] JP 2020-35095 A Summary of the Invention [Problem to be solved by the invention]

[0006] When generating an annotation model to be used for automatic annotation, if the number of training data used to train the annotation model is insufficient, the accuracy of the automatic annotation may also be insufficient. In order to collect the number of training data required to generate an annotation model with good accuracy, an annotator may need to annotate the data manually, which may increase the annotation cost.

[0007] In view of the above problems, one objective of the present disclosure is to provide an automatic annotation technique for reducing annotation costs. [Means for solving the problem]

[0008] One aspect of the present disclosure relates to a data generation support device having a data acquisition unit that acquires data to be annotated, a model selection unit that selects an annotation model from a first machine learning model and a second machine learning model based on a predetermined indicator, an annotation processing unit that performs annotation processing using the selected annotation model, and a user interface unit that provides annotated data generated by the annotation processing. Effect of the Invention

[0009] According to the present disclosure, an automatic annotation technique can be provided to reduce annotation costs. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating a data generation support system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram illustrating an initial stage of a data generation assistance process according to one embodiment of the present disclosure. [Diagram 3] FIG. 3 is a schematic diagram illustrating a later stage of a data generation assisted process according to one embodiment of the present disclosure. [Figure 4] FIG. 4 is a block diagram illustrating a hardware configuration of a data generation assistance device according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 is a block diagram illustrating a functional configuration of a data generation assistance device according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating the accuracy of an initial annotation model and a later annotation model according to an embodiment of the present disclosure. [Figure 7] FIG. 7 is a schematic diagram illustrating an initial stage of a data generation assistance process based on the number of data acquisitions according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a schematic diagram illustrating a later stage of a data generation assistance process based on the number of data acquisitions according to one embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating the switching timing of an annotation model according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a flow chart illustrating an initial stage of a data generation assistance process according to one embodiment of the present disclosure. [Figure 11] FIG. 11 is a flow chart illustrating later stages of a data generation assisted process according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] In the following embodiment, a data generation support system is disclosed that generates annotated data to be used as training data for a machine learning model.

[0013] [Data generation support system] 1, the data generation support system 10 includes a user terminal 20, a training data database (DB) 50 for an early model, a training data DB 60 for a later model, and a data generation support device 100. As shown in the figure, the user terminal 20, the training data database (DB) 50 for an early model, the training data DB 60 for a later model, and the data generation support device 100 can be communicatively connected to each other.

[0014] The data generation support device 100 adds annotations to the received data to be annotated, as described in more detail below. Specifically, when the data generation support device 100 receives the data to be annotated, the data generation support device 100 generates annotated data using an annotation model, and provides the generated annotated data to the user terminal 20. For example, when the annotation model receives image data, the annotation model may be image data that shows a human face detected in the image data by a rectangular area (hereinafter, may be referred to as a bounding box) or the like.

[0015] A data generation support device 100 according to an embodiment described later uses different annotation models in an early stage and a later stage of annotation processing. Specifically, the data generation support device 100 selectively uses an annotation model 30 for an early stage (hereinafter, may be referred to as an early annotation model or an early model) and an annotation model 40 for a later stage (hereinafter, may be referred to as a later annotation model or a later model) to generate annotated data. Here, the early annotation model 30 and / or the later annotation model 40 may be provided in the data generation support device 100, or may be provided in an external server or the like. In addition, the early annotation model 30 and / or the later annotation model 40 may be trained by an external server or the like, or may be trained by the data generation support device 100.

[0016] The user terminal 20 may be any type of information processing device such as a personal computer (PC), and instructs the data generation support device 100 to generate annotated data and acquires the annotated data from the data generation support device 100. The user checks the appropriateness of the acquired annotated data on the user terminal 20, and when the user determines that the annotated data generated by the data generation support device 100 is appropriate, the user notifies the data generation support device 100 to use the annotated data as training data. On the other hand, when the user determines that the annotated data generated by the data generation support device 100 is not appropriate, the user corrects the annotation of the data on the user terminal 20 and returns the corrected data to the data generation support device 100. The data generation support device 100 stores the annotated data determined to be appropriate and / or the corrected data acquired from the user terminal 20 in the early model training data DB 50 and / or the later model training data DB 60 as training data for the annotation model.

[0017] The initial model training data DB50 stores training data for training the initial annotation model 30. In the following embodiment, the data generation support device 100 stores in the initial model training data DB50 data that has been corrected by the user among the generated annotated data. Specifically, the initial model training data DB50 may store data that has been annotated by the data generation support device 100 and that has been determined to be inappropriate by the user and whose annotation has been corrected. That is, data that has been annotated by the data generation support device 100 and that has been determined to be appropriate by the user is not stored in the initial model training data DB50. The stored initial model training data may be used to train the initial annotation model 30.

[0018] The training data DB60 for late models stores training data for training the late annotation model 40. In the following embodiment, the data generation support device 100 stores both the generated annotated data and the data corrected by the user in the training data DB60 for late models. That is, the training data DB60 for late models may store data annotated by the data generation support device 100 and data that is determined to be inappropriate by the user and whose annotation has been corrected. The stored training data for late models may be used to train the late annotation model 40.

[0019] [Summary of the Disclosure] The data generation support device 100 uses the initial annotation model 30 and the later annotation model 40 to perform annotation processing on data to be annotated, and generates annotated data. The annotated data is provided to a user terminal 20, and the appropriateness of the annotation is determined by the user. The annotated data and / or data whose annotations have been corrected by the user can be used to train the initial annotation model 30 and the later annotation model 40.

[0020] In MLOps, when data is collected for updating or retraining an operational machine learning model, in the early stage of data collection when the amount of collected data is not sufficient to obtain an annotation model with good accuracy, the data generation support device 100 performs annotation processing using an initial annotation model 30 that can achieve a certain degree of accuracy even in the early stage when the amount of data is not sufficient. After that, when the amount of data necessary to achieve good accuracy is collected, the data generation support device 100 transitions from the early stage to the later stage, and switches the annotation model from the initial annotation model 30 to the later annotation model 40 to perform annotation processing.

[0021] Typically, the initial annotation model 30 may be composed of a machine learning model that can achieve a certain degree of accuracy even with a small amount of training data, and / or a publicly available machine learning model. On the other hand, even if the initial annotation model 30 is trained using a large amount of training data, it is not possible to improve the accuracy according to the amount of training data. On the other hand, the later annotation model 40 cannot achieve good accuracy unless it is trained using a certain amount of data, but when it is trained using a large amount of training data, it may achieve significantly higher accuracy than the initial annotation model 30.

[0022] For example, in the initial stage, as shown in Fig. 2, when the data generation assistance device 100 receives image data to be annotated, it uses the initial annotation model 30 to generate image data showing a face region by a bounding box, and provides the generated annotated image data to the user terminal 20. The user checks the annotated image data, and if the annotated image data is appropriate, notifies the data generation assistance device 100 that the annotated image data is appropriate as training data. On the other hand, if the annotated image data is not appropriate, the user corrects the annotations of the annotated image data, and returns the image data with the corrected annotations to the data generation assistance device 100.

[0023] Here, the data generation support device 100 stores image data corrected by the user in both the training data DB50 for the early model and the training data DB60 for the late model, while storing image data that has been appropriately annotated only in the training data DB60 for the late model. The early annotation model 30 may be trained by the training data stored in the training data DB50 for the early model, and the late annotation model 40 may be trained by the training data stored in the training data DB60 for the late model. In this way, the early annotation model 30 is trained using image data whose annotations have been corrected by the user, not image data that has been appropriately annotated by the data generation support device 100. On the other hand, the late annotation model 40 is trained using both image data that has been appropriately annotated by the data generation support device 100 and image data whose annotations have been corrected by the user. In this way, the initial annotation model 30 is trained using image data whose annotations have been corrected by a user, rather than using annotated image data generated by the data generation support device 100, in order to avoid overlearning of the initial annotation model 30.

[0024] Thereafter, as the training of the initial annotation model 30 and the later annotation model 40 progresses, the data generation assistance device 100 transitions from the initial stage to the later stage, switches from the initial annotation model 30 to the later annotation model 40, and executes the annotation process. Specifically, in the later stage, as shown in FIG. 3, when the data generation assistance device 100 receives data to be annotated, it uses the later annotation model 40 to generate image data showing a face region by a bounding box, and provides the generated annotated image data to the user terminal 20. The user checks the annotated image data, and if the annotated image data is appropriate, notifies the data generation assistance device 100 that the annotated image data can be used as training data. On the other hand, if the annotated image data is not appropriate, the user corrects the annotation of the annotated image data and returns the image data with the corrected annotation to the data generation assistance device 100. The data generation support device 100 stores the annotated image data and the corrected image data in the later model training data DB 60. After switching to the later annotation model 40, the data generation support device 100 does not need to store the image data with the corrected annotation in the initial model training data DB 50.

[0025] Here, the data generation support device 100 may be realized by a computing device such as a server, a personal computer (PC), a smartphone, a tablet, etc., and may have a hardware configuration such as that shown in Fig. 4. That is, the data generation support device 100 has a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, which are interconnected via a bus B.

[0026] The programs or instructions for implementing the various functions and processes in the data generation assistance device 100 may be stored in a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or a flash memory. When the storage medium is set in the drive device 101, the programs or instructions are installed from the storage medium via the drive device 101 into the storage device 102 or the memory device 103. However, the programs or instructions do not necessarily have to be installed from the storage medium, and may be downloaded from any external device via a network or the like.

[0027] The storage device 102 is realized by a hard disk drive or the like, and stores installed programs or instructions as well as files, data, etc. used in executing the programs or instructions.

[0028] The memory device 103 is realized by a random access memory, a static memory, or the like, and when a program or an instruction is activated, reads and stores the program or instruction, data, or the like from the storage device 102. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory storage medium.

[0029] The processor 104 may be realized by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuitry, etc., which may be composed of one or more processor cores, and performs various functions and processing of the data generation support device 100 in accordance with programs, instructions, data such as parameters necessary to execute the programs or instructions, etc. stored in the memory device 103.

[0030] The user interface (UI) device 105 may be composed of input devices such as a keyboard, mouse, camera, microphone, etc., output devices such as a display, speaker, headset, printer, etc., and input / output devices such as a touch panel, and realizes an interface between the user and the data generation support device 100. For example, the user operates the data generation support device 100 by operating a GUI (Graphical User Interface) displayed on a display or touch panel using a keyboard, mouse, etc.

[0031] The communication device 106 is realized by various communication circuits that execute wired and / or wireless communication processing with an external device, the Internet, a LAN (Local Area Network), a cellular network, or other communication network.

[0032] However, the above-described hardware configuration is merely an example, and the data generation assistance device 100 according to the present disclosure may be realized by any other appropriate hardware configuration.

[0033] [Data generation support device] Next, a data generation assistance device 100 according to an embodiment of the present disclosure will be described. Fig. 5 is a block diagram showing the data generation assistance device 100 according to an embodiment of the present disclosure.

[0034] 5, the data generation support device 100 includes a data acquisition unit 110, a model selection unit 120, an annotation processing unit 130, and a user interface unit 140. Each functional unit of the data acquisition unit 110, the model selection unit 120, the annotation processing unit 130, and the user interface unit 140 may be realized by a processor 104 executing a computer program stored in a memory device 103 of the data generation support device 100.

[0035] The data acquisition unit 110 acquires data to be annotated. For example, in MLOps, data collected from an on-site environment in which a machine learning model is operated may be provided to the data acquisition unit 110. For example, the data acquisition unit 110 may acquire data that has been misjudged by an operating machine learning model among data collected in the on-site environment as data to be annotated. By using such misjudged data to update or retrain the operating machine learning model, it may be possible to acquire a machine learning model that is more suitable for the on-site environment.

[0036] The model selection unit 120 selects an annotation model from a plurality of machine learning models based on a predetermined index. For example, the model selection unit 120 may determine which of the initial annotation model 30 and the later annotation model 40 is to be used for the annotation process. Here, the initial annotation model 30 and the later annotation model 40 may be machine learning models with different accuracy in the annotation process.

[0037] For example, the predetermined index may be related to the number of data to be annotated. The number of data to be annotated required to realize annotation processing with a predetermined accuracy may be different for the initial annotation model 30 and the later annotation model 40. The initial annotation model 30 may be, for example, a published model such as GLIP (Grounded Language-Image Pre-training). Alternatively, the initial annotation model 30 may be any machine learning model trained using a small amount of training data by a training method such as few-shot learning. On the other hand, the later annotation model 40 may be any machine learning model that cannot achieve sufficient accuracy with a small amount of training data but can achieve high accuracy when trained with a sufficient amount of training data. The later annotation model 40 may be used as a machine learning model in operation.

[0038] 6, in an early stage where the number of training data generated from data to be annotated is insufficient, the early annotation model 30 can perform annotation processing with higher accuracy than the later annotation model 40. On the other hand, even if the number of training data increases, the accuracy of the early annotation model 30 does not improve as much as that of the later annotation model 40, and in a later stage where the number of training data generated from data to be annotated is sufficient, the later annotation model 40 can perform annotation processing with higher accuracy than the early annotation model 30. In other words, when trained with data to be annotated that is equal to or greater than a predetermined threshold, the later annotation model 40 can realize annotation processing with higher accuracy than the initial annotation model 30.

[0039] In one embodiment, the model selection unit 120 may select the initial annotation model 30 when the number of data to be annotated is less than a predetermined threshold, and may select the later annotation model 40 when the number of data to be annotated is equal to or greater than the predetermined threshold. As shown in FIG. 6, in the initial stage, the accuracy of the initial annotation model 30 is higher than that of the later annotation model 40, but as the number of training data increases, the accuracy of the later annotation model 40 may become higher than that of the initial annotation model 30. In this manner, the model selection unit 120 may select the initial annotation model 30 before acquiring data to be annotated of a number of data such that the accuracy of the later annotation model 40 is higher than that of the initial annotation model 30, and may select the later annotation model 40 after acquiring data to be annotated of a number of data such that the accuracy of the later annotation model 40 is higher than that of the initial annotation model 30.

[0040] The annotation processing unit 130 executes the annotation process using the selected annotation model. For example, when the data to be annotated is image data, the annotation processing unit 130 may add annotations to the image data to be annotated. When the annotation process is to detect a human face area captured in the image data, the annotation processing unit 130 may add a bounding box to the human face area in the image data to be annotated. However, the annotation process according to the present disclosure is not limited to the detection of a human face area.

[0041] The user interface unit 140 provides the annotated data generated by the annotation process. Then, the user interface unit 140 receives feedback on the provided data. Specifically, the user interface unit 140 provides the annotated data to the user terminal 20, and acquires the result of the user's judgment on the suitability of the annotated data from the user terminal 20. For example, when the user receives image data to which a bounding box indicating a human face region is added from the user interface unit 140, the user checks the position and / or size of the bounding box and judges whether the added bounding box properly surrounds the human face region captured in the image data. When the user judges that the bounding box is properly added, the user notifies the user interface unit 140 that the annotation process is proper. On the other hand, when the user judges that the bounding box is not properly added, the user may correct the annotated image data, add a bounding box that surrounds the human face region, and return the corrected image data to the user interface unit 140.

[0042] When the user interface unit 140 is notified by the user terminal 20 that the annotation process was appropriate, the user interface unit 140 adopts the annotated image data as training data and stores the annotated image data in the training data DB 60 for the later model. On the other hand, when the annotation process is inappropriate and corrected image data is acquired from the user terminal 20, the user interface unit 140 adopts the image data corrected by the user as training data and stores the corrected image data in both the training data DB 50 for the early model and the training data DB 60 for the later model. That is, the user interface unit 140 accepts image data in which annotations have been corrected for the image data provided to the user terminal 20 as feedback, and the corrected image data is used for training the early annotation model 30 and the later annotation model 40. In addition, image data for which positive feedback has been received for the image data provided to the user terminal 20 is used for training the later annotation model 40. Here, only image data corrected by the user is stored in the training data DB50 for the initial model, and appropriately annotated image data is not stored in the training data DB50 for the initial model, in order to avoid overlearning of the initial annotation model 30 by training using appropriately annotated image data.

[0043] The early annotation model 30 and the late annotation model 40 can be appropriately trained using training data stored in the early model training data DB 50 and the late model training data DB 60, respectively. Note that the fully trained late annotation model 40 may be used not only for annotation processing of the training data, but also to replace an operational machine learning model.

[0044] 7 and 8 are schematic diagrams illustrating a data generation support process based on the number of data acquisitions according to an embodiment of the present disclosure. In the illustrated embodiment, the model selection unit 120 determines whether to use the initial annotation model 30 or the later annotation model 40 for the annotation process based on the number of data to be annotated.

[0045] In an early stage where the number of acquired data to be annotated is less than a predetermined threshold, the model selection unit 120 selects the initial annotation model 30 as the annotation model to be used in the annotation process, as shown in Fig. 7. Here, the predetermined threshold for the number of acquired data may be determined based on the number of training data required for the later annotation model 40 to achieve a predetermined accuracy.

[0046] Here, the predetermined accuracy may be set so that the work time required for the user to correct the annotated data generated using the data generation support device 100 is shorter than the work time required for the user to obtain the annotated data without using the data generation support device 100. In other words, the predetermined accuracy may be an accuracy that allows the user to feel the benefit of using the data generation support device 100. In addition, the predetermined threshold for the number of data acquisitions may be determined based on the number of training data that can make the accuracy of the initial annotation model 30 and the later annotation model 40 equal.

[0047] As shown in the figure, when data to be annotated is received, the annotation processing unit 130 generates annotated data from the received data to be annotated, using the initial annotation model 30 selected by the model selection unit 120. If the user determines that the annotated data is appropriate, the user interface unit 140 stores the annotated data in the training data DB 60 for the later model, and if the user determines that the annotated data is inappropriate and modifies the annotation, the user interface unit 140 stores the data modified by the user in both the training data DB 50 for the early model and the training data DB 60 for the later model.

[0048] Then, when the number of collected data reaches or exceeds a predetermined threshold value as the annotation process progresses, the data generation support process transitions to a later stage shown in Fig. 8, where the model selection unit 120 switches the annotation model from the initial annotation model 30 to the later annotation model 40, and as shown in the figure, the annotation processing unit 130 uses the later annotation model 40 to generate annotated data from the received data to be annotated. If the user determines that the annotated data is appropriate, the user interface unit 140 stores the annotated data in the training data DB 60 for the later model, and if the user determines that the annotated data is inappropriate and modifies the annotation, the user interface unit 140 stores the data modified by the user in the training data DB 60 for the later model.

[0049] In one embodiment, the predetermined index is related to the detection accuracy or evaluation index of the annotation process, and the model selection unit 120 may select the initial annotation model 30 when the detection accuracy or evaluation index of the initial annotation model 30 is higher than that of the later annotation model 40, and may select the later annotation model 40 when the detection accuracy or evaluation index of the initial annotation model 30 is lower than that of the later annotation model 40. Here, the accuracy of the annotation process may be calculated based on detection accuracy such as TP (True Positive), FP (False Positive), FN (False Negative) and / or TN (True Negative). Alternatively, the accuracy of the annotation process may be calculated based on evaluation index such as IoU (Intersection over Union) and / or mAP (mean Average Precision).

[0050] In the above-described embodiment, a single initial annotation model 30 is used in the initial stage, but the present disclosure may use a plurality of different initial annotation models 30. For example, two types of annotation models, a publicly available annotation model (hereinafter referred to as initial annotation model candidate #1) and an annotation model based on an algorithm (e.g., Few-shot Learning, etc.) suitable for a small number of training data (hereinafter referred to as initial annotation model candidate #2), may be used as the initial annotation model 30. Typically, the initial annotation model candidate #1 has already been trained by the operator of the public model, and when the data generation support device 100 uses the initial annotation model candidate #1, training data for training the initial annotation model candidate #1 is not required. On the other hand, the initial annotation model candidate #2 needs to be trained using training data stored in the initial model training data DB 50, but annotation processing with a certain degree of accuracy can be realized with a smaller amount of training data than training the later annotation model 40.

[0051] In this way, when a plurality of different initial annotation models 30 are used, the model selection unit 120 may switch the annotation model at a switching timing as shown in FIG. 9. That is, as shown in the figure, the model selection unit 120 initially selects the initial annotation model candidate #1, and the annotation processing unit 130 executes the annotation process using the initial annotation model candidate #1. Thereafter, as the collection of training data progresses and the initial annotation model candidate #2 and the later annotation model 40 are trained by the collected training data, a time may come when the accuracy of the initial annotation model candidate #2 exceeds the accuracy of the initial annotation model candidate #1. The model selection unit 120 may execute switching from the initial annotation model candidate #1 to the initial annotation model candidate #2 at this timing as shown in FIG. 9.

[0052] As the collection of training data progresses further and the initial annotation model candidate #2 and the later annotation model 40 are trained using the collected training data, a time may come when the accuracy of the later annotation model 40 exceeds the accuracy of the initial annotation model candidate #2. The model selection unit 120 may switch from the initial annotation model candidate #2 to the later annotation model 40 at this time.

[0053] [Data generation support processing] Next, a data generation assistance process according to an embodiment of the present disclosure will be described. The data generation assistance process may be executed by the data generation assistance device 100, and more specifically, may be realized by one or more processors 104 of the data generation assistance device 100 executing one or more programs or instructions stored in one or more memory devices 103. Fig. 10 is a flowchart showing an early-stage data generation assistance process according to an embodiment of the present disclosure, and Fig. 11 is a flowchart showing a later-stage data generation assistance process according to an embodiment of the present disclosure.

[0054] First, the initial stage of the data generation support process will be described with reference to FIG.

[0055] 10, in step S101, the data generation assistance device 100 acquires data to be annotated. For example, in an annotation process for adding a bounding box to a human face region in image data, the data to be annotated may be image data in which a human face is captured.

[0056] In step S102, the data generation support device 100 initially trains the initial annotation model 30 and the later annotation model 40. For example, the data generation support device 100 may initially train the initial annotation model 30 and the later annotation model 40 using training data initially stored in the initial model training data DB 50 and the later model training data DB 60. Note that the initial annotation model 30 and the later annotation model 40 do not necessarily need to be trained by the data generation support device 100, and may be trained by, for example, an external server.

[0057] In step S103, the data generation support device 100 compares the accuracy of the initial annotation model 30 and the later annotation model 40. If the accuracy of the initial annotation model 30 is higher than that of the later annotation model 40 (S103: Yes), the data generation support process proceeds to step S104, and on the other hand, if the accuracy of the initial annotation model 30 is equal to or lower than that of the later annotation model 40 (S103: No), the data generation support process proceeds to step S105 and may proceed to a later stage described later.

[0058] In step S104, the data generation support device 100 uses the initial annotation model 30 to add annotations to the data to be annotated. For example, the data generation support device 100 may add a bounding box to a human face area in the image data to be annotated. Then, the data generation support device 100 provides the annotated data to the user terminal 20.

[0059] In step S106, the data generation support device 100 acquires a judgment result of the suitability of the annotated data from the user terminal 20. If feedback is acquired from the user terminal 20 that the annotated data is appropriate and the annotation does not need to be corrected (S106: No), the data generation support process proceeds to step S108. On the other hand, if feedback is acquired from the user terminal 20 that the annotated data is not appropriate and the annotation needs to be corrected (S106: Yes), the data generation support process proceeds to step S107.

[0060] In step S107, the data generation support device 100 acquires the data with the corrected annotation from the user terminal 20. For example, the user may add an appropriate bounding box to a human face region in the image data to be annotated by deleting, resizing, moving, or the like, the annotation added by the initial annotation model 30 in step S104.

[0061] In step S108, the data generation support device 100 stores the data annotated by the initial annotation model 30 and the data with the corrected annotations acquired from the user terminal 20 in the training data DB 60 for the later model.

[0062] In step S109, the data generation support device 100 stores the data with the corrected annotations acquired from the user terminal 20 in the initial model training data DB 50. Note that in order to avoid overlearning of the initial annotation model 30, data that has been appropriately annotated by the initial annotation model 30 is not stored in the initial model training data DB 50.

[0063] Next, the later stage of the data generation support process will be described with reference to FIG.

[0064] As shown in FIG. 11, in step S201, the data generation support device 100 switches the annotation model used in the annotation process from the initial annotation model 30 to the later annotation model 40.

[0065] In step S202, the data generation assistance device 100 acquires data to be annotated. As in step S101, for example, in an annotation process for adding a bounding box to a human face region in image data, the data to be annotated may be image data in which a human face is captured.

[0066] In step S203, the data generation support device 100 uses the later annotation model 40 to add annotations to the data to be annotated. For example, the data generation support device 100 may add a bounding box to a human face region in the image data to be annotated. Then, the data generation support device 100 provides the annotated data to the user terminal 20.

[0067] In step S204, the data generation support device 100 acquires the suitability judgment result of the annotated data from the user terminal 20. If feedback is acquired from the user terminal 20 that the annotated data is appropriate and the annotation does not need to be corrected (S204: No), the data generation support process ends. On the other hand, if feedback is acquired from the user terminal 20 that the annotated data is not appropriate and the annotation needs to be corrected (S204: Yes), the data generation support process proceeds to step S205.

[0068] In step S205, the data generation support device 100 acquires the data with the corrected annotation from the user terminal 20. For example, the user may add an appropriate bounding box to a human face area in the image data to be annotated by deleting, resizing, moving, or the like, the annotation added by the later annotation model 40 in step S204.

[0069] In step S206, the data generation support device 100 stores the data annotated by the later annotation model 40 and the data with corrected annotations acquired from the user terminal 20 in the later model training data DB 60.

[0070] According to the above-described embodiment, in MLOps, when data for updating or retraining an operational machine learning model is collected, in the early stage of data collection when the amount of collected data is not sufficient to obtain an annotation model with good accuracy, the data generation support device 100 performs annotation processing using the initial annotation model 30 that can achieve a certain degree of accuracy even in the early stage when the amount of data is not sufficient. After that, when the amount of data necessary to achieve good accuracy is collected, the data generation support device 100 performs annotation processing using the later annotation model 40.

[0071] In addition, the following supplementary notes are provided in relation to the above description. (Appendix 1) A data acquisition unit that acquires data to be annotated; a model selection unit that selects an annotation model from a first machine learning model and a second machine learning model based on a predetermined index; an annotation processing unit that executes annotation processing by using the selected annotation model; a user interface unit that provides annotated data generated by the annotation process; A data generation support device having the above configuration. (Appendix 2) 2. The data generation assistance device according to claim 1, wherein the data to be annotated is image data. (Appendix 3) The predetermined index is related to the number of data to be annotated, The data generation support device described in Appendix 1 or 2, wherein the number of data to be annotated required to achieve annotation processing with a specified accuracy is different for the first machine learning model and the second machine learning model. (Appendix 4) The data generation support device of any one of appendices 1 to 3, wherein when trained with a number of data to be annotated that is equal to or greater than a predetermined threshold, the second machine learning model realizes the annotation processing with higher accuracy than the first machine learning model. (Appendix 5) The data generation assistance device according to claim 4, wherein the model selection unit selects the first machine learning model when the number of data to be annotated is less than the predetermined threshold, and selects the second machine learning model when the number of data to be annotated is equal to or greater than the predetermined threshold. (Appendix 6) The data generation assistance device of any one of Appendices 3 to 5, wherein the specified accuracy is set so that the work time required for a user to correct annotated data generated using the data generation assistance device is shorter than the work time required for the user to obtain data annotated by the user without using the data generation assistance device. (Appendix 7) The predetermined index is related to a detection accuracy or an evaluation index, The data generation support device of any one of Appendices 1 to 6, wherein the model selection unit selects the first machine learning model when a detection accuracy or evaluation index of the first machine learning model is higher than a detection accuracy or evaluation index of the second machine learning model, and selects the second machine learning model when the detection accuracy or evaluation index of the first machine learning model is lower than the detection accuracy or evaluation index of the second machine learning model. (Appendix 8) 8. The data generation assistance device according to any one of claims 1 to 7, wherein the user interface unit receives feedback on the provided data. (Appendix 9) the user interface unit receives, as the feedback, data corrected with respect to the provided data; The data generation assistance device of claim 8, wherein the corrected data is used to train the first machine learning model and the second machine learning model. (Appendix 10) The data generation assistance device according to claim 8 or 9, wherein data that receives positive feedback regarding the provided data is used to train the second machine learning model. (Appendix 11) Obtaining data to be annotated; selecting an annotation model from the first machine learning model and the second machine learning model based on a predetermined metric; performing an annotation process using the selected annotation model; and providing data generated by the annotation process; A computer-implemented data generation assistance method comprising: (Appendix 12) Obtaining data to be annotated; selecting an annotation model from the first machine learning model and the second machine learning model based on a predetermined metric; performing an annotation process using the selected annotation model; and providing data generated by the annotation process; A program that causes a computer to execute the following.

[0072] Although the examples of the present disclosure have been described in detail above, the present disclosure is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present disclosure described in the claims. [Explanation of symbols]

[0073] 10 Data Generation Support System 20 User terminal 30 Initial annotation model 40 Late annotation model 50 Training data DB for initial models 60 Training data DB for later models 100 Data generation support device 110 Data Acquisition Section 120 Model Selection Section 130 Annotation Processing Unit 140 User Interface Section

Claims

1. A data acquisition unit that acquires data to be annotated; a model selection unit that selects an annotation model from the first machine learning model and the second machine learning model based on a predetermined index; an annotation processing unit that executes annotation processing by using the selected annotation model; a user interface unit that provides annotated data generated by the annotation process; A data generation support device having the above configuration.

2. The data generation assistance device according to claim 1 , wherein the data to be annotated is image data.

3. The predetermined index is related to the number of data to be annotated, The data generation assistance device according to claim 1 , wherein the number of data to be annotated required to realize annotation processing with a predetermined accuracy differs between the first machine learning model and the second machine learning model.

4. The data generation assistance device according to claim 3 , wherein when trained with a number of data to be annotated equal to or greater than a predetermined threshold, the second machine learning model realizes the annotation process with higher accuracy than the first machine learning model.

5. 5. The data generation assistance device according to claim 4, wherein the model selection unit selects the first machine learning model when the number of acquired data to be annotated is less than the predetermined threshold, and selects the second machine learning model when the number of acquired data to be annotated is equal to or greater than the predetermined threshold.

6. 4. The data generation assistance device according to claim 3, wherein the specified accuracy is set so that the work time required for a user to correct annotated data generated using the data generation assistance device is shorter than the work time required for the user to obtain data annotated by the user without using the data generation assistance device.

7. The predetermined index is related to a detection accuracy or an evaluation index, 2. The data generation support device according to claim 1, wherein the model selection unit selects the first machine learning model when a detection accuracy or evaluation index of the first machine learning model is higher than a detection accuracy or evaluation index of the second machine learning model, and selects the second machine learning model when the detection accuracy or evaluation index of the first machine learning model is lower than the detection accuracy or evaluation index of the second machine learning model.

8. The data generation assistance device according to claim 1 , wherein the user interface unit receives feedback on the provided data.

9. the user interface unit receives, as the feedback, data corrected with respect to the provided data; The data generation assistance device according to claim 8 , wherein the corrected data is used to train the first machine learning model and the second machine learning model.

10. The data generation assistance device according to claim 8 , wherein the data for which positive feedback has been received regarding the provided data is used for training the second machine learning model.

11. Obtaining data to be annotated; selecting an annotation model from the first machine learning model and the second machine learning model based on a predetermined metric; performing an annotation process using the selected annotation model; and providing data generated by the annotation process; A computer-implemented data generation assistance method comprising:

12. Obtaining data to be annotated; selecting an annotation model from the first machine learning model and the second machine learning model based on a predetermined metric; performing an annotation process using the selected annotation model; and providing data generated by the annotation process; A program that causes a computer to execute the following.

Images