Annotation device, method, program, and annotation system

Abstract

To increase the efficiency of automatic annotation.SOLUTION: An annotation device according to one embodiment of the present invention comprises: an inference unit which, by using a general-purpose model that, when an image is inputted, outputs at least some of regions of the image and a label of the region, infers the region and the label from the image; an annotation unit that assigns the label to the region of the image and performs annotation in accordance with an indication to correct at least one of the region and the label; and a model generation unit that generates a model specialized in annotation by using an image for which the annotation was performed from among the inputted images.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an annotation device, a method, a program, and an annotation system. [Background technology]

[0002] It has been known that at least a partial region of an image (e.g., a rectangle representing an object in the case of object detection, or the entire image in the case of image classification) and information about that region can be inferred from the image using a machine learning model. To improve the accuracy of such machine learning models, annotation is sometimes performed on the training data (i.e., labels are assigned to at least a partial region of the image). In addition to methods in which annotation is performed manually by humans, methods in which annotation is performed by computers (also known as automatic annotation) are also known. Summary of the Invention [Problem to be solved by the invention]

[0003] However, automatic annotation using general-purpose machine learning models often fails to infer the correct regions and labels, requiring manual correction by humans, making automatic annotation inefficient.

[0004] Therefore, an object of the present invention is to improve the efficiency of automatic annotation. [Means for solving the problem]

[0005] An annotation device according to one embodiment of the present invention comprises an inference unit that uses a general-purpose model that outputs at least a portion of an area of ​​the image and a label for that area when the image is input, to infer the area and the label from the image; an annotation unit that performs annotation by assigning the label to the area of ​​the image in accordance with instructions to modify at least one of the area and the label; and a model generation unit that generates a model specialized for annotation using the annotated image from the input images. [Effects of the Invention]

[0006] According to the present invention, the efficiency of automatic annotation can be improved. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram for explaining an overview of automatic annotation. [Figure 2] 1 is a diagram illustrating an overall configuration according to an embodiment of the present invention. [Figure 3] 1 is a diagram illustrating a hardware configuration of an annotation device according to an embodiment of the present invention. [Figure 4] 1 is a functional block diagram of an annotation system according to an embodiment of the present invention. [Figure 5] FIG. 10 is a sequence diagram illustrating annotation processing according to an embodiment of the present invention. [Figure 6] FIG. 10 is a sequence diagram illustrating annotation processing according to an embodiment of the present invention. [Figure 7] FIG. 10 is a sequence diagram illustrating annotation processing according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram illustrating annotations of a behavior estimation model according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] <Summary> Figure 1 is a diagram for explaining an overview of automatic annotation. In automatic annotation, information about objects in images (a group of images or a video) (specifically, a rectangle indicating the area of ​​the object, the object's label, and reliability) is inferred from the images (a group of images or a video) using a general-purpose model (in the example of Figure 1, an object detection model is illustrated). If the object information is not inferred correctly, a human performs manual annotation (i.e., corrects the information about the object that was not inferred correctly). A dataset is generated and stored from the images (a group of images or a video) and the object information.

[0010] Conventionally, an object detection model is trained (retrained) using the dataset generated as described above and the dataset used to generate a general-purpose model (object detection model), and the retrained object detection model is then used for automatic annotation. However, training an object detection model requires a high-performance server, and the more datasets there are, the longer the training time becomes. In other words, there is a trade-off between improving model accuracy and reducing training time.

[0011] Therefore, we train an object detection model for automatic annotation using only datasets generated with images from the target environment or situation.

[0012] In this case, the model will be overtrained and specialized for the target environment or situation, which will reduce the accuracy of normal object detection, but this is not a problem for automatic annotation purposes.Finally, once the generation of a dataset for the target environment or situation is complete, training can be performed using the dataset used to generate the general-purpose model, making it possible to train a model that can make inferences with high accuracy even for normal object detection.

[0013] <Overall structure> 2 is a diagram showing the overall configuration of an embodiment of the present invention. The annotation system 1 can include an annotation device 10 and a user terminal 20. The annotation device 10 and the user terminal 20 are connected to each other so as to be able to communicate with each other via an arbitrary network.

[0014] <<Annotation device>> The annotation device (for example, a server) 10 is a device that annotates learning data for a model (for example, an object detection model, an image classification model). The annotation device 10 is one or more computers.

[0015] Specifically, when an image is input, the annotation device 10 uses a general-purpose model that outputs at least a partial region of the image (for example, a rectangle indicating an object in the case of object detection, or the entire image in the case of image classification) and a label (also called a tag or metadata) for the region, and infers at least a partial region of the image and a label for the region from the image. The annotation device 10 also performs annotation by assigning a label to the region of the image in accordance with an instruction to modify at least one of the region and the label (specifically, an instruction input by the user 21 to the user terminal 20). The annotation device 10 also generates a model specialized for annotation using annotated images from the images input to the general-purpose model.

[0016] [Model] Here, the model will be described.

[0017] A general-purpose model is not a model specialized for a specific field, but is a model that can be used in a variety of fields. A general-purpose model is a model that, when an image is input, outputs at least a partial region of the image (not limited to a rectangle, but may be a polygon, segmentation, keypoint, or the entire image) and information about the region (i.e., a label). For example, a general-purpose model is a general-purpose object detection model, image classification model, etc.

[0018] Annotation-specialized models are suitable for annotating the training data for the general-purpose models described above.

[0019] <<User terminal>> The user terminal 20 is a terminal (for example, a personal computer, a smartphone, etc.) operated by a user.

[0020] Although the annotation device 10 and the user terminal 20 have been described as separate devices in this specification, the annotation device 10 and the user terminal 20 may be implemented as a single device.

[0021] The devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, the annotation device (e.g., server) 10 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0022] <Hardware configuration> 3 is a hardware configuration diagram of an annotation device (for example, a server) 10 according to an embodiment of the present invention. The same applies to the user terminal 20.

[0023] As shown in Figure 3, the annotation device 10 is constructed by a computer, and as shown in Figure 3, it is equipped with a CPU 1001, a ROM 1002, a RAM 1003, a HD 1004, an HDD (Hard Disk Drive) controller 1005, a display 1006, an external device connection I / F (Interface) 1007, a network I / F 1008, a data bus 1009, a keyboard 1010, a pointing device 1011, a DVD-RW (Digital Versatile Disk Rewritable) drive 1013, and a media I / F 1015.

[0024] Of these, the CPU 1001 controls the overall operation of the annotation device 10. The ROM 1002 stores programs, such as IPL, used to drive the CPU 1001. The RAM 1003 is used as a work area for the CPU 1001. The HD 1004 stores various data, such as programs. The HDD controller 1005 controls the reading and writing of various data from and to the HD 1004 under the control of the CPU 1001. The display 1006 displays various information, such as a cursor, menu, window, text, or image. The external device connection I / F 1007 is an interface for connecting various external devices. In this case, the external devices are, for example, USB (Universal Serial Bus) memories, printers, etc. The network I / F 1008 is an interface for data communication using a communication network. The bus line 1009 is an address bus, data bus, etc. for electrically connecting the components, such as the CPU 1001, shown in FIG. 3.

[0025] The keyboard 1010 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 1011 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 1013 controls reading and writing of various data from a DVD-RW 1012, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 1015 controls reading and writing (storing) of data from a recording medium 1014, such as a flash memory.

[0026] <Functional configuration> FIG. 4 is a functional block diagram of the annotation system 1 according to one embodiment of the present invention.

[0027] The annotation device 10 can include an annotation processing unit 100 (including an inference unit 101 and an annotation unit 102), a dataset generation unit 103, a model generation unit 104, a model storage unit 105, a content storage unit 106, and a dataset storage unit 107. The annotation device 10 can function as the inference unit 101, the annotation unit 102, the dataset generation unit 103, and the model generation unit 104 by executing a program.

[0028] The user terminal 20 can include a UI (user interface) unit 201. The user terminal 20 can function as the UI unit 201 by executing a program.

[0029] <<Annotation device>> The inference unit 101 uses a general-purpose model that, when an image is input, outputs at least a partial area of ​​the image and a label for that area, to infer at least a partial area of ​​the image and a label for that area from the image stored in the content storage unit 106.

[0030] Specifically, the inference unit 101 inputs an image into a general-purpose model, and outputs at least a partial region of the image and a label for the region. Note that the reliability may also be output together with the region and label.

[0031] The annotation unit 102 performs annotation by assigning the label to the region of the image in accordance with an instruction to modify at least one of the region of the image and the label of the region.

[0032] Specifically, the annotation unit 102 displays the result of the inference by the inference unit 101 (at least a partial region of the image and a label for that region) on the user terminal 20. The annotation unit 102 acquires at least one of a correction of the region and a correction of the label input by the user 21 to the user terminal 20 (when the reliability is low, etc.). The annotation unit 102 assigns a label to at least a partial region of the image and performs annotation in accordance with the instruction for correction acquired from the user terminal 20.

[0033] If the user 21 has not made any corrections, the annotation unit 102 performs annotation by assigning the label inferred by the inference unit 101 to the region inferred by the inference unit 101.

[0034] The dataset generation unit 103 stores the images stored in the content storage unit 106 and the information on the regions and labels annotated by the annotation unit 102 as a set in the dataset storage unit 107. Note that the dataset may include not only datasets corrected by the user 21, but also datasets that were not corrected by the user 21.

[0035] The model generation unit 104 generates a model specialized for annotations using annotated images from among the images input to the general-purpose model. The model generation unit 104 performs training using only annotated images (i.e., the dataset in the dataset storage unit 107) to generate a model specialized for annotations.

[0036] For example, the model generation unit 104 generates a model specialized for annotations (that is, generates a new model) by learning using only images that have been annotated.

[0037] For example, the model generation unit 104 generates a model specialized for annotations by retraining a general-purpose model using only images that have been annotated.

[0038] [When generating an annotation-specific model according to instructions from the user device] For example, when the accuracy of the model exceeds a threshold, the user terminal 20 is notified. For example, when instructed by the user terminal 20, the model generation unit 104 generates a model specialized for the annotation. Note that the threshold may be set by the user 21. This will be described in detail with reference to FIG. 6.

[0039] [When generating annotation-specific models automatically] For example, when the accuracy of the model exceeds a threshold, the model generation unit 104 generates a model specialized for the annotation. Note that the threshold may be set by the user 21. This will be described in detail with reference to FIG. 7.

[0040] The accuracy of the model is, for example, the number of images (or the number of objects) corrected by the user 21 / the total number of images (or the number of objects).

[0041] The model storage unit 105 stores a general-purpose model and a model specialized for annotations.

[0042] Images (still images or moving images) of any object are stored in the content storage unit 106. In the case of moving images, all frames may be stored, or sampled images may be stored.

[0043] The dataset storage unit 107 stores the dataset generated by the dataset generation unit 103. For example, the dataset storage unit 107 stores "reference information to images stored in the content storage unit 106," "rectangles and labels indicating objects in the images," and "reliability."

[0044] [Annotation-specialized model] The inference unit 101 uses a model specialized for annotation to infer from the image at least a partial region of the image and a label for that region, and the annotation unit 102 performs annotation by assigning the label to the region of the image in accordance with an instruction to correct at least one of the partial region of the image and the label for that region. Note that if no correction is made by the user 21, the annotation unit 102 performs annotation by assigning the label inferred by the inference unit 101 to the region inferred by the inference unit 101.

[0045] [General-purpose model] The generic model is retrained using the annotated image and an image other than the annotated image (the image used to generate the generic model), and the retrained generic model is used to infer from the image at least some regions of the image and the labels of those regions.

[0046] <<User terminal>> The UI unit 201 provides a user interface. Specifically, the UI unit 201 acquires and displays various data from the annotation device 10.

[0047] <Method> FIG. 5 is a sequence diagram showing annotation processing according to one embodiment of the present invention.

[0048] In step 101 (S101), the user 21 inputs into the user terminal 20 an instruction to register content.

[0049] In step 102 (S102), the user terminal 20 stores the content designated by the user 21 in the content storage unit 106 of the annotation device 10.

[0050] In step 103 (S103), the user 21 inputs into the user terminal 20 an instruction to execute automatic annotation.

[0051] In step 104 (S104), the user terminal 20 requests the annotation device 10 (the inference unit 101) to perform automatic annotation. Specifically, the user terminal 20 transmits to the inference unit 101 information on the model and content for which automatic annotation is desired to be performed.

[0052] In step 105 (S105), the inference unit 101 acquires from the model storage unit 105 a model for executing the automatic annotation requested by the user terminal 20 in S104.

[0053] In step 106 (S106), the inference unit 101 acquires from the content storage unit 106 the content (image) for which automatic annotation is to be performed, as requested by the user terminal 20 in S104.

[0054] In step 107 (S107), the inference unit 101 infers at least a partial region of the image and the label of that region by inputting the image acquired in S106 into the model acquired in S105, and stores the inference results.

[0055] In step 108 (S108), the user 21 inputs an instruction to the user terminal 20 requesting that the inference result be displayed.

[0056] In step 109 (S109), the user terminal 20 acquires the inference result from the annotation device 10.

[0057] In step 110 (S110), the user terminal 20 displays the inference result acquired in S109.

[0058] If the inference result needs to be corrected, steps 111 and 112 are executed.

[0059] In step 111 (S111), the user 21 inputs into the user terminal 20 at least one of a correction to the area and a correction to the label.

[0060] In step 112 (S112), the user terminal 20 transmits the corrections input in S111 to the annotation device 10 (annotation unit 102). Thereafter, the annotation unit 102 performs annotation by assigning labels to regions of the image in accordance with the correction instructions.

[0061] In step 113 (S113), the user 21 inputs to the user terminal 20 that the confirmation of the inference result has been completed.

[0062] In step 114 (S114), the user terminal 20 requests the annotation device 10 (data set generation unit 103) to store the inference result and the correction.

[0063] In step 115 (S115), the data set generation unit 103 stores the data set in the data set storage unit 107.

[0064] S108 to S115 are repeated.

[0065] In step 116 (S116), the user 21 inputs an instruction to the user terminal 20 requesting training of a model specialized for annotations.

[0066] In step 117 (S117), the user terminal 20 requests the annotation device 10 (model generation unit 104) to train a model specialized for annotation. Specifically, the user terminal 20 transmits information about the dataset to the model generation unit 104.

[0067] In step 118 (S118), the model generation unit 104 acquires the dataset stored in the dataset storage unit 107.

[0068] In step 119 (S119), the model generation unit 104 performs training using only images that have been annotated, and generates a model specialized for annotations.

[0069] In step 120 (S120), the model generation unit 104 stores the trained model (the model specialized for annotations) in the model storage unit 105.

[0070] 6 is a sequence diagram showing annotation processing according to an embodiment of the present invention [when an annotation specialized model is generated in response to an instruction from a user terminal].

[0071] In step 201 (S201), the user 21 inputs into the user terminal 20 an instruction to register content.

[0072] In step 202 (S202), the user terminal 20 stores the content designated by the user 21 in the content storage unit 106 of the annotation device 10.

[0073] In step 203 (S203), the user 21 inputs into the user terminal 20 an instruction to execute automatic annotation.

[0074] In step 204 (S204), the user terminal 20 requests the annotation device 10 (the inference unit 101) to perform automatic annotation. Specifically, the user terminal 20 transmits to the inference unit 101 information on the model and content for which automatic annotation is desired to be performed.

[0075] In step 205 (S205), the inference unit 101 acquires from the model storage unit 105 a model for executing the automatic annotation requested by the user terminal 20 in S204.

[0076] In step 206 (S206), the inference unit 101 acquires from the content storage unit 106 the content (image) for which automatic annotation is to be performed, as requested by the user terminal 20 in S204.

[0077] In step 207 (S207), the inference unit 101 infers at least a partial region of the image and the label of that region by inputting the image acquired in S206 into the model acquired in S205, and stores the inference result.

[0078] In step 208 (S208), the user 21 inputs an instruction to the user terminal 20 requesting that the inference result be displayed.

[0079] In step 209 (S209), the user terminal 20 acquires the inference result from the annotation device 10.

[0080] In step 210 (S210), the user terminal 20 displays the inference result acquired in S109.

[0081] If the inference result needs to be corrected, steps 211 and 212 are executed.

[0082] In step 211 (S211), the user 21 inputs into the user terminal 20 at least one of a correction to the area and a correction to the label.

[0083] In step 212 (S212), the user terminal 20 transmits the corrections input in S211 to the annotation device 10 (annotation unit 102). Thereafter, the annotation unit 102 performs annotation by assigning labels to regions of the image in accordance with the correction instructions.

[0084] In step 213 (S213), the user 21 inputs to the user terminal 20 that the confirmation of the inference result has been completed.

[0085] In step 214 (S214), the user terminal 20 requests the annotation device 10 (data set generation unit 103) to store the inference result and the correction.

[0086] In step 215 (S215), the data set generation unit 103 stores the data set in the data set storage unit 107.

[0087] In step 216 (S216) and step 217 (S217), the data set generator 103 determines whether the accuracy of the model (e.g., the number of images (or the number of objects) modified by the user 21 / the total number of images (or the number of objects)) exceeds a threshold. If the accuracy of the model exceeds the threshold, steps 218 and 219 are executed.

[0088] In step 218 (S218), the data set generation unit 103 notifies the user terminal 20 that the accuracy of the model exceeds the threshold.

[0089] In step 219 (S219), the user terminal 20 displays a notification that the accuracy of the model exceeds the threshold (that is, a notification that training of a model specialized for annotations is required).

[0090] In step 220 (S220), the user 21 inputs an instruction to the user terminal 20 requesting training of a model specialized for annotations.

[0091] In step 221 (S221), the user terminal 20 requests the annotation device 10 (model generation unit 104) to train a model specialized for annotation. Specifically, the user terminal 20 transmits information about the dataset to the model generation unit 104.

[0092] In step 222 (S222), the model generation unit 104 acquires the dataset stored in the dataset storage unit 107.

[0093] In step 223 (S223), the model generation unit 104 performs training using only annotated images to generate a model specialized for annotations.

[0094] In step 224 (S224), the model generation unit 104 stores the trained model (model specialized for annotation) in the model storage unit 105. After that, automatic annotation is performed using the model specialized for annotation.

[0095] S208 to S224 are repeated.

[0096] 7 is a sequence diagram showing annotation processing according to an embodiment of the present invention [when an annotation specialization model is automatically generated].

[0097] In step 301 (S301), the user 21 inputs into the user terminal 20 an instruction to register content.

[0098] In step 302 (S302), the user terminal 20 stores the content designated by the user 21 in the content storage unit 106 of the annotation device 10.

[0099] In step 303 (S303), the user 21 inputs into the user terminal 20 an instruction to execute automatic annotation.

[0100] In step 304 (S304), the user terminal 20 requests the annotation device 10 (the inference unit 101) to perform automatic annotation. Specifically, the user terminal 20 transmits to the inference unit 101 information on the model and content for which automatic annotation is desired to be performed.

[0101] In step 305 (S305), the inference unit 101 acquires from the model storage unit 105 a model for executing the automatic annotation requested by the user terminal 20 in S304.

[0102] In step 306 (S306), the inference unit 101 acquires from the content storage unit 106 the content (image) for which automatic annotation is to be performed, as requested by the user terminal 20 in S304.

[0103] In step 307 (S307), the inference unit 101 infers at least a partial area of ​​the image and the label of that area by inputting the image acquired in S306 into the model acquired in S305, and stores the inference results.

[0104] In step 308 (S308), the user 21 inputs an instruction to the user terminal 20 requesting that the inference result be displayed.

[0105] In step 309 (S309), the user terminal 20 acquires the inference result from the annotation device 10.

[0106] In step 310 (S310), the user terminal 20 displays the inference result acquired in S309.

[0107] If the inference result needs to be corrected, steps 311 and 312 are executed.

[0108] In step 311 (S311), the user 21 inputs into the user terminal 20 at least one of a correction to the area and a correction to the label.

[0109] In step 312 (S312), the user terminal 20 transmits the corrections input in S311 to the annotation device 10 (annotation unit 102). Thereafter, the annotation unit 102 performs annotation by assigning labels to regions of the image in accordance with the correction instructions.

[0110] In step 313 (S313), the user 21 inputs to the user terminal 20 that the confirmation of the inference result has been completed.

[0111] In step 314 (S314), the user terminal 20 requests the annotation device 10 (data set generation unit 103) to store the inference result and the correction.

[0112] In step 315 (S315), the data set generation unit 103 stores the data set in the data set storage unit 107.

[0113] In step 316 (S316) and step 317 (S317), the data set generation unit 103 determines whether the accuracy of the model (for example, the number of images (or the number of objects) modified by the user 21 / the total number of images (or the number of objects)) exceeds a threshold. If the accuracy of the model exceeds the threshold, steps 318 to 325 are executed.

[0114] In step 318 (S318), the data set generation unit 103 requests the model generation unit 104 to train a model specialized for annotations.

[0115] In step 319 (S319), the model generation unit 104 acquires the dataset stored in the dataset storage unit 107.

[0116] In step 320 (S320), the model generation unit 104 performs training using only annotated images to generate a model specialized for annotations.

[0117] In step 321 (S321), the model generation unit 104 stores the trained model (a model specialized for annotations) in the model storage unit 105.

[0118] Automatic annotation is then performed using a model specialized for annotation.

[0119] In step 322 (S322), the dataset generation unit 103 requests the inference unit 101 to perform automatic annotation. Specifically, the dataset generation unit 103 transmits to the inference unit 101 information on the model and content for which automatic annotation is desired to be performed (i.e., information on the content for which the user 21 has not yet confirmed the inference results).

[0120] In step 323 (S323), the inference unit 101 acquires from the model storage unit 105 a model for executing the automatic annotation requested by the data set generation unit 103 in S322.

[0121] In step 324 (S324), the inference unit 101 acquires from the content storage unit 106 the content (image) for which automatic annotation is to be performed, as requested by the data set generation unit 103 in S322.

[0122] In step 325 (S325), the inference unit 101 infers at least a partial region of the image and the label of that region by inputting the image acquired in S324 into the model acquired in S323, and stores the inference result.

[0123] S308~S325 are repeated.

[0124] 8 is a diagram illustrating annotation of an activity estimation model according to one embodiment of the present invention. The present invention may be used not only for automatic annotation of an object detection model, but also for automatic annotation of a posture estimation model (specifically, a model that outputs the posture of a body in an image (skeleton information) when an image is input), an activity estimation model (specifically, a model that classifies an activity performed in an image when an image is input), etc.

[0125] Since there are often no general-purpose models for behavior estimation models, humans perform annotation manually for the first few times, train the behavior estimation model using that data, and then perform automatic annotation using the trained behavior estimation model.From then on, the process follows the same flow as automatic annotation for object detection models.

[0126] <Effects> In this way, in one embodiment of the present invention, a model specialized for annotation is generated using only correctly annotated images, thereby reducing the need for human correction of annotations and increasing the efficiency of automatic annotation.

[0127] Therefore, even in special environments or situations (such as detecting hands wearing gloves (which may vary in color depending on the factory or assembly process) and inferring hand posture) where general-purpose models (such as general-purpose hand detection models or models that infer hand posture) would require a lot of manual annotation corrections, automatic annotation can create highly accurate learning data.

[0128] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each function described above. [Explanation of symbols]

[0129] 1. Annotation System 10 Annotation Device 20 User terminal 100 Annotation Processing Unit 101 Reasoning section 102 Annotation section 103 Dataset Generation Unit 104 Model Generation Unit 105 Model Memory Unit 106 Content storage unit 107 Dataset Storage 201 UI section [Prior art documents] [Patent documents]

[0130] [Patent Document 1] Japanese Patent Application Publication No. 2022-553732

Claims

1. an inference unit that infers a region and a label from an image using a general-purpose model that outputs, when an image is input, at least a partial region of the image and a label of the region; an annotation unit that performs annotation by assigning the label to the region of the image in accordance with an instruction to modify at least one of the region and the label; a model generation unit that generates a model specialized for the annotation using the annotated image from among the input images; An annotation device comprising:

2. The annotation device according to claim 1 , wherein the model generation unit generates a model specialized for the annotation by learning using only the images on which the annotation has been made.

3. The annotation device according to claim 1 , wherein the model generation unit generates a model specialized for the annotation by re-learning the general-purpose model using only the images on which the annotation has been performed.

4. The annotation device according to claim 1 , wherein when the accuracy of the model exceeds a threshold, a notification is sent to a user terminal.

5. The annotation device according to claim 4 , wherein the model generation unit generates a model specialized for the annotation when instructed by the user terminal.

6. The annotation device according to claim 1 , wherein the model generation unit generates a model specialized for the annotation when accuracy of the model exceeds a threshold.

7. The annotation device according to claim 4 , wherein the threshold value is set by a user.

8. The annotation device according to claim 1 , wherein the inference unit infers the region and the label from the image using a model specialized for the annotation.

9. The generic model is retrained using the annotated image and an image other than the annotated image; The annotation device of claim 1 , wherein the retrained generic model is used to infer the regions and the labels from the images.

10. A method performed by an annotation device, comprising: Inferring regions and labels from an image using a generic model that receives an image and outputs at least a portion of a region of the image and a label of the region; annotating the region of the image by assigning the label to the region in accordance with an instruction to modify at least one of the region and the label; generating a model specific to the annotation using the annotated image from among the input images; A method comprising:

11. The annotation device A process of inferring a region and a label from an image using a general-purpose model that, when an image is input, outputs at least a partial region of the image and a label of the region; a process of annotating the region of the image by assigning the label to the region in accordance with an instruction to modify at least one of the region and the label; A process of generating a model specialized for annotations using the annotated images from among the input images. A program to execute.

12. An annotation system including an annotation device and a user terminal, The annotation device an inference unit that uses a general-purpose model that outputs, when an image is input, at least a partial region of the image and a label of the region, to infer the region and the label from the image, and presents the inference result on the user terminal; an annotation unit that performs annotation by assigning the label to the region of the image in accordance with an instruction from the user terminal to modify at least one of the region and the label; a model generation unit that generates a model specialized for the annotation using the annotated image from among the input images; An annotation system equipped with

Images