Annotation support device, method, and program

Abstract

An annotation support device, annotation support method, and program for efficiently creating high-quality training data for AI learning are provided. [Solution] The annotation support device of the present invention provides centralized support for annotation work management by comprising a recording unit that records project information, work target data, worker information, and annotation results, an output unit that outputs specifications and standard operating procedures, an execution unit that automatically evaluates the quality of workers' work and determines whether the work can be completed, an allocation unit that automatically allocates appropriate data to workers, and an output unit that aggregates and visualizes progress and quality. This reduces the labor required for complex management work in large-scale annotation projects, suppresses variations in work quality, and enables efficient and highly accurate data creation.

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for supporting annotation work in creating training data used in machine learning and the like, and a program for causing a computer to execute the method. [Background technology]

[0002] In recent years, with the evolution of AI models such as discriminative AI and generative AI, the importance of "data-centric AI development," which improves the quality of the training data used by AI to learn, has increased, rather than "model-centric AI development," which improves the performance of the model itself.

[0003] In this type of data-centric AI development, the task of creating training data, i.e., annotation, accounts for more than 80% of the overall work, and is a major bottleneck in the operational phase of AI development. Traditionally, annotation has been performed using open-source annotation tools or cloud-based products. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2022 / 185363 [Non-patent literature]

[0005] [Non-Patent Document 1] Open source annotation tool CVAT, URL https: / / github.com / cvat-ai / cvat (Retrieved August 12, 2025) [Non-patent document 2] From Model-centric to Data-centric AI,URL https: / / www.youtube.com / watch?v=06-AZXmwHjo (searched on August 12, 2025) [Non-patent document 3] Mohammad Hossein Jarrahi, Ali Memariani, Shion Guha, The Principles of Data-Centric AI (DCAI), Communications of the ACM, Volume 66, Issue 8Pages 84 - 92, July 25, 2023 Summary of the Invention [Problem to be solved by the invention]

[0006] However, conventional annotation tools and cloud services were designed primarily for annotating small amounts of data during the Proof of Concept (PoC) stage, and were not suitable for annotating large amounts of data, resulting in issues such as a decline in data quality and delays in the process.

[0007] Furthermore, to efficiently carry out large-scale annotation work, it is essential that the work be divided among multiple people. However, conventional tools lack the functionality and mechanisms to handle multiple workers, which leads to complicated communication and operation processes, a decline in quality due to work variations, and operation that is dependent on the individual, resulting in issues with scalability.

[0008] To solve the above problems, projects are being carried out by combining multiple tools, such as using Excel or spreadsheets for work management, various chat tools for communication, and PowerPoint or Word for creating specifications. However, information and work locations are scattered, resulting in operational costs and communication gaps, leading to reduced quality and work delays. In particular, when annotating training data required for computer vision, efficient work instructions, inspection, and information sharing for unstructured data such as images, videos, and point cloud data are extremely important.

[0009] To solve the above issues, traditionally, projects have been carried out by combining multiple tools, such as using Excel or spreadsheets for work management, various chat tools for communication, and PowerPoint or Word for creating specifications.

[0010] However, these tools were independent of each other, and the location of information and work was dispersed, which increased operational costs and created communication gaps, resulting in reduced quality and work delays.

[0011] In particular, when annotating training data in the field of computer vision, it is extremely important to provide efficient work instructions, inspection, and information sharing for unstructured data such as images, videos, and point cloud data.

[0012] Furthermore, some existing annotation support systems have a function to support collaborative work.

[0013] However, in machine learning technology and related industries, emphasis is still placed on the Proof of Concept (PoC) and research and development phases, and there are only a limited number of cases where AI development has reached the operational stage. As a result, existing systems are limited to supporting annotation work on a certain scale and are insufficient for comprehensively managing the entire project.

[0014] Specifically, there has not previously been a system for integrated management of the entire annotation project process, including requirements definition, specification creation, annotator training, quality control, and progress management, and for realizing continuous, large-scale annotation work.

[0015] Therefore, the present invention aims to provide an annotation support device, method, and program that can comprehensively manage processes such as quality, progress, allocation, and inspection when multiple people are performing large-scale annotation work, and can create high-quality and efficient training data. [Means for solving the problem]

[0016] An annotation support device for supporting annotation work according to a first embodiment of the present invention includes: a recording unit that records at least project information, work target data, worker information, and annotation results; a first output unit that outputs specifications and standard operating procedures based on the project information; a first execution unit that evaluates the quality of the work performed by the worker based on the worker information and a plurality of sample annotation results and determines whether or not the work can be started; a first allocation unit that allocates work target data to workers based on a determination result by the first execution unit; a second output unit that outputs a user interface that enables annotation work to be performed on the work target data; a second allocation unit that samples the annotation results after the annotation work is completed on a lot-by-lot basis to extract an inspection object, and allocates a next worker based on the inspection result of the inspection object; a second execution unit that aggregates work progress and work quality; a third output unit that visualizes the aggregated results; A fourth output unit that outputs a thumbnail-format user interface that displays a list of annotation results. Equipped with The second execution unit executes a re-education process for the worker when the defect rate of the inspection target is equal to or greater than a specified value.

[0017] An annotation support device according to a second embodiment of the present invention is an annotation support device according to the first embodiment, characterized in that the re-training process is executed in a format that presents feedback based on a comparison of the sample annotation results performed by the worker with correct data.

[0018] An annotation support device according to a third embodiment of the present invention is an annotation support device according to the first or second embodiment, characterized in that the re-training process presents optimized sample data using past evaluation history when a worker who has previously failed onboarding is onboarded again.

[0019] An annotation support device according to a fourth embodiment of the present invention is an annotation support device according to any of the first to third embodiments, characterized in that the first execution unit has a function of calculating a score by comparing correct data with the annotation result and determining whether a worker is permitted to work based on the score.

[0020] An annotation support device according to a fifth embodiment of the present invention is an annotation support device according to the first to fourth embodiments, characterized in that the first allocation unit has a function of allocating the work target data based on the priority of the work target data and the load status of the worker.

[0021] An annotation support device according to a sixth embodiment of the present invention is an annotation support device according to any of the first to fifth embodiments, characterized in that the second execution unit has a function of automatically sampling the annotation results on a lot-by-lot basis and performing quality evaluation.

[0022] An annotation support device according to a seventh embodiment of the present invention is an annotation support device according to any of the first to sixth embodiments, characterized in that the second output unit has the function of receiving input of inspection results, updating the approval status, and notifying the next worker.

[0023] An annotation support device according to an eighth embodiment of the present invention is an annotation support device according to any of the first to seventh embodiments, characterized in that the third output unit has a function of visually displaying work progress and quality information aggregated by worker or annotation class.

[0024] An annotation support device according to a ninth embodiment of the present invention is an annotation support device according to any of the first to eighth embodiments, characterized in that the first output unit has a function of outputting specifications and standard operating procedures as an editable screen in Markdown format.

[0025] An annotation support device according to a tenth embodiment of the present invention is an annotation support device according to any of the first to ninth embodiments, characterized in that the recording unit records the work history of each worker and makes it available for re-evaluation or re-training by the first execution unit.

[0026] A method for supporting annotation work according to an eleventh embodiment of the present invention includes: A step of recording at least project information, work target data, worker information, and annotation results; outputting specifications and standard operating procedures based on the project information; evaluating the quality of the work performed by the worker based on the worker information and a plurality of sample annotation results, and determining whether or not the work can be started; assigning work target data to the worker based on the determination result; outputting a user interface that enables annotation work to be performed on the work target data; a step of sampling the annotation results after the annotation work is completed on a lot-by-lot basis to extract an inspection object, and assigning the next worker based on the inspection result of the inspection object; a step of aggregating work progress and work quality; A step of visualizing the aggregated results; a step of outputting a user interface in thumbnail format that displays a list of annotation results; Including, If the defect rate of the inspection target is equal to or greater than a specified value, a re-education process for the worker is executed.

[0027] A method for supporting annotation work according to a twelfth embodiment of the present invention is a method for supporting annotation work according to the eleventh embodiment, characterized in that the re-training process is carried out in a form that presents feedback based on a comparison with correct data for the sample annotation results performed by the worker.

[0028] A method for supporting annotation work according to a thirteenth embodiment of the present invention is a method for supporting annotation work according to the eleventh or twelfth embodiment, characterized in that the re-training process presents optimized sample data using past evaluation history when a worker who has previously failed onboarding is onboarded again.

[0029] A method for supporting annotation work according to a fourteenth embodiment of the present invention is a method for supporting annotation work according to the eleventh to thirteenth embodiments, characterized in that the step of evaluating the quality of the worker includes a step of calculating a score by comparing correct data with the annotation result and determining the worker's work permission based on the score.

[0030] A method for supporting annotation work according to a fifteenth embodiment of the present invention is a method for supporting annotation work according to the eleventh to fourteenth embodiments, characterized in that the step of allocating the work target data is executed based on the priority of the work target data and the worker's workload.

[0031] A method for supporting annotation work according to a sixteenth embodiment of the present invention is a method for supporting annotation work according to any of the eleventh to fifteenth embodiments, characterized in that it further includes a step of sampling the annotation results on a lot-by-lot basis and performing quality evaluation.

[0032] A method for supporting annotation work according to a seventeenth embodiment of the present invention is a method for supporting annotation work according to any of the eleventh to sixteenth embodiments, characterized in that it includes steps of accepting input of inspection results, updating the approval status, and notifying the next worker.

[0033] A method for supporting annotation work according to an 18th embodiment of the present invention is a method for supporting annotation work according to any of the 11th to 17th embodiments, characterized in that it further includes a step of recording and visualizing the work progress, quality, and work history of each worker.

[0034] A program according to a nineteenth embodiment of the present invention is characterized in that it causes a computer to execute the method for supporting annotation work according to any one of the eleventh to eighteenth embodiments. [Effects of the Invention]

[0035] This invention enables both quality control and work efficiency in annotation work. First, by comparing the correct data with the work results, the quality of the work can be quantitatively evaluated, allowing for an objective understanding of the capabilities of the workers and reducing human variation. Furthermore, by automatically assigning the data to be worked on to the workers based on the evaluation results, it is possible to distribute the workload and reduce management costs.

[0036] Furthermore, efficient and standardized quality control is achieved by automatically sampling and inspecting the quality of each lot based on the annotation results. In addition, by updating the approval status based on the inspection results and automating notifications to the next process worker, delays in the work process and communication efforts can be reduced.

[0037] Furthermore, this invention makes it possible to visualize work progress and quality information on a dashboard, making it easy to grasp the status of the entire project or individual workers and to detect abnormalities.In addition, specifications and standard operating procedures can be managed centrally in Markdown format, etc., making it possible to standardize work rules and quickly train new members.

[0038] In addition, the work history of each worker is recorded, which can be used for continuous evaluation and retraining, and the intuitive user interface that displays annotation results in thumbnail format improves the efficiency and visibility of inspection work.

[0039] These features enable cloud-based and web-based operation, enabling scalable and efficient operation even for large-scale annotation projects. As a result, it is possible to realize a data creation system on a scale that goes beyond the PoC level and is effective in eliminating bottlenecks in AI development as a whole. [Brief explanation of the drawings]

[0040] [Figure 1] 1 is a functional block diagram showing the overall configuration of a system including an annotation support device according to an embodiment of the present invention. [Figure 2] 1 is a schematic configuration diagram of a computer system that functions as an annotation support device according to an embodiment of the present invention. [Figure 3] 10 is a flowchart showing the overall processing flow in the annotation support device. [Figure 4] 10 is a flowchart showing the flow of a project setting process. [Figure 5] 10 is a flowchart showing the flow of a process for evaluating the work quality of a worker. [Figure 6] 10 is a flowchart showing the flow of an automatic assignment process. [Figure 7] 10 is a flowchart showing the flow of a process for lot-based quality control. [Figure 8]10 is a flowchart showing the flow of sampling processing. [Figure 9] 10 is a flowchart showing the flow of inspection work processing. [Figure 10] 10 is a flowchart showing the flow of a process for visualizing work progress and quality. [Figure 11] FIG. 10 is a diagram illustrating an example of annotation work data. [Figure 12] FIG. 10 is a diagram showing an example of member data in annotation work. [Figure 13] FIG. 10 is a diagram illustrating an example of production plan data for annotation work. [Figure 14] FIG. 10 illustrates an example of a user interface that displays the project progress status. [Figure 15] FIG. 10 is a diagram illustrating an example of a user interface that displays progress status by user and annotation class. [Figure 16] FIG. 10 illustrates an example of a user interface that displays the project progress status. [Figure 17] FIG. 10 is a diagram showing an example of a user interface that displays an automatic determination (setting screen) of activity quality. [Figure 18] FIG. 10 is a diagram illustrating an example of a user interface that displays a screen showing the results of the judgment of the quality of work. [Figure 19] FIG. 10 is a diagram showing an example of a user interface that displays automatic determination and comparison with correct data on the work screen. [Figure 20] This figure shows an example of a user interface that displays a comment and marking screen for efficiently performing inspections on the cloud. [Figure 21] FIG. 10 is a diagram showing an example of a user interface that displays thumbnails for each image, allowing efficient inspection of a large amount of annotation data. [Figure 22] FIG. 10 is a diagram showing an example of a user interface that displays thumbnails for each annotation, allowing efficient inspection of a large amount of annotation data. [Figure 23]FIG. 1 is a diagram illustrating an overview of random sampling processing. DETAILED DESCRIPTION OF THE INVENTION

[0041] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will now be described with reference to the accompanying drawings. The individual embodiments of the present invention are not independent and can be appropriately combined with each other for implementation.

[0042] 1 is a functional block diagram showing the overall configuration of a system including an annotation support device according to one embodiment of the present invention. The system for supporting annotation work comprises terminals 1-1 to 1-N (N is a natural number) used by users, and a computer system 2 connected to these terminals 1-1 to 1-N via a communication network CN.

[0043] The terminals 1-1 to 1-N are devices used by users who perform annotation work, and include, for example, multi-function mobile phones (so-called smartphones), tablets, notebook computers, desktop computers, etc. These terminals are collectively referred to as terminals 1.

[0044] The computer system 2 functions as an annotation support device by executing the annotation support program of the present invention. For example, it is used or managed by an administrator of an organization (including a company) that manages annotation target data. The computer system 2 executes processing in response to a request from the terminal 1 and provides the results to the terminal 1. The computer system 2 may be configured as a single computer or may be configured as multiple computers. An example configured as a single computer will be described below.

[0045] Fig. 2 is a diagram showing the schematic configuration of a computer system that functions as an annotation support device according to one embodiment of the present invention. As shown in Fig. 2, the computer system 2 includes an input interface 21, a communication module 22, a storage device 23, a memory 24, an output interface 25, and a processor 26. The input interface 21 accepts operational inputs from an administrator of the computer system 2 and outputs signals corresponding to the accepted inputs to the processor 26. The communication module 22 is connected to a communication line network CN and performs data communication with the terminal 1. Note that this communication may be wired or wireless, but the present embodiment will be described assuming a wired connection.

[0046] The storage device 23 is, for example, a storage device, and stores various data and programs that are read and executed by the processor 26. The memory 24 is a storage area for temporarily storing data and programs, and is configured as a volatile memory, for example, RAM (Random Access Memory). The output interface 25 enables connection to an external device and has the function of outputting signals to the external device. The processor 26 loads a program stored in the storage device 23 into the memory 24 and executes a series of instructions contained in the program, thereby operating as the following functional blocks: That is, a first output unit 261, a first execution unit 262, a first allocation unit 263, a second output unit 264, a second allocation unit 265, a second execution unit 266, a third output unit 267, a fourth output unit 268, and a recording unit 269.

[0047] The system is equipped with a first output unit 261 that displays the overall specifications and work procedure manual, a first execution unit 262 that refers to the recording unit 269 and automatically evaluates the work quality, a first allocation unit 263 that refers to the recording unit 269 and automatically assigns a worker who passes the evaluation, a second output unit 264 that displays an annotation work screen, a second allocation unit 265 that refers to the recording unit 269 after the work and automatically assigns the next worker, a second execution unit 266 that compiles and records the overall work progress and quality status, a third output unit 267 that visualizes the recorded work progress and quality status, a fourth output unit 268 that displays annotation data in a thumbnail UI (user interface), and a recording unit 269 that saves various settings and transaction data such as project information, members, approval workflow settings, production plans, annotation work target data, work progress, and annotation results.

[0048] Fig. 3 is a flowchart showing the overall processing flow in the annotation support device. The overall flow involves setting up a project, allocating lots to the data to be worked on, and performing annotation on a lot-by-lot basis until all lots are completed. Fig. 3 shows the overall processing procedure for performing annotation work, showing a series of steps from the initial project setup to annotation execution and review. First, prior to annotation work, input of basic information about the project, such as the project name, type of data to be worked on, work scope, person in charge, deadline, specifications, and work procedure, is accepted (step S101).

[0049] Next, annotation work begins and continues (step S102) until annotation processing for all specified data is completed (steps S102 to S105). Work allocation, quality checks, progress updates, etc. are also included in this phase.

[0050] The annotation support device (computer system 2) receives the results of annotations such as label information, bounding boxes, and classification attributes made by the workers (users who perform annotation work) via the terminals 1-1 to 1-N for each target data (step S103). Annotations can be made manually or semi-automatically using an auxiliary tool.

[0051] After the worker has completed the annotation work, the annotation support device has a reviewer check the quality of each piece of data and suggest corrections, and records the review information (approval / rejection / comments, etc.) in the recording unit 269 (step S104).

[0052] Based on the review results, annotations are corrected or added, and a loop process may be implemented to improve quality (step S105).Then, when annotation and review of all data is completed and the data is finally approved and finalized, the project is terminated.

[0053] FIG. 4 is a flowchart showing the flow of the project setting process. In the project setting (step S101), after creating a new project, documents and manuals such as specifications and standard operating procedures are uploaded, working members are added, annotation classes and production plans are set, the data to be annotated is uploaded, sample annotation is performed, and the data is used to evaluate the work quality of the planned workers and perform onboarding. This flowchart shows the preparation phase in the annotation support device, from the initial project setup to the worker quality evaluation. The contents of each step (S201 to S208) are explained in order below. Each step (S201 to S208) is executed by the annotation support device, which accepts input from terminals 1-1 to 1-N via the communication line network CN. These steps are preparatory work for smoothly and with high quality in the subsequent annotation work.

[0054] The annotation support device receives input of basic meta information of the project, such as the project name, purpose, schedule, target data type, and person in charge, and registers it in the recording unit 269 (step S201). At this time, the first output unit 261 of the annotation support device can display the overall specifications and work procedure manual.

[0055] The annotation support device receives a document (specification / standard operating procedure) that describes the definition of the annotation target, class design, attribute information, input rules, work flow, etc., and registers it in the recording unit 269. The document is input in Markdown format or by file upload (step S202).

[0056] The annotation support device receives input of user account information of the worker (member) who will perform the annotation work, and registers it in the recording unit 269. This information includes the role (annotator / inspector / manager, etc.), affiliation, skill information, etc. (step S203).

[0057] The annotation support device receives an input of an annotation class, which is a class for annotations registered by the user (step S204). For the annotation class, for example, a classification name such as "person," "vehicle," or "building," a hierarchical structure, a display color, and the like can be set.

[0058] The annotation support device receives input of a production plan, such as the amount of data to be processed within the project period, lot size, work schedule, priority, and inspection standards, and sets the production plan (step S205).

[0059] The annotation support device receives input of annotation target data such as images, audio, and text to be actually annotated (step S206). Methods for inputting annotation target data include file upload and linking with external storage.

[0060] The annotation support device performs an annotation test based on sample data for the registered worker, and receives input of annotations (sample annotations) made by the worker on the annotation work data for work quality evaluation (step S207). The sample annotations become reference data to be used for subsequent worker evaluation, training, judgment, etc.

[0061] Finally, the annotation support device (first execution unit 262) evaluates the quality of the sample annotations of registered workers based on the degree of agreement between the content of the sample annotations and the correct data (e.g., IoU (Intersection over Union), which is an index showing the degree to which two areas overlap) (step S208). Only workers whose scores exceed a certain standard can proceed to the actual work. In this way, when steps S201 to S208 are completed, the device is ready to proceed to the annotation work phase.

[0062] 5 is a flowchart showing the process flow for evaluating the quality of a worker's work. This flowchart shows a series of processes in which the annotation support device (for example, the first execution unit 262) evaluates the accuracy of sample annotations made by a worker and grants work authority based on the results. In the evaluation of the worker's work quality (onboarding), the correct answer data to be used in the evaluation is registered, a worker is assigned, the worker performs annotation, and the correct answer data is compared with the work data to automatically calculate a score, thereby determining whether the worker's work quality is pass or fail, and only workers who pass are allowed to work.

[0063] The annotation support device compares the annotation data entered by the worker (user) with the pre-registered correct answer data and calculates the score using IoU (step S301). This process is performed automatically, and the data is tallied at the granularity of individual labels, data sets, etc.

[0064] The annotation support device determines whether the score calculated in step S301 is equal to or greater than the standard score specified by the administrator or project settings (step S302). If the standard is not met ("No" in S302), the worker is reevaluated and can be retested or return to the training phase.

[0065] If the score exceeds the reference value ("Yes" in S302), the annotation support device grants "work permission authority" to the user information (step S303), which allows the user to participate in the actual annotation work.

[0066] Figure 6 is a flowchart showing the flow of the automatic assignment process. This flowchart shows the series of processes that occur when a worker logs in to the annotation system, from when the annotation support device dynamically extracts and assigns unassigned data to when the work screen is displayed. After the worker onboarding is complete, each worker logs in to the application and presses a button to start work. The unassigned data is automatically retrieved and assigned based on priority and displayed on the work screen.

[0067] The annotation support apparatus acquires currently unassigned annotation target data for a logged-in worker from the recording unit 269 (step S401). The data to be acquired is often limited to data associated with a project or a class.

[0068] The annotation support device automatically selects a task from the unassigned annotation target data acquired in step S401 based on the priority information (step S402). For example, tasks with upcoming deadlines or strict quality standards are given priority.

[0069] The annotation support device checks the number of jobs (number of tasks) already assigned to the worker and determines whether the upper limit of the job size set by the administrator or worker (user) has been reached (step S403). If the upper limit has not been reached ("Yes" in S403), the device returns to S401 and attempts to select an additional task. If the upper limit has been reached ("No" in S403), the device proceeds to the next process.

[0070] The annotation support device (first assignment unit 263) assigns a worker to the selected task as a person in charge (step S404). The assignment result is registered in the recording unit 269 and becomes a target for future progress and quality management.

[0071] The annotation support device (second output unit 264) generates and displays an annotation work screen based on the assigned work content (step S405). The target data, guidelines, tool UI, etc. are displayed on the annotation work screen, and the worker can begin specific input work. The worker is now ready to start work, and the series of automatic assignment processes is completed.

[0072] The automatic assignment shown in Figure 6 represents one of the features of this invention: "priority-based dynamic allocation" and "load balancing through job size control." This is an important process for achieving both efficiency and fairness in annotation work.

[0073] 7 is a flowchart showing the flow of lot-based quality control processing. This flowchart shows the flow of work management and quality inspection / improvement processing on a lot-by-lot basis by the annotation support device.

[0074] The annotation support device automatically determines and records the number of data items per lot and the classification unit based on the project settings (step S501). A lot is the smallest unit of work, inspection, and delivery.

[0075] The annotation support device divides the entire data set into lots and sets the data structure so that it can assign lots to workers and manage inspection targets (step S502). The workers perform annotation work on the assigned lots, and the annotation support device receives the results and stores them in the recording unit 269 (step S503).

[0076] The annotation support device randomly extracts a certain percentage of the data in the submitted lot and presents it to the inspector (step S504), and also records the inspection results (pass / fail, comments, corrections, etc.).

[0077] The annotation support device automatically calculates the defect rate based on the inspected data and determines whether it exceeds a specified value set by the manager (step S505). If the defect rate exceeds the specified value ("Yes" in step S505), the annotation support device presents suggestions for correction of the specifications and standard operating procedures to the manager to prevent recurrence, and records and manages them in the recording unit 269 as an editing history (step S506). In addition, the annotation support device automatically implements a reevaluation and re-education process (onboarding) for the workers of the corresponding lot according to the work quality (step S507).

[0078] If it is determined that the defect rate is less than the specified value ("No" in step S505), the lot is approved as having been inspected, and the annotation support device executes delivery processing (attaching a delivery flag, external collaboration processing, etc.) (step S508). Then, quality control and approval processing for each lot are completed, and processing for that lot ends.

[0079] The lot-based quality control process shown in Figure 7 is one example of how annotation operations that meet the quality standards of the present invention can be realized, and the annotation support device can perform integrated control of lot management, automatic inspection, and dynamic training.

[0080] Figure 8 is a flowchart showing the flow of the sampling process. This flowchart shows a series of processing flows from when the annotation support device acquires the data to be sampled, to when the data is extracted and displayed based on the specified sampling conditions. In this processing flow, the submitted annotation data can be automatically sampled and inspected based on the lot information assigned in advance.

[0081] To perform inspection and quality check, the annotation support device acquires a group of data to be sampled from the annotated data stored in the recording unit 269 (step S601). The target can be selected based on conditions such as a specific lot, worker, or class.

[0082] The annotation support device extracts target data by filtering based on a predefined sampling size (e.g., 10% of the lot) (step S602). The filtering may include random extraction, extraction in order of importance, edge case detection, etc.

[0083] Based on the extracted sampling data, the annotation support device (second output unit 264) displays a work screen for the inspector (step S603). This screen includes a thumbnail display of the target data, label information, an input field for comments for inspection, and a UI for selecting approval / rejection. Then, the necessary sampling display process is completed, and the inspector is ready to begin inspection work.

[0084] The sampling process shown in FIG. 8 corresponds to support for efficient sampling inspection as part of the lot quality control of the present invention, and can contribute to reducing the burden on both workers and managers and maintaining quality.

[0085] Figure 9 is a flowchart showing the flow of inspection work processing. This flowchart shows how the annotation support device controls the inspection process and automates the flow transition according to the inspection results and notifications to the person in charge of the next process. During the inspection process, the device accepts comments and markings on the screen from each user in charge of the inspection work, as well as input of the inspection results, automatically advances the approval workflow, and automatically notifies each person in charge via email and various communication tools.

[0086] The annotation support device (second output unit 264) displays the work results, including the annotation data to be inspected (e.g., images, bounding boxes, attributes, etc.), to the inspector in a list format or individual format on a confirmation screen (step S701).

[0087] The annotation support device receives input from an inspector (step S702). The input includes judgment results such as "pass," "requires correction," or "fail," as well as correction instructions, comments, markings, etc. The input contents are saved in the recording unit 269 and are also used for history management.

[0088] The annotation support device automatically updates the approval status (e.g., "approved," "returned," "re-inspection," etc.) for the relevant data or lot based on the inspection results received in step S702 (step S703). The updated results are also reflected on the management screen and dashboard.

[0089] After updating the approval status, the annotation support device (second assignment unit 265) determines the user in charge of the next process (e.g., correction person, approver, delivery person, etc.) and automatically sends a notification (step S704). Notification methods include email, app notification, chat integration (Slack or Teams), etc. Once the series of inspection processes and preparations for integration to the next process are complete, the processing flow ends. This process is one of the important functions that realizes efficient inspection work and automation of the approval process in the annotation support device, and can support a scalable and high-quality operational system.

[0090] Figure 10 is a flowchart showing the process flow for visualizing work progress and quality. This flowchart shows the series of processes in which the annotation support device acquires the work history of the target project, automatically aggregates the data along multiple analysis axes, and records the results. Regarding work progress and quality status, the device acquires the latest work information, aggregates it, and records it so that it can be displayed on the dashboard, triggered by a scheduled job or user input.

[0091] The annotation support device acquires work information for the target project stored in the recording unit 269 (step S801). This work information includes progress for each worker, the number of annotations, inspection results, scores related to quality evaluation (quality scores), and the like.

[0092] The annotation support device automatically aggregates the acquired work information by multiple criteria (aggregation axes), such as by user, by class, by lot, and by date (step S802). This allows immediate calculation of, for example, "Worker A's quality score this week" or "Class X defect rate."

[0093] The annotation support device (second execution unit 266) records the aggregated results in the recording unit 269 (step S803), which can then be used for subsequent dashboard display, CSV output, re-training triggers, etc. The results are also stored as history for comparison with past results and trend analysis. Through the above processing, basic data for visualizing and analyzing the operational status of the project is prepared. This processing flow is an important process that corresponds to the quantitative visualization of work status and automation of quality control in this invention, and is one of the core functions for improving the efficiency of large-scale annotation operations.

[0094] Fig. 11 shows an example of annotation work data, and the table shown in Fig. 11 shows an example of the recording structure of work data managed in the annotation support device. This table is an example of the configuration of a database table that records the processing status, person in charge, approval result, etc. of each data unit in the annotation work, and reflects the information stored in the recording unit 269 of the annotation support device.

[0095] "Identifier" is a unique name or ID for each piece of annotation target data (e.g., "Data 1") and is used to uniquely identify the processing target. "Data tag" indicates the name of the lot (unit of work) to which the target data belongs (e.g., "Lot 1", "Lot 2", etc.). It is used as a unit of inspection and delivery. "Annotator" indicates the ID or identification information of the worker who performed the annotation work on the data. "Reviewer" indicates the ID or identification information of the person who inspected (reviewed) the annotation results.

[0096] "Approver" indicates the ID or identification information of the person or system that determined the final status, such as approval or rejection, based on the inspection results. "Status" indicates the processing status of each piece of data (for example, "approved" or "rejected") and is used to determine whether subsequent processes can be carried out depending on whether approval has been granted. "Priority" indicates the processing priority of the target data (for example, "high," "medium," or "low") and is used for job allocation order and inspection priority.

[0097] The annotation support device can utilize the information recorded in tables such as those shown in Fig. 11 for, for example, progress management, quality control, job allocation optimization, ensuring traceability, and determining whether or not to deliver on a lot-by-lot basis. In progress management, the status column makes it possible to grasp the status of work in progress, approved, returned, etc. in real time. In quality control, it is possible to analyze the inspection results and return history for each reviewer and understand the tendencies of workers and reviewers. In job allocation optimization, it is possible to automatically assign high-priority data to workers on a preferential basis.

[0098] By linking and recording the annotators, reviewers, and approvers, it becomes possible to verify and improve quality at a later date, ensuring traceability. The logic for determining whether or not to deliver is based on whether all data within the same lot has been approved, making it possible to determine whether or not to deliver on a lot-by-lot basis.

[0099] This configuration enables the annotation support device to realize scalable and highly manageable large-scale annotation operations. The database is also linked to the front UI and dashboard, enabling real-time visualization of progress and quality.

[0100] Fig. 12 shows an example of member data for annotation work, and the table shown in Fig. 12 is an example of the configuration of a database table for recording and managing information, roles, test evaluation results, etc. of each user participating in a project. In other words, this table shows an example of a member database for managing basic information and work authority of each user involved in a project in an annotation support device.

[0101] "Identifier" is an ID or user name that uniquely identifies a user and is used for internal reference and log recording. "Name" indicates the user's display name or real name (hide in the example in Figure 12). "Email address" indicates the email address used for notification collaboration with the user and login authentication. "Permissions" indicates the user's role classification. Role classifications include, for example, "annotator" who performs annotation work, "reviewer" who performs inspection work, "approver" who makes approval decisions after review, and "owner" who oversees overall project management. "Test result" indicates the pass / fail result of a pre-skill check (onboarding test) conducted on workers (mainly annotators). A score above a certain level is considered "pass" and work permissions are enabled. This may be omitted for reviewers and above.

[0102] The annotation support device can control the UI display and the range of operations that can be performed according to the user's "authority" information based on the member information recorded in the table shown in Fig. 12, thereby realizing role-based authority control. In addition, the annotation support device can assign appropriate tasks, such as allowing annotators to access the annotation work screen and reviewers to access the inspection screen, thereby realizing work allocation and workflow control.

[0103] Furthermore, the annotation support system can check the annotator's prior test results ("pass" or "fail" in the table) and encourage retraining (retesting) as necessary, enabling skill checks and / or quality assurance. The annotation support system can also automatically send progress reports, return notifications, approval requests, etc. via email address, providing notification and alert functions. The user information table shown in Figure 12 provides basic information for improving project management efficiency, ensuring work quality, and realizing security control (access permission management).

[0104] Fig. 13 is a diagram showing an example of production plan data for annotation work. The table shown in Fig. 13 is an example of the configuration of a database table for recording and managing daily work plans and results of workers in annotation work, etc. In other words, this table shows an example of a production management database for recording and visualizing daily production plans and results in an annotation support device or project management system.

[0105] "Date" indicates the annotation work date or planned work date. "Plan" indicates the number of annotation work (or processes) planned for the relevant day. This is a number entered in advance as a production plan. "Results" indicates the number of work (annotations, inspections, etc.) actually completed on the relevant day.

[0106] Production plan data such as that shown in Figure 13 can be used for a variety of purposes, including daily progress monitoring, resource optimization, monthly / weekly report output, bottleneck detection, and quality-linked analysis. For example, the annotation support device can compare and analyze the degree of achievement against the daily plan, visualize progress management in real time, and enable daily progress monitoring. The annotation support device can also automatically adjust worker assignments and scheduling if actual results deviate from the plan, enabling resource optimization. Furthermore, the annotation support device can aggregate the "actual results" of the production plan data and display them in graphs or on a dashboard, enabling monthly / weekly report output.

[0107] The annotation support system can detect bottlenecks by utilizing production plan data, and if actual results consistently fall short of the plan, it triggers a review of the process and training system.The annotation support system also correlates and evaluates the number of actual results with the defect rate and return rate, making it possible to grasp the impact of excessive work and workload, and enabling quality-linked analysis.

[0108] The structure of the production planning data table shown in Figure 13 can also be expanded by project, worker, or lot, and by linking it with the dashboard function and visualization module of the annotation support device, management efficiency and transparency can be significantly improved.

[0109] Fig. 14 is a diagram showing an example of a user interface that displays the project progress status. The example screen of the project progress status shown in Fig. 14 is an example showing a visualization interface of the project progress in the third output unit 267 of the annotation support device, and is a dashboard function that allows the project manager and team members to grasp the work status in real time.

[0110] The "Start Annotation" button in the top group of buttons is a button that accepts an action for the annotator to start work, and the "Start Approval" button is a button that accepts an action for the reviewer or approver to enter the review / inspection phase. In the tab menu (for switching display modes), the "Progress" tab (selected in the example in Figure 14) displays the current overall progress. The "User Performance" tab displays the results and quality of each user so that they can be checked, and "Statistics" displays statistical information such as the number of tasks and distribution by category.

[0111] In the progress information overview panel, "Progress: 12%" indicates the current completion ratio of the total workload, "Submitted: 7" indicates the number of annotation data submitted by workers so far, "Approved: 4" indicates the number of submitted data that have been inspected and approved, and "Rejected: 0" indicates the number of data that were returned / rejected as a result of review (a quality indicator).

[0112] FIG. 15 is a diagram showing an example of a user interface that displays progress status by user and annotation class. FIG. 15 is an example of a progress status display screen (progress management dashboard) by user and annotation class, displayed by the third output unit 267 of the annotation support device. This screen can aggregate the number of new data, the number of submitted data, the number of approved data, etc. by worker or label class, and visualize the progress and quality status. The purpose is to grasp the work progress by task and class (category) in real time.

[0113] The "Task Progress" section (progress management by user) displays classification units such as "Assigned," "All," or "Unassigned," and can also display individual users. "Progress" indicates the percentage of completion (submitted + approved) of the entire task (12% in this example), "New" indicates the number of unprocessed or assigned data items that have not yet been started, "Submitted" indicates the number of items that have completed annotation and been submitted, "Approved" indicates the number of items that have been deemed approved by the reviewer, "Skipped" indicates the number of items that were skipped due to being excluded from processing or manual exclusion, "Rejected" indicates the number of items that were returned or rejected (a value that serves as an indicator of quality control), and "Total" indicates the total number of items assigned to the corresponding user or category. In the example in Figure 15, of the 33 items in the "Unassigned" status, 22 are new, 7 have been submitted, and 4 have been approved (a progress rate of 12%).

[0114] In the "Annotation Progress" section (progress by annotation class), "Name" indicates the annotation class name (e.g., traffic light, car, background, etc.) (equivalent to the classification label), "New" indicates the number of data in that class for which work has not yet begun, "Submitted" indicates the number of data for which annotation results have been submitted but not yet approved, "Approved" indicates the number of data in that class that has been approved, "Skipped / Rejected" indicates the number of data in that class that has been skipped or sent back, and "Total" indicates the total number of data for each class (e.g., 22 traffic lights, 34 cars, etc.). For example, in the "Car" class, there are 2 new data and 32 approved, for a total of 34 data. "Background" and "Green Stone" are also managed separately, making it possible to understand imbalances and delays in progress for each class.

[0115] The progress display screen by user and annotation class, as shown in Figure 15, allows task managers to see at a glance which users or classes are progressing or lagging behind. Furthermore, the approval rate / rejection rate can be used as a quality indicator to identify those who should be retrained and to determine resource reallocation. In this way, quantitative evaluation of project progress makes it possible to identify delivery delays and overproduction before they occur.

[0116] 16 shows an example of a user interface that displays the project progress status, and is a screen (dashboard) that visualizes quality information for each worker and overall, output by the third output unit 267 of the annotation support device. This screen displays the accuracy, rejection rate, average work time, etc. in chronological order, enabling management that contributes to re-training decisions and work allocation adjustments.

[0117] In the top tab menu (screen mode switch), of the "Progress" / "User Performance" / "Statistics" tabs, "User Performance" is selected in the example in Figure 16. This screen displays information on quality and efficiency, such as work accuracy, rejection rate, and work time. As work quality indicators (overall aggregate), "Accuracy" is a numerical value indicating the accuracy of the submitted annotation results (e.g., based on IoU comparison with the correct data), "Rejection Rate" is the percentage of submitted tasks that were "rejected" by the reviewer (a value that serves as an indicator of quality control), and the number of rejections is the cumulative number of rejected tasks. Note that in the example in Figure 16, all are displayed as 0% / 0, indicating that there is no track record yet.

[0118] In the accuracy trend graph (line graph) displayed at the top, the horizontal axis is the date (March 9th to March 20th, 2025) and the vertical axis is accuracy (%). The accuracy trend graph displays the daily change in accuracy as a line graph, and is used to check the results of an increase in the number of workers or training.

[0119] The average time / task trend graph displayed at the bottom plots the average time taken for each task (one data processing) by day, and can be used to determine the proficiency and efficiency of workers, as well as whether resources are insufficient or excessive. In the example in Figure 16, all times are "00:00:00", indicating that the work time has not yet been recorded.

[0120] In the quality list table by person in charge (bottom), "person in charge" is the identifier of the user who performed the work, "accuracy" is the average accuracy for each individual, "rejection rate / number of rejections" is the number of returned cases and their percentage, "average time / task" is the average time taken to complete each task, "average time / annotation" is the average work time per label, and "total time" is the total work time for the target period. Note that in the example in Figure 16, all items are in an initial state with no records, and the display is updated as records are accumulated.

[0121] The screen shown in Fig. 16, which visualizes quality information for each worker and overall, can be used to visualize and quantitatively evaluate work quality, as a trigger indicator for retraining and feedback, to extract highly accurate users, to detect excessive work or slowdowns in speed, and to reallocate resources and optimize scheduling.

[0122] 17 is an example of a user interface that displays the automatic determination (setting screen) of work quality, and shows a setting screen related to the evaluation of the quality of a worker's work executed by the first execution unit 262 of the annotation support device. On this screen, threshold information used for the automatic evaluation of work results and the work permission determination process can be flexibly adjusted by specifying an IoU threshold, pixel distance, pass score, etc. In onboarding (initial evaluation) or continuous quality evaluation, it is possible to set thresholds for automatically determining whether annotation results are correct or not and whether work permission is granted.

[0123] In the example settings screen shown in Figure 17, the "Settings" tab is selected among the "Results," "Tasks," and "Settings" tabs. Regarding the details of the settings, "Enable Onboarding" can be toggled on / off using a checkbox. When enabled, a preliminary quality assessment (test) is conducted on the worker, and work permission is automatically determined based on the results. The "IoU Threshold" (75% in the example in Figure 17) is a threshold for determining the accuracy of rectangles, polygons, segmentations, etc. based on "Intersection over Union." For example, a match of 75% or more is considered correct. The "Pixel Threshold" (15 pixels in the example in Figure 17) determines whether a keypoint or linear annotation is correct if the distance is within this value. The "Pass Score" (80% in the example in Figure 17) is the pass criterion for the entire task. If the accuracy rate for each annotation is 80% or higher, the entire task is deemed "passed." "Show correct annotations" (checked in the example in Figure 17) is a setting that visually displays pre-registered correct annotations when the onboarding results are displayed, and is useful for learning and feedback. The "Save" operation button at the bottom records the above settings and can be applied to the actual judgment process (onboarding and quality evaluation).

[0124] On the setting screen shown in Fig. 17, the annotation support device can control parameters to realize functions such as worker quality checks and automatic work permission determination (onboarding test), automatic scoring during continuous quality monitoring, quantitative pass / fail determination based on IoU·pixel difference, and educational support by displaying feedback (visualization of correct answers).

[0125] FIG. 18 is an example of a user interface that displays a work quality assessment result screen, showing the results of work quality automatically evaluated by the first execution unit 262 of the annotation support device. This screen allows the user to check the quality assessment status, number of passed tasks, and total number of tasks for each worker in a list format, providing information that contributes to work permission decisions, training management, and quality maintenance. The example quality assessment result screen shown in FIG. 18 displays a list of onboarding assessment results. Evaluation information that is mainly used to verify the skills of new workers and determine whether or not they can continue working is provided in a list format.

[0126] Of the "Results," "Tasks," and "Settings" tabs on the quality assessment results screen shown in Figure 18, the "Results" tab is selected in the example in Figure 18. The evaluation targets are users (annotators) who have taken or will take onboarding (work skill test), and the administrator can check the pass / fail results of the automatically assessed work quality score on a per-user basis.

[0127] In the "Results" section of the quality assessment results screen, "Responsible Person" is the identification name of the worker (annotator) being assessed, and is clickable to transition to a detailed screen, etc. "Status" indicates the current assessment status (e.g., "New" indicates that the test has not yet started or completed), "Number of Passed Tasks" indicates the number of work tasks that have been judged as passed (those that have received a score above the set pass score), and "Total Number of Tasks" indicates the total number of assessment tasks that the worker has received (the value that serves as the basis for calculating the progress rate and determining whether work is permitted). In the example display in Figure 18, all workers are in "New" status, which is the initial state in which assessment has not yet been carried out.

[0128] The quality assessment results screen shown in Figure 18 can be used for operational management purposes such as determining whether to grant work permission (pass / fail score), managing training status (extracting those who have not completed onboarding), recording performance on a task-by-task basis (comparing the number of passes and pass rates), and providing basic information for continuous quality assessment (basis for determining whether to reassess or retrain).

[0129] FIG. 19 is a diagram showing an example of a user interface that displays an automatic determination and a comparison with the correct data on the work screen. FIG. 19 shows how the degree of coincidence (IoU) between the correct data and the work data is automatically calculated and visually compared on the work screen in a work quality evaluation performed by the first execution unit 262 of the annotation support device. This screen allows the worker or manager to intuitively grasp the match score for each individual label, which can be used for work permission determination and feedback. This work screen is mainly used during onboarding evaluation and quality checks, and allows the degree of coincidence between the worker's annotations and the correct data (reference labels) to be visually and numerically confirmed.

[0130] In the annotation display area in the center of the work screen, two rectangular annotations are displayed superimposed on the image of the bird to be annotated. The two rectangular annotations are an annotation (virtual label) entered by the worker that does not enclose the lower part (tail) of the bird, and a correct annotation (reference label) registered in the annotation support device that encloses the entire bird (including the lower part (tail)). The annotation support device automatically calculates an accuracy score based on the degree of overlap between the two (IoU: Intersection over Union).

[0131] "Score: 100%" is displayed in the upper right corner of the work screen, indicating that the degree of coincidence of the annotations in this task has been determined to be 100%. "IoU: 84.8477" is displayed within the annotation frame, clearly indicating that the IoU score between the target labels is approximately 84.8%. In this way, the annotation support device (first execution unit 262) can automatically calculate the IoU between the correct answer data and the work data to calculate the score. In addition, on the work screen, workers and project managers can check the comparison between the correct answer data and the work data using a GUI.

[0132] The right sidebar of the work screen displays a list of annotation classes, with target labels (for example, "bird," "cat," and "other") color-coded. The annotation list panel allows the worker to select, display, and check information for each label they created. The display can also be switched on / off and sorted, making it useful for review and learning.

[0133] The user interface shown in FIG. 19 can be used for purposes such as labor savings and objectivity through automatic evaluation processing (first execution unit 262), support for work permit decisions (whether the pass score is met), and training and review through comparative visualization using a GUI. In this way, the re-training process is executed in a format that provides feedback based on a comparison of the sample annotation results performed by the worker with the correct data. In addition, the re-training process can also present optimized sample data using past evaluation history when a worker who previously failed the onboarding test is re-onboarded.

[0134] FIG. 20 shows an example of a user interface displaying a comment and marking screen for efficient inspection on the cloud. The inspection user interface shown in FIG. 20 is displayed by the second output unit 264 of the annotation support device. By displaying the annotation result display area and the corresponding comments, visual review and correction instructions can be efficiently performed. Comments have functions such as assigning the responsible person, history management, and display ON / OFF, making it suitable for quality control on the cloud. The example shown in FIG. 20 shows a review screen with a comment and marking function on the cloud for inspection work. This screen is a GUI that allows reviewers and approvers to check the quality of annotation results and visually add comments and correction instructions, contributing to improved accuracy and speed of quality control.

[0135] In the central display area of ​​the inspection user interface shown in Fig. 20, annotation images are displayed in different colors by class, and segmentation annotations are drawn against a background image (e.g., a traffic scene) in different colors by class (e.g., cars = blue, sidewalks = brown, utility poles = green, curbs = purple, etc.). In the example in Fig. 20, comment numbers (No. 1 to 5) are displayed in balloon format on the object being inspected, indicating that they are associated with the corresponding location.

[0136] The right panel of the inspection user interface shown in Figure 20 displays a list of comments and tag information, while the "Annotation Class" section displays a list of currently displayed annotation classes (cars, curbs, sidewalks, utility poles, etc.). Each display can be toggled on and off, allowing users to focus on reviewing and correcting specific classes. In the "Comments" section, each comment is recorded with the name of the person in charge and the time stamp, clearly identifying who made the comment, when, and what it was. By using mentions (e.g., @hiroki.fujiwara), users can make comments and correction requests to specific workers on the cloud. Comment content, such as specific correction instructions (e.g., "Please circle the area up to the side mirror"), can contribute to improving inspection quality. The "Save" button at the bottom records comment input or review results in the recording unit 269, while the "Cancel" button cancels unsaved comments or operations.

[0137] The inspection user interface shown in Figure 20 is displayed on the user's device from the annotation support device, allowing remote reviewers / annotators to review and point out errors in real time. Furthermore, the visualization link for comments links the target position on the annotation image with the comment, preventing misunderstanding of the points raised. The inspection user interface's history management and designated person in charge record who pointed out what, when, and can also be used for trail management and educational feedback. Approval flow integration also allows for a design in which pointed out and corrected data is passed on as is to the next approver.

[0138] FIG. 21 shows an example of a user interface that displays thumbnails on an image-by-image basis, enabling efficient inspection of large amounts of annotation data. The thumbnail-format user interface shown in FIG. 21 is output by the fourth output unit 268 of the annotation support device, allowing users to visually check, inspect, and manage large amounts of annotation image data in a list format. This screen visually displays annotated and unannotated image data, allowing users to see the processing status, annotation content, and approval status for each image at a glance. This allows reviewers to efficiently grasp the data to be inspected and quickly confirm the work and issue correction instructions. The user interface shown in FIG. 21 significantly improves the efficiency of review, especially on an image-by-image basis, during the quality check and approval process for annotation work.

[0139] A list of thumbnail images is displayed in the central display area of ​​the user interface shown in Figure 21, with the file name overlaid on each image, for example, 1_trafficimage_sam.jpg, 2_trafficimage_handwork.jpg, FYI00203167.jpg, etc. A preview of the segmentation display is overlaid on annotated images, allowing for a quick visual confirmation during inspection work. The menu (navigation bar) displayed on the left side of the display screen is integrated with project management functions such as "Dashboard," "Annotation Class," "Members," and "Settings," making it possible to perform operations from inspection to reassignment and setting changes in a single go.

[0140] The features and advantages of the user interface shown in Figure 21 are that it displays a list of many images at once, allowing users to visually grasp which images have been annotated and which have not yet been processed. This user interface also allows users to move to a detailed screen by clicking on a thumbnail, where annotation content and comments can be immediately confirmed, speeding up the review cycle. Furthermore, the structure makes it easy to spot annotation omissions and errors by viewing the entire image at a glance, and the filter function allows users to narrow down the results by person in charge and status, making it ideal for dividing tasks among reviewers.

[0141] FIG. 22 shows an example of a user interface that displays thumbnails on an annotation-by-annotation basis, enabling efficient inspection of large amounts of annotation data. The annotation-by-annotation thumbnail display screen shown in FIG. 22 is provided by the fourth output unit 268 of the annotation support device. This screen displays a list of images extracted for each annotation label, allowing inspectors to check the shape, range, accuracy, etc. of the annotation at a glance. Furthermore, the ability to narrow down by condition and save search conditions enables efficient quality checks of large amounts of data. In this way, this screen lists large amounts of annotation results by individual annotation (label), providing a mechanism for visual inspection and management.

[0142] In the central display area of ​​the annotation-based thumbnail display screen shown in Figure 22, a list of thumbnails by annotation is displayed, and each image is displayed by extracting individual annotations within the image (for example, each annotation class "car"). If there are multiple annotations within the same image, each is displayed separately. At the bottom of the image, the target annotation class (for example, "car") and the original image file name (for example, 1_trafficimage_sam.jpg) are displayed, clearly indicating the source of the annotation.

[0143] The pull-down box at the top of the thumbnail display screen allows for conditional filtering and narrowing. By setting conditions such as "annotation," "contains," and "enter a value," you can narrow down your search by specific class, attribute, or work status. For example, you can narrow down your search by a specific class (cars only), unapproved status, or work performed by a specific user. Additionally, the "Add Condition," "Save Condition," and "Saved Condition" options located below the pull-down box allow you to create reusable sets of search and confirmation conditions. Predefining check and review targets streamlines routine inspection work. Other features include the "Group by Task" switch in the upper right corner of the screen, which allows you to switch between image and annotation display. The zoom slider adjusts the thumbnail size, and the number of items displayed can be adjusted using the checkbox (50 items per page in the example shown in Figure 22).

[0144] The annotation-based thumbnail display screen shown in Figure 22 visualizes a list at the annotation granularity level, making it possible to check quality on a label-by-label basis, even when a single image contains multiple labels. This screen is also ideal for identifying defects by classification; for example, by extracting only the annotation class "car," it is possible to simultaneously detect variations or omissions in the annotation range. Furthermore, when used in conjunction with filters, it streamlines mass inspection, and conditional search and saving supports periodic reviews and quality assurance workflows. In this way, this screen is ideal for reviewers and administrators, allowing them to quickly check annotations by specific workers or unapproved annotations.

[0145] Fig. 23 is a diagram showing an overview of random sampling processing. As one form of quality evaluation processing in the annotation support device according to the present invention, in an example in which random sampling processing is performed on all data as shown in Fig. 23, (a) shows all annotation data including the sampling target, (b) shows the data distribution by annotator based on the sampling results, and (c) shows the distribution of the number of annotations by class included in the sampling target. This type of sampling processing according to the present invention enables fair quality inspection targeting a variety of classes and operators.

[0146] The example shown in Figure 23 illustrates the execution of a random sampling process (lot inspection) as part of the quality evaluation process in the annotation support device, and shows the case where "30% (= 3 items) of 10 data items are selected as sampling targets."

[0147] (a) The overall data lists a total of 10 annotation data (Data 1 to Data 10), and each row shows information such as the "annotator name" (A to C), "priority" (high or blank), "annotation data" (number of each annotation class (e.g., "car," "person," "animal") for each data), and "sampling target" (three randomly selected data (in the example in Figure 23, Data 1, Data 5, and Data 10) are marked with a "★").

[0148] (b) Data Distribution (by Annotator) is a table showing the distribution of data by annotator during sampling. In the example in Figure 23, sampling was performed with consideration given to the balance of data by annotator, with one item extracted from each of annotators A, B, and C. By ensuring a balanced extraction, fair and comprehensive quality evaluation is possible.

[0149] (c) Data distribution (by annotation class) is a table showing the breakdown of annotation classes contained in the three sampled data. The breakdown is as follows: cars: 5, animals: 4, people: 3. Covering a wide range of classes ensures that quality assessment is not biased towards a specific label, making it effective for checking the overall accuracy of annotations.

[0150] As shown in Fig. 23, the sampling process of the present invention makes it possible to visualize the quality inspection process by clearly indicating the criteria and balance by which inspection data is selected from all data, and by sampling at equal rates for multiple annotators, it is possible to fairly evaluate the work quality. Furthermore, the sampling process of the present invention can also detect accuracy degradation for individual labels by intentionally incorporating data containing multiple annotation classes.

[0151] A system including an annotation support device according to one embodiment of the present invention can achieve both quality control and work efficiency in annotation work. First, by quantitatively evaluating the work quality by comparing the correct data with the work results, it is possible to objectively grasp the capabilities of workers and reduce human variation. Furthermore, by automatically assigning work target data to workers based on the evaluation results, it is possible to distribute the workload and reduce management costs.

[0152] Furthermore, efficient and standardized quality control is achieved by automatically sampling and inspecting the quality of each lot based on the annotation results. In addition, by updating the approval status based on the inspection results and automating notifications to the next process worker, delays in the work process and communication efforts can be reduced.

[0153] Furthermore, this invention makes it possible to visualize work progress and quality information on a dashboard, making it easy to grasp the status of the entire project or individual workers and to detect abnormalities.In addition, specifications and standard operating procedures can be managed centrally in Markdown format, etc., making it possible to standardize work rules and quickly train new members.

[0154] In addition, the work history of each worker is recorded, which can be used for continuous evaluation and retraining, and the intuitive user interface that displays annotation results in thumbnail format improves the efficiency and visibility of inspection work.

[0155] These features enable cloud-based and web-based operation, enabling scalable and efficient operation even for large-scale annotation projects. As a result, it is possible to realize a data creation system on a scale that goes beyond the PoC level and is effective in eliminating bottlenecks in AI development as a whole. [Industrial Applicability]

[0156] According to the present invention, it is possible to provide multiple workers involved in annotation work with integrated services such as quality evaluation-based judgment of work acceptability, automatic allocation of appropriate data, lot-by-lot quality control, and visualization of work results, and thus the present invention can be widely used in industrial fields where large-scale and high-precision annotation work is required. Specifically, the present invention can be applied to the creation of training data, which forms the foundation of all AI technologies such as image recognition, natural language processing, speech recognition, and anomaly detection, and is useful for companies developing AI, contract data preparation companies, research institutions, educational institutions, and the like. Therefore, the present invention contributes to improving the efficiency and quality of data preparation processes in the AI ​​and data science fields, and is extremely useful industrially. [Explanation of symbols]

[0157] 1: Terminal 1-1~1-N: Terminal 2: Computer Systems 21: Input interface 22: Communication module 23: Storage device 24: Memory 25: Output interface 26: Processor 261: First output section 262: First Executive Division 263: 1st allocation department 264: Second output section 265:Second Assignment Department 266: Second Executive Division 267: Third output section 268: 4th output section 269: Recording Department CN: communication line network

Claims

1. An annotation support device that supports annotation work, a recording unit that records at least project information, work target data, worker information, and annotation results; a first output unit that outputs specifications and standard operating procedures based on the project information; a first execution unit that evaluates the quality of the work performed by the worker based on the worker information and a plurality of sample annotation results and determines whether or not the work can be started; a first allocation unit that allocates work target data to workers based on a determination result by the first execution unit; a second output unit that outputs a user interface that enables annotation work to be performed on the work target data; a second allocation unit that samples the annotation results after the annotation work is completed on a lot-by-lot basis to extract an inspection object, and allocates a next worker based on the inspection result of the inspection object; a second execution unit that aggregates work progress and work quality; a third output unit that visualizes the aggregated results; a fourth output unit that outputs a user interface in thumbnail format that displays a list of annotation results; Equipped with The second execution unit executes a re-education process for a worker when the defect rate of the inspection target is equal to or greater than a specified value.

2. The annotation support device according to claim 1 , wherein the re-training process is performed in a manner that provides feedback based on a comparison of sample annotation results performed by the worker with correct answer data.

3. The annotation support device according to claim 1 or 2, wherein the re-training process presents sample data optimized using a past evaluation history when a worker who previously failed the onboarding test is to undergo onboarding again.

4. The annotation support device according to claim 1 , wherein the first execution unit has a function of calculating a score by comparing correct answer data with the annotation result and determining whether or not a worker is permitted to perform the work based on the score.

5. The annotation support device according to claim 1 , wherein the first allocation unit has a function of allocating the work target data based on a priority of the work target data and a load status of a worker.

6. The annotation support device according to claim 1 , wherein the second execution unit has a function of automatically sampling the annotation results in units of lots and performing quality evaluation.

7. The annotation support device according to claim 1 , wherein the second output unit has a function of receiving an input of an inspection result, updating an approval status, and notifying the next worker.

8. The annotation support device according to claim 1 , wherein the third output unit has a function of visually displaying work progress and quality information aggregated by worker or by annotation class.

9. The annotation support device according to claim 1 , wherein the first output unit has a function of outputting specifications and standard operating procedures as editable screens in Markdown format.

10. The annotation support device according to claim 1 , wherein the recording unit records a work history for each worker and makes the work history available for re-evaluation or re-training by the first execution unit.

11. A method for supporting annotation work, comprising: A step of recording at least project information, work target data, worker information, and annotation results; outputting specifications and standard operating procedures based on the project information; evaluating the quality of the work performed by the worker based on the worker information and a plurality of sample annotation results, and determining whether or not the work can be started; assigning work target data to the worker based on the determination result; outputting a user interface that enables annotation work to be performed on the work target data; a step of sampling the annotation results after the annotation work is completed on a lot-by-lot basis to extract an inspection object, and assigning the next worker based on the inspection result of the inspection object; a step of aggregating work progress and work quality; A step of visualizing the aggregated results; a step of outputting a user interface in thumbnail format that displays a list of annotation results; Including, The annotation support method executes a re-education process for the worker when the defect rate of the inspection target is equal to or greater than a specified value.

12. The annotation support method according to claim 11 , wherein the re-training process is performed by providing feedback based on a comparison of a sample annotation result performed by the worker with correct answer data.

13. The annotation support method according to claim 11 , wherein the re-training process presents sample data optimized using a past evaluation history to a worker who previously failed the onboarding test when the worker is onboarded again.

14. The annotation support method according to claim 11 , wherein the step of evaluating the quality of the worker includes a step of calculating a score by comparing correct answer data with the annotation result, and determining whether the worker is permitted to work based on the score.

15. The annotation support method according to claim 11 , wherein the step of allocating the work target data is executed based on the priority of the work target data and a load status of the worker.

16. The annotation support method according to claim 11 , further comprising the step of sampling the annotation results on a lot-by-lot basis and evaluating their quality.

17. The annotation support method according to claim 11 , further comprising the steps of accepting an input of an inspection result, updating an approval status, and notifying the next worker of the update.

18. The annotation support method according to claim 11 or 12, further comprising the step of recording and visualizing the work progress, quality, and work history for each worker.

19. A program for causing a computer to execute the annotation support method according to any one of claims 11 to 17.

20. A program for causing a computer to execute the annotation support method according to claim 18.

Images