Annotation quality assurance support device, method, and program

Abstract

In conventional annotation work, visual review was the norm, but as data volume increased and annotation requirements became more complex, the reviewer's workload increased, resulting in problems such as a decline in quality and delivery delays.In addition, mechanical rule-based checks were not responsive enough, making it difficult to achieve flexible quality assurance in combination with human judgment. [Solution] The annotation quality assurance support device, method, and program of the present invention automatically evaluate annotation results based on check rules defined on a project-by-project basis, and provide feedback to the worker at each level of error, warning, and information. It also records the worker's response to the rule results, and, if necessary, makes judgments using AI such as visual language models, enabling flexible and efficient quality confirmation.

Description

[Technical Field]

[0001] To implement artificial intelligence, annotation work is required, which teaches the AI ​​to identify objects in images (videos and still images) and where the boundaries between the objects and the background are. Because untrained AI cannot identify objects, annotation work is often performed by humans drawing the outlines of objects on the screen. This invention relates to technology that automatically evaluates and ensures the quality of annotation results when creating training data used in machine learning and AI development. In particular, this technology relates to annotation support technology that automatically inspects the results of annotation work on images, text, etc. using AI models such as rule-based checks and visual language models, thereby reducing the review burden and stabilizing quality. [Background technology]

[0002] In recent years, as machine learning and deep learning have become more practical, the preparation of training data to be used in training AI models has become increasingly important, and annotation work is widely carried out on a variety of data, including images, text, and audio. Annotation is a task that requires accurate and consistent data assignment based on specialized rules, and is performed manually in many development sites.

[0003] Conventionally, the quality of annotation results was generally checked visually by reviewers, with quality assured through reviews by multiple people, double checks, recording of revision history, etc. Furthermore, cloud-based annotation support tools and open-source annotation management software offer functions that allow users to enter review comments, manage approval status, visualize progress, etc. through a user interface.

[0004] Additionally, some systems have introduced rule-based quality checks on annotation results, which check for formatting errors, missing entries, inconsistencies, etc. based on mechanical criteria. Such rule-based checks are effective for certain routine checks and contribute to improving work efficiency. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2022 / 185363 [Patent Document 2] Japanese Patent Application Laid-Open No. 2024-144030 [Non-patent literature]

[0006] [Non-Patent Document 1] Open source annotation tool CVAT, URL https: / / github.com / cvat-ai / cvat (Retrieved August 4, 2025) Summary of the Invention [Problem to be solved by the invention]

[0007] Conventional technologies provide a mechanism for reviewers to visually check annotation results during the annotation workflow. However, as the amount of data increases and annotation requirements become more complex, the reviewer's workload increases nonlinearly, resulting in issues such as a decline in annotation quality and delays in the training data creation process.

[0008] In addition, while some systems existed that performed mechanical rule-based checks, it was difficult to flexibly change the rules according to annotation requirements, and there was no integrated system in place that combined quality checks with human judgment, making it difficult to apply to actual projects.

[0009] Behind these challenges lies the technical difficulty of designing a system with the versatility and abstraction necessary to accommodate all annotation requirements and operational situations. Other technical challenges that needed to be resolved included the difficulty of providing user-friendly feedback that did not impair user efficiency and the difficulty of ensuring real-time performance. Furthermore, conventional rule-based mechanical checking mechanisms were unable to replace the evaluation of semantic consistency, which was previously performed by human visual inspection, limiting the automation and scalability of quality assurance.

[0010] Therefore, the present invention aims to solve these problems and provide an annotation quality assurance support device, a quality assurance support method, and a quality assurance support program that function as an automatic quality assurance mechanism (AutoQA) that efficiently and stably realizes quality control in annotation work. [Means for solving the problem]

[0011] An annotation quality assurance support device for evaluating annotation quality according to a first embodiment of the present invention, comprising: a first recording unit that records setting information of rules including check conditions for annotation quality assurance; a first output unit that outputs a setting screen for setting annotation quality assurance based on the setting information; a first execution unit that, during annotation work, refers to the setting information and performs a quality check on an annotation work screen; a second output unit that outputs a feedback screen that presents the result of the quality check to the worker; a second recording unit that records the worker's confirmation status of the results presented on the feedback screen; The second execution unit performs judgment processing using an AI model on the annotation results. The present invention is characterized by comprising:

[0012] An annotation quality assurance support device according to a second embodiment of the present invention is an annotation quality assurance support device according to the first embodiment, characterized in that the setting information is capable of setting a check level for each rule that includes at least an error, a warning in the annotation, and information.

[0013] An annotation quality assurance support device according to a third embodiment of the present invention is an annotation quality assurance support device according to the first or second embodiment, characterized in that the first execution unit has a function of performing the quality check at the completion of each step of the workflow during annotation work, or as a process performed collectively on annotated data.

[0014] An annotation quality assurance support device according to a fourth embodiment of the present invention is an annotation quality assurance support device according to any of the first to third embodiments, characterized in that the second output unit classifies the results of the quality check into at least errors, warnings, and information and outputs them.

[0015] An annotation quality assurance support device according to a fifth embodiment of the present invention is an annotation quality assurance support device according to any of the first to fourth embodiments, characterized in that the second execution unit refers to the worker's confirmation status and does not proceed to the next workflow until all errors have been resolved and all warnings have been confirmed.

[0016] An annotation quality assurance support device according to a sixth embodiment of the present invention is an annotation quality assurance support device according to any of the first to fifth embodiments, characterized in that it generates quality assurance feedback based on the output results of an AI model including at least one of an object detection model, an area detection model, a visual language model, and a large-scale language model.

[0017] An annotation quality assurance support device according to a seventh embodiment of the present invention is an annotation quality assurance support device according to the sixth embodiment, characterized in that the second output unit is configured to display the output results from the AI ​​model in association with the worker's annotations, so that the worker or reviewer can immediately check the content.

[0018] An annotation quality assurance support method for evaluating annotation quality according to an eighth embodiment of the present invention. A method for supporting annotation work, the annotation quality assurance support method being executed by a computer, comprises: recording setting information of rules including check conditions for annotation quality assurance; outputting a setting screen for setting annotation quality assurance based on the setting information; a step of performing a quality check on an annotation work screen by referring to the setting information during annotation work; outputting a feedback screen that presents the results of the quality check to the worker; a step of recording the worker's confirmation status regarding the results presented on the feedback screen; A step of performing judgment processing using an AI model on the annotation results. The present invention is characterized by comprising:

[0019] An annotation quality assurance support method according to a ninth embodiment of the present invention is an annotation quality assurance support method according to the eighth embodiment, characterized in that the setting information is capable of setting a check level for each rule that includes at least an error, a warning in the annotation, and information.

[0020] An annotation quality assurance support method according to the tenth embodiment of the present invention is an annotation quality assurance support method according to the eighth or ninth embodiment, characterized in that the quality check is performed after execution of a process that advances the workflow in annotation work.

[0021] An annotation quality assurance support method according to an 11th embodiment of the present invention is an annotation quality assurance support method according to any of the 8th to 10th embodiments, characterized in that the results of the quality check are classified into at least errors, warnings, and information and presented on the feedback screen.

[0022] An annotation quality assurance support method according to a twelfth embodiment of the present invention is an annotation quality assurance support method according to any of the eighth to eleventh embodiments, characterized in that it refers to the worker's confirmation status and does not allow the workflow to proceed to the next step until the annotation worker has completed resolving all errors and checking all warnings.

[0023] An annotation quality assurance support method according to a thirteenth embodiment of the present invention is an annotation quality assurance support method according to any one of the eighth to twelfth embodiments, wherein one of the AI ​​models is a visual language model; The method further includes a step of processing prompts according to annotation requirements and annotated images as inputs into the visual language model, and generating quality assurance feedback based on the output results of the visual language model.

[0024] An annotation quality assurance support method according to a 14th embodiment of the present invention is an annotation quality assurance support method according to the 13th embodiment, characterized in that the feedback screen displays the output results from the AI ​​model in correspondence with the worker's annotations, and is configured so that the worker or reviewer can intuitively confirm the content by highlighting or selecting the corresponding annotation in response to a user interface operation on an item in the list.

[0025] An annotation quality assurance support program according to a fifteenth embodiment of the present invention is characterized in that it causes a computer to execute the annotation quality assurance support method according to any of the eighth to fourteenth embodiments. [Effects of the Invention]

[0026] This invention automates or semi-automates the quality check process for annotation results, significantly reducing the reviewer's workload. Specifically, by replacing the conventional visual review with automatic quality assessment based on check rules defined for each project, it is possible to perform efficient and consistent quality assurance even for large amounts of data.

[0027] Furthermore, this invention makes judgments and provides feedback to users according to the level of error, warning, and information, and it is possible to record the confirmation status and accumulate a response history, thereby improving the transparency and traceability of quality control.

[0028] Furthermore, auxiliary checking functions using visual language models, etc., make it possible to detect contextual and semantic errors that are difficult to detect using rule-based methods, thereby complementing human judgment and improving the accuracy of quality assurance.

[0029] As a result, the quality of training data creation will be stabilized, work progress will be visualized, and quality assurance will be standardized, greatly contributing to the efficiency and quality improvement of the entire annotation process. [Brief explanation of the drawings]

[0030] [Figure 1] 1 is a block diagram showing the hardware configuration of a system including an annotation quality assurance support device according to an embodiment of the present invention, which is configured from a computer system connected to a user terminal via a communication network. [Figure 2] 1 is a block diagram showing the hardware configuration of an annotation quality assurance support device (computer system) according to the present invention, which includes an input interface, a communication module, a storage device, a memory, an output interface, and a processor. [Figure 3]1 is a flowchart showing a process flow of workflow control executed by the annotation quality assurance support device, which shows a flow of controlling rule execution and error and warning checks according to the progress of annotation work. [Figure 4] This is a flowchart showing the steps for setting quality assurance rules to be applied to a project, including selecting rules, entering application conditions, and checking levels. [Figure 5] 1 is a diagram showing the structure of a database related to quality assurance rule settings used in the annotation quality assurance support device according to the present invention, in which information such as the type of rule, applicable target, check conditions, and level set for each project is recorded in table format. [Figure 6] This figure shows the structure of a database that records the results of user confirmation and response to warnings detected by the quality assurance process in the annotation quality assurance support device according to the present invention. The target data, applicable rules, warning contents, user response history, etc. are recorded in table format. [Figure 7] 1 is a diagram showing a screen for setting quality assurance rules in the annotation quality assurance support device according to the present invention, in which it is possible to set for each rule whether application is on or off, the rule name, check level, detailed rule content, check type, and execution action. [Figure 8] 1 is a diagram showing a detailed setting screen for quality assurance rules in the annotation quality assurance support device according to the present invention, which allows the user to set in detail the check level to be applied to each rule and the conditions under which the rule is applied. [Figure 9] This is a diagram showing a user interface screen for annotation and review work in the annotation quality assurance support device according to the present invention. For items detected by automatic quality assurance, the status, type, annotation target, and corresponding message are displayed in a list format, allowing the user to check and respond. [Figure 10]1 is a diagram illustrating an example of a quality check process using a visual language model (VLM) to perform visual judgment closer to that of a human in the annotation quality evaluation support device according to the present embodiment. The diagram illustrates an example of the configuration of the quality check process, in which a prompt and annotated images are input, classify them using the visual language model, and detect errors in the annotation classes. DETAILED DESCRIPTION OF THE INVENTION

[0031] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will now be described with reference to the accompanying drawings. The individual embodiments of the present invention are not independent and can be appropriately combined with each other for implementation.

[0032] Fig. 1 is a block diagram showing the hardware configuration of a system including an annotation quality assurance support device according to one embodiment of the present invention. As shown in Fig. 1, the system S includes terminals 1-1 to 1-N (N is a natural number) used by users, and a computer system 2 connected to these terminals 1-1 to 1-N via a communication network CN.

[0033] Terminals 1-1 to 1-N are devices used by users who perform annotation work, and include, for example, multi-function mobile phones (so-called smartphones), tablets, laptops, and desktop computers. These terminals are collectively referred to as terminal 1. Terminal 1 may have the same configuration regardless of its status (reviewer, annotator, etc.), but it does not have to be the exact same model. Role permissions set according to the project are assigned based on the ID and password used when terminal 1 logs in to computer system 2.

[0034] The computer system 2 functions as an annotation quality assurance support device by executing the annotation quality assurance support program of the present invention. For example, it is used or managed by an administrator of an organization (including a company) that manages annotation target data. The computer system 2 executes processing in response to a request from the terminal 1 and provides the results to the terminal 1. The computer system 2 may be configured as a single computer or may be configured with multiple computers. An example configured as a single computer will be described below.

[0035] Fig. 2 is a block diagram showing the hardware configuration of an annotation quality assurance support device (computer system) according to the present invention. As shown in Fig. 2, the computer system 2 includes an input interface 21, a communication module 22, a storage device 23, a memory 24, an output interface 25, and a processor 26. The input interface 21 accepts operational inputs from an administrator of the computer system 2 and outputs signals corresponding to the accepted inputs to the processor 26. The communication module 22 is connected to a communication line network CN and performs data communication with the terminal 1. Note that this communication may be wired or wireless, but the present embodiment will be described assuming a wired connection.

[0036] The storage device 23 is, for example, a storage device, and stores various data and programs that are read and executed by the processor 26. The memory 24 is a storage area for temporarily storing data and programs, and is configured as a volatile memory, for example, RAM (Random Access Memory). The output interface 25 enables connection to an external device and has the function of outputting signals to the external device. The processor 26 loads a program stored in the storage device 23 into the memory 24 and executes a series of instructions contained in the program, thereby operating as the following functional blocks: That is, a first recording unit 261, a first output unit 262, a first execution unit 263, a second recording unit 264, a second output unit 265, and a second execution unit 266.

[0037] The first recording unit 261 is a recording means for recording basic information related to annotation quality assurance. Specifically, it stores project information, annotation target data, worker information, quality assurance rules, etc. in a database. This recording unit aggregates and manages information required for subsequent quality assurance processing and output functions.

[0038] The first output unit 262 is an output means that provides a user interface (UI) for setting annotation quality assurance rules. Through this UI, the user can set the check target, check method (rules), check level, execution action, etc. For example, the first output unit 262 provides a user-friendly graphical user interface (GUI) that also allows switching between enabling and disabling various rules.

[0039] The first execution unit 263 is a functional unit that automatically checks annotation results based on set quality assurance rules. For example, it automatically detects errors such as missing labels, inconsistent coordinates, and incorrect classes, and generates warnings for each annotation. This reduces the burden of review work and enables early quality detection.

[0040] The second recording unit 264 is a recording unit that holds records of the quality assurance results generated by the first execution unit 263, i.e., the detected warnings and findings, and the worker's response to them (checked, corrected, ignored, etc.). The recorded content is also used for subsequent analysis processing and progress / quality assurance.

[0041] The second output unit 265 is a part that outputs a user interface that provides feedback on warning information and evaluation results to annotators and reviewers. It provides functions such as displaying a list of warnings, checking them with thumbnails, and adding comments, helping annotators to visually check and deal with information related to quality.

[0042] The second execution unit 266 is a processing unit that compiles and analyzes statistical information on work quality and progress status based on the warning information and response results accumulated in the second recording unit 264. For example, it performs trend analysis on a per-user basis, the number of occurrences by rule, and per-project basis, contributing to the visualization of quality control and continuous improvement.

[0043] 3 is a flowchart showing the processing flow of workflow control executed by the annotation quality assurance support device. As shown in the flowchart of FIG. 3, the annotation quality assurance support device according to the present invention supports quality control in annotation work by executing the following processes.

[0044] First, in step S101, the device accepts automatic quality assurance rules (hereinafter also referred to as AutoQA rules) input by the user. These rules include check targets, application conditions, thresholds, importance levels, actions, etc., and these settings are recorded in the database of the first recording unit 261.

[0045] Next, in step S102, the following processes are repeated until all annotation work on the annotation target data is completed (steps S102 to S111). In step S103, annotation input by the worker is accepted, and the input annotation information is recorded in the database of the first recording unit 261.

[0046] Then, in step S104, the process to advance the workflow for project management is executed. For example, the transition to the review process, the status change, and the assignment of work to the next person in charge are automatically executed. In step S105, the annotation results are automatically checked based on the AutoQA rules set in S101, and the judgment results (presence or absence of errors or warnings, etc.) are displayed on the GUI.

[0047] Next, in step S106, the annotation quality assurance support device determines whether the annotation resulting from the annotation work contains any errors. If it is determined that the annotation contains errors ("Yes" in S106), the annotation content is deemed inappropriate, and the workflow returns to step S103, prompting the worker to re-input (correct) the content. If the annotation does not contain any (fatal) errors, it is determined in step S107 whether there are any warnings corresponding to minor problems (e.g., deviations from the recommended conditions in the guidelines).

[0048] If a warning exists in the annotation ("Yes" in S107), the worker is asked in step S108 whether or not to make corrections. If corrections are to be made ("Yes" in S108), the workflow returns to S103 again, where re-input (correction) processing is prompted and annotation input by the worker is accepted. Alternatively, if no corrections are to be made in step S108 ("No" in S108), a confirmation process is executed for each warning in step S109, and a determination such as "Confirmed" or "Pending" is recorded in the second recording unit 264.

[0049] If it is determined in step S107 that there is no warning in the annotation ("No" in S107), or after confirmation processing for each warning is performed in step S109, the process proceeds to step S110, where the annotation data is saved and registered in the second recording unit 264 as confirmed data.

[0050] Finally, in step S111, control is returned to S102 to repeat the same process for the next annotation target data. In this way, according to the present invention, quality assurance is performed automatically at each phase of the annotation work, and intervention for correction or confirmation is encouraged, thereby making it possible to improve both the quality and efficiency of annotation.

[0051] 4 is a flowchart showing the process of setting automatic quality assurance rules (AutoQA rules) executed by the annotation quality assurance support device according to the present invention. The device executes the following procedure based on user operations to register various rules used in quality checks and set application conditions.

[0052] In step S201, the annotation quality evaluation support device first receives input of "rules to be used" from the user. Each rule is a unit of check items used in the automatic quality assurance process, and includes, for example, "label mismatch," "duplicate annotation," and "no attribute input."

[0053] Next, in step S202, based on the input rules, the device acquires or updates a list of "all rules to be used" for the entire project. This rule list explicitly identifies all rules to be subsequently set. Note that the processing of steps S202 to S205 is repeated the number of times equal to the number of rules to be registered (setting loop for each rule). After setting for all rules is complete, the rule setting processing ends.

[0054] Next, in step S203, the annotation quality evaluation support device accepts input of the "check level" for each rule. The check level indicates the importance of each rule and the strength of the warning. For example, it is classified as "error," "warning," or "information," and is reflected in the result display and notification priority. This allows the device to determine how strictly the rule should be applied to the annotation data.

[0055] Furthermore, in step S204, the device accepts input of the "application conditions" for each rule. The application conditions consist of the attributes of the target data to which the rule applies, the annotation class, the operator's authority, the type of target file, etc. This allows for flexible customization of rule application.

[0056] Finally, in step S205, based on the input setting information, the annotation quality assurance support device registers and saves the currently set rule in the second recording unit 264 as an automatic quality assurance rule (AutoQA rule).

[0057] 5 is a diagram showing an example of the configuration of a database that records the settings of the automatic quality assurance rules (AutoQA rules) used in an annotation quality assurance support device according to one embodiment of the present invention. As shown in this diagram, the rule setting database manages the following items for each rule: · Identifier: Indicates the unique name or number of the rule (e.g., Rule 1, Rule 2, etc.). Check content explanation (text): An explanation of how the rule checks annotation data (e.g., "Check for annotations that exist in the specified range," "Check for unfilled areas," etc.). ON / OFF: A setting value that indicates whether the rule is enabled or disabled. When ON, the rule is applied, and when OFF, it is not applied. Check level: Classification according to the severity of the violation, such as "Error," "Warning," and "Information." Check conditions: The conditions for determining whether the rule is violated are described, such as annotation class, coordinate range, area, distance, etc.

[0058] For example, rule 1 determines that an error has occurred if the annotation class is A or B and the coordinates are within a specified range, or if the class is C or D and the coordinates fall within a different range. Rule 4 outputs a warning if the area of ​​the annotation is less than 100 pixels. Rule 5 checks whether the annotation class is E or F and the distance from the ground exceeds 0.05 m, and notifies the user of this information.

[0059] The rule setting database configured in this way is displayed and managed by the first recording unit 261 and the first output unit 262, and the user can dynamically set and change the on / off status and detailed conditions of each rule via the GUI. This configuration enables flexible and detailed quality checks according to the quality standards of each project.

[0060] 6 is a diagram showing an example of a confirmation result database that records the results of reviewers checking or responding to warnings detected by the automatic quality assurance process in an annotation quality assurance support device according to one embodiment of the present invention. This confirmation result database records, for each annotation, the status of the confirmation process performed by the user in response to "warning"-level issues from the quality check results automatically executed based on the various rules defined in FIG. 5, for example.

[0061] The database contains at least the following information items: Task: Indicates the unit of work that is the subject of quality assurance. Identifiers for images or data files to be annotated are mainly used. For example, "Image 1" and "Image 2" correspond to task units. Annotation: Information that identifies the individual annotation targets included in each task. They are distinguished by symbols or names (e.g., A, B, C, etc.) and indicate the unit of annotation in which the warning occurred. Rule: The name or summary of the quality check rule that was applied when the warning was detected. For example, the rule content is recorded in text, such as "Check for annotations that are smaller than a specified area." Verification status: Indicates the reviewer's verification process result for the warning. For example, "OK" indicates that the target annotation was deemed problem-free. Other statuses such as "Fixed" and "Unverified" can also be set. Confirming user: User identification information of the reviewer (confirmer) who confirmed the warning. Symbols (e.g., a, b) and user IDs are recorded. This information is used to ensure traceability of the confirmation process.

[0062] As described above, with the database configuration shown in Fig. 6, the annotation quality assurance support system according to the present invention ensures traceability of the results of automatic quality assurance processing, and also makes it possible to centrally manage and visualize the history of confirmation and response to warnings across the entire project. This reduces the review load, prevents oversights, and improves the efficiency of the quality assurance process.

[0063] Terminal 1 logged in with the authority of a project manager (PM) or reviewer provides a function for displaying a list of error occurrence histories for each annotator who is currently working or has completed work on the project being reviewed by that user. This error history display screen is configured to visualize, for each annotator, what type of error has occurred and how frequently, in which annotation class or work unit. This allows, for example, if errors are occurring disproportionately only among a specific annotator, even though they are working based on the same specifications, to determine that the annotator's interpretation of the specifications or judgment criteria may deviate from the contents of the specifications.

[0064] Terminal 1, logged in as a PM or reviewer, can display a history of which annotators made what kinds of errors in the project they are reviewing. Each annotator works based on the same specifications, but if errors are concentrated among a particular annotator, it is assumed that the annotator's understanding deviates from the specifications. Therefore, the annotator's individual capabilities can be improved by utilizing the education, advice, and onboarding functions.

[0065] In other words, the annotation quality assurance support device of the present invention can encourage annotators to correct their individual understanding and improve their work quality by applying re-education, individual guidance, and supplementary onboarding functions to the annotators. The onboarding functions include, for example, highlighting relevant sections of the specifications, presenting correct annotations for similar cases, providing feedback on the reasons for errors, and presenting interactive confirmation questions. This helps annotators autonomously recognize and correct errors in their understanding of the specifications.

[0066] Furthermore, if the number of errors is higher than normal and present across the entire team, it is likely that the specifications differ from what the annotators normally consider to be correct, or that the criteria for determining errors differ from the normal understanding of the specifications. In such cases, if you have PM authority, you can revise the specifications in the system, or you can modify the application conditions (described below) to prevent errors from being detected. Such modifications affect the progress and quality of the entire annotation project, and therefore cannot be made by annotator authority.

[0067] When multiple annotators make similar errors, it's possible that the problem lies not just with the individual annotator, but also with the wording or content of the specifications themselves. This could be because there's a discrepancy between the specifications' instructions and the annotator's common sense, or because the error detection criteria based on the specifications are vague or overly strict. In such cases, a user with PM authority can improve the overall quality of annotation work by revising or adding to the relevant section of the specifications via the system. It's also possible to adjust the quality standards to better reflect actual conditions by relaxing or adjusting the error detection criteria (thresholds, target attributes, detection criteria, etc.) without changing the specifications.

[0068] These revisions to specifications and changes to evaluation criteria have a significant impact on the quality and progress plan of the entire project, so they cannot be performed by annotators and are only permitted for users with PM or reviewer privileges.This operational design allows for flexible responses and the effectiveness of annotation work while maintaining control over quality management.

[0069] Figure 7 shows the setting screen for automatic quality assurance rules (Auto QA rules) in the annotation quality assurance support system. This screen displays a list of quality inspection rules that are automatically applied to annotation data, and allows you to enable / disable each rule (ON / OFF), set the rule name, check level, detailed description of the rule, type of check target, and perform action operations to access detailed settings.

[0070] Specifically, the following components are included: ON / OFF: A switch that allows the user to toggle the application of each rule. When turned ON, the rule will be automatically applied during subsequent annotation and review work. Rule name: The name of the quality check rule to be applied. For example, "duplicate annotations" or "annotations not touching the ground" should be used to concisely describe the check content. Check Level: Indicates how serious a violation of a rule is considered to be in quality inspection. The level is mainly distinguished as "Error" or "Warning." Rule details: A description of what conditions each rule checks. For example, "Checks whether it is in contact with the ground" or "Checks whether the cuboids overlap." Check type: Indicates the type of object to which the rule applies. On this screen, all checks are specified on an "annotation class" basis. Action: An action that allows the user to move to the detailed settings screen for each rule. This is represented by a gear icon and a link to "Detailed Settings."

[0071] This setting screen closely corresponds to the database configuration for quality assurance rule settings, as shown in Figure 5, and functions as an interface that allows users to easily enable / disable rules, check their contents, and edit their settings through a GUI. Furthermore, the enablement status and check level of each rule also affect the subsequent automatic check process (AutoQA process) and the check result recording shown in Figure 6. Therefore, this screen is one of the central components that enables flexible control and management of the entire quality assurance flow.

[0072] Figure 8 shows the detailed setting screen for the automatic quality assurance rules (Auto QA rules) in the annotation quality assurance support system. This screen allows users to define basic information for each rule and set application conditions. In particular, the GUI allows users to flexibly customize the annotation conditions to be applied to each rule and the check level (Error / Warning, etc.) in the event of a violation.

[0073] The following items can be set in the "Basic Information" section in Figure 8. Rule Name: The name of the rule to be automatically checked (e.g. "Duplicate Annotations"). Rule details: A description of what the rule checks (e.g. "Check if cuboids overlap"). Check Level: Defines the level at which the system will report a violation of this rule. Typical options are "Error" and "Warning".

[0074] In "Application Conditions" in Figure 8, you can specify the conditions (such as the attributes of the target annotation) under which the defined rule will be applied. By entering a condition set, you can set the meta attributes of the annotation (e.g., "annotation class contains XX") as the condition, and each rule can have one or more condition sets, which are applied using OR logic. The condition field, operator (contains, is equal to, is not equal to, etc.), and value can be selected from the GUI, making it possible to define advanced rules even without specialized knowledge. This setting allows for flexible and fine-grained inspection control, such as "apply the 'small area' check only to annotations of class A."

[0075] Such condition changes affect the progress and quality of the entire project, so they can be implemented by the PM, or by the reviewer's proposal and approved by the PM. As mentioned above, if the history of specific errors shows that errors occur frequently under certain conditions, minor adjustments to the applicable conditions can reduce the frequency of minor errors, prevent unnecessary checks and rework, and improve the efficiency of annotation work.

[0076] The definition, saving, and reading of rules are performed by the first recording unit 261, first output unit 262, and first execution unit 263 of the annotation quality assurance support device, and Figure 8 is an example of a user interface that handles part of the setting process.

[0077] The detailed settings interface shown in Figure 8 is an element that realizes "flexible quality assurance according to annotation requirements that vary from project to project," which is the core of this invention.Unlike conventional static, rule-based quality checks, this interface has the advantage of allowing users to design and apply inspection rules according to their own tasks.

[0078] Furthermore, the GUI-based, intuitive design allows non-engineers to define rules, improving on-site applicability, scalability, and maintainability. This also helps to resolve issues with conventional technology, such as the difficulty of modifying rules and the increased burden on users.

[0079] 9 shows the user interface during review work in the annotation quality assurance support device, and in particular, the operation screen for managing feedback on errors or warnings extracted by automatic quality assurance (Auto QA) and for user confirmation processing. The operation screen for the confirmation processing plays a central role in the interactive processing of the entire quality confirmation flow, in cooperation with each unit of the annotation quality assurance support device (first recording unit 261, first output unit 262, first execution unit 263, second recording unit 264, second output unit 265, and second execution unit 266 shown in FIG. 2).

[0080] The dialog box in the upper left of the screen shown in Figure 9 displays the error / warning detections, and after the Auto QA rules are executed, the quality issues detected are displayed in list format. Each line displays the status (error / warning), type (e.g., out-of-range annotation, isolated region), identifier of the annotation in question (e.g., car #d625), and the detected message. This is based on the rules defined in the Auto QA Rule Settings (see Figure 5) and the results recorded by the QA process (see Figure 6).

[0081] The dialog can accept confirmation operations from the user, and as stated in the speech bubble at the top, control is in place to prevent the process from progressing to the next workflow step, particularly for quality issues displayed with a "warning" status, unless the user performs the "confirmed" operation. This behavior corresponds to the flowchart in Figure 3 (steps S107 to S109, etc.), and controls are in place to limit or allow automatic progression depending on the check level.

[0082] The screen shown in Figure 9 is an example of an interactive annotation link display, where annotations where problems have been detected are visually highlighted, allowing users to immediately identify the detected location. As shown in the speech bubble at the bottom, an "association" with the target annotation has been made, and users can click on the relevant location to check the details.

[0083] Status management after a confirmation operation is, for example, as shown in Figure 6. The details of the confirmation operation performed by the user are recorded as the "Confirmation Status" and "Confirmation User" shown in Figure 6 and are maintained as part of quality assurance traceability. This makes it possible to clearly show who responded to which warning and how in subsequent quality reviews and reports.

[0084] The "Tags" section displayed on the left side of Figure 9 is a user interface for managing and checking the attribute information attached to the target annotation. "Annotation Class" is attribute information that assigns categorical meaning to the annotation, and in Figure 9, it is labeled "car." This is basic information for applying "annotation class" as a condition in the set rules (see Figures 5 and 8), and is used for rule operations such as "apply duplication check to annotations of the car class."

[0085] An "eye icon" is displayed next to each tag, which is a UI element that controls whether the annotation is visible or invisible. This allows users to work while increasing visibility by switching the display on or off for each tag. Annotation instances (e.g., car #0001 d625) are displayed in a list as "selected annotations." These correspond to the target annotations detected by the check, and correspond, for example, to the annotations and message fields shown in the dialog in the upper left of Figure 9.

[0086] FIG. 10 is a diagram showing an example of quality check processing using a visual language model (VLM) to make a visual judgment closer to that of a human being in the annotation quality evaluation support device according to this embodiment.

[0087] In this embodiment, in addition to the annotation target image and its annotation information, a prompt based on predetermined annotation requirements is input, and a visual language model, which is one of the AI ​​models, is made to perform an inference process on the validity of the annotation class, thereby automatically verifying the class classification result. The AI ​​model can include one or more of an object detection model, a region detection model, a visual language model, and a large-scale language model.

[0088] As shown in Figure 10, the prompt given as input specifies the class of the object to be classified (e.g., car, bus, truck, pedestrian, bicycle, motorcycle), and then describes the criteria and policy for classification judgment. For example, rules such as "A person riding a bicycle should be classified as 'bicycle', not 'pedestrian'" and "A person not riding a vehicle should be classified as 'pedestrian'" are described. This prompt can be edited and configured according to the project's specific annotation requirements.

[0089] At the same time, the target image has already been annotated, and in this example, it contains an annotation that classifies it as "truck." This image and its annotation information (i.e., the annotated image) are input into a visual language model (VLM) along with the above prompt. The VLM can be an integrated visual language model such as Qwen, or any other AI model that can determine the validity of annotations, such as an object detection model, region detection model, or large-scale language model. This enables a judgment that is close to that of a human, based on both visual features and contextual information.

[0090] The visual language model analyzes the objects contained in the input image according to the prompts and automatically infers the validity of the specified annotation class. If a discrepancy is detected, such as when the model predicts a "bus" but the annotation is a "truck," the device generates a warning message about the discrepancy and notifies the user.

[0091] As shown on the right side of Figure 10, the AutoQA result displays a warning message for the annotation (#d34f) stating, "The annotation class may be incorrect (expected: bus, current: truck)." This message is listed in the review interface shown in Figure 9 along with other rule-based detection results, and is presented to the user for confirmation and correction.

[0092] In this way, by utilizing the validity estimation of annotation classes by VLM, it is possible to mechanically supplement misclassifications that are often overlooked by manual visual inspection, contributing to the efficiency of the quality assurance process and the reduction of errors.In addition, by setting prompts, annotation requirements can be flexibly communicated to the model, enabling check processing to be performed in accordance with the specifications and policies of each project. [Industrial Applicability]

[0093] The annotation quality assurance support device, annotation quality assurance support method, and annotation quality assurance support program according to the present invention enable stable and efficient annotation quality assurance in the creation and management of training data used in machine learning and deep learning. They are particularly useful in industrial fields that require the preparation of highly accurate annotation data in a wide range of fields, such as autonomous driving, medical image analysis, robotics, natural language processing, video analysis, and satellite image recognition, which involve the construction of large-scale datasets.

[0094] Furthermore, by combining rule-based static check processing with dynamic and flexible decision-support processing using AI such as visual language models, this invention makes it possible to build an advanced quality control system while reducing manual quality assurance work. This not only improves the efficiency, labor savings, and quality of annotation work, but also provides additional value such as recording, reuse, and improved traceability of quality assurance results.

[0095] Therefore, the present invention can be incorporated into a variety of business systems, such as AI development support platforms, data annotation tools, quality control systems, and business flow management solutions, and is extremely useful in industry. [Explanation of symbols]

[0096] 1: Terminal 1-1~1-N: Terminal 2: Computer Systems 21: Input interface 22: Communication module 23: Storage device 24: Memory 25: Output interface 26: Processor 261: First Recording Section 262: First output section 263: First Executive Division 264: Second recording section 265: Second output section 266: Second Executive Division CN: communication line network

Claims

1. An annotation quality assurance support device that evaluates annotation quality, comprising: a first recording unit that records setting information of rules including check conditions for annotation quality assurance; a first output unit that outputs a setting screen for setting annotation quality assurance based on the setting information; a first execution unit that, during annotation work, refers to the setting information and performs a quality check on an annotation work screen; a second output unit that outputs a feedback screen that presents the result of the quality check to the worker; a second recording unit that records the worker's confirmation status of the results presented on the feedback screen; A second execution unit that performs judgment processing using an AI model on the annotation results; An annotation quality assurance support device including:

2. 2. The annotation quality assurance support device according to claim 1, wherein the setting information allows setting of check levels including at least an error, a warning in the annotation, and information for each rule.

3. The annotation quality assurance support device according to claim 1 or 2, wherein the first execution unit has a function of executing the quality check at the completion of each step of a workflow during annotation work, or as a process executed collectively on annotated data.

4. The annotation quality assurance support device according to claim 1 , wherein the second output unit classifies the results of the quality check into at least an error, a warning, and information and outputs the results.

5. The annotation quality assurance support device described in claim 1 or 2, wherein the second execution unit refers to the status of error resolution and warning confirmation by the worker, and allows transition to the next workflow step when all errors have been resolved, and further, depending on the project settings, allows progression to the next workflow step even if warning confirmation has not been completed.

6. The annotation quality assurance support device described in claim 1 or 2, wherein the second execution unit generates quality assurance feedback based on output results from an AI model including at least one of an object detection model, a region detection model, a visual language model, and a large-scale language model.

7. The annotation quality assurance support device of claim 6, wherein the second output unit is configured to display the output results from the AI ​​model in association with the worker's annotations so that the worker or reviewer can immediately confirm the contents.

8. 1. A computer-implemented method for supporting annotation quality assurance for evaluating annotation quality, comprising: recording setting information of rules including check conditions for annotation quality assurance; outputting a setting screen for setting annotation quality assurance based on the setting information; a step of performing a quality check on an annotation work screen by referring to the setting information during annotation work; outputting a feedback screen that presents the results of the quality check to the worker; a step of recording the worker's confirmation status regarding the results presented on the feedback screen; A step of performing a judgment process using an AI model on the annotation results. An annotation quality assurance support method, including:

9. The annotation quality assurance support method according to claim 8 , wherein the setting information allows setting of check levels including at least an error, a warning in the annotation, and information for each rule.

10. The annotation quality assurance support method according to claim 8 , wherein the quality check is performed after a process for advancing a workflow in an annotation task is performed.

11. The annotation quality assurance support method according to claim 8 , wherein the results of the quality check are presented on the feedback screen after being classified into at least an error, a warning, and information.

12. 9. The annotation quality assurance support method according to claim 8, wherein the confirmation status of the worker is referenced, and the workflow is not allowed to proceed to the next step until the annotation worker has completed eliminating errors and checking warnings.

13. one of the AI ​​models is a visual language model; The annotation quality assurance support method of claim 8, further comprising a step of processing a prompt according to annotation requirements and an annotated image as input into the visual language model, and generating quality assurance feedback based on the output results of the visual language model.

14. The annotation quality assurance support method of claim 13, wherein the feedback screen displays the output results of the AI ​​model in association with the worker's annotations, and is configured to highlight or select the corresponding annotation in response to a user interface operation on an item in the list, thereby allowing the worker or reviewer to intuitively confirm the content.

15. A program for causing a computer to execute the annotation quality assurance support method according to any one of claims 8 to 14.

Images