System and method for carrying out data annotation on supervised data
By introducing permission management module, front-end page and back-end server into the data labeling system, it supports multi-role collaboration and double-blind auditing, solving the problem of time-consuming and labor-intensive data and difficulty in ensuring the consistency of results, and achieving high-quality data Provide labeling and model training data.
Patent Information
- Application Number
- CN202411405858.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2025-02-11
AI Technical Summary
The existing technology has problems such as time-consuming and labor-intensive, result consistency and reliability in the data annotation process. Especially in the development of large-scale language models, the acquisition of high-quality data depends on precise and meticulous data annotation work. .
Provide a system and method for data annotation of supervised data. Through the combination of permission management module, front-end page and back-end server, users can log in and execute data annotation tasks in different roles, including task creation, annotation execution, auditing and Double-blind retrieval and other steps to ensure the quality and consistency of data labeling.
It realizes high-quality annotation of sample data, provides high-quality training data for model training, thereby promoting self-learning and continuous optimization of the model, and improving the efficiency and reliability of data annotation.
Smart Images
Figure CN119415946A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data labeling, and in particular to a system and method for labeling supervised data. Background Art
[0002] In the field of deep learning, especially in the development of large-scale language models, the quality and quantity of data have become key factors in determining model performance and generalization capabilities. However, obtaining high-quality data is not easy, and it relies on accurate and detailed data labeling. The labeling process is not only time-consuming and labor-intensive, but also easily affected by the subjectivity of the labelers, making it difficult to ensure the consistency and reliability of the labeling results.
[0003] To address these challenges, the industry has developed a series of technologies and tools to assist in data annotation. This includes developing detailed annotation specifications, using efficient annotation tools, and implementing multi-person collaboration and real-time quality control mechanisms. These technologies have improved the efficiency and quality of data annotation to a certain extent, but there are still some shortcomings. Summary of the invention
[0004] In response to the needs and shortcomings of current technological development, the present invention provides a system and method for labeling supervised data. By labeling sample data, high-quality training data is provided for model training, thereby promoting self-learning and continuous optimization of the model.
[0005] In a first aspect, the present invention provides a system for labeling supervised data. The technical solution adopted to solve the above technical problems is as follows:
[0006] A system for labeling supervised data, comprising:
[0007] The permission management module is used to support users to log in to the front-end page as task creators, annotation team members or double-blind team members, and limit each role to only access and operate functions and data within its permission range;
[0008] The front-end page is built with the Vue.js front-end framework, including the task creation page, the annotation tool interface, and the task management page accessed through the browser. The task creator creates and starts the annotation task through the task creation page, the annotation team members and the double-blind team members execute and submit the annotation task through the annotation tool interface, and the task creator reviews the annotation task through the task management page and puts the annotation task on the shelf or double-blindly returns it;
[0009] The backend server is built using the Spring Cloud microservice framework to provide a RESTful API interface for the front-end page to call for data interaction. It is also used to integrate with the Hive database to store and manage large-scale sample data sets.
[0010] The data storage module uses MySQL to store structured data and Hive as the big data storage and query engine to process large-scale sample data sets.
[0011] Optionally, the task creator involved creates and starts the labeling task through the task creation page, including:
[0012] After the task creator logs in to the front-end page through the permission management module, he enters the task creation page and performs the following operations: create a new labeling task, select the Hive data set to be labeled, set the double-blind data ratio, specify the labeling team and double-blind team members, and configure the label system used in the labeling process. Then start the labeling task, which automatically appears in the labeling task list on the front-end page.
[0013] Further optionally, the members of the annotation team and the double-blind team involved perform and submit the annotation task through the annotation tool interface, specifically including:
[0014] The annotation team members and the double-blind team members log in to the front-end page through the permission management module respectively, obtain the annotation task list, select an unexecuted annotation task, and then enter the annotation tool interface. The Web-based annotation tool performs data annotation work on the selected annotation task. During the annotation process, the annotator selects labels, changes labels or makes annotations on the data, and submits the annotation task after completing the data annotation work.
[0015] Optionally, the task creators involved review the labeling tasks through the task management page and perform operations such as listing or double-blind rejection of the labeling tasks, including:
[0016] The task creator logs in to the front-end page through the permission management module, obtains the labeling task list, selects a submitted labeling task, and then enters the task management page to review the original data and labeling results, calculates the consistency of the double-blind data, generates a double-blind consistency rate, and compares it with the preset threshold: when the double-blind consistency rate exceeds the preset threshold, the data corresponding to the labeling task will be put on the shelf; when the double-blind consistency rate does not exceed the preset threshold, the data corresponding to the labeling task will be double-blindly rejected.
[0017] Preferably, the labeling tasks involved include four states: unexecuted, in progress, completed and rejected, wherein the labeling tasks started by the task creator are displayed in the unexecuted state, the labeling tasks selected by the labeling team members and the double-blind team members are displayed in the in progress state, the labeling tasks submitted by the labeling team members and the double-blind team members are displayed in the completed state, the labeling tasks listed by the task creator are not displayed in the labeling list, and the labeling tasks rejected by the task creator are displayed in the rejected state;
[0018] The task creator logs in to the front-end page through the permission management module, obtains the annotation task list, tracks the status of the annotation tasks in the annotation task list, modifies the task information, and deletes the tasks.
[0019] In a second aspect, the present invention provides a method for labeling supervised data based on the system described in the first aspect. The technical solution adopted to solve the above technical problem is as follows:
[0020] A method for labeling supervised data, based on the system of the first aspect, comprises the following steps:
[0021] S1. The user logs in to the system as a task creator, enters the task creation page through the front-end page, creates and starts a labeling task, and the labeling task automatically appears in the labeling task list on the front-end page;
[0022] S2. The user logs in to the system as a member of the annotation team or a double-blind team, obtains a list of annotation tasks, selects an unexecuted annotation task, and then enters the annotation tool interface to execute and submit the annotation task;
[0023] S3. The user logs in to the system as a task creator, obtains the labeling task list through the front-end page, selects a submitted labeling task, and then enters the task management page to review the labeling task, and performs a listing or double-blind return operation on the data corresponding to the labeling task;
[0024] S4. For the double-blind returned labeling tasks, return to execute steps S2-S3 until the task creator puts the data corresponding to the labeling task on the shelf.
[0025] Optionally, execute step S1. After the task creator enters the task creation page, perform the following operations: create a new labeling task, select the Hive data set to be labeled, set the double-blind data ratio, specify the labeling team and double-blind team members, and configure the label system used in the labeling process, and then start the labeling task. The labeling task automatically appears in the labeling task list on the front-end page.
[0026] Further optionally, step S2 is executed. After the annotation team members and the double-blind team members enter the annotation tool interface, the Web-based annotation tool performs data annotation work on the selected annotation task. During the annotation process, the annotators select labels, change labels or make label remarks for the data, and submit the annotation task after completing the data annotation work.
[0027] Further optionally, step S3 is executed. After the task creator enters the task management page, he / she conducts a review based on the original data and the annotation results, calculates the consistency of the double-blind data, generates a double-blind consistency rate, and compares it with a preset threshold: when the double-blind consistency rate exceeds the preset threshold, the data corresponding to the annotation task is put on the shelf; when the double-blind consistency rate does not exceed the preset threshold, the data corresponding to the annotation task is double-blindly rejected.
[0028] Preferably, the labeling tasks include four states: unexecuted, in progress, completed and rejected. Among them, the labeling tasks started by the task creator are displayed in the unexecuted state, the labeling tasks selected by the labeling team members and the double-blind team members are displayed in the executed state, the labeling tasks submitted by the labeling team members and the double-blind team members are displayed in the completed state, the labeling tasks listed by the task creator are not displayed in the labeling list, and the labeling tasks rejected by the task creator are displayed in the rejected state.
[0029] The task creator can track the status of the annotation tasks in the annotation task list, modify the task information, and delete the tasks through the annotation task list on the front-end page.
[0030] Compared with the prior art, the system and method for labeling supervised data of the present invention have the following beneficial effects:
[0031] The present invention realizes the full-chain management of sample data acquisition, task distribution, data labeling and result processing, providing high-quality training data for large language model training, thereby promoting self-learning and continuous optimization of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Attached Figure 1 is a system architecture diagram of Embodiment 1 of the present invention;
[0033] Attached Figure 2 is a flowchart of a method according to Embodiment 2 of the present invention;
[0034] Attached Figure 3 It is a detailed flow chart of data annotation of the present invention. DETAILED DESCRIPTION
[0035] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.
[0036] Embodiment 1:
[0037] Combined with Figure 1 This embodiment proposes a system for labeling supervised data, which includes:
[0038] The permission management module is used to support users to log in to the front-end page as task creators, annotation team members or double-blind team members, and limit each role to only access and operate functions and data within its permission range;
[0039] The front-end page is built with the Vue.js front-end framework, including the task creation page, the annotation tool interface, and the task management page accessed through the browser. The task creator creates and starts the annotation task through the task creation page, the annotation team members and the double-blind team members execute and submit the annotation task through the annotation tool interface, and the task creator reviews the annotation task through the task management page and puts the annotation task on the shelf or double-blindly returns it;
[0040] The backend server is built using the Spring Cloud microservice framework to provide a RESTful API interface for the front-end page to call for data interaction. It is also used to integrate with the Hive database to store and manage large-scale sample data sets.
[0041] The data storage module uses MySQL to store structured data and Hive as the big data storage and query engine to process large-scale sample data sets.
[0042] In this embodiment, combined with the attached Figure 3 The task creator creates and starts the labeling task through the task creation page, including:
[0043] After the task creator logs in to the front-end page through the permission management module, he enters the task creation page and performs the following operations: create a new labeling task, select the Hive data set to be labeled, set the double-blind data ratio, specify the labeling team and double-blind team members, and configure the label system used in the labeling process. Then start the labeling task, which automatically appears in the labeling task list on the front-end page.
[0044] In this embodiment, combined with the attached Figure 3 , the annotation team members and double-blind team members perform and submit annotation tasks through the annotation tool interface, including:
[0045] The annotation team members and the double-blind team members log in to the front-end page through the permission management module respectively, obtain the annotation task list, select an unexecuted annotation task, and then enter the annotation tool interface. The Web-based annotation tool performs data annotation work on the selected annotation task. During the annotation process, the annotator selects labels, changes labels or makes annotations on the data, and submits the annotation task after completing the data annotation work.
[0046] In this embodiment, combined with the attached Figure 3 The task creator reviews the annotation task through the task management page and performs operations such as listing or double-blind rejection of the annotation task, including:
[0047] The task creator logs in to the front-end page through the permission management module, obtains the labeling task list, selects a submitted labeling task, and then enters the task management page to review the original data and labeling results, calculates the consistency of the double-blind data, generates a double-blind consistency rate, and compares it with the preset threshold: when the double-blind consistency rate exceeds the preset threshold, the data corresponding to the labeling task will be put on the shelf; when the double-blind consistency rate does not exceed the preset threshold, the data corresponding to the labeling task will be double-blindly rejected.
[0048] It should be added that the labeling tasks include four states: unexecuted, in progress, completed, and rejected. Among them, the labeling tasks started by the task creator are displayed in the unexecuted state, the labeling tasks selected by the labeling team members and the double-blind team members are displayed in the in progress state, the labeling tasks submitted by the labeling team members and the double-blind team members are displayed in the completed state, the labeling tasks listed by the task creator are not displayed in the labeling list, and the labeling tasks rejected by the task creator are displayed in the rejected state. The task creator logs in to the front-end page through the permission management module, obtains the labeling task list, and tracks the status of the labeling tasks in the labeling task list, modifies the task information, and deletes the tasks.
[0049] It should be added that the consistency of double-blind data can be calculated using cross-validation, confusion matrix, Kappa coefficient or rule-based methods.
[0050] Embodiment 2:
[0051] Combined with Figure 2 , 3 This embodiment proposes a method for data labeling of supervised data, which is based on the authority management module, front-end page, back-end server and data storage module of the system described in the first embodiment, and includes the following implementation steps:
[0052] S1. The user logs in to the system as a task creator, enters the task creation page through the front-end page, and performs the following operations: create a new labeling task, select the Hive data set to be labeled, set the double-blind data ratio, specify the labeling team and double-blind team members, and configure the label system used in the labeling process, and then start the labeling task. The labeling task automatically appears in the labeling task list on the front-end page.
[0053] S2. The user logs in to the system as a member of the labeling team or a double-blind team member, obtains a list of labeling tasks, selects an unexecuted labeling task, and then enters the labeling tool interface. The web-based labeling tool performs data labeling for the selected labeling task. During the labeling process, the labeler selects labels, changes labels, or makes label remarks for the data. After completing the data labeling work, the labeling task is submitted.
[0054] S3. The user logs in to the system as a task creator, obtains the labeling task list through the front-end page, selects a submitted labeling task, and then enters the task management page to review the original data and labeling results, calculates the consistency of the double-blind data, generates a double-blind consistency rate, and compares it with the preset threshold: when the double-blind consistency rate exceeds the preset threshold, the data corresponding to the labeling task is put on the shelf; when the double-blind consistency rate does not exceed the preset threshold, the data corresponding to the labeling task is double-blindly rejected.
[0055] S4. For the double-blind returned labeling tasks, return to execute steps S2-S3 until the task creator puts the data corresponding to the labeling task on the shelf.
[0056] It should be added that the labeling tasks include four states: unexecuted, in progress, completed, and rejected. Among them, the labeling tasks started by the task creator are displayed in the unexecuted state, the labeling tasks selected by the labeling team members and the double-blind team members are displayed in the in progress state, the labeling tasks submitted by the labeling team members and the double-blind team members are displayed in the completed state, the labeling tasks listed by the task creator are not displayed in the labeling list, and the labeling tasks rejected by the task creator are displayed in the rejected state. The task creator logs in to the front-end page through the permission management module, obtains the labeling task list, and tracks the status of the labeling tasks in the labeling task list, modifies the task information, and deletes the tasks.
[0057] It should be added that the consistency of double-blind data can be calculated using cross-validation, confusion matrix, Kappa coefficient or rule-based methods.
[0058] In summary, the system and method for labeling supervised data of the present invention provide high-quality training data for model training by labeling sample data, thereby promoting self-learning and continuous optimization of the model.
[0059] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.
Claims
1. A system for labeling supervised data, characterized in that: It includes: The permission management module is used to support users to log in to the front-end page as task creators, annotation team members or double-blind team members, and limit each role to only access and operate functions and data within its permission range; The front-end page is built with the Vue.js front-end framework, including the task creation page, the annotation tool interface, and the task management page accessed through the browser. The task creator creates and starts the annotation task through the task creation page, the annotation team members and the double-blind team members execute and submit the annotation task through the annotation tool interface, and the task creator reviews the annotation task through the task management page and puts the annotation task on the shelf or double-blindly returns it; The backend server is built using the Spring Cloud microservice framework to provide a RESTful API interface for the front-end page to call for data interaction. It is also used to integrate with the Hive database to store and manage large-scale sample data sets. The data storage module uses MySQL to store structured data and Hive as the big data storage and query engine to process large-scale sample data sets.
2. A system for labeling supervised data according to claim 1, characterized in that: The task creator creates and starts the labeling task through the task creation page, including: After the task creator logs in to the front-end page through the permission management module, he enters the task creation page and performs the following operations: create a new labeling task, select the Hive data set to be labeled, set the double-blind data ratio, specify the labeling team and double-blind team members, and configure the label system used in the labeling process. Then start the labeling task, which automatically appears in the labeling task list on the front-end page.
3. A system for labeling supervised data according to claim 2, characterized in that: The annotation team members and double-blind team members perform and submit annotation tasks through the annotation tool interface, including: The annotation team members and the double-blind team members log in to the front-end page through the permission management module respectively, obtain the annotation task list, select an unexecuted annotation task, and then enter the annotation tool interface. The Web-based annotation tool performs data annotation work on the selected annotation task. During the annotation process, the annotator selects labels, changes labels or makes annotations on the data, and submits the annotation task after completing the data annotation work.
4. A system for labeling supervised data according to claim 3, characterized in that: The task creator reviews the annotation task through the task management page and performs operations such as listing or double-blind rejection of the annotation task, including: The task creator logs in to the front-end page through the permission management module, obtains the labeling task list, selects a submitted labeling task, and then enters the task management page to review the original data and labeling results, calculates the consistency of the double-blind data, generates a double-blind consistency rate, and compares it with the preset threshold: when the double-blind consistency rate exceeds the preset threshold, the data corresponding to the labeling task will be put on the shelf; when the double-blind consistency rate does not exceed the preset threshold, the data corresponding to the labeling task will be double-blindly rejected.
5. A system for labeling supervised data according to claim 4, characterized in that: The labeling tasks include four states: unexecuted, in progress, completed, and rejected. The labeling tasks started by the task creator are displayed in the unexecuted state, the labeling tasks selected by the labeling team members and double-blind team members are displayed in the in progress state, the labeling tasks submitted by the labeling team members and double-blind team members are displayed in the completed state, the labeling tasks listed by the task creator are not displayed in the labeling list, and the labeling tasks rejected by the task creator are displayed in the rejected state. The task creator logs in to the front-end page through the permission management module, obtains the annotation task list, tracks the status of the annotation tasks in the annotation task list, modifies the task information, and deletes the tasks.
6. A method for labeling supervised data, characterized in that: It is based on the system as claimed in claim 1, comprising the following steps: S1. The user logs in to the system as a task creator, enters the task creation page through the front-end page, creates and starts a labeling task, and the labeling task automatically appears in the labeling task list on the front-end page; S2. The user logs in to the system as a member of the annotation team or a double-blind team, obtains a list of annotation tasks, selects an unexecuted annotation task, and then enters the annotation tool interface to execute and submit the annotation task; S3. The user logs in to the system as a task creator, obtains the labeling task list through the front-end page, selects a submitted labeling task, and then enters the task management page to review the labeling task, and performs a listing or double-blind return operation on the data corresponding to the labeling task; S4. For the double-blind returned labeling tasks, return to execute steps S2-S3 until the task creator puts the data corresponding to the labeling task on the shelf.
7. A method for labeling supervised data according to claim 6, characterized in that: Execute step S1. After the task creator enters the task creation page, perform the following operations: create a new labeling task, select the Hive data set to be labeled, set the double-blind data ratio, specify the labeling team and double-blind team members, and configure the label system used in the labeling process. Then start the labeling task. The labeling task automatically appears in the labeling task list on the front-end page.
8. The method for labeling supervised data according to claim 7, characterized in that: Execute step S2. After the annotation team members and the double-blind team members enter the annotation tool interface, the web-based annotation tool performs data annotation work on the selected annotation task. During the annotation process, the annotator selects labels, changes labels, or makes annotations on the data. After completing the data annotation work, the annotation task is submitted.
9. A method for labeling supervised data according to claim 8, characterized in that: Execute step S3. After the task creator enters the task management page, he / she will review the original data and the annotation results, calculate the consistency of the double-blind data, generate a double-blind consistency rate, and compare it with the preset threshold: when the double-blind consistency rate exceeds the preset threshold, the data corresponding to the annotation task will be put on the shelf; when the double-blind consistency rate does not exceed the preset threshold, the data corresponding to the annotation task will be double-blind rejected.
10. A method for labeling supervised data according to claim 9, characterized in that: The labeling tasks include four states: unexecuted, in progress, completed, and rejected. The labeling tasks started by the task creator are displayed in the unexecuted state, the labeling tasks selected by the labeling team members and double-blind team members are displayed in the in progress state, the labeling tasks submitted by the labeling team members and double-blind team members are displayed in the completed state, the labeling tasks listed by the task creator are not displayed in the labeling list, and the labeling tasks rejected by the task creator are displayed in the rejected state. The task creator can track the status of the annotation tasks in the annotation task list, modify the task information, and delete the tasks through the annotation task list on the front-end page.