Data annotation method, device and equipment based on large model and medium

Through the two data annotation and consistency rate calculation methods, the problem of low accuracy of large model annotation data is solved, the model training effect is improved, and the output accuracy is achieved.

CN119046682APending Publication Date: 2024-11-29SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411157431.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The prior art is difficult to improve the accuracy of the labeled data of large models, which affects the model training effect.

Method used

Through the two data annotation and consistency rate calculation method, the data set to be marked and double-blind are marked using the label to be used to obtain the first marked data and the second marked data, and determine whether the data is put on the shelves based on the consistency rate result. Model training.

Benefits of technology

It improves the accuracy of output results after large model training and improves the accuracy of labeled data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046682A_ABST
    Figure CN119046682A_ABST
Patent Text Reader

Abstract

The invention discloses a data annotation method and device based on a large model, equipment and a medium, and relates to the technical field of data annotation, and the method comprises the following steps: annotating all data in a to-be-annotated data set by using a to-be-used label to obtain first annotated data; performing double-blind labeling on partial data in the to-be-labeled data set by using the to-be-used label to obtain second labeled data; the partial data is data determined from the to-be-labeled data set based on a preset labeling proportion; performing consistency rate calculation on the first labeled data and the second labeled data to obtain a consistency rate result; and determining whether the first labeled data is to-be-shelved data based on the consistency rate result, and shelving the first labeled data if the first labeled data is to-be-shelved data, so that a large model can use the shelved data to train. In this way, labeled data with higher accuracy can be provided for large model training.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data annotation technology, and in particular to a data annotation method, device, equipment and medium based on a large model. Background Art

[0002] Big model is short for Large Language Model (LLM), which is generally used to understand and generate human language. The model parameters of big model are relatively more than those of general models. In addition, the design and training of big model are aimed at providing more powerful and accurate model performance to cope with more complex and larger data sets or tasks. In order to have more powerful and accurate model performance, high-quality labeled data is essential. The data labeling involved refers to providing labeled data sets for big models.

[0003] From the above, it can be seen that how to improve the accuracy of the labeled data provided for large models is an urgent problem to be solved. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a data labeling method, device, equipment and medium based on a large model, which can improve the accuracy of labeled data based on two data labeling and consistency calculation, so that the large model can be trained based on the labeled data with higher accuracy, thereby improving the accuracy of the output results of the trained large model. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a data annotation method based on a large model, comprising:

[0006] Using the labels to be used to label all data in the to-be-labeled data set to obtain first labeled data;

[0007] Double-blind labeling is performed on part of the data in the to-be-labeled data set using the to-be-labeled label to obtain second labeled data; the part of the data is data determined from the to-be-labeled data set based on a preset labeling ratio;

[0008] Calculating the consistency rate of the first labeled data and the second labeled data to obtain a consistency rate result;

[0009] Based on the consistency rate result, it is determined whether the first labeled data is data to be put on the shelf. If the first labeled data is data to be put on the shelf, the first labeled data is put on the shelf so that the large model can use the put-on data for training.

[0010] Optionally, before obtaining the first labeled data and the second labeled data, the method further includes:

[0011] Creating a labeling task and marking the task status of the labeling task as a pending labeling state;

[0012] Starting the labeling task to switch the task status of the labeling task from a to-be-labeled state to a labeling state, and triggering the step of labeling all data in the to-be-labeled data set with the labels to be used;

[0013] Correspondingly, after obtaining the first labeled data and the second labeled data, the method further includes:

[0014] The task state of the labeling task is switched from a labeling in progress state to a labeling completed state, and the step of calculating the consistency rate of the first labeled data and the second labeled data is triggered.

[0015] Optionally, the labeling task is a task constructed based on a plurality of first member identifiers, a plurality of second member identifiers, the data set to be labeled and the labels to be used;

[0016] Correspondingly, the step of labeling all data in the to-be-labeled data set using the to-be-used labels to obtain first labeled data includes:

[0017] After the labeling task is started, all data in the to-be-labeled data set are allocated to the first members corresponding to the first member identifiers respectively according to a preset allocation ratio, and the labeled data uploaded by the first members is obtained after the allocated data is labeled with the to-be-used labels, so as to obtain the first labeled data;

[0018] Correspondingly, the step of using the to-be-used label to double-blindly label part of the data in the to-be-labeled data set to obtain second labeled data includes:

[0019] After starting the labeling task, part of the data in the data set to be labeled is allocated to the second members corresponding to the several second member identifiers respectively, and the labeled data uploaded by the second members after labeling the part of the data with the labels to be used is obtained to obtain the second labeled data.

[0020] Optionally, before labeling all the data in the to-be-labeled data set with the labels to be used, the method further includes:

[0021] Obtaining a data set import instruction, wherein the data set import instruction includes a database address, login information, a data table name, and fields to be imported;

[0022] After determining a target database based on the database address and logging into the target database based on the login information, determining a target data table corresponding to the data table name from the logged-in target database;

[0023] The data to be imported corresponding to the field to be imported is determined from the target data table to obtain a data set to be imported, and the data set to be imported is imported into a local preset cache so that the data set to be labeled is selected from the preset cache.

[0024] Optionally, before labeling all the data in the to-be-labeled data set with the labels to be used, the method further includes:

[0025] Acquire user login information, and determine a target user role and a target user authority corresponding to the target user role based on the user login information; wherein different user roles correspond to different user authorities;

[0026] If the target user authority includes tag management authority, when a tag online instruction is obtained, a tag online operation is performed on the offline tag corresponding to the tag online instruction, so as to select the tag to be used from the online tags.

[0027] Optionally, the calculating the consistency rate of the first labeled data and the second labeled data to obtain a consistency rate result includes:

[0028] Target labeled data corresponding to part of the data in the to-be-labeled data set is determined from the first labeled data, and a consistency rate is calculated based on the target labeled data and the second labeled data to obtain a consistency rate result.

[0029] Optionally, the determining whether the first labeled data is data to be put on the shelf based on the consistency rate result includes:

[0030] Determine whether the consistency rate result is greater than a preset consistency rate threshold;

[0031] If the consistency rate result is greater than the preset consistency rate threshold, determining that the first labeled data is the data to be put on the shelf, and changing the labeling identifier of the data set to be labeled from the first identifier to the second identifier;

[0032] The first identifier is used to indicate that the data set to be labeled is in an unlabeled state; and the second identifier is used to indicate that the data set to be labeled is in a labeled state.

[0033] In a second aspect, the present application provides a data annotation device based on a large model, comprising:

[0034] A first labeling module, used to label all data in the to-be-labeled data set using the to-be-used labels to obtain first labeled data;

[0035] A second labeling module is used to perform double-blind labeling on part of the data in the to-be-labeled data set using the to-be-used label to obtain second labeled data; the part of the data is data determined from the to-be-labeled data set based on a preset labeling ratio;

[0036] A consistency rate calculation module, used for calculating the consistency rate of the first labeled data and the second labeled data to obtain a consistency rate result;

[0037] The data uploading module is used to determine whether the first labeled data is the data to be put on the shelf based on the consistency rate result. If the first labeled data is the data to be put on the shelf, the first labeled data is put on the shelf so that the large model can use the put-on data for training.

[0038] In a third aspect, the present application provides an electronic device, including:

[0039] Memory, used to store computer programs;

[0040] A processor is used to execute the computer program to implement the aforementioned large model-based data labeling method.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned large model-based data labeling method.

[0042] In the present application, all data in the data set to be labeled are labeled with the labels to be used to obtain first labeled data; part of the data in the data set to be labeled is double-blind labeled with the labels to be used to obtain second labeled data; the partial data is data determined from the data set to be labeled based on a preset labeling ratio; the consistency rate is calculated for the first labeled data and the second labeled data to obtain a consistency rate result; based on the consistency rate result, it is determined whether the first labeled data is the data to be put on the shelf, and if the first labeled data is the data to be put on the shelf, the first labeled data is put on the shelf so that the large model can use the data after being put on the shelf for training. It can be seen that in the process of data labeling based on the large model, the present application needs to calculate the consistency rate based on the first labeled data obtained after labeling all the data in the data set to be labeled and the second labeled data obtained after double-blind labeling of part of the data in the data set to be labeled, so as to determine whether the large model can use the first labeled data for training based on the consistency rate result. In this way, this solution improves the accuracy of the labeled data based on two data labeling and consistency calculation, so that the large model can be trained based on the more accurate labeled data, thereby improving the accuracy of the output results of the trained large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 A flow chart of a data annotation method based on a large model disclosed in this application;

[0045] Figure 2 This is an indication of a labeling task disclosed in this application;

[0046] Figure 3 A labeling task label disclosed in this application indicates the intent;

[0047] Figure 4 A labeling task status representation disclosed in this application;

[0048] Figure 5 A data set disclosed in this application is intended to represent;

[0049] Figure 6 A data representation scheme disclosed in this application;

[0050] Figure 7 A data annotation flow chart disclosed in this application;

[0051] Figure 8 This is a schematic diagram of the structure of a data annotation device based on a large model disclosed in this application;

[0052] Fig. 9 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Large models are commonly used to understand and generate human language, and the design and training of large models are intended to provide more powerful and accurate model performance to cope with more complex and larger data sets or tasks. In order to have more powerful and accurate model performance, high-quality labeled data is essential. How to improve the accuracy of labeled data provided for large models is a problem that needs to be solved urgently. To this end, the present application provides a data labeling method based on a large model, which can improve the accuracy of labeled data based on two data labelings and the calculation of consistency rate, so that the large model can be trained based on the more accurate labeled data, thereby improving the accuracy of the output results of the trained large model.

[0055] See also Figure 1 As shown, the embodiment of the present invention discloses a data annotation method based on a large model, including:

[0056] Step S11: label all data in the to-be-labeled data set using the labels to be used to obtain first labeled data.

[0057] In this embodiment, before annotating all the data in the data set to be annotated with the labels to be used, it is necessary to first create a labeling task and mark the task status of the labeling task as the to-be-annotated state; wherein the labeling task is specifically a task constructed based on a number of first member identifiers, a number of second member identifiers, a data set to be annotated, and labels to be used. Specifically, in the process of creating the labeling task, the data administrator needs to select a number of first member identifiers, two second member identifiers, a preset labeling ratio, labels to be used, and select the data set to be annotated from the local preset cache. It is understandable that when the data administrator selects the first member and the second member, the identifiers corresponding to the first member and the second member are selected.

[0058] In one specific implementation, the identifiers corresponding to the first member and the second member may be a unique string of numbers; in another specific implementation, the identifiers corresponding to the first member and the second member may be a unique string of English letters. It can be understood that the identifiers corresponding to the above members can be determined according to actual needs, and any information that can characterize the identity of the members can be applicable to this embodiment, and no specific limitation is imposed herein.

[0059] In this embodiment, after the labeling task is created, the labeling task can be started based on the task start instruction issued by the data administrator. It is worth mentioning that the purpose of starting the labeling task is to distribute the data to be labeled to different members so that different members can label the data to be labeled with the labels to be used. Specifically, the labeling task is started and the task status of the labeling task is switched from the to-be-labeled state to the labeling state, and the step of labeling all the data in the to-be-labeled data set with the labels to be used is triggered, so as to obtain the first labeled data corresponding to all the data in the to-be-labeled data set.

[0060] In this embodiment, labeling all the data in the data set to be labeled with the label to be used to obtain the first labeled data may include: after creating and starting the labeling task, all the data in the data set to be labeled needs to be distributed to the first members corresponding to several first member identifiers according to a preset distribution ratio, and the labeled data uploaded by the first member after labeling the allocated data with the label to be used is obtained to obtain the first labeled data. In a specific embodiment, the preset distribution ratio can be 1:1:1:1; wherein the number of digits 1 is the number of first member identifiers, that is, all the data in the data set to be labeled is evenly distributed to the first members corresponding to several first member identifiers. It is worth mentioning that the preset distribution ratio can also be set according to actual needs, which will not be described one by one here.

[0061] It should be noted that the creation and opening of labeling tasks are specifically performed through the labeling task management module. Among them, the labeling task management module is responsible for the management of labeling tasks, and its main functions include creating data labeling tasks, opening data labeling tasks, labeling data, submitting data labeling tasks, calculating the consistency rate of labeled data, and putting labeled data on the shelves.

[0062] It is worth mentioning that only after obtaining the corresponding instructions issued by the data administrator can the data annotation task be created and started based on the instructions, that is, the data administrator has the authority to create and start the data annotation task. Among them, the data administrator and the members who annotate the data (including the first member corresponding to the first member identifier and the second member corresponding to the second member identifier) ​​are all created by the super administrator (i.e., the system administrator) through the user and role management module.

[0063] Among them, for the user and role management module, in one embodiment, it can be constructed using the open source Ruo Yi framework. The Ruo Yi framework is a simple and easy-to-use open source framework based on Spring Boot (i.e., an open source application framework on the Java platform) and Bootstrap (i.e., a Web framework). It is a lightweight open source framework suitable for the rapid development of small to medium-sized projects. It provides rich functions and components to improve development efficiency. It is worth mentioning that the user and role management module can also be built using other frameworks according to actual needs. Any framework that can establish the user and role management module can be applied to this embodiment, and it is not specifically limited here.

[0064] In the user and role management module, the default user admin, whose role is super administrator, has the authority to log in to the user and role management module and to create and maintain other roles and users. It is worth mentioning that the other roles and users created by the default user admin do not have the authority to operate the user and role management module, and different roles correspond to different permissions.

[0065] In a specific implementation, the default user admin with the role of super administrator can create three different roles, namely data administrator, first member and second member, and grant them different permissions. It is understandable that the number of data administrators is not limited here, and can be one person or multiple people. The number of first members is not less than one person, and the number of second members is two people. Among them, the data administrator is a role that manages data and tasks, and has the following permissions: responsible for the import and export of data, the creation and maintenance of labels, and the creation and allocation of labeling tasks; the system administrator is a role that manages users, roles, and labels; the first member and the second member are roles responsible for labeling data.

[0066] It is worth mentioning that the difference between the first member and the second member is that the first member needs to use the labels to be used to label all the data to be labeled in the data set to be labeled, while the second member needs to use the labels to be used to perform double-blind labeling on part of the data to be labeled in the data set to be labeled. The purpose of using double-blind labeling to label the data to be labeled is to eliminate the possible deviation in the accuracy of the labeled data due to subjective bias and personal preference in the consciousness of the participants. Double-blind labeling is to let two members participating in double-blind labeling label the same data to be labeled during the labeling process.

[0067] It is worth mentioning that when selecting the first member and the second member, the two members should be different. In this way, the reliability of the results obtained by the participants in labeling the data to be labeled can be ensured, which can effectively improve the accuracy of the labeled data.

[0068] In this embodiment, the tags to be used when creating the labeling task are specifically selected from the online tags. The label management module is responsible for the online operation of the tags, and the label management module also includes management operations such as adding, deleting, and offline of tags; wherein each tag corresponds to a tag identifier. In addition, the tag is a mark and description added to the data, which is a kind of labeling, annotation and understanding of the data by humans or systems. For example, the tags can include life, military, entertainment, parent-child, correct, wrong, etc. By adding tags to the data, the embodiment of the present application can make the data easier to understand and use, while improving the availability and value of the data. Data tags are an important input for machine learning and artificial intelligence applications. Labeling the data before inputting the data can improve the classification ability and prediction accuracy of the model. It is worth mentioning that the data administrator and the system administrator have the management authority over the tags, and the online and offline operations of the data administrator and the system administrator on the tags determine whether the tags can be used.

[0069] In this embodiment, the specific operations for the online listing of tags and the selection of tags to be used are: obtaining user login information, and determining the target user role and the target user permissions corresponding to the target user role based on the user login information; wherein different user roles correspond to different user permissions; if the target user permissions include tag management permissions, then when the tag online instruction is obtained, the tag online operation is performed on the offline tag corresponding to the tag online instruction, so as to select the tag to be used from the online tags.

[0070] In a specific implementation, the target user's rights can be identified based on the user's account. For example, when assigning an account to a user, the first five digits of the serial number correspond to roles with different rights. It is worth mentioning that any information that can uniquely characterize the corresponding rights of the user can be applied to this embodiment, and it is not specifically limited here. In addition, the identifier corresponding to the tag can be a unique string of numbers or a unique string of English letters. It can be understood that the identifier corresponding to the tag can be determined according to actual needs, and any information that can characterize the tag can be applied to this embodiment, and it is not specifically limited here.

[0071] In this embodiment, the selection of labels to be used is specifically that the data administrator uses the label management module to select the labels to be used, so that the first member and the second member can select appropriate labels from the labels to be used according to the characteristic information corresponding to the data to be labeled during the data labeling process. In a specific implementation, the characteristic information is the content, question and answer corresponding to the data to be labeled; the content is the text content corresponding to the data to be labeled, which comes from crawler collection and other system collation; the question is the question raised based on the above text content; the answer is the answer to the question obtained based on the content and question. It can be understood that the embodiment of the present application can also use the type corresponding to the data to be labeled as the characteristic information corresponding to the above data to be labeled, which will not be listed one by one here.

[0072] In this embodiment, the dataset to be annotated required when creating the annotation task is specifically selected from the local preset cache of the system, and the dataset stored in the preset cache is specifically managed by the dataset management module. The functions of the dataset management module include importing and exporting datasets. It is worth mentioning that the data administrator has access and operation rights to the dataset management module. Among them, a dataset is a collection of similar data, and a dataset can contain multiple similar data.

[0073] In this embodiment, the data set import function refers to importing the data set to be imported from the target database into the local preset cache when receiving the data set import instruction. Specifically, before annotating all the data in the data set to be annotated with the tags to be used, it can also include: obtaining the data set import instruction, the data set import instruction includes the database address, login information, data table name and the field to be imported; after determining the target database based on the database address and logging in to the target database based on the login information, determining the target data table corresponding to the data table name from the logged in target database; determining the data to be imported corresponding to the field to be imported from the target data table to obtain the data set to be imported, and importing the data set to be imported into the local preset cache, so as to select the data set to be annotated from the preset cache. It can be understood that a data set can contain multiple data, and one data belongs to only one data set.

[0074] In a specific implementation, the target database may be a MySQL database (i.e., a relational database management system). It is worth mentioning that the target database may also be replaced by other tools capable of storing data, such as hive (i.e., a data warehouse tool), according to actual needs. Any data pool capable of storing data and supporting data import and data export functions may be applicable to this embodiment and is not specifically limited herein.

[0075] Step S12: double-blind labeling is performed on part of the data in the to-be-labeled data set using the to-be-used label to obtain second labeled data; the part of the data is data determined from the to-be-labeled data set based on a preset labeling ratio.

[0076] In this embodiment, after the labeling task is created and started, it is necessary to double-blind label part of the data in the to-be-labeled data set using the to-be-used labels, so as to obtain the second labeled data. Specifically, after the labeling task is started, part of the data in the to-be-labeled data set is allocated to the second members corresponding to the second member identifiers, and the labeled data uploaded by the second members after labeling part of the data using the to-be-used labels is obtained, so as to obtain the second labeled data.

[0077] It is worth mentioning that part of the data is data determined from the data set to be labeled based on a preset labeling ratio. In one implementation, the preset labeling ratio may be one-half, that is, one-half of all the data in the data set to be labeled is allocated to the second members corresponding to the plurality of second member identifiers. It is worth mentioning that the preset labeling ratio may also be set according to actual needs, which will not be described one by one here.

[0078] Among them, the specific process when the second member performs double-blind labeling on part of the data in the to-be-labeled data set is as follows: after any member of the second member labels part of the data in the to-be-labeled data set using the to-be-used label, the first labeled data is obtained; after another member of the second member labels part of the data in the to-be-labeled data set using the to-be-used label, the second labeled data is obtained. Then, the double-blind consistency rate is calculated for the first labeled data and the second labeled data to obtain the double-blind consistency rate result. If the double-blind consistency rate result is higher than the preset double-blind consistency rate threshold, the first labeled data or the second labeled data is used as the second labeled data. If the double-blind consistency rate result is lower than or equal to the preset double-blind consistency rate threshold, the arbitration personnel can use the to-be-used label to re-label the data to be labeled in the to-be-labeled data set corresponding to the inconsistent labeled data or part of the data in the to-be-labeled data set to obtain the third labeled data, and determine the second labeled data corresponding to part of the data in the to-be-labeled data set from the first labeled data, the second labeled data and the third labeled data using the principle of minority obeys majority. Among them, the inconsistent labeled data is the inconsistent labeled data in the first labeled data and the second labeled data.

[0079] Of course, after obtaining the first labeled data and the second labeled data, the arbitrator can directly use the labels to be used to re-label the data to be labeled in the data set to be labeled corresponding to the inconsistent labeled data to obtain the third labeled data, and determine the second labeled data using the principle of minority obeys majority.

[0080] Step S13: Calculate the consistency rate of the first labeled data and the second labeled data to obtain a consistency rate result.

[0081] In this embodiment, after obtaining the first labeled data uploaded by the first member after labeling the assigned data with the label to be used, and the second labeled data uploaded by the second member after labeling part of the data with the label to be used, that is, after obtaining the labeled data submitted by different members, it is necessary to use a preset consistency rate algorithm to calculate the consistency rate of the first labeled data and the second labeled data. Specifically, calculating the consistency rate of the first labeled data and the second labeled data to obtain a consistency rate result can include: determining the target labeled data corresponding to part of the data in the to-be-labeled data set from the first labeled data, and calculating the consistency rate based on the target labeled data and the second labeled data to obtain a consistency rate result.

[0082] It is worth mentioning that after obtaining the first labeled data and the second labeled data submitted by the first member and the second member, the task status corresponding to the labeling task is determined to be a labeling completion status. Specifically, after obtaining the first labeled data and the second labeled data, it can also include: switching the task status of the labeling task from a labeling in progress state to a labeling completion state, and triggering a step of calculating the consistency rate of the first labeled data and the second labeled data.

[0083] Step S14: determine whether the first labeled data is data to be put on the shelf based on the consistency rate result; if the first labeled data is data to be put on the shelf, put the first labeled data on the shelf so that the large model can use the put-on data for training.

[0084] In this embodiment, after the consistency rate calculation is performed on the first labeled data and the second labeled data to obtain the consistency rate result, it is necessary to determine whether the first labeled data is the data to be put on the shelf based on the consistency rate result. Specifically, determining whether the first labeled data is the data to be put on the shelf based on the consistency rate result may include: judging whether the consistency rate result is greater than a preset consistency rate threshold; if the consistency rate result is greater than the preset consistency rate threshold, determining that the first labeled data is the data to be put on the shelf, and putting the first labeled data on the shelf, and changing the labeling identifier of the data set to be labeled from the first identifier to the second identifier; wherein the first identifier is used to indicate that the data set to be labeled is in an unlabeled state; and the second identifier is used to indicate that the data set to be labeled is in a labeled state. If the consistency rate result is not greater than the preset consistency rate threshold, the labeling failure information is returned, and the first member and the second member are instructed to re-label the data to be labeled in the data set to be labeled corresponding to the labeling failure information after obtaining the labeling failure information, so as to obtain new first labeled data and second labeled data, and the consistency rate calculation is performed on the first labeled data and the second labeled data again.

[0085] In a specific implementation, the first identifier used to indicate that the dataset to be labeled is in an unlabeled state may be the number 0, and the second identifier used to indicate that the dataset to be labeled is in a labeled state may be the number 1. It is understandable that the first identifier and the second identifier may also be set according to actual needs, and any information that can be used to indicate the labeling state of the dataset is applicable to this embodiment, and they are not listed one by one here.

[0086] Among them, if the consistency rate result corresponding to the first labeled data is greater than the preset consistency rate threshold, the first labeled data is determined as high consistency rate data; it can be understood that the higher the consistency rate corresponding to the labeled data, the higher the quality corresponding to the labeled data, the higher the accuracy of the data labeling, and the more suitable it is to participate in the training of the large model.

[0087] In this embodiment, after the labeled data is put on the shelves, the labeled data can be separately classified as the put on the shelves data, and the put on the shelves labeled data can be stored in the target data set, and the target data set can be exported based on the data set export instruction issued by the data administrator. It is worth mentioning that the export of the data set refers to exporting the target data set to a preset storage file or a preset database when the data set export instruction is received, so that the large model can obtain the target data set from the preset storage file or the preset database, and train based on the labeled data in the target data set. Among them, in one specific embodiment, the preset storage file is a CSV file (i.e., Comma Separated Values, a plain text file), and in another specific embodiment, the preset database is a MySQL database; it can be understood that any preset storage file or preset database that can store data and support data import and data export functions can be set according to actual needs, and will not be repeated here.

[0088] It is worth mentioning that when the large model is trained using labeled data, the large model will output predicted labels corresponding to the labeled data based on the text content, questions, and answers of the labeled data, and compare the predicted labels with the true labels of the labeled data to determine the error between the predicted labels and the true labels, thereby obtaining the error results, and adjusting the parameters of the large model based on the error results, so that the large model tends to converge and stabilize.

[0089] In this embodiment, the data storage module stores a labeling task table, a labeling task label table, a labeling task status table, a data set table, and a data table. Among them, the labeling task table is used to record the relevant information of each labeling task, such as Figure 2 As shown, Figure 2 is a labeling task table. Figure 2The relevant information of each labeling task recorded in is as follows: primary key identifier, creator, creation time, updater, update time, task name, description, status, dataset to be labeled, first member identifier, second member identifier, preset labeling ratio, double-blind consistency rate, total data, labeled data, whether to delete, operation manual name, operation manual path and label identifier. Among them, the primary key identifier is used to uniquely identify a record, and duplicate information cannot appear, and the information is not allowed to be empty; the creator, creation time, updater and update time are the creator, creation time and updater and update time of the labeling task information; the status describes the task status, including the status to be labeled, the status in the process of labeling and the status completed; the total data amount describes the total amount of data contained in the labeling task; whether to delete indicates whether the labeling task has been deleted, the number 2 indicates that the labeling task has been deleted, and the number 0 indicates that the labeling task has not been deleted; the operation manual name and operation manual path are used to determine the location of the operation manual; the operation manual is used to guide users on how to create labeling tasks, start labeling tasks, manage labeling tasks and import and export data.

[0090] In this embodiment, the labeling task label table is used to store label related information used in the labeling task. Figure 3 As shown, Figure 3 The tag-related information used in the labeling task recorded in is as follows: auto-increment primary key, task name, tag identifier, tag value, whether to participate in double-blind labeling, whether to delete, creator, creation time, updater, and update time; among them, the tag value is used to represent the specific information of the tag; whether to participate in double-blind calculation is used to determine whether the tag can be used to participate in double-blind labeling. The number 1 represents that the tag can be used to participate in double-blind labeling, and the number 0 represents that the tag cannot be used to participate in double-blind labeling.

[0091] In this embodiment, the labeling task status table is used to record the task status corresponding to each labeling task. Figure 4 As shown, Figure 4 The task status related information corresponding to each labeling task recorded in is as follows: label identifier, task name, user identifier, task status, creator, creation time, updater, update time, whether to delete; among them, the user identifier is used to represent the user's identity information, including the first member identifier and the second member identifier.

[0092] In this embodiment, the data set table is used to record the relevant information of each data set. Figure 5 As shown, Figure 5The relevant information of each dataset recorded in is as follows: dataset ID, task ID, creator, creation time, updater, update time, dataset name, dataset source, total data, dataset type, whether labeled, whether available, and whether deleted; among them, whether available indicates whether the dataset is available. The number 0 indicates that the dataset is not available, and the number 1 indicates that the dataset is available.

[0093] In this embodiment, each data set corresponds to a data table, and the data table is used to store relevant information of all data in a data set, such as Figure 6 As shown, Figure 6 The relevant information of all data in a data set recorded in is as follows: data identifier, status, creator, creation time, updater, update time, data set identifier, text content, question, answer, first member identifier, first labeled data, second member identifier, second labeled data, data type, annotation status, first member annotation label, label identifier, second member annotation label, label identifier, data remark, second member remark, whether deleted, whether corrected; among them, the data identifier is used to represent each piece of data; the status describes the data status, including unlabeled status and labeled status; the data set identifier is used to represent each data set; the annotation status describes the data annotation status, the number 1 represents the labeled status, and the number 0 represents the unlabeled status; the first member annotation label and the label identifier need to be separated by a comma; correspondingly, the second member annotation label and the label identifier need to be separated by a comma; whether corrected describes whether the data has been corrected, the number 1 represents corrected, and the number 0 represents uncorrected.

[0094] In the present application, in the process of data labeling based on the large model, it is necessary to label all the data in the data set to be labeled with the labels to be used to obtain the first labeled data; use the labels to be used to double-blindly label part of the data in the data set to be labeled to obtain the second labeled data; the part of the data is data determined from the data set to be labeled based on a preset labeling ratio; the consistency rate of the first labeled data and the second labeled data is calculated to obtain a consistency rate result; based on the consistency rate result, it is determined whether the first labeled data is the data to be put on the shelf. If the first labeled data is the data to be put on the shelf, the first labeled data is put on the shelf so that the large model can use the data after being put on the shelf for training. In this way, this scheme improves the accuracy of the labeled data through two data labeling and consistency rate calculation, so as to provide the large model with more accurate labeled data, thereby improving the accuracy of the output results of the large model.

[0095] See also Figure 7 As shown, the embodiment of the present invention discloses a data annotation method based on a large model, including:

[0096] In this embodiment, a labeling task is created based on a task creation instruction issued by a data administrator, wherein the labeling task is a task constructed based on a number of first member identifiers, a number of second member identifiers, a data set to be labeled, and labels to be used. After the labeling task is created, the task status of the labeling task is marked as a to-be-labeled state; then the labeling task can be started based on a task start instruction issued by the data administrator, at which time the task status of the labeling task is switched from a to-be-labeled state to a labeling state, and all the data in the to-be-labeled data set is allocated to the first members corresponding to the number of first member identifiers according to a preset allocation ratio, and part of the data in the to-be-labeled data set is allocated to the second members corresponding to the number of second member identifiers. After the above steps are completed, the first member and the second member label the allocated data to be labeled to obtain the corresponding first labeled data and second labeled data, and then submit the first labeled data and the second labeled data.

[0097] Correspondingly, after obtaining the first labeled data and the second labeled data submitted by the first member and the second member, the task status of the labeling task is switched from the labeling in progress state to the labeling completed state, and the consistency rate of the first labeled data and the second labeled data is calculated to obtain the consistency rate result. Then, based on the consistency rate result, it is determined whether the first labeled data is the data to be put on the shelf. If the first labeled data is the data to be put on the shelf, the first labeled data is put on the shelf so that the large model can use the labeled data after being put on the shelf for training.

[0098] It can be seen that in the process of data annotation based on the large model, the present application needs to calculate the consistency rate based on the first labeled data obtained after annotating all the data in the data set to be labeled and the second labeled data obtained after double-blind annotation of part of the data set to be labeled, so as to determine whether the large model can be trained using the first labeled data based on the consistency rate result. In this way, this solution improves the accuracy of the labeled data based on two data annotations and the calculation of the consistency rate, so that the large model can be trained based on the labeled data with higher accuracy, thereby improving the accuracy of the output results of the trained large model.

[0099] See also Figure 8 As shown, the present application also provides a data annotation device based on a large model, comprising:

[0100] A first labeling module 11 is used to label all data in the to-be-labeled data set using the to-be-used labels to obtain first labeled data;

[0101] A second labeling module 12 is used to perform double-blind labeling on part of the data in the to-be-labeled data set using the to-be-used label to obtain second labeled data; the part of the data is data determined from the to-be-labeled data set based on a preset labeling ratio;

[0102] A consistency rate calculation module 13, used for calculating the consistency rate of the first labeled data and the second labeled data to obtain a consistency rate result;

[0103] The data uploading module 14 is used to determine whether the first labeled data is the data to be put on the shelf based on the consistency rate result. If the first labeled data is the data to be put on the shelf, the first labeled data is put on the shelf so that the large model can use the put-on data for training.

[0104] As can be seen from the above, before the data annotation based on the large model is performed, the embodiment of the present application needs to calculate the consistency rate based on the first annotated data obtained after annotating all the data in the data set to be annotated and the second annotated data obtained after double-blind annotation of part of the data set to be annotated, so as to determine whether the large model can be trained using the first annotated data based on the consistency rate result. In this way, the accuracy of the annotated data is improved by adopting the scheme based on two data annotations and consistency rate calculation, so that the large model can be trained based on the annotated data with higher accuracy, and the accuracy of the output results of the trained large model is improved.

[0105] In some specific implementations, the first marking module 11 may specifically be:

[0106] A task creation unit, used for creating a labeling task and marking the task status of the labeling task as a pending labeling status;

[0107] A task starting unit, used to start the labeling task, so as to switch the task state of the labeling task from a to-be-labeled state to a labeling state, and trigger the step of labeling all data in the to-be-labeled data set with the labels to be used;

[0108] Correspondingly, the consistency rate calculation module 13 may specifically be:

[0109] A task status switching unit, used to switch the task status of the labeling task from a labeling in progress state to a labeling completed state, and trigger the step of calculating the consistency rate of the first labeled data and the second labeled data;

[0110] In some specific implementations, the first marking module 11 may specifically be:

[0111] A first labeling subunit is used to distribute all the data in the to-be-labeled data set to the first members corresponding to the first member identifiers respectively according to a preset distribution ratio after starting the labeling task, and obtain the labeled data uploaded by the first members after labeling the allocated data with the to-be-used labels, so as to obtain the first labeled data;

[0112] Accordingly, the second marking module 12 may specifically be:

[0113] The second labeling subunit is used to distribute part of the data in the to-be-labeled data set to the second members corresponding to the several second member identifiers respectively after starting the labeling task, and obtain the labeled data uploaded by the second members after labeling the part of the data with the to-be-used labels, so as to obtain second labeled data.

[0114] In some specific implementations, the first marking module 11 may specifically be:

[0115] An import instruction acquisition unit, used to acquire a data set import instruction, wherein the data set import instruction includes a database address, login information, a data table name, and fields to be imported;

[0116] a target data table determining unit, configured to determine a target database based on the database address, and after logging into the target database based on the login information, determine a target data table corresponding to the data table name from the logged-in target database;

[0117] The data set acquisition unit is used to determine the data to be imported corresponding to the field to be imported from the target data table to obtain the data set to be imported, and import the data set to be imported into a local preset cache so as to select the data set to be labeled from the preset cache.

[0118] In some specific implementations, the first marking module 11 may specifically be:

[0119] A user rights determination unit, configured to obtain user login information, and determine a target user role and a target user rights corresponding to the target user role based on the user login information; wherein different user roles correspond to different user rights;

[0120] The tag online unit is used to, if the target user's authority includes tag management authority, perform a tag online operation on the offline tag corresponding to the tag online instruction when a tag online instruction is obtained, so as to select the tag to be used from the online tags.

[0121] In some specific implementations, the consistency rate calculation module 13 may specifically be:

[0122] The consistency rate calculation subunit is used to determine the target labeled data corresponding to the part of the data in the to-be-labeled data set from the first labeled data, and perform consistency rate calculation based on the target labeled data and the second labeled data to obtain a consistency rate result.

[0123] In some specific implementations, the data uploading module 14 may specifically be:

[0124] A result judgment unit, used to judge whether the consistency rate result is greater than a preset consistency rate threshold;

[0125] In some specific implementations, the data uploading module 14 may specifically be:

[0126] An identification modification unit is used to determine that the first labeled data is the data to be put on the shelf if the consistency rate result is greater than the preset consistency rate threshold, and to modify the labeling identification of the data set to be labeled from the first identification to the second identification; wherein the first identification is used to indicate that the data set to be labeled is in an unlabeled state; and the second identification is used to indicate that the data set to be labeled is in a labeled state.

[0127] Furthermore, the present application also discloses an electronic device. Fig. 9 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input and output interface 25 and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the data annotation method based on the large model disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment can specifically be an electronic computer.

[0128] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0129] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0130] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the large model-based data labeling method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0131] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed method for labeling data based on a large model is implemented. The specific steps of the method can be referred to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0132] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0133] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0134] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0135] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0136] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data annotation method based on a large model, It is characterized in that include: Using the labels to be used to label all data in the to-be-labeled data set to obtain first labeled data; Using the to-be-used label to double-blindly label part of the data in the to-be-labeled data set to obtain second labeled data; The partial data is data determined from the to-be-annotated data set based on a preset annotation ratio; Calculating the consistency rate of the first labeled data and the second labeled data to obtain a consistency rate result; Based on the consistency rate result, it is determined whether the first labeled data is data to be put on the shelf. If the first labeled data is data to be put on the shelf, the first labeled data is put on the shelf so that the large model can use the put-on data for training.

2. According to the data annotation method based on the large model in claim 1, It is characterized in that Before obtaining the first labeled data and the second labeled data, the method further includes: Creating a labeling task and marking the task status of the labeling task as a pending labeling state; Starting the labeling task to switch the task status of the labeling task from a to-be-labeled state to a labeling state, and triggering the step of labeling all data in the to-be-labeled data set using the labels to be used; Correspondingly, after obtaining the first labeled data and the second labeled data, the method further includes: The task state of the labeling task is switched from a labeling in progress state to a labeling completed state, and the step of calculating the consistency rate of the first labeled data and the second labeled data is triggered.

3. The data annotation method based on a large model according to claim 2, It is characterized in that The labeling task is a task constructed based on a plurality of first member identifiers, a plurality of second member identifiers, the data set to be labeled and the labels to be used; Correspondingly, the step of labeling all data in the to-be-labeled data set using the to-be-used labels to obtain first labeled data includes: After the labeling task is started, all data in the to-be-labeled data set are allocated to the first members corresponding to the first member identifiers respectively according to a preset allocation ratio, and the labeled data uploaded by the first members is obtained after the allocated data is labeled with the to-be-used labels, so as to obtain the first labeled data; Correspondingly, the step of using the to-be-used label to double-blindly label part of the data in the to-be-labeled data set to obtain second labeled data includes: After starting the labeling task, part of the data in the data set to be labeled is allocated to the second members corresponding to the several second member identifiers respectively, and the labeled data uploaded by the second members after labeling the part of the data with the labels to be used is obtained to obtain the second labeled data.

4. The data annotation method based on a large model according to claim 1, It is characterized in that Before labeling all the data in the to-be-labeled data set using the to-be-used labels, the method further includes: Obtaining a data set import instruction, wherein the data set import instruction includes a database address, login information, a data table name, and fields to be imported; After determining a target database based on the database address and logging into the target database based on the login information, determining a target data table corresponding to the data table name from the logged-in target database; The data to be imported corresponding to the field to be imported is determined from the target data table to obtain a data set to be imported, and the data set to be imported is imported into a local preset cache so that the data set to be labeled is selected from the preset cache.

5. The data annotation method based on a large model according to claim 1, It is characterized in that Before labeling all the data in the to-be-labeled data set using the to-be-used labels, the method further includes: Acquire user login information, and determine a target user role and a target user authority corresponding to the target user role based on the user login information; wherein different user roles correspond to different user authorities; If the target user authority includes tag management authority, when a tag online instruction is obtained, a tag online operation is performed on the offline tag corresponding to the tag online instruction, so as to select the tag to be used from the online tags.

6. The data annotation method based on a large model according to claim 1, It is characterized in that The calculating the consistency rate of the first labeled data and the second labeled data to obtain a consistency rate result includes: Target labeled data corresponding to part of the data in the to-be-labeled data set is determined from the first labeled data, and a consistency rate is calculated based on the target labeled data and the second labeled data to obtain a consistency rate result.

7. The data annotation method based on a large model according to any one of claims 1 to 6, It is characterized in that The determining whether the first labeled data is data to be put on the shelf based on the consistency rate result includes: Determine whether the consistency rate result is greater than a preset consistency rate threshold; If the consistency rate result is greater than the preset consistency rate threshold, determining that the first labeled data is the data to be put on the shelf, and changing the labeling identifier of the data set to be labeled from the first identifier to the second identifier; The first identifier is used to indicate that the data set to be labeled is in an unlabeled state; and the second identifier is used to indicate that the data set to be labeled is in a labeled state.

8. A data annotation device, It is characterized in that include: A first labeling module, used to label all data in the to-be-labeled data set using the to-be-used labels to obtain first labeled data; A second labeling module is used to perform double-blind labeling on part of the data in the to-be-labeled data set using the to-be-used label to obtain second labeled data; The partial data is data determined from the to-be-annotated data set based on a preset annotation ratio; A consistency rate calculation module, used for calculating the consistency rate of the first labeled data and the second labeled data to obtain a consistency rate result; The data uploading module is used to determine whether the first labeled data is the data to be put on the shelf based on the consistency rate result. If the first labeled data is the data to be put on the shelf, the first labeled data is put on the shelf so that the large model can use the put-on data for training.

9. An electronic device, It is characterized in that include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the large model-based data labeling method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the steps of the large model-based data labeling method as described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Method, system and device for marking data in quality inspection

    CN111581195A

  • Data annotation method and device, equipment and storage medium

    CN115732061A

  • Data processing method and device, electronic equipment and storage medium

    CN115829305A

  • Data annotation method and device, computer equipment and storage medium

    CN116167457A

  • Method and device for determining data annotation quality for large language model training

    CN117216051A