Method, device and electronic equipment for selecting personnel for labeling professional data
The method and device select annotators based on capability assessment and feature construction to match their professionalism with the task, improving annotation accuracy and efficiency in medical data labeling for medical AI.
Patent Information
- Application Number
- CN202210113924.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-30
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-01-30
AI Technical Summary
Medical artificial intelligence requires high-accuracy annotation of unstructured medical data, such as electronic medical records, which is challenging due to the complexity and expertise needed for medical terms, necessitating the adaptation of appropriate annotators to improve annotation efficiency and accuracy.
A method and device for selecting annotators based on capability assessment, constructing user and entity type features, and assigning tasks to annotators with matching professionalism, ensuring high-quality labeling results.
Improves the accuracy and credibility of medical annotation by matching annotators with the task requirements, reducing review rounds and enhancing annotation efficiency, thereby assisting intelligent algorithms in better recognition.
Smart Images

Figure CN114446431B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of artificial intelligence application technology, and in particular, relates to a method, device and electronic equipment for selecting personnel to annotate professional data. Background Art
[0002] At present, medical artificial intelligence technology has been widely used in various fields of medicine. Its performance driven by medical big data is remarkable and has become an increasingly important auxiliary means for medical practitioners. The use of medical artificial intelligence technology is based on a large amount of structured data to train models that can assist in prediction or classification.
[0003] Medical data such as electronic medical records are very important in medical resources. They complete the information record of a series of processes from the patient's visit to the doctor after the onset of symptoms, the doctor's diagnosis, the formulation of treatment plans, and the patient's treatment operations such as surgery. It is the first-hand data for understanding the patient's cause of illness, treatment status, and prognosis. It contains a lot of important information and can be used to provide a data foundation for medical artificial intelligence technology. However, electronic medical records are realized by clinicians' descriptions and records, usually presented in irregular text, that is, unstructured text, and cannot be directly applied to data-driven model training. It is necessary to first complete the extraction of fixed category content such as disease, surgery, experiment and other data.
[0004] In view of this, large-scale data training in medical artificial intelligence technology requires the annotation of medical terms as a prerequisite. Accordingly, identifying different types of medical terms in unstructured medical data (such as electronic medical records), such as diseases and operations, has become an important issue that medical artificial intelligence needs to solve. The applicant's research found that medical data usually has the characteristics of high knowledge expertise, high complexity, and wide data coverage. In addition, medical artificial intelligence technology requires that the annotated medical terms have high accuracy and high requirements for medical annotators. Therefore, it is very important for this field to adapt the appropriate annotators to the tasks to be annotated and have the same professionalism as the tasks to be annotated, so as to help improve the accuracy of medical annotation, reduce the number of review rounds, and improve annotation efficiency. Summary of the invention
[0005] In view of this, the present application provides a method, device and electronic device for selecting labeling personnel for professional data, which are used to adapt suitable labeling personnel who correspond to the task to be labeled in terms of professionalism, so as to improve the accuracy and credibility of the labeling results and assist the intelligent algorithm to achieve better recognition effects.
[0006] The specific technical solutions are as follows:
[0007] A method for selecting a professional data labeler, comprising:
[0008] Capacity assessment is performed on different annotators in the annotator set to obtain capability assessment results corresponding to each annotator; the capability assessment results corresponding to the annotators can be used to characterize the annotation quality of the annotators for each entity type in multiple entity types of professional data in a predetermined field, and / or the annotation quality of each text in multiple test texts;
[0009] According to the corresponding ability assessment results of the annotation personnel, the user characteristics of the annotation personnel are constructed;
[0010] Constructing entity type features for each entity type in the plurality of entity types; and / or constructing text topic features for each test text in the plurality of test texts;
[0011] Selecting at least one target annotator for task annotation from the annotator set for the task to be annotated according to the user characteristics of the annotators, the entity type characteristics of each entity type and / or the text subject characteristics of each test text, and the task characteristics of the task to be annotated;
[0012] Assign the task to be labeled to the selected target labeler.
[0013] Optionally, the capability assessment of different labelers in the labeling personnel set is performed to obtain the capability assessment results corresponding to each labeler, including:
[0014] Setting a plurality of test texts of professional data in the predetermined field, each test text corresponding to one or more entity types in the predetermined field;
[0015] Obtaining annotation results obtained by each annotator in the annotator group for entity annotation of the plurality of test texts;
[0016] According to the annotation results of each annotator, the quality index value of each annotator's annotation quality for each entity type and / or the quality index value of the annotation quality for each test text is determined to obtain the corresponding ability evaluation result of each annotator.
[0017] Optionally, constructing user characteristics of the annotators according to the capability evaluation results corresponding to the annotators includes:
[0018] Constructing user features of the annotators according to the quality index value of the annotation quality of each entity type and / or the quality index value of the annotation quality of each test text;
[0019] Among them, the user features of the annotators are multi-dimensional features arranged in order, and each dimensional feature includes: the correspondence between the corresponding entity type and the quality indicator value generated by the annotator in the corresponding entity type, and / or the correspondence between the text identifier of each test text and the quality indicator value generated by the annotator in the corresponding test text.
[0020] Optionally, before constructing the user features of the annotators, the following is also included:
[0021] The labeling personnel whose corresponding quality index values do not meet the set index conditions are filtered out from the labeling personnel set.
[0022] Optionally, constructing entity type features of each entity type in the multiple entity types includes:
[0023] According to the entity type structure of each entity type in the multiple entity types, construct a feature representation of each entity type to obtain an entity type feature of each entity type;
[0024] The step of constructing a text topic feature of each test text in the plurality of test texts includes:
[0025] According to the topic distribution of each test text in the multiple test texts, a text topic feature of each test text is constructed.
[0026] Optionally, the entity type structure of the entity type includes: extracting entity objects and labeling entity labels from knowledge data corresponding to the entity type, and constructing a tree-type entity labeling system structure based on object relationships of each entity object;
[0027] According to the entity type structure of each entity type in the multiple entity types, construct a feature representation of each entity type, including:
[0028] According to the entity label contained in each node in the entity type structure of the entity type, a vector representation of the entity type is constructed through a vector space model as the entity type feature of the entity type.
[0029] Optionally, the task features of the task to be labeled include: entity type features and / or text topic features of the task to be labeled;
[0030] The selecting at least one target annotator for task annotation from the annotator set for the task to be annotated according to the user characteristics of each annotator, the entity type characteristics of each entity type and / or the text subject characteristics of each test text, and the task characteristics of the task to be annotated includes:
[0031] According to the entity type features of the task to be labeled and the entity type features corresponding to the entity type in the user features of each labeler, the similarity between the entity type of the task to be labeled and the entity type in the user features of each labeler is calculated; the labeler whose corresponding entity type similarity and entity type quality index value meet the first screening condition is determined as the target labeler;
[0032] Alternatively, based on the text topic features of the task to be labeled and the text topic features corresponding to the test text in the user features of each labeler, the text topic similarity between the task to be labeled and the test text in the user features of each labeler is calculated; and the labelers whose corresponding text topic similarity and test text quality index values meet the second screening condition are determined as target labelers.
[0033] Optionally, after assigning the task to be labeled to the selected target labeler, the method further includes:
[0034] Obtaining and reviewing the labeling results of the target labeling personnel on the task to be labeled, and obtaining a quality indicator value of the labeling quality of the target labeling personnel on the task to be labeled;
[0035] Based on the quality indicator values obtained from the review, the user characteristics of the target labelers are updated, so that labelers for subsequent labeling tasks are selected based on the updated user characteristics.
[0036] A device for selecting a person to label professional data, comprising:
[0037] A capability evaluation module is used to evaluate the capabilities of different annotators in the annotator set, and obtain capability evaluation results corresponding to each annotator; the capability evaluation results corresponding to the annotators can be used to characterize the annotation quality of the annotators for each entity type in multiple entity types of professional data in a predetermined field, and / or the annotation quality of each text in multiple test texts;
[0038] The first feature construction module is used to construct the user features of each labeling personnel according to the capability evaluation results corresponding to each labeling personnel;
[0039] A second feature construction module is used to construct entity type features of each entity type in the plurality of entity types; and / or to construct text topic features of each test text in the plurality of test texts;
[0040] A labeler selection module, configured to select at least one target labeler for task labeling from the labeler set for the task to be labeled according to the user characteristics of each labeler, the entity type characteristics of each entity type and / or the text subject characteristics of each test text, and the task characteristics of the task to be labeled;
[0041] The task assignment module is used to assign the task to be labeled to the selected target labeler.
[0042] An electronic device, comprising:
[0043] A memory for storing at least one set of instructions;
[0044] A processor is used to call and execute the instruction set in the memory, and implement the method for selecting labeling personnel for professional data as described in any one of the above items by executing the instruction set.
[0045] According to the above scheme, the method, device and electronic device for selecting annotation personnel for professional data provided by the present application construct user characteristics of each annotation personnel based on the ability assessment of the annotation personnel, and construct entity type characteristics of each entity type and / or text topic characteristics corresponding to each test text, and select at least one target annotation personnel for task annotation for the task to be annotated based on the user characteristics of each annotation personnel, the entity type characteristics of each entity type and / or the text topic characteristics of each test text, and the task characteristics of the task to be annotated.
[0046] Therefore, this application proposes to adapt the task to be processed to suitable labeling personnel whose professionalism corresponds to the task to be labeled from multiple dimensions and aspects (such as user characteristics, entity type characteristics, text subject characteristics, etc.), so that the appropriate task can be pushed to the appropriate labeling personnel, so that the labeling personnel are familiar with and proficient in the pushed content to be labeled, and can complete the task to be labeled quickly, efficiently and accurately, thereby improving the accuracy and credibility of the labeling results, and can assist the intelligent algorithm to achieve better recognition effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0048] Figure 1 This is a flowchart of the method for selecting annotators for professional data provided by this application;
[0049] Figure 2 This is a diagram of the implementation process of the capability assessment of the annotation personnel provided by this application;
[0050] Figure 3 It is another flowchart diagram of the method for selecting annotators of professional data provided by this application;
[0051] Figure 4 It is a composition structure diagram of the device for selecting personnel for labeling professional data provided by this application;
[0052] Figure 5 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0054] The present application discloses a method, device and electronic device for selecting annotators for professional data. The method can be used to select annotators for the task to be annotated, and the annotators are suitable and professionally consistent with the task to be annotated, in accordance with the annotating needs of professional data in predetermined fields such as medicine / pharmaceuticals, biology, optoelectronics / photoconductors, and space technology. The present application embodiment will mainly use the annotation of medical data as an example to illustrate the solution.
[0055] Medical terminology annotation is a prerequisite for large-scale data training. It can be used as a training set for neural network algorithm training, or as a test set for medical semantic type recognition in the form of a gold standard. Regardless of the form in which it appears, the annotated medical terms are required to have high accuracy. Therefore, there are high requirements for medical annotators. In addition, different medical annotators may have learned and practiced different things, so they are good at annotating semantic types or medical content. In view of this, this application aims to find annotators with corresponding focuses for the task to be annotated, so as to help improve the accuracy of medical annotation, reduce the number of review rounds, improve annotation efficiency, and then improve the accuracy and credibility of the annotation results, and assist intelligent algorithms to achieve better recognition effects.
[0056] The method disclosed in the embodiments of the present application can be used in many general or special computing device environments or configurations, such as: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, etc.
[0057] See also Figure 1 The method for selecting annotators for professional data disclosed in the embodiment of the present application specifically includes the following processing steps:
[0058] Step 101: Conduct capability assessment on different labelers in the labeling personnel set to obtain capability assessment results corresponding to each labeler; the capability assessment results corresponding to the labelers can be used to characterize the labeling quality of the labelers for each entity type in multiple entity types of professional data in a predetermined field, and / or the labeling quality of each text in multiple test texts.
[0059] The ability assessment of the annotators is used to estimate and evaluate the annotators' ability to annotate professional data in the predetermined field, and then to profile the subdivided professional fields and the degree of expertise of different annotators in the predetermined field. The predetermined field professional data can be, but is not limited to, the relevant data in the fields of medicine / pharmaceuticals, biology, optoelectronics / photoconductors, space technology, etc., such as electronic medical records in the medical field.
[0060] See also Figure 2 ,The process of evaluating the capabilities of different labelers can be further implemented as:
[0061] Step 201 : setting test texts of multiple entity types of professional data in a predetermined field, wherein each entity type corresponds to multiple different test texts.
[0062] Exemplarily, this embodiment sets a test text containing M entity types according to the annotation requirements of medical data. The M entity types include but are not limited to disease, medicine, surgery, experiment and other entity types, which are used to represent various sub-professional fields in the medical field. The M entity types correspond to N test texts in total, and each test text corresponds to one or more entity types of the M entity types. M and N are integers greater than 1 respectively.
[0063] Each test text is used as unstructured data to be annotated for the ability assessment of annotators. In addition, for each test text, a corresponding annotation standard is formulated for it as the basis for evaluation. For example, the results of the expert annotation of each test text are regarded as the gold standard of the test text, which is used as the basis for the subsequent ability assessment of annotators.
[0064] Step 202: Obtain annotation results obtained by each annotator in the annotator set for entity annotation of the test texts of the multiple entity types.
[0065] Each test text is assigned to different annotators in the annotator pool, and each annotator conducts the annotating. Each annotator needs to annotate N test texts cumulatively, which corresponds to a total of M entity types. After the annotation is completed, the annotation results of each annotator are collected.
[0066] Step 203: Determine the quality index value of each annotator's annotation quality for each entity type and / or the quality index value of each test text's annotation quality based on the annotation results of each annotator, and obtain the corresponding capability evaluation result of each annotator.
[0067] Afterwards, the annotation standards corresponding to each test text are further used as the comparison basis to analyze and count the relevant quality index values of each annotator's annotation quality. This includes but is not limited to analyzing and counting the quality index values of each annotator's annotation quality for each entity type in N test texts, and / or the quality index values of the annotation quality for each test text, and correspondingly obtaining the corresponding ability evaluation results of each annotator.
[0068] The quality index value of the annotation quality of each entity type by the annotator may be, but is not limited to, the F1 value, accuracy or recall rate of the annotation of each entity type by the annotator; similarly, the quality index value of the annotation quality of each test text by the annotator may be, but is not limited to, the F1 value, accuracy or recall rate of the annotation of each test text under each entity type by the annotator. The larger the quality index value, the better the annotation quality and the higher the annotation ability of the annotator for the corresponding entity type or the corresponding test text.
[0069] The definitions and formulas of the F1 value, precision or recall rate can be found in the relevant description of the prior art and will not be described in detail here.
[0070] Step 102: construct user characteristics of the annotation personnel according to the corresponding capability assessment results of the annotation personnel.
[0071] Optionally, before constructing the user characteristics of the labelers, first filter out the labelers whose corresponding quality index values do not meet the set index conditions from the labeler set. Specifically, for example, filter out the labelers whose F1 value / accuracy / recall rate for the entity type / test text (or the average F1 value / accuracy / recall rate for each entity type / test text) is less than a set threshold, etc., so as to preliminarily screen out labelers with relatively high labeling ability in the labeler set for task allocation of tasks to be labeled.
[0072] On this basis, for each annotator who has not been filtered out, corresponding user features are constructed for each annotator according to the quality index value of the annotator's annotation quality for each entity type and / or the quality index value of the annotation quality for each test text.
[0073] Among them, the user features of the annotators are multi-dimensional features arranged in order, and each dimensional feature includes: the corresponding relationship between the corresponding entity type and the quality indicator value generated by the annotator in the corresponding entity type, and / or the corresponding relationship between the text identifier of each test text and the quality indicator value generated by the annotator in the corresponding test text.
[0074] For ease of understanding, the following example is given: In this example, the accuracy of the annotator corresponding to each entity type is set to the F1 value corresponding to the previous capability assessment statistics, and the value of F1 is set to 100% (the F1 value of the annotator for a certain entity type is 0, which means that the annotator has not tested the text annotation of this entity type. For this case, the accuracy is assumed to be 100% for the first time, which is equivalent to giving the annotator in this case a chance to try). On this basis, for each unfiltered annotator, its user characteristics are represented as {(A, P1), (B, P2), (C, P3)...}. Among them, A, B, C... respectively represent different entity types, such as disease, medicine, surgery, test, etc., and P1, P2, P3... respectively represent the annotator's annotation F1 value of the corresponding entity type in A, B, C... The order of A, B, C... is arranged in descending order by the annotator's annotation F1 value of each entity type.
[0075] It is easy to understand that in the above example, the user features of each annotator include multidimensional features sorted in descending order of F1 value, and each dimensional feature includes: the corresponding relationship between the corresponding entity type and the quality indicator value (such as F1 value) generated by the annotator in the corresponding entity type.
[0076] Step 103: construct entity type features for each entity type in the plurality of entity types; and / or construct text topic features for each test text in the plurality of test texts.
[0077] In addition to user features, the embodiment of the present application also constructs other relevant features such as entity type features of entity types and text topic features of test texts.
[0078] The construction process of entity type features and text topic features is as follows:
[0079] 11) Entity type characteristics
[0080] This embodiment constructs a feature representation of each entity type according to the entity type structure of each entity type in the multiple entity types, and obtains the entity type feature of each entity type.
[0081] The entity type structure of each entity type includes: extracting entity objects and labeling entity labels from the knowledge data corresponding to the entity type, and constructing a tree-type entity label system structure based on the object relationship of each entity object. The tree structure of the entity label system corresponding to the entity type reflects the object relationship of each entity object of the entity type, such as the include / included relationship, parallel relationship, derivative relationship, etc.
[0082] Based on this, in practical applications, the entity type structure of each entity type can be determined first. There are two main ways to determine the entity type structure. One is manual determination, that is, according to actual needs, relevant objects are extracted from existing medical data such as medical comprehensive vocabulary, professional medical vocabulary, medical ontology, and medical knowledge graph, and entity objects are labeled with entity labels. An entity label system is constructed based on the object relationship of each entity object, or the label system is directly constructed by experts, so as to recommend appropriate entity type labeling tasks to suitable labelers in the later stage. The other is to train a large amount of text content through unsupervised learning to generate a bottom-up labeling system for medical entity types and obtain the entity type structure of entity types.
[0083] Afterwards, according to the entity label corresponding to each node in the entity type structure of the entity type, a vector representation of the entity type is constructed through a vector space model (Word2vec) as the entity type feature of the entity type, such as two entity types V1 and V2.
[0084] 12) Text topic features
[0085] This embodiment constructs a text topic feature of each test text according to the topic distribution of each test text in multiple test texts.
[0086] Specifically, a large number of texts are trained through the LDA (Latent Dirichlet allocation) algorithm to learn the topic distribution characteristics of a large number of texts, and on this basis, the topic distribution of each test text is constructed, and then a topic-based vector is constructed for the test text. Among them, each topic of the test text is a feature of the vector, and the feature value is the probability value of the topic in the test text. Each topic is normalized, and finally the text topic feature of the test text is obtained.
[0087] Step 104: Select at least one target annotator for task annotation from the set of annotators for the task to be annotated based on the user characteristics of each annotator, the entity type characteristics of each entity type and / or the text topic characteristics of each test text, and the task characteristics of the task to be annotated.
[0088] There are many ways to select target labelers for task labeling, mainly including:
[0089] 21) Selection based on similarity with the entity type in the annotator's user features:
[0090] In this method, the task to be labeled (entity type to be labeled) is first vectorized to obtain the entity type features of the task to be labeled, and based on the entity type features of the task to be labeled and the entity type features corresponding to the entity type in the user features of each labeler, the similarity between the entity type of the task to be labeled and the entity type in the user features of each labeler is calculated. After that, the labelers whose corresponding entity type similarity and entity type quality index values meet the first screening condition are determined as target labelers.
[0091] Specifically, the similarity between two entity types can be calculated based on the feature vector of the entity type to be labeled and the feature vector of each entity type in the user features of the labeler. The calculation formula is designed as follows:
[0092]
[0093] Among them, a i ,b i are the i-th feature components of the two entity types, w i is the weight of the i-th feature of the two entity types, 1≤i≤k, and i is an integer. i The value can be set based on manual experience / understanding of the business, or it can be trained using a machine learning algorithm.
[0094] Optionally, specifically, labelers whose corresponding entity type similarity is in topk1 and whose quality index value corresponding to each entity type in topk1 is in topk2 can be selected as target labelers for task labeling of the task to be processed. That is, labelers whose entity type similarity is high and whose labeling quality index value (such as F1 value) for the entity type is high are selected as target labelers suitable for the task to be labeled for task assignment.
[0095] The screening condition based on topk1 and topk2 can be used as the first screening condition mentioned above.
[0096] k1 and k2 are both positive integers and can be freely set according to actual needs.
[0097] 22) Selection based on the topic similarity of the test text in the annotator's user features:
[0098] In this method, the topic of the task to be labeled (text to be labeled) is first predicted by the LDA model to obtain the text topic features of the task to be labeled, and based on the text topic features of the task to be labeled and the text topic features corresponding to the test text in the user features of each labeler, the text topic similarity between the task to be labeled and the test text in the user features of each labeler is calculated, and then the labelers whose corresponding text topic similarity and test text quality index values meet the second screening condition are determined as target labelers.
[0099] Optionally, specifically, annotators whose corresponding text topic similarity is in topk3 and whose annotation quality index value corresponding to each test text in topk3 is in topk4 can be selected as target personnel for task annotation of the task to be processed. That is, annotators who have high text topic similarity with the test text and high annotation quality index value (such as F1 value) of the test text are selected as target annotators who are suitable for the personnel to be annotated for task assignment.
[0100] The screening condition based on topk3 and topk4 can be used as the second screening condition mentioned above.
[0101] K3 and k4 are both positive integers and can be freely set according to actual needs.
[0102] It should be noted that when implementing the present application, it is not limited to using the above-mentioned method 21) or 22) to select the target labeling personnel. The above two methods can also be combined to select suitable target labeling personnel for the task to be labeled. Specifically, for example, the target labeling personnel selected by method 21) and the target labeling personnel selected by method 22) are intersected, and the labeling personnel in the intersection of the two are used as the target labeling personnel selected for the task to be labeled, etc.
[0103] Step 105: assign the task to be labeled to the selected target labeler.
[0104] Finally, the task to be labeled is pushed to the selected target labeler for text labeling.
[0105] The embodiment of the present application evaluates the capabilities of labelers based on a number of test texts of multiple different entity types, so as to identify and assess the professional sub-fields with higher labeling accuracy and recall rates of different labelers, and profile the labeling capabilities of different labelers in different professional sub-fields. Based on this, the labeling tasks in the same or similar fields as their expertise are recommended / pushed, thereby improving the accuracy and efficiency of the overall labeling task, and providing a strong guarantee for the labeling results for the intelligent algorithm.
[0106] According to the above scheme, it can be seen that the method of the embodiment of the present application constructs the user characteristics of each annotation personnel based on the ability assessment of the annotation personnel, and constructs the entity type characteristics of each entity type and / or the text topic characteristics corresponding to each test text, and selects at least one target annotation personnel for task annotation for the task to be annotated based on the user characteristics of each annotation personnel, the entity type characteristics of each entity type and / or the text topic characteristics of each test text, and the task characteristics of the task to be annotated.
[0107] Therefore, this application proposes to adapt the task to be processed to suitable labeling personnel whose professionalism corresponds to the task to be labeled from multiple dimensions and aspects (such as user characteristics, entity type characteristics, text subject characteristics, etc.), so that the appropriate task can be pushed to the appropriate labeling personnel, so that the labeling personnel are familiar with and proficient in the pushed content to be labeled, and can complete the task to be labeled quickly, efficiently and accurately, thereby improving the accuracy and credibility of the labeling results, and can assist the intelligent algorithm to achieve better recognition effects.
[0108] In one embodiment, see Figure 3 The flowchart of the method for selecting a professional data annotator is provided. The method for selecting a professional data annotator disclosed in this application may further include, after step 105:
[0109] Step 106: Obtain and review the labeling results of the target labeler for the labeling task to be labeled, and obtain the quality index value of the labeling quality of the target labeler for the labeling task to be labeled;
[0110] Step 107: based on the quality indicator value obtained through the review, the user characteristics of the target labeler are updated, so as to select labelers for subsequent labeling tasks based on the updated user characteristics.
[0111] Optionally, in this embodiment, after the target labeler completes the labeling of the content of the task to be labeled, the labeling results of the target labeler for the task to be labeled are obtained, and the labeling results are reviewed, such as by a reviewer or automatically reviewed based on relevant artificial intelligence methods.
[0112] After the review is completed, the target annotator's accuracy, recall or F1 value and other annotation quality indicators for the assigned task are further calculated, and the target annotator's user characteristics in the entity category and other related data such as the sorting order are updated. Subsequently, the next task recommendation will be made based on the updated user characteristics, that is, the next task to be annotated will be selected based on the updated user characteristics, and the annotator who is suitable and professionally consistent with the task to be annotated will be selected for the next task to be annotated.
[0113] This embodiment verifies the annotation results of the assigned tasks by the annotation personnel, calculates the annotation quality index value of the task, and feeds it back to the user characteristics and updates the user characteristics, so that the user characteristics of the annotation personnel can be increasingly accurate and complete, and can accurately portray the field categories and proficiency levels of the subdivided professional fields that different annotation personnel are good at in specific fields, thereby providing a strong guarantee for recommending / pushing field annotation tasks that are the same or similar to their fields of expertise to different annotation personnel, and improving the accuracy and efficiency of the overall annotation tasks.
[0114] Corresponding to the above method, the present application embodiment also discloses a device for selecting a professional data annotator, such as Figure 4 As shown, the device comprises:
[0115] The capability evaluation module 401 is used to evaluate the capabilities of different annotators in the annotator set, and obtain capability evaluation results corresponding to each annotator; the capability evaluation results corresponding to the annotators can be used to characterize the annotation quality of the annotators for each entity type in multiple entity types of professional data in a predetermined field, and / or the annotation quality of each text in multiple test texts;
[0116] The first feature construction module 402 is used to construct the user features of each labeling personnel according to the capability evaluation results corresponding to each labeling personnel;
[0117] The second feature construction module 403 is used to construct entity type features of each entity type in the multiple entity types; and / or to construct text topic features of each test text in the multiple test texts;
[0118] The labeling personnel selection module 404 is used to select at least one target labeling personnel for task labeling from the labeling personnel set for the task to be labeled according to the user characteristics of each labeling personnel, the entity type characteristics of each entity type and / or the text subject characteristics of each test text, and the task characteristics of the task to be labeled;
[0119] The task assignment module 405 is used to assign the task to be labeled to the selected target labeler.
[0120] In one implementation, the capability assessment module 401 is specifically configured to:
[0121] Setting a plurality of test texts of professional data in the predetermined field, each test text corresponding to one or more entity types in the predetermined field;
[0122] Obtaining annotation results obtained by each annotator in the annotator group for entity annotation of the plurality of test texts;
[0123] According to the annotation results of each annotator, the quality index value of each annotator's annotation quality for each entity type and / or the quality index value of the annotation quality for each test text is determined to obtain the corresponding ability evaluation result of each annotator.
[0124] In one implementation, the first feature construction module 402 is specifically configured to:
[0125] Constructing user features of the annotators according to the quality index value of the annotation quality of each entity type by the annotators and / or the quality index value of the annotation quality of each test text under each entity type;
[0126] Among them, the user features of the annotators are multi-dimensional features arranged in order, and each dimensional feature includes: the correspondence between the corresponding entity type and the quality indicator value generated by the annotator in the corresponding entity type, and / or the correspondence between the text identifier of each test text and the quality indicator value generated by the annotator in the corresponding test text.
[0127] In one embodiment, the above device further comprises:
[0128] The personnel filtering module is used to filter out the labeling personnel whose corresponding quality index values do not meet the set index conditions from the labeling personnel set before constructing the user characteristics of the labeling personnel.
[0129] In one embodiment, the second feature construction module 403, when constructing the entity type feature of each entity type in the multiple entity types, is specifically used to: construct a feature representation of each entity type according to the entity type structure of each entity type in the multiple entity types to obtain the entity type feature of each entity type;
[0130] The second feature construction module 403 is specifically used to construct the text theme feature of each test text in the multiple test texts according to the theme distribution of each test text in the multiple test texts when constructing the text theme feature of each test text in the multiple test texts.
[0131] In one embodiment, the entity type structure of the entity type includes: extracting entity objects and labeling entity labels from knowledge data corresponding to the entity type, and constructing a tree-type entity label system structure based on the object relationship of each entity object;
[0132] The second feature construction module 403, when constructing a feature representation of each entity type according to the entity type structure of each entity type in the multiple entity types, is specifically used to:
[0133] According to the entity label contained in each node in the entity type structure of the entity type, a vector representation of the entity type is constructed through a vector space model as the entity type feature of the entity type.
[0134] In one embodiment, the task features of the task to be labeled include: entity type features and / or text topic features of the task to be labeled;
[0135] The marking personnel selection module 404 is specifically used for:
[0136] According to the entity type features of the task to be labeled and the entity type features corresponding to the entity type in the user features of each labeler, the similarity between the entity type of the task to be labeled and the entity type in the user features of each labeler is calculated; the labeler whose corresponding entity type similarity and entity type quality index value meet the first screening condition is determined as the target labeler;
[0137] Alternatively, based on the text topic features of the task to be labeled and the text topic features corresponding to the test text in the user features of each labeler, the text topic similarity between the task to be labeled and the test text in the user features of each labeler is calculated; and the labelers whose corresponding text topic similarity and test text quality index values meet the second screening condition are determined as target labelers.
[0138] In one embodiment, the above device further comprises:
[0139] User feature update module, used to:
[0140] After assigning the task to be labeled to the selected target labeler, obtaining and reviewing the labeling result of the task to be labeled by the target labeler, and obtaining a quality index value of the labeling quality of the task to be labeled by the target labeler;
[0141] Based on the quality indicator values obtained from the review, the user characteristics of the target labelers are updated, so that labelers for subsequent labeling tasks are selected based on the updated user characteristics.
[0142] As for the device for selecting labeling personnel for professional data disclosed in the embodiment of the present application, since it corresponds to the method for selecting labeling personnel for professional data disclosed in the above method embodiment, the description is relatively simple. For relevant similarities, please refer to the description of the above method embodiment, which will not be described in detail here.
[0143] The present application also discloses an electronic device, which may be, but is not limited to, a device in a variety of general or special computing device environments or configurations, such as a personal computer, a server computer, a handheld device or a portable device, a tablet device, a multi-processor device, etc.
[0144] The electronic device has a structure such as Figure 5 As shown, including:
[0145] Memory 501, used for storing computer instruction sets;
[0146] The computer instruction set in the memory 501 may be implemented in the form of a computer program.
[0147] The processor 502 is used to implement the method for selecting annotators for professional data disclosed in the above method embodiment by executing a computer instruction set.
[0148] The processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices.
[0149] In addition, the electronic device may also include components such as a communication interface and a communication bus. The memory, the processor and the communication interface communicate with each other through the communication bus.
[0150] The communication interface is used for communication between electronic devices and other devices. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.
[0151] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0152] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0153] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.
[0154] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0155] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for selecting professional data labelers, characterized in that: include: Capacity assessment is performed on different annotators in the annotator set to obtain capability assessment results corresponding to each annotator; the capability assessment results corresponding to the annotators can be used to characterize the annotation quality of the annotators for each entity type in multiple entity types of professional data in a predetermined field, and / or the annotation quality of each text in multiple test texts; According to the corresponding ability assessment results of the annotation personnel, the user characteristics of the annotation personnel are constructed; Constructing entity type features for each entity type in the plurality of entity types; and / or constructing text topic features for each test text in the plurality of test texts; Selecting at least one target annotator for task annotation from the annotator set for the task to be annotated according to the user characteristics of the annotators, the entity type characteristics of each entity type and / or the text subject characteristics of each test text, and the task characteristics of the task to be annotated; Assigning the task to be labeled to a selected target labeler; The ability assessment of different labelers in the labeling personnel set is performed to obtain the ability assessment results corresponding to each labeler, including: Setting a plurality of test texts of professional data in the predetermined field, each test text corresponding to one or more entity types in the predetermined field; Obtaining annotation results obtained by each annotator in the annotator group for entity annotation of the plurality of test texts; According to the annotation results of each annotator, determine the quality index value of each annotator's annotation quality for each entity type and / or the quality index value of each test text's annotation quality, and obtain the corresponding ability evaluation result of each annotator; The user characteristics of the labeling personnel are constructed according to the corresponding capability evaluation results of the labeling personnel, including: Constructing user features of the annotators according to the quality index value of the annotation quality of each entity type and / or the quality index value of the annotation quality of each test text; The user features of the annotators are multi-dimensional features arranged in order, and each dimensional feature includes: a corresponding relationship between a corresponding entity type and a quality indicator value generated by the annotator in the corresponding entity type, and / or a corresponding relationship between a text identifier of each test text and a quality indicator value generated by the annotator in the corresponding test text; Before constructing the user features of the annotated personnel, the method further includes: Filter out from the set of labeling personnel those labeling personnel whose corresponding quality index values do not meet the set index conditions; The step of constructing entity type features of each entity type in the plurality of entity types includes: According to the entity type structure of each entity type in the multiple entity types, construct a feature representation of each entity type to obtain an entity type feature of each entity type; The step of constructing a text topic feature of each test text in the plurality of test texts includes: Constructing a text topic feature of each test text according to the topic distribution of each test text in the plurality of test texts; The entity type structure of the entity type includes: extracting entity objects and labeling entity labels from knowledge data corresponding to the entity type, and constructing a tree-type entity labeling system structure based on the object relationship of each entity object; According to the entity type structure of each entity type in the multiple entity types, a feature representation of each entity type is constructed, including: According to the entity label contained in each node in the entity type structure of the entity type, a vector representation of the entity type is constructed through a vector space model as an entity type feature of the entity type; The task features of the task to be labeled include: entity type features and / or text topic features of the task to be labeled; The selecting at least one target annotator for task annotation from the annotator set for the task to be annotated according to the user characteristics of each annotator, the entity type characteristics of each entity type and / or the text subject characteristics of each test text, and the task characteristics of the task to be annotated includes: According to the entity type features of the task to be labeled and the entity type features corresponding to the entity type in the user features of each labeler, the similarity between the entity type of the task to be labeled and the entity type in the user features of each labeler is calculated; the labeler whose corresponding entity type similarity and entity type quality index value meet the first screening condition is determined as the target labeler; Or, according to the text theme features of the task to be labeled and the text theme features corresponding to the test text in the user features of each labeler, the text theme similarity between the task to be labeled and the test text in the user features of each labeler is calculated; and the labeler whose corresponding text theme similarity and test text quality index value meet the second screening condition is determined as the target labeler; After assigning the task to be labeled to the selected target labeler, the method further includes: Obtaining and reviewing the labeling results of the target labeling personnel on the task to be labeled, and obtaining a quality indicator value of the labeling quality of the target labeling personnel on the task to be labeled; Based on the quality indicator values obtained from the review, the user characteristics of the target labelers are updated, so that labelers for subsequent labeling tasks are selected based on the updated user characteristics.
2. A device for selecting professional data labelers, characterized in that: include: The capability assessment module is used to assess the capabilities of different annotators in the annotation personnel set and obtain the capability assessment results corresponding to each annotator; The corresponding ability evaluation results of the annotators can be used to characterize the annotation quality of the annotators for each entity type in multiple entity types of professional data in a predetermined field, and / or the annotation quality of each text in multiple test texts; The first feature construction module is used to construct the user features of each labeling personnel according to the capability evaluation results corresponding to each labeling personnel; A second feature construction module is used to construct entity type features of each entity type in the plurality of entity types; and / or to construct text topic features of each test text in the plurality of test texts; A labeler selection module, configured to select at least one target labeler for task labeling from the labeler set for the task to be labeled according to the user characteristics of each labeler, the entity type characteristics of each entity type and / or the text subject characteristics of each test text, and the task characteristics of the task to be labeled; A task assignment module, used to assign the to-be-annotated task to a selected target annotator; The capability assessment module is specifically used for: Setting a plurality of test texts of professional data in the predetermined field, each test text corresponding to one or more entity types in the predetermined field; Obtaining annotation results obtained by each annotator in the annotator group for entity annotation of the plurality of test texts; According to the annotation results of each annotator, determine the quality index value of each annotator's annotation quality for each entity type and / or the quality index value of each test text's annotation quality, and obtain the corresponding ability evaluation result of each annotator; The first feature building module is specifically used to: Constructing user features of the annotators according to the quality index value of the annotation quality of each entity type by the annotators and / or the quality index value of the annotation quality of each test text under each entity type; The user features of the annotators are multi-dimensional features arranged in order, and each dimensional feature includes: a corresponding relationship between a corresponding entity type and a quality indicator value generated by the annotator in the corresponding entity type, and / or a corresponding relationship between a text identifier of each test text and a quality indicator value generated by the annotator in the corresponding test text; The device also includes a personnel filtering module, which is used to filter out the labeling personnel whose corresponding quality index values do not meet the set index conditions from the labeling personnel set before constructing the user characteristics of the labeling personnel; The second feature construction module, when constructing the entity type feature of each entity type in the multiple entity types, is specifically used to: construct a feature representation of each entity type according to the entity type structure of each entity type in the multiple entity types to obtain the entity type feature of each entity type; The second feature construction module, when constructing the text topic feature of each test text in the multiple test texts, is specifically used to: construct the text topic feature of each test text according to the topic distribution of each test text in the multiple test texts; The entity type structure of the entity type includes: extracting entity objects and labeling entity labels from knowledge data corresponding to the entity type, and constructing a tree-type entity labeling system structure based on the object relationship of each entity object; The second feature construction module, when constructing a feature representation of each entity type according to the entity type structure of each entity type in the multiple entity types, is specifically used to: According to the entity label contained in each node in the entity type structure of the entity type, a vector representation of the entity type is constructed through a vector space model as an entity type feature of the entity type; The task features of the task to be labeled include: entity type features and / or text topic features of the task to be labeled; The labeling personnel selection module is specifically used for: According to the entity type features of the task to be labeled and the entity type features corresponding to the entity type in the user features of each labeler, the similarity between the entity type of the task to be labeled and the entity type in the user features of each labeler is calculated; the labeler whose corresponding entity type similarity and entity type quality index value meet the first screening condition is determined as the target labeler; Or, according to the text theme features of the task to be labeled and the text theme features corresponding to the test text in the user features of each labeler, the text theme similarity between the task to be labeled and the test text in the user features of each labeler is calculated; and the labeler whose corresponding text theme similarity and test text quality index value meet the second screening condition is determined as the target labeler; The device also includes a user feature updating module, which is used to: After assigning the task to be labeled to the selected target labeler, obtaining and reviewing the labeling result of the task to be labeled by the target labeler, and obtaining a quality index value of the labeling quality of the task to be labeled by the target labeler; Based on the quality indicator values obtained from the review, the user characteristics of the target labelers are updated, so that labelers for subsequent labeling tasks are selected based on the updated user characteristics.
3. An electronic device, characterized in that: include: A memory for storing at least one set of instructions; A processor is used to call and execute the instruction set in the memory, and implement the method for selecting labeling personnel for professional data as described in claim 1 by executing the instruction set.
