A low-cost entity annotation method and system based on user behavior analysis
By collecting user documents and revision records using a state machine to optimize the annotation dataset, and by using a NER model to replace the state machine, the problem of scarce annotation data is solved, achieving low-cost and efficient entity annotation. The model accuracy continuously improves with the number of times users use it.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUZHOU UNIV ZHICHENG COLLEGE
- Filing Date
- 2023-03-22
- Publication Date
- 2026-06-09
AI Technical Summary
In entity recognition tasks, the scarcity of labeled training data makes it difficult to achieve high-quality NER models. Furthermore, existing methods for reducing the workload of labeled data either suffer from insufficient accuracy or require a large amount of manual labeling.
By utilizing state machines to provide document formatting services, collecting user historical documents and entity recognition results, optimizing the labeled dataset by combining user revision records, and replacing state machines with NER models, user behavior analysis is used to reduce the workload of labeling and improve labeling accuracy.
With a relatively small amount of annotation work, a high annotation accuracy rate was achieved. As the number of users increases and the number of annotated datasets grows, the model accuracy continues to improve and the service quality is automatically upgraded.
Smart Images
Figure CN116776878B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a low-cost entity annotation method and system based on user behavior analysis. Background Technology
[0002] In the field of artificial intelligence, there is a type of task called Entity Recognition (NER), which is used to extract entity objects from natural language. To achieve NER modeling, sufficient training sample data with labeled entities is first required.
[0003] Taking intelligent paper typesetting applications as an example, the system needs to automatically identify dozens of different elements in a document, such as the title, author, first-level headings, and figure numbers. After the content is identified, its format can be standardized and adjusted. Due to the arbitrary nature of document authors' writing, ordinary state machines cannot accurately identify these entities, while a well-trained NER (Non-Executable Formatting) system can adapt to various common writing problems. Figure 1 In the "2.13D model generation effect", the state machine cannot identify whether the title number should be 2.1 or 2.13, while the NER model can.
[0004] To obtain a well-trained NER, these entities must be labeled on a large corpus (e.g., Figure 1 In this context, various types of entities are labeled with their positions and entity types within sentences. However, in practical applications, the scarcity of labeled data, requiring manual annotation and being extremely costly, hinders the development of high-quality NER models.
[0005] Common methods for reducing the workload of data annotation include: first, manually annotating a small amount of data, then building a Negative Errata (NER) model based on this data. Next, using the NER model to annotate the unannotated corpus. However, since this NER model is trained on a small amount of data, its accuracy is insufficient, often requiring manual correction of the annotation results. Another approach is to build highly efficient platforms, such as providing user-friendly interfaces and importing dictionaries and auxiliary tools during the annotation process, to minimize the time spent on manual annotation.
[0006] Chinese patent application number 202210451731.2 discloses a training method and annotation method for an annotation model based on self-learning annotation. This method first uses a small number of manually annotated samples to train the model, and then uses the model to annotate the unannotated corpus, forming new annotated data. Due to insufficient model training, samples with low confidence are manually annotated again to prevent mislabeling. This significantly reduces the workload compared to manual annotation of all samples. However, the main problem with this method is that the confidence threshold is difficult to control. If the confidence threshold is set too high, the number of manual annotations will increase dramatically, failing to achieve the goal of reducing annotations. If the confidence threshold is too low, the accuracy of the data annotation will decrease. Summary of the Invention
[0007] The purpose of this invention is to provide a low-cost entity annotation method and system based on user behavior analysis, which can achieve a high annotation accuracy with less annotation workload.
[0008] To achieve the above objectives, the technical solution adopted by this invention is: a low-cost entity annotation method based on user behavior analysis, comprising the following steps:
[0009] S1. Data Collection: Utilize state machines to provide document formatting services, collect user historical documents, entity recognition results, and user revision records;
[0010] S2. Data annotation: Generate an annotated dataset based on entity recognition results, use user revision records to find suspicious erroneous annotations and reconfirm them, and optimize the annotated dataset;
[0011] S3. Model Update: Train the NER model using the labeled dataset. When the accuracy of the NER model exceeds that of the state machine, replace the state machine in step S1 with the NER model.
[0012] Furthermore, in step S1, data collection includes the following steps:
[0013] S101. The user uploads the document to be formatted to the system;
[0014] S102. The system uses a state machine to identify entities in the document;
[0015] S103. The system performs format checks and corrections on the document according to the identified entities, and generates a standardized document for users to download; at the same time, it records the original document, correction records, and the entities identified in the document into the document library;
[0016] S104. The user downloads the document formatted by the system, checks the document format, finds content that the system misidentifies, repairs and adjusts the downloaded document, and adjusts other content in the document that needs to be adjusted.
[0017] S105. The user submits repeated formatting to the system until the service is no longer needed.
[0018] Further, in step S2, the user data in the document library is read and traversed. For each user data, the following operation is performed:
[0019] S201. For the user's first uploaded document and the entity list identified by the system, construct labeled data samples;
[0020] S202. Examine the revision history in subsequent user submissions to identify potential entity identification errors;
[0021] S203. Check all suspicious erroneous annotation records and fix errors in the annotation dataset.
[0022] Furthermore, in step S202, user behavior recognition rules are formulated to reduce interference in the process of detecting potential entity recognition errors; the user behavior recognition rules include:
[0023] ① The historical versions of this paragraph contain at least one entity;
[0024] ② The user performs a rollback operation on the same paragraph;
[0025] ③ Different entity recognition results appeared in different versions of the same paragraph.
[0026] The present invention also provides a low-cost entity annotation system based on user behavior analysis, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.
[0027] Compared with existing technologies, this invention has the following advantages: It provides a low-cost entity annotation method and system based on user behavior analysis. This method does not require annotating the original dataset, nor does it require a trade-off between annotation workload and accuracy thresholds, achieving high accuracy with less annotation workload. As the number of service sessions increases, the number of annotations required decreases, while the amount of annotation dataset increases. The system developed based on this method will automatically upgrade its service layout quality as user usage increases, the amount of annotated data increases, the model becomes more accurate, the service becomes higher quality, and more users participate. Attached Figure Description
[0028] Figure 1 It is the entity annotation result in the document in the existing technology.
[0029] Figure 2This is a flowchart illustrating the method implementation of an embodiment of the present invention.
[0030] Figure 3 This is a flowchart of data collection in an embodiment of the present invention.
[0031] Figure 4 This is a schematic diagram of a paper uploaded by a user in an embodiment of the present invention.
[0032] Figure 5 This is a schematic diagram illustrating the use of a state machine to identify entities in a paper in an embodiment of the present invention.
[0033] Figure 6 This is a schematic diagram illustrating the generation of specification documents in an embodiment of the present invention.
[0034] Figure 7 This is a flowchart of data annotation in an embodiment of the present invention.
[0035] Figure 8 This is an example diagram of the user's original document fragment and the corresponding constructed annotation data in an embodiment of the present invention.
[0036] Figure 9 This is an example diagram of a document downloaded and revised by a user in an embodiment of the present invention.
[0037] Figure 10 This is an example diagram illustrating the repair of errors in the labeled dataset in an embodiment of the present invention. Detailed Implementation
[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0039] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.
[0040] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0041] like Figure 2 As shown, this embodiment provides a low-cost entity annotation method based on user behavior analysis, including the following steps:
[0042] S1. Data Collection: Utilize state machines to provide document formatting services, collect user historical documents, entity recognition results, and user revision records;
[0043] S2. Data annotation: Generate an annotated dataset based on entity recognition results, use user revision records to find suspicious erroneous annotations and reconfirm them, and optimize the annotated dataset;
[0044] S3. Model Update: Train the NER model using the labeled dataset. When the accuracy of the NER model exceeds that of the state machine, replace the state machine in step S1 with the NER model.
[0045] The following example, using the formatting of a thesis document, illustrates how to implement this method to obtain low-cost annotation data.
[0046] (a) Data collection
[0047] The system initially used a state machine to construct a scheme for recognizing entities, and then corrected the user's paper format (font, indentation, etc.) based on the recognition results. Since the paper itself has a relatively fixed format, the state machine itself has a certain degree of accuracy.
[0048] like Figure 3 As shown, the data collection process is as follows:
[0049] S101. The user uploads the paper to be formatted to the system. The uploaded paper has many formatting irregularities, such as... Figure 4 As shown.
[0050] S102. The system uses a state machine to identify entities in the paper, such as... Figure 5 As shown.
[0051] S103. The system will perform format checks and corrections on the paper according to the identified entities, and generate a standardized document for users to download, such as... Figure 6 As shown. Simultaneously, the original document, correction record, and entities identified in the document are recorded in the document library.
[0052] S104. Users download the paper after it has been formatted by the system, check the paper format, and find any content that the system has misidentified. They then repair and adjust the downloaded paper, as well as adjust other content that needs to be adjusted.
[0053] S105. The user submits repeated formatting to the system until the service is no longer needed.
[0054] (ii) Data labeling
[0055] Iterate through and read user data from the document library, such as... Figure 7 As shown, the following operations are performed for each user's data:
[0056] S201. For the user's first uploaded document and the entity list identified by the system, construct labeled data samples.
[0057] like Figure 8 The image shows an example of a user's original document fragment and the corresponding constructed annotation data (using the BIOES annotation method).
[0058] In the example above, four entities appeared, with the last two being incorrectly identified. They should have been part of the main text, but were treated as chapter numbers and titles.
[0059] 1. Named Entity Recognition Task
[0060] An entity refers to an objectively existing and distinguishable thing; it can be a specific person, event, or object, or it can be a concept. Named Entity Recognition (NER) aims to identify named entities in text. Specific categories emerge in particular domains, such as drug names and diseases in the medical field.
[0061] For example: Xiaoming goes to school at 8:00 AM.
[0062] In this sentence, "Xiaoming" is a personal name, "8 a.m." is a time entity, and "school" is a location entity. "Go" and "go to class" are non-entities.
[0063] 2. BIOES data annotation method
[0064] The letters in BIOES represent:
[0065] B indicates the start of the entity.
[0066] I represents the middle of the entity.
[0067] O indicates outside the entity.
[0068] E indicates the end of the entity.
[0069] S represents a single character / word that is independently called an entity.
[0070] Xiao Ming goes to school at 8 o'clock in the morning for class.
[0071] B-PER,E-PER,B-TIM,I-TIM,I-TIM,E-TIM,O,B-LOC,E-LOC,O,O
[0072] PER represents a person's name, TIM represents a time, and LOC represents a location.
[0073] S202. Examine the revision history in subsequent user submissions to identify potential entity identification errors.
[0074] Due to an entity recognition error, the document downloaded by the user (referred to as version V1.1) contains the following: Figure 9 The formatting is incorrect as shown. Users typically correct this formatting issue in subsequent submissions.
[0075] Therefore, the system can obtain a revision log: "The font of Chapter 1 Introduction..." has been changed from 18pt to 12pt.
[0076] To reduce interference in the detection of potential entity recognition errors, user behavior recognition rules are established to record suspected annotation errors. A user must perform the following actions upon detecting a potential entity recognition error for the error to be considered genuine. This is because user revisions are not necessarily due to system errors; the user may simply be adjusting their paper. Without filtering and recognition, misjudgments are likely. Only revision records meeting the following conditions will be analyzed to determine whether the original annotations need to be corrected.
[0077] User behavior recognition rules may include (but are not limited to):
[0078] ① The historical versions of this paragraph contain at least one entity.
[0079] ② The user performs a rollback operation on the same paragraph.
[0080] ③ Different entity recognition results appeared in different versions of the same paragraph.
[0081] S203. Check all suspicious erroneous annotation records and fix errors in the annotation dataset, such as... Figure 10 As shown. [Optional] Domain experts categorize and summarize annotation error records to improve the state machine's recognition capabilities.
[0082] (III) Model Update
[0083] Using a labeled dataset, a NER model is trained. Once the NER model's accuracy surpasses that of the state machine, the state machine in step S1 is replaced. During this process, as the training dataset size gradually increases and the model accuracy continuously improves, the workload requiring domain expert annotation gradually decreases.
[0084] The annotation scheme provided by this invention does not require a pre-annotated minimum dataset, but it does require that the corpus to be annotated possesses a relatively high recognition accuracy even when used with a state machine. It collects the corpus using a server that provides free services, with the user's knowledge. Initial annotation is performed using a state machine, followed by supplementary annotation through the user's revision process, thus solving the problem of inaccurate corpus annotation by the state machine.
[0085] This embodiment also provides a low-cost entity annotation system based on user behavior analysis, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.
[0086] This invention utilizes the imperfect service provided to users, leveraging their feedback on service outcomes. By comparing user revisions between two versions and applying pre-defined user behavior filtering rules, it identifies suspicious system entity recognition errors. The invention first uses a state machine to provide entity annotations for the corpus, then identifies user behavior that corrects the annotated data, resulting in a more accurate annotated dataset. This annotated dataset is then used to train a model to annotate new corpus entities, replacing the state machine. The annotation results for new corpus entities are then further improved through this iterative process. This iterative process allows for the annotation of more data with less manual intervention.
[0087] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0088] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.
Claims
1. A low-cost entity annotation method based on user behavior analysis, characterized in that, Includes the following steps: S1. Data Collection: Utilize state machines to provide document formatting services, collect user historical documents, entity recognition results, and user revision records; S2. Data annotation: Generate an annotated dataset based on entity recognition results, use user revision records to find suspicious erroneous annotations and reconfirm them, and optimize the annotated dataset; S3. Model Update: Train the NER model using the labeled dataset. When the accuracy of the NER model exceeds that of the state machine, replace the state machine in step S1 with the NER model.
2. The method for low-cost entity annotation based on user behavior analysis according to claim 1, characterized in that, In step S1, data collection includes the following steps: S101. The user uploads the document to be formatted to the system; S102. The system uses a state machine to identify entities in the document; S103. The system performs format checks and corrections on the document according to the identified entities, and generates a standardized document for users to download; at the same time, it records the original document, correction records, and the entities identified in the document into the document library; S104. The user downloads the document formatted by the system, checks the document format, finds content that the system misidentifies, repairs and adjusts the downloaded document, and adjusts other content in the document that needs to be adjusted. S105. The user submits repeated formatting to the system until the service is no longer needed.
3. The method for low-cost entity annotation based on user behavior analysis according to claim 2, characterized in that, In step S2, the user data in the document library is read and traversed. For each user data, the following operation is performed: S201. For the user's first uploaded document and the entity list identified by the system, construct labeled data samples; S202. Examine the revision history in subsequent user submissions to identify potential entity identification errors; S203. Check all suspicious erroneous annotation records and fix errors in the annotation dataset.
4. The method for low-cost entity annotation based on user behavior analysis according to claim 3, characterized in that, In step S202, user behavior recognition rules are formulated to reduce interference in the process of detecting potential entity recognition errors; the user behavior recognition rules include: ① The paragraph's historical versions contain at least one entity; ② The user performs a rollback operation on the same paragraph; ③ Different entity recognition results appeared in different versions of the same paragraph.
5. A low-cost entity annotation system based on user behavior analysis, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the steps of the method as described in any one of claims 1-4.