Data annotation system and data annotation method

Abstract

The present disclosure relates to a data annotation system and a data annotation method, which comprises the following steps: establishing an annotation specification, and generating at least one question according to the annotation specification; annotating at least one answer in a text document according to the question through a terminal device; converting the question and the answer into annotated data according to an annotation format; determining whether the number of questions in the annotated data is greater than or equal to a threshold value; and generating a reading comprehension data set according to the annotated data if the number of questions is greater than or equal to the threshold value.

Description

Technical Field

[0001] The present disclosure relates to a data annotation system, which can annotate text files and adjust a data analysis model according to the annotated results. Background Art

[0002] In the field of machine learning and deep learning, computers use data analysis models to analyze and interpret files. Since the establishment of data analysis models requires a large amount of training data, and the quality of training data is crucial to the accuracy of the model, how to efficiently and systematically integrate training data without spending a lot of money will be an important topic worth exploring. Summary of the invention

[0003] One embodiment of the present disclosure is a data annotation method, comprising the following steps: establishing an annotation specification, generating at least one question according to the annotation specification. Annotating at least one answer in a text file according to the question through a terminal device. Converting the questions and answers into annotation data according to an annotation format. Determining whether the number of questions in the annotation data is greater than or equal to a threshold value. If the number of questions is greater than or equal to the threshold value, generating a reading comprehension data set according to the annotation data.

[0004] One embodiment of the present disclosure is a data annotation method, comprising the following steps: annotating at least one set of analysis data in the original data according to the annotation specification through a terminal device, wherein each set of analysis data includes at least one question and at least one answer. Converting the analysis data into annotated data according to the annotation format. Determining whether the number of the at least one set of analysis data in the annotated data is greater than or equal to a threshold value through a server. Using the annotated data as training data for a data analysis model to train the data analysis model.

[0005] One embodiment of the present disclosure is a data annotation system, comprising a server and a terminal device. The server stores raw data and annotation specifications. The terminal device is connected to the server to receive the raw data and annotation specifications. The terminal device is used to generate at least one question based on the annotation specifications and the raw data, and annotate at least one answer in the raw data. The terminal device is also used to convert the question and the answer into annotated data. When the number of questions in the annotated data is greater than or equal to a threshold value, the server generates a reading comprehension data set based on the annotated data.

[0006] Accordingly, through the annotation specifications, it can be ensured that the annotation methods of the analysis data, questions and answers generated by the terminal device have a unified level, so that the reading comprehension data set generated by the server can effectively train the data analysis model. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1Schematic diagram of a data annotation system according to some embodiments of the present disclosure.

[0008] Figure 2A and Figure 2B It is a schematic diagram of original files and analysis data according to some embodiments of the present disclosure.

[0009] Figure 3 The present invention is a flowchart of the steps of a data annotation method according to some embodiments of the present disclosure.

[0010] The reference numerals are described as follows:

[0011] 100: Data Annotation System

[0012] 110: Server

[0013] 111: Processor

[0014] 111a: Data Analysis Model

[0015] 112: Storage unit

[0016] 112a: Labeling Specifications

[0017] 112b: Threshold value

[0018] 120: Terminal device

[0019] D1: Raw data

[0020] D2: Test data

[0021] D3: Analyze Data

[0022] S301-S311: Steps

[0023] P1: Test Procedure

[0024] P2: Labeling Procedure DETAILED DESCRIPTION

[0025] The following will illustrate multiple embodiments of the present invention with the accompanying drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is, in some embodiments of the present invention, these practical details are not necessary. In addition, in order to simplify the drawings, some conventional structures and elements will be shown in a simple schematic manner in the drawings.

[0026] In this document, when an element is referred to as "connected" or "coupled", it may refer to "electrically connected" or "electrically coupled". "Connected" or "coupled" may also be used to indicate that two or more elements cooperate or interact with each other. In addition, although the terms "first", "second", etc. are used in this document to describe different elements, the terms are only used to distinguish between elements or operations described by the same technical terms. Unless the context clearly indicates otherwise, the terms do not specifically refer to or imply an order or sequence, nor are they used to limit the present invention.

[0027] See also Figure 1 , which is a schematic diagram of a data annotation system 100 of some embodiments of the present disclosure. The data annotation system 100 includes a server 110 and at least one terminal device 120. The server 110 stores original data D1 and annotation specifications 112a. In some embodiments, the server 110 includes a processor 111 and a storage unit 112. The processor 111 is used to perform data operations. The processor 160 can be a central processing unit (CPU), a system on chip (SoC), an application processor, or a processing chip or controller with specific functions. In addition, the processor 111 can analyze and interpret the file through a data analysis model 111a.

[0028] The storage unit 112 is used to store the original data D1, the test data D2, the annotation specification 112a and the threshold value 112b. In some embodiments, the storage unit 112 can be implemented as a read-only memory, a flash memory, a floppy disk, a hard disk, a USB flash disk, a database accessible by a network, or a storage medium or device with the same function that can be easily thought of by a person skilled in the art. The original data D1 includes a text file, such as the text content of an article or a web page. In other embodiments, the original data D1 may also include a picture file, an audio file or a video file.

[0029] The terminal device 120 is connected to the server 110 to receive the original data D1 and the annotation specification 112a. Each terminal device 120 generates at least one set of analysis data according to the annotation specification 112a and the original data D1. Each set of analysis data includes a question and a corresponding answer. That is, the terminal device 120 is used to generate at least one question and at least one answer, and each question will correspond to at least one answer.

[0030] The annotation specification 112a includes a plurality of rules for analyzing the original data D1, so that the analysis data generated by the terminal device can have a consistent level. In some embodiments, the annotation specification 112a includes question extraction rules, answer extraction rules, answer annotation rules, character sorting rules, and formatting rules. The annotation specification 112a can be established in the form of program instructions so that the terminal device 120 can analyze the original data D1 accordingly. In other embodiments, the annotation specification 112a can also include colloquial description text so that the user of the terminal device can perform manual inspection.

[0031] Here, the content of the annotation specification 112a is illustrated as an example. The "question extraction rule" may include "questions must be clear", "sentences cannot be simply converted into questions", etc. In addition, in some embodiments, the terminal device 120 will extract the corresponding content in the text file for any question (or all questions) in the analysis data. For example, for a text file related to "motion controller", the question extracted by the terminal device 120 is "Which functions do customers have unresolvable technical problems when modifying the horizontal packaging machine?", and the corresponding answer is "rotary cutting, meshing and material length offset compensation functions". At this time, the terminal device 120 will also mark the text paragraph "The customer wants to use the motion controller to modify the horizontal packaging machine and complete the origin return, inching and fixed-point motion functions of the servo axis by himself, but there are technical problems that cannot be solved when using the rotary cutting, meshing and material length offset compensation functions..." in the text file. In one embodiment, because the text paragraph is used to explain the most relevant content of the problem, the answer will also appear in the text paragraph.

[0032] Furthermore, in some embodiments, the "answer extraction rules" may include "the answer is concise and clear" and "must appear in the marked text paragraph". In addition, since the answer may appear multiple times in a text document, the "answer marking rules" may include the position of the answer in the text document or text paragraph. That is, the terminal device 120 will mark the position of the answer in the original data D1. The "appearance position" can be the order in which the answers are arranged in the text paragraph, or it can be the order in which the answers appear in the text document (such as the third appearance of the term "material length offset compensation function"). The terminal device 120 is used to calculate the number of times the answer appears in the text document, and then mark the order of the answers corresponding to the text paragraphs (such as: the position of the third appearance corresponds to the answer).

[0033] In the aforementioned embodiment, the terminal device 120 marks the text paragraph corresponding to the question in the original data D1, and marks the location where the answer appears in the text file. In other embodiments, if the original data D1 is not a text file, the terminal device 120 can mark the audio paragraph or image coordinates corresponding to the question. Similarly, the terminal device 120 can also mark the video playback time where the answer appears.

[0034] The "character arrangement rule" may include "presenting all numbers as Arabic numerals" to unify the text format generated by the terminal device 120. Similarly, the "formatting rule" may include: "separate the captured text paragraphs, questions, and answers with line breaks". The above description is only an example of the annotation specification 112a, and the content of the present disclosure is not limited to this. In addition, the above embodiment is presented in a colloquial description, and in other embodiments, it can be converted into a programming language or other instruction set that can be recognized by the terminal device 120.

[0035] After generating multiple sets of analysis data, the terminal device 120 can convert the questions and answers in the multiple sets of analysis data into annotation data according to a pre-stored annotation format (or provided by the server 110). Then, the terminal device 120 transmits the annotation data to the server 110. When the server 110 determines that the number of questions in the received annotation data is greater than the threshold value 112b (such as 500 or 2000), the server 110 generates a reading comprehension data set based on the annotation data. The reading comprehension data set is a data set that integrates all questions and corresponding answers in the annotation data according to the file format of the data analysis model 111a. The annotation data or the reading comprehension data set can be used as training data for the data analysis model 111a, so that the server 110 can adjust and train the data analysis model 111a.

[0036] In one embodiment, the aforementioned “marked format” refers to the format of the file, which is also a universal standard for communication across devices and programming languages, such as JSON format, HTML format, etc. By marking the format, the server 110 and the terminal device 120 can transmit and analyze data through the same parsing program or parsing module (such as web pages, APPs, analysis programs, etc.), thereby avoiding reading errors.

[0037] Accordingly, since the data annotation system 100 first organizes the original data D1 according to the annotation specification 112a to generate the analysis data D3 consisting of questions and answers, the analysis data D3 generated between different terminal devices 120 will be able to maintain the same analysis level according to the annotation specification 112a, so that the server 110 can efficiently and time-savingly generate a reading comprehension data set to perform machine learning and deep learning on the data analysis model 111a to improve the accuracy of its analysis.

[0038] In some embodiments, the data analysis model 111a is used to perform machine reading and analysis. That is, after the server 110 receives a text file (e.g., a user uploads a paper), the data analysis model 111a can be used to operate the text file. Then, when the server 110 receives a question about the text file (e.g., a user raises a question in the paper), the server 110 can analyze the question through the data analysis model 111a and obtain the corresponding answer from the text file.

[0039] The data analysis model 111a requires a large amount of training data for machine learning or deep learning to establish its operation database. One training method is to manually annotate the original data D1. However, the analysis data (i.e., questions and answers) generated by manual annotation often have problems such as too large a level difference and inconsistent formats, which makes it impossible to effectively train the data analysis model 111a. If professionals are hired to perform annotation, there is a problem of high cost. The present disclosure establishes an annotation specification 112a so that the terminal device 110 can generate analysis data based on it, thereby ensuring the level of questions and answers in the analysis data. Similarly, in other embodiments of the present disclosure, based on similar concepts, the user of the terminal device 120 can review the original document, and then annotate the questions, answers and corresponding text paragraphs through the terminal device 120 to generate analysis data.

[0040] Please refer to Figure 2A and Figure 2B As shown, it is a schematic diagram of a text file in the original data D1, and a schematic diagram of the analysis data D3. For example, the original data D1 is a short article about "asteroid satellites". After receiving the original data D1, the terminal device 120 will generate multiple questions (such as: "What is the definition of an asteroid satellite?") according to the annotation specification 112a. Then, in response to the questions, the corresponding answers are marked in the original data D1 (such as: "An asteroid orbiting another asteroid"). In some embodiments, the questions and answers marked by the terminal device 120 may not be spoken descriptions, but this does not affect the data annotation system 100 from generating annotation data. The data annotation system 100 trains the data analysis model 111a based on the annotation data (or a reading comprehension data set integrated from the annotation data). Therefore, as long as the processor 111 of the server 110 can recognize the questions and answers in the analysis data D3, the data analysis model 111a can be trained.

[0041] In one embodiment, the terminal device 120 marks the analysis data D3 in the original data D1 according to the context analysis module. The context analysis module is set with context analysis parameters to analyze the text content in the original data D1. For example, when the terminal device 120 receives the original data D1, the terminal device 120 will analyze the text document in the original data D1 according to the context analysis parameters to extract multiple key sentences. Then, the terminal device 120 analyzes these key sentences according to the marking specification 112a to obtain multiple sets of analysis data D3. Since people in this field understand the composition and operation of the context analysis module, they will not be further described here.

[0042] As mentioned above, after the terminal device 120 converts the analysis data D3 into the annotation data and transmits the annotation data to the server 110, the server 110 will first determine whether the annotation data conforms to the annotation specification 112a (because the annotation result of the terminal device 120 may still contain questions or answers that do not conform to the annotation specification 112a). In the case where the annotation data does not conform to the annotation specification 112a, the server 110 will adjust the context parsing parameters. The terminal device 120 will update the question according to the annotation specification 112a and the adjusted context parsing parameters (i.e., regenerate new analysis data D3 according to the annotation specification 112a again).

[0043] See also Figure 3 , is a flow chart of a data annotation method of some embodiments of the present disclosure. The data annotation method comprises steps S301 to S311. In step S301, the server 110 first establishes an annotation specification 112a. As mentioned above, the annotation specification 112a may include question extraction rules, answer extraction rules, answer annotation rules, character arrangement rules, and formatting rules. In some embodiments, the annotation specification 112a may be edited and generated by an expert through one of the terminal devices 120 and uploaded to the storage unit of the server 110.

[0044] In some embodiments, the data annotation method includes a detection procedure P1 and an annotation procedure P2. Before performing the formal annotation procedure P2, in order to ensure that the annotation capability of the terminal device 120 meets the standard, the annotation capability of the terminal device 120 can be tested through the detection procedure P1. In step S302, the server 110 transmits test data D2 to the terminal device 120. The test data D2 is similar to the aforementioned original file and may include a text file, a picture file, an audio file, or a video file. The terminal device 120 annotates the test data D2 to generate multiple sets of test analysis content. Each set of test analysis content includes test questions and test answers.

[0045] In step S303, after the terminal device 120 annotates the test data D2, the server 110 receives the test analysis content sent back by the terminal device 120 to analyze the annotation capability of the terminal device 120. If the annotation capability is not as expected, it is likely that the annotation specification 112a is not complete, so the server 110 will also determine whether the annotation specification 112a needs to be adjusted. In some embodiments, the server 110 may store multiple groups of manual analysis content that have been manually annotated in advance, so that the server 110 can compare the similarity between the manual analysis content and the test analysis content to determine the annotation capability of the terminal device 120. In other embodiments, the server 110 can determine whether the annotation capability of the terminal device 120 is stable and uniform based on the similarity between each group of test analysis content.

[0046] After the server 110 analyzes the annotation capability of the terminal device 120, if the server 110 determines that the annotation specification 112a needs to be adjusted, it returns to step S301 and readjusts the annotation specification 112a, for example, adjusting the parameter range of the annotation specification 112a. If the server 110 determines that the annotation specification 112a does not need to be adjusted, then in step S304, it further determines whether it is necessary to adjust the annotation parameters in the terminal device 120? This is because the parameters used to perform the annotation function in the terminal device 120 will also affect its annotation capability.

[0047] If the server 110 determines that the annotation parameters in the terminal device 120 need to be adjusted, the process proceeds to step S305 to adjust the terminal device 120. In some embodiments, the terminal device 120 analyzes the text file in the test data D2 according to the context parsing parameters in the context parsing module to generate test questions and answers. Therefore, the server 110 readjusts the context parsing parameters (e.g., re-updates the database, re-performs deep learning on the context parsing module).

[0048] If the server 110 determines that there is no need to adjust the annotation parameters in the terminal device 120, it means that the "test program P1" has passed and the formal "annotation program P2" can be executed. In step S306, the terminal device 120 receives the original data D1 transmitted by the server 110, and annotates at least one set of analysis data D3 in the original data D1 (such as: text file, image file, audio file or video). Each set of analysis data D3 includes at least one question and at least one answer. As mentioned above, the data annotation system 100 of the present disclosure is used to generate annotation data based on questions and answers, and uses the annotation data (or reading comprehension data set) as training data for the data analysis model 111a, so that the server 110 can adjust and train the data analysis model 111a. The annotation data can be generated by multiple terminal devices 120 respectively, and then the server 110 uniformly performs machine learning and deep learning on the data analysis model 111a. Therefore, in step S306 , each terminal device 120 is used to mark at least one set of analysis data D3 , and the server 110 can receive multiple sets of analysis data D3 from multiple terminal devices 120 .

[0049] In other embodiments, the terminal device 120 may mark multiple corresponding candidate answers for a question, for example, listing three answers and respectively listing the predicted accuracy rates after analysis (eg, answer A has an accuracy rate of 80%, answer B has an accuracy rate of 60%, etc.).

[0050] In some embodiments, the terminal device 120 first generates a plurality of questions according to the marking specification 112a, and then marks the corresponding answers from the original content to obtain a plurality of sets of analysis data D3.

[0051] In step S307 , the terminal device 120 converts all questions and corresponding answers in the analysis data D3 into annotated data according to the annotation format, and uploads the annotated data to the server 110 .

[0052] In some embodiments, the "original data D1" includes multiple text files (or image files, audio files) to be analyzed, and the terminal device 120 can generate multiple sets of analysis data D3 based on each text file (or image file, audio file). For example, the server 110 transmits 5,000 files (i.e., "original data D1", which may include text files, image files, audio files) to the terminal device 120. The terminal device 120 will generate at least one question and answer for each file.

[0053] In step S308, the server 110 first determines whether the annotated data conforms to the annotated format. Although the terminal device 120 converts the analysis data D3 into the annotated data according to the annotated format, in actual operation, the terminal device 120 may still produce errors during the conversion. This error may be due to the poor quality of the analysis data D3 produced in step S306. Therefore, after receiving the annotated data, the server 110 will perform another format check. If the server 110 determines that the annotated data does not conform to the annotated format, it proceeds to step S309, adjusts the annotation parameters (such as: context analysis parameters) in the terminal device 120, and then returns to step S306. The terminal device 120 regenerates the analysis data D3 based on the adjusted annotation parameters, converts it into annotation data, and then uploads it to the server 110 to re-perform step S308 inspection.

[0054] If the server 110 determines that the annotated data conforms to the annotated format, then in step S310, the server 110 further determines whether the number of the questions in the annotated data is greater than the threshold value 112b. In some embodiments, the server 110 determines the total number of all questions (e.g., whether it is greater than 5,000). In other embodiments, if the original data D1 contains 3,000 text files, the server 110 will determine whether the number of questions in each text file is greater than 5.

[0055] In step S310, if the server 110 determines that the number of questions in the annotated data is less than the threshold value 112b, it means that the server 110 determines that the terminal device 120 has not completed the data annotating process, and returns to step S306; if the server 110 determines that the number of questions in the annotated data is greater than the threshold value 112b, then the process proceeds to step S311, and the server 110 generates a reading comprehension data set based on the annotated data. The server 110 can use the annotated data as training data for the data analysis model 111a to train the data analysis model 111a.

[0056] The various elements, method steps or technical features in the aforementioned embodiments may be combined with each other and are not limited to the order of textual description or the order of presentation of the drawings in the present disclosure.

[0057] Although the present disclosure has been disclosed in the above embodiments, it is not intended to limit the present disclosure. Any professional in the field can make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be determined by the appended claims.

Claims

1. A data annotation method, comprising: Establishing a labeling specification, generating at least one question according to the labeling specification; Using a terminal device, marking at least one answer in a text document according to the question; Converting the question and the answer into a labeled data according to a labeled format; Determining whether the number of the at least one question in the labeled data is greater than or equal to a threshold value; as well as If the number of the at least one question is greater than or equal to the threshold value, generating a reading comprehension data set according to the annotated data; The data annotation method also includes: receiving a test analysis content transmitted by the terminal device; Analyzing a marking capability of the terminal device according to the test analysis content; When the annotation capability of the terminal device does not meet an expected value, adjusting the annotation specification, The terminal device marks the at least one question according to a context analysis parameter, wherein the context analysis parameter is used to analyze the text document to extract a plurality of key sentences. The data marking method further includes: Determine whether the annotation data complies with the annotation specification; If the annotation data does not conform to the annotation specification, adjusting the context parsing parameter; and The at least one question is updated according to the annotation specification and the context parsing parameter.

2. The data labeling method according to claim 1, further comprising: The reading comprehension data set is used as training data of a data analysis model through a server to train the data analysis model.

3. The data labeling method according to claim 1, wherein the data labeling method further comprises: Through the terminal device, a section of text content in the text file is obtained based on the question.

4. The data labeling method according to claim 3, further comprising: In the text content, multiple corresponding candidate answers are marked, and the predicted accuracy after analysis is listed respectively.

5. A data annotation method, comprising: Using a terminal device, annotating at least one set of analysis data in an original data according to a marking specification, wherein each set of analysis data includes a question and an answer; According to a marking format, converting the analysis data into a marked data; Determining, by a server, whether the amount of the analysis data in the annotated data is greater than a threshold value; as well as If it is greater than the threshold value, the labeled data is used as training data of a data analysis model to train the data analysis model; The data annotation method also includes: receiving a test analysis content transmitted by the terminal device; Analyzing a marking capability of the terminal device according to the test analysis content; When the annotation capability of the terminal device does not meet an expected value, adjusting the annotation specification, The terminal device marks at least one question according to a context analysis parameter, wherein the context analysis parameter is used to analyze the original data to extract a plurality of key sentences. The data marking method further includes: Determining whether the at least one set of analysis data in the labeled data complies with the labeling specification; If the at least one set of analysis data does not meet the annotation specification, adjusting the context parsing parameter; as well as The at least one set of analysis data is updated according to the annotation specification and the context parsing parameter.

6. The data labeling method as claimed in claim 5, wherein the original data comprises a text file, and the data labeling method further comprises: Through the terminal device, a section of text content in the text file is obtained according to the question in the at least one set of analysis data.

7. The data labeling method according to claim 6, further comprising: Analyzing the text document according to a text parsing parameter to extract a plurality of key sentences; and The key sentence is analyzed according to the annotation specification to obtain the at least one set of analysis data.

8. A data annotation system, comprising: A server storing raw data and a labeling specification; a terminal device connected to the server to receive the original data and the annotation specification, wherein the terminal device is used to generate at least one question according to the annotation specification and the original data, and annotate at least one answer in the original data; and the terminal device is also used to convert the at least one question and the at least one answer into an annotated data, and when the number of the at least one question in the annotated data is greater than or equal to a threshold value, the server generates a reading comprehension data set according to the annotated data; The server is further configured to receive a test analysis content transmitted by the terminal device, and analyze a marking capability of the terminal device according to the test analysis content; if the marking capability of the terminal device does not meet an expected value, the server adjusts the marking specification. When the server determines that at least one question in the labeled data does not comply with the labeling specification, the server controls the terminal device to adjust a context analysis parameter in the terminal device, and the terminal device analyzes the original data according to the context analysis parameter to update the at least one question and the at least one answer.

9. A data labeling system as described in claim 8, wherein the terminal device is used to convert the at least one question and the at least one answer into the labeling data according to a labeling format; and when the server receives the labeling data, the server is also used to determine whether the labeling data conforms to the labeling format.

10. The data annotation system as claimed in claim 8, wherein the server is further used to use the reading comprehension data set as training data for a data analysis model to train the data analysis model.

11. The data tagging system as claimed in claim 8, wherein the original data comprises a text file, and the terminal device is further used to obtain a text content in the text file according to one of the questions.

12. The data labeling system as claimed in claim 11, wherein the terminal device is also used to label multiple corresponding candidate answers and list the predicted accuracy rates after analysis respectively.

Images