Keyword labeling auditing system and auditing method

Abstract

The application provides a keyword marking auditing system and an auditing method, belongs to the technical field of natural language processing, is applied to the auditing process after keyword marking, and solves the problem of lacking a special marking auditing system and method in the prior art; the system comprises a data generation module and an auditing interaction module; the data generation module is used for acquiring marking data of keywords and outputting the marking data of the keywords to the auditing interaction module; the auditing interaction module is used for receiving and displaying the marking data of the keywords and providing interactive operation for keyword marking auditing; in the auditing method, firstly, whether a focus keyword meets the requirements is audited, then, according to two modes of context priority and label priority, context auditing and label auditing processes of the focus keyword are completed, and finally, the auditing of all keywords in the auditing work is completed; the application is customized for the auditing task of keyword marking, is easy to use in the auditing process, and can complete the keyword marking auditing work with high quality and high efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and is applied to the auditing process after keyword labeling, specifically a keyword labeling auditing system and an auditing method. BACKGROUND

[0002] In the processes of word segmentation, named entity recognition and keyword extraction in natural language processing, labeling high-quality keywords can greatly improve the performance of system algorithms. Auditing the keyword labeling results is an important means to ensure high-quality labeling. The quality of labeling auditing depends on professional knowledge and keyword context information, especially in the auditing of corpus labeling in specific fields.

[0003] The efficiency of keyword labeling auditing depends on auditing tools, auditing methods and labeling tools, etc. In the prior art, attention is often only paid to the process of how to label keywords, but the process of strictly auditing the labeled keywords in specific fields is ignored. There are many systems that can implement keyword labeling, such as Doccano, SUTDAnnotator, etc., but there are almost no research or mention of systems that can audit labeled results with high quality and efficiency. When facing an auditing task, the keyword labeling system is usually directly used for auditing work, or some auditing functions are customized based on the keyword labeling system. Therefore, auditing is often ignored in the field of natural language processing, especially in various labeling work; however, this part of the content is particularly important in specific fields and cannot be ignored.

[0004] In the field of transport aviation pilot training, flight training evaluation is a strict evaluation of the skills and abilities of flight students, and the evaluation process and results both include a large amount of text corpus. The evaluation of the "nine core competencies" is an important part of the transport aviation pilot skill life cycle management system, including "application of knowledge, procedure execution and compliance with regulations, flight path management-automatic flight, flight path management-hand flight, communication, leadership and teamwork, situational awareness and information management, workload management, problem solving and decision making"; this evaluation work requires a large number of high-quality labeled keywords, and the auditing process belongs to the technical field of natural language processing. As the field experts who audit various types of keyword labeling in the "nine core competencies", the existing keyword labeling tools and the simple addition of auditing methods based on the labeling tools are not very friendly to them. Therefore, for various types of work involving keyword labeling auditing in specific fields, it is necessary to design a dedicated and easy-to-use auditing system to improve the willingness of field experts to participate in auditing, and to customize or match efficient auditing methods as needed, ultimately improving the quality and efficiency of the auditing work. SUMMARY

[0005] In order to guarantee the labeling quality by the keyword labeling review mode, and according to professional knowledge and keyword context information in the review process, the application discloses a keyword labeling review system and a review method. The application is specially customized for the keyword labeling review task, and can review the keyword labeling with high quality and high efficiency.

[0006] The application adopts the following technical solutions to achieve the purpose:

[0007] The keyword labeling review system comprises a data generation module and a review interaction module; the data generation module is used for acquiring keyword labeling data and outputting the keyword labeling data to the review interaction module; and the review interaction module is used for receiving and displaying the keyword labeling data, and providing interactive operation for keyword labeling review.

[0008] The keyword labeling data refers to the keyword labeling result of the labeling personnel.

[0009] Specifically, the keyword labeling data acquired by the data generation module comprises a keyword, a keyword context, a selected keyword label and a candidate keyword label.

[0010] The keyword context refers to a text containing the keyword; the selected keyword label refers to a label assigned to the keyword in the labeling process; the candidate keyword label refers to a label not yet assigned to the keyword in the labeling process; each keyword in the keyword labeling data has corresponding context, selected label and candidate label, and the context, selected label and candidate label can be of any number.

[0011] Further, the review interaction module comprises a keyword interaction module, a context interaction module and a label interaction module; the data generation module is used for outputting a sequence composed of multiple keywords in the keyword labeling data to the keyword interaction module, and outputting context information of a focus keyword to the context interaction module and outputting selected labels and candidate labels of the focus keyword to the label interaction module according to the focus keyword set by the keyword interaction module.

[0012] The focus keyword refers to a keyword selected by keyword interaction operation in the keyword sequence.

[0013] Specifically, the keyword interaction module is used for displaying the keyword sequence and providing keyword interaction operation; the keyword interaction operation comprises deleting a keyword from the keyword sequence and setting a keyword in the keyword sequence as the focus keyword.

[0014] Specifically, the context interaction module is configured to display the context of the focus keyword and provide a context interaction operation; the context interaction operation includes deleting the context of the focus keyword.

[0015] Specifically, the label interaction module is configured to display the selected label and the candidate label of the focus keyword and provide a label interaction operation; the label interaction operation includes changing the selected label of the focus keyword to the candidate label and changing the candidate label of the focus keyword to the selected label.

[0016] The application also discloses a keyword labeling auditing method, which comprises the following steps:

[0017] The keyword labeling auditing system is configured to acquire and display the labeling data of the keyword and select an auditing mode; the auditing mode includes a context priority auditing mode and a label priority auditing mode.

[0018] The keyword interaction module is configured to set the focus keyword in the keyword sequence.

[0019] The context interaction module is configured to display the context of the focus keyword.

[0020] The label interaction module is configured to display the selected label and the candidate label of the focus keyword.

[0021] The focus keyword is audited to determine whether it is a keyword meeting the requirements; if yes, the context of the focus keyword and / or the selected label and the candidate label of the focus keyword are audited according to the selected auditing mode.

[0022] After the auditing of one focus keyword is completed according to the selected auditing mode, another keyword in the keyword sequence that has not been audited is set as the focus keyword, and the auditing process is repeated until the auditing of all the keywords in the keyword sequence is completed.

[0023] Specifically, the focus keyword is audited to determine whether it is a keyword meeting the requirements; if not, the focus keyword is deleted from the keyword sequence, and another keyword is set as the focus keyword.

[0024] Further, the context priority auditing mode specifically includes auditing the context of the focus keyword to determine whether it is correct according to the focus keyword and the selected label of the focus keyword; if yes, no interaction operation is performed, and the labeling result is retained; if not, the context irrelevant to the focus keyword and the selected label of the focus keyword is deleted.

[0025] Further, the label priority review mode is specifically: according to the focus keyword and the context of the focus keyword, whether the selected label and the candidate label of the focus keyword are correct is reviewed, if correct, no interactive operation is performed, and the labeling result is reserved; if incorrect, the selected label of the focus keyword is changed to the candidate label, and the correct label in the candidate label of the focus keyword is changed to the selected label.

[0026] In summary, due to the adoption of the technical solution, the beneficial effects of the present application are as follows:

[0027] 1. The system of the present application is specially customized for the review task of keyword labeling, and the business process of keyword review is remodeled by driving the review process with keywords, thereby improving the review efficiency.

[0028] 2. In the review process, the present application only provides keywords, labels and contexts of the keywords for review, and most of the redundant information irrelevant to keyword labeling review is shielded, thereby improving the review efficiency.

[0029] 3. The review interaction of the present application mainly relies on deletion or point selection operation, which reduces the review complexity and improves the usability of the review system. In the implementation of the review, the main work can be completed by only clicking the mouse, which can further improve the willingness of field experts to review keyword labeling while improving the review efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 FIG. 1 is a structural schematic diagram of the system of the present application;

[0031] Figure 2 FIG. 3 is a schematic diagram of a layout mode of the review interaction module. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0034] Embodiment 1

[0035] AsFigure 1 As shown in the figure, the keyword labeling auditing system comprises a data generation module and an auditing interaction module; the data generation module is configured to obtain keyword labeling data and output the keyword labeling data to the auditing interaction module; the auditing interaction module is configured to receive and display the keyword labeling data and provide interactive operations for keyword labeling auditing.

[0036] In this embodiment, the keyword labeling data refers to keyword labeling results of the labeling personnel.

[0037] The keyword labeling data obtained by the data generation module comprises a keyword, a context of the keyword, selected labels of the keyword, and candidate labels of the keyword; wherein the keyword, the selected labels of the keyword, and the candidate labels of the keyword can be used as basic labeling data, and the context of the keyword can be used as additional labeling data.

[0038] In this embodiment, the context of the keyword refers to a text containing the keyword; the selected labels of the keyword refer to labels that have been assigned to the keyword in the labeling process; the candidate labels of the keyword refer to labels that have not been assigned to the keyword in the labeling process; each keyword in the keyword labeling data has a corresponding context, selected labels, and candidate labels, and the context, the selected labels, and the candidate labels can be of any number.

[0039] As shown in the figure, Figure 1 and Figure 2 As shown in the figure, the auditing interaction module comprises a keyword interaction module, a context interaction module, and a label interaction module; the data generation module is configured to output a sequence comprising a plurality of keywords in the keyword labeling data to the keyword interaction module, and output context information of a focus keyword to the context interaction module and output selected labels and candidate labels of the focus keyword to the label interaction module according to the focus keyword set by the keyword interaction module; the focus keyword refers to a keyword selected by keyword interaction operation in the keyword sequence.

[0040] The keyword interaction module is configured to display the keyword sequence and provide keyword interaction operations; the keyword interaction operations comprise deleting a keyword from the keyword sequence and setting a keyword in the keyword sequence as the focus keyword.

[0041] The context interaction module is configured to display the context of the focus keyword and provide context interaction operations; the context interaction operations comprise deleting the context of the focus keyword.

[0042] The label interaction module is configured to display the selected labels and the candidate labels of the focus keyword and provide label interaction operations; the label interaction operations comprise changing the selected labels of the focus keyword to the candidate labels and changing the candidate labels of the focus keyword to the selected labels.

[0043] In this embodiment, Figure 2 The given audit interaction module layout can be specifically applied to the hardware screen of the audit terminal to display the keyword labeling data, facilitating the field experts to view and perform the audit work. Figure 2 In the keyword interaction module, K i represents the i-th keyword; when K i is set as the focus keyword, C i,j in the context interaction module represents the j-th context of the focus keyword K i ; T i,k in the label interaction module represents the k-th label of the focus keyword K i ; at the corresponding positions of each keyword and each context, a clickable delete key is provided to perform the keyword interaction operation and the context interaction operation through clicking; at the position of each label, a checkable check box is provided to realize whether to check through clicking, and the checked label becomes the selected label of the focus keyword, and the unchecked label becomes the candidate label of the focus keyword.

[0044] Embodiment 2

[0045] Based on the system of Embodiment 1, this embodiment specifically introduces the keyword labeling audit method, and the process of the method is summarized as follows:

[0046] Through the keyword labeling audit system, the keyword labeling data is acquired and displayed, and the audit mode is selected; the audit mode includes a context priority audit mode and a label priority audit mode;

[0047] Through the keyword interaction module, the focus keyword in the keyword sequence is set;

[0048] Through the context interaction module, the context of the focus keyword is displayed;

[0049] Through the label interaction module, the selected label and the candidate label of the focus keyword are displayed;

[0050] The focus keyword is audited to determine whether it is a keyword meeting the requirements, if yes, the context of the focus keyword and / or the selected label and the candidate label of the focus keyword are audited according to the selected audit mode; if not, the focus keyword is deleted from the keyword sequence, and another keyword is set as the focus keyword;

[0051] After the audit of one focus keyword is completed according to the selected audit mode, another keyword in the keyword sequence which has not been audited is set as the focus keyword, and the audit process is repeated until the audit of all the keywords in the keyword sequence is completed.

[0052] In the embodiment, the context priority review mode specifically refers to: according to the focus keyword and the selected label of the focus keyword, reviewing whether the context of the focus keyword is correct; if correct, no interactive operation is performed, and the labeling result is retained; if incorrect, the context of the focus keyword at this time is deleted; the label priority review mode specifically refers to: according to the focus keyword and the context of the focus keyword, reviewing whether the selected label and the candidate label of the focus keyword are correct, if correct, no interactive operation is performed, and the labeling result is retained; if incorrect, the selected label of the focus keyword is changed to the candidate label, and the correct label in the candidate label of the focus keyword is changed to the selected label.

[0053] In order to better understand the keyword labeling review system and the review method in the embodiment, the embodiment takes the keyword labeling review of flight training evaluation in the training field of transport aviation pilots as an example to specifically show the implementation process of the method, and introduces the context priority review mode and the label priority review mode.

[0054] In the training field of transport aviation pilots, in the evaluation system of Professionalism Lifecycle Management System (PLM), nine core competencies are used as candidate labels of keywords. The nine core competencies include: application of knowledge, procedure execution and compliance with regulations, flight path management-automatic flight, flight path management-hand flight, communication, leadership and teamwork, situational awareness and information management, workload management, problem solving and decision making; the keyword labeling review process of this kind is particularly strict, and is related to aviation safety, so the method of the embodiment can obtain higher quality review effect.

[0055] In the embodiment, the corpus used by the natural language processing for keyword labeling review is the comments of teachers obtained by students during flight training in China Civil Aviation Flight College as the corpus for labeling keywords

[0056] The embodiment specifically shows the review process performed by the selected context priority review mode.

[0057] Step (1), the data generation module obtains basic labeling data: from the labeling result of the corpus , the N keywords that have been labeled and the selected label of each keyword are obtained;

[0058] Step (2), the data generation module obtains additional labeling data: the context of each keyword is obtained, specifically: from the corpus In the embodiment, a text containing the keyword is intercepted as the context of the keyword; the candidate label of the keyword is obtained, specifically, the nine core competencies that are not the selected label of the keyword are taken as the candidate label of the keyword; the labeling data of the keyword can refer to Table 1.

[0059] In step (3), the data generation module outputs the keyword: from the N keywords, K keywords are randomly selected to form a sequence, which is output to the keyword interaction module; part of the keywords in the keyword sequence are shown in Table 1, including the keywords of posture and throttle collection;

[0060] Table 1: Labeling data of the keyword

[0061]

[0062] In step (4), the keyword interaction module sets the focus keyword: from the keyword sequence displayed by the keyword interaction module, the "throttle collection" is selected as the focus keyword;

[0063] In step (5), the data generation module outputs other data: the data generation module obtains the focus keyword "throttle collection" determined by the keyword interaction module; the context "cyclic check and put fast, speed, quickly collect throttle, timely supplement after speed adjustment" of the focus keyword "throttle collection" is output to the context interaction module; the selected label "flight path management-manual flight" of the focus keyword "throttle collection" is output to the label interaction module; the candidate labels "application of knowledge", "program execution and compliance with regulations", "flight path management-automatic flight", "communication", "leadership and team cooperation", "situation awareness and information management", "workload management" and "problem solving and decision making" of the focus keyword "throttle collection" are output to the label interaction module;

[0064] In step (6), the keyword interaction module audits the keyword: if the focus keyword is not a keyword after auditing, the keyword interaction module deletes the focus keyword; if the focus keyword "throttle collection" is a keyword after auditing, the context priority audit is continued;

[0065] In step (7), the context priority audit (audited by the context interaction module): according to the focus keyword "throttle collection" and its selected label "flight path management-manual flight", the context is audited; after auditing, the context "throttle collection speed adjustment is insufficient in speed control" of the focus keyword "throttle collection" does not conform to the selected label "flight path management-manual flight" in this place, and the context is deleted;

[0066] In step (8), after the context priority audit of the keyword "throttle collection" is implemented, the data after auditing is shown in Table 2.

[0067] Table 2 Contextual Prior Review Result Example

[0068]

[0069] Example 3

[0070] Based on Example 2, this example specifically demonstrates the review process conducted by the selected label prior review mode.

[0071] In this example, the content of steps (1) to (3) is the same as in Example 2.

[0072] Step (4), the keyword interaction module sets the focus keyword: from the keyword sequence displayed by the keyword interaction module, "pose" is selected as the focus keyword;

[0073] Step (5), the data generation module outputs other data: the data generation module obtains the focus keyword "pose" determined by the keyword interaction module; the context of the focus keyword "pose" "must retract the plane, cannot continue to go down, pose action must be fast when sinking fast, cannot wait" is output to the context interaction module; the selected label of the focus keyword "pose" "flight path management-automatic flight" is output to the label interaction module; the candidate labels of the focus keyword "pose" "application of knowledge", "program execution and compliance with regulations", "flight path management-hand flight", "communication", "leadership and teamwork", "situational awareness and information management", "workload management", and "problem solving and decision making" are output to the label interaction module;

[0074] Step (6), the keyword interaction module reviews the keyword: if the focus keyword is not a keyword after review, the keyword interaction module deletes the focus keyword; if the focus keyword "pose" is a required keyword after review, the label prior review continues;

[0075] Step (7), label prior review (reviewed by the label interaction module): according to the focus keyword "pose" and its context "must retract the plane, cannot continue to go down, pose action must be fast when sinking fast, cannot wait", the label is reviewed; after review, the label of the focus keyword "pose" "flight path management-automatic flight" is inconsistent with the context "must retract the plane, cannot continue to go down, pose action must be fast when sinking fast, cannot wait" at this point; combined with the context, the label of the keyword "pose" at this point should be "situational awareness and information management"; the label of the focus keyword "pose" "flight path management-automatic flight" is set as a candidate label, and the candidate label of the focus keyword "pose" "situational awareness and information management" is set as a selected label;

[0076] Step (8), the keyword "posture" is labeled to implement label priority review, and the reviewed data is shown in Table 3.

[0077] Table 3 Label priority review result example

[0078]

[0079] In summary of the above embodiments, the keyword labeling review system and the review method disclosed by the application are easy to use in the technical field of natural language processing. For the work of large amount of language text and strict review, the review efficiency and the review quality of the application meet the high standard requirements in the industry, reshape the business process of keyword review, and improve the review efficiency.

Claims

1. A keyword annotation and review system, characterized in that: The system comprises a data generation module and an audit interaction module; the data generation module is configured to obtain labeled data of keywords and output the labeled data of the keywords to the audit interaction module; and the audit interaction module is configured to receive and display the labeled data of the keywords and provide interactive operations for the labeled audit of the keywords. The audit interaction module comprises a keyword interaction module, a context interaction module and a label interaction module; the data generation module is configured to output a sequence of keywords from the labeled data of the keywords to the keyword interaction module, and output context information of a focus keyword to the context interaction module and output selected labels and candidate labels of the focus keyword to the label interaction module according to the focus keyword set by the keyword interaction module. The keyword interaction module is configured to display the sequence of keywords and provide keyword interaction operations. The keyword interaction operations comprise deleting a keyword from the sequence of keywords and setting a keyword in the sequence of keywords as the focus keyword. The context interaction module is configured to display the context of the focus keyword and provide context interaction operations; and the context interaction operations comprise deleting the context of the focus keyword. The label interaction module is configured to display the selected labels and the candidate labels of the focus keyword and provide label interaction operations; and the label interaction operations comprise changing the selected labels of the focus keyword to the candidate labels and changing the candidate labels of the focus keyword to the selected labels. 2.The keyword tagging review system of claim 1, wherein, The labeled data of the keywords obtained by the data generation module comprises the keywords, the context of the keywords, the selected labels of the keywords and the candidate labels of the keywords.

3. A keyword tagging review method of the keyword tagging review system according to claim 1, characterized by, The system comprises: The keyword labeling and auditing system is configured to obtain and display the labeled data of the keywords and select an auditing mode. The auditing mode comprises a context-first auditing mode and a label-first auditing mode. The keyword interaction module is configured to set a focus keyword in the sequence of keywords. The context interaction module is configured to display the context of the focus keyword. The label interaction module is configured to display the selected labels and the candidate labels of the focus keyword. The focus keyword is audited to determine whether it is a keyword meeting the requirements; if yes, the context of the focus keyword and / or the selected labels and the candidate labels of the focus keyword are audited according to the selected auditing mode. After the auditing of one focus keyword is completed according to the selected auditing mode, another keyword in the sequence of keywords is set as the focus keyword, and the auditing process is repeated until the auditing of all the keywords in the sequence of keywords is completed. 4.The keyword tagging review method of claim 3, wherein: The focus keyword is audited to determine whether it is a keyword meeting the requirements; if no, the focus keyword is deleted from the sequence of keywords, and another keyword is set as the focus keyword. 5.The keyword tagging review method of claim 3, wherein, The context-first auditing mode is specifically configured to audit the context of the focus keyword according to the focus keyword and the selected labels of the focus keyword to determine whether the context is correct; if yes, no interaction operation is performed, and the labeling result is retained; if no, the context irrelevant to the focus keyword and the selected labels of the focus keyword is deleted. 6.The keyword tagging review method of claim 3, wherein, The label preferential review method specifically comprises: according to the focus keyword and the context of the focus keyword, checking whether the selected label and the candidate label of the focus keyword are correct, if correct, no interactive operation is performed, and the labeling result is reserved; if incorrect, the selected label of the focus keyword is changed to the candidate label, and the correct label in the candidate label of the focus keyword is changed to the selected label.

Images