An imbalanced risk event labeling method and system

Abstract

The application provides an unbalanced risk event labeling method and system, relates to the technical field of data labeling, and performs rough screening on original data through a rough screening layer of a prescreening model to obtain candidate risk event data; the candidate risk data are screened through a fine screening layer of the prescreening model, a risk event type identification result is used to determine that an event sequence is automatically labeled by a fine screening execution model, and prescreening data to be labeled is obtained; the prescreening data to be labeled is imported into a labeling platform, and data type labeling specification is determined according to the prescreening data to be labeled; the prescreening data to be labeled is labeled or corrected according to the labeling specification through a multi-person labeling channel with labeling consistency, the labeled data is audited, and the labeled data passing the audit is used as a final result, so that the technical problem that the number of sample data of unbalanced risk events is greatly different and the accuracy of small-class classification results is low is solved. The amount of each type of labeling data is balanced, and the accuracy of the labeling result is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data labeling, and in particular to a method and system for labeling imbalance risk events. Background Art

[0002] Research in AI-related fields, especially those utilizing machine learning and deep learning methods, largely relies on guided approaches, requiring a certain amount of high-quality, manually annotated training data. However, even with large amounts of manually annotated data, it is found that real-world data is often imbalanced, meaning the number of annotated samples for one or more classes is far smaller than the number of samples for other classes. Consequently, using this annotated data to train models results in lower accuracy for small class classifications. Currently, there is a lack of high-quality, manually annotated data, and even fewer that can be used for research. This scarcity of annotated data contrasts sharply with the vast amount of text data. Summary of the Invention

[0003] In order to solve the above problems, the present application provides a method and system for labeling unbalanced risk events, which solves the technical problem in the existing technology that there is a large difference in the number of sample data for unbalanced risk events, resulting in low accuracy of small category classification results when training models with labeled data.

[0004] In view of the above problems, the present application provides a method and system for labeling imbalance risk events.

[0005] In the first aspect, the present application provides a method for labeling unbalanced risk events, which includes: obtaining original data, performing a coarse screening of risk events on the original data according to the coarse screening layer of a pre-screening model to obtain candidate risk event data; identifying the risk event type of the candidate risk data through the fine stratification of the pre-screening model, and using the risk event type identification result to determine the fine stratification execution model to automatically label the event sequence to obtain pre-screened data to be labeled; importing the pre-screened data to be labeled into a labeling platform, and determining the data type labeling specification based on the pre-screened data to be labeled; labeling or correcting the pre-screened data to be labeled according to the labeling specification through a multi-person labeling channel with labeling consistency to obtain labeled data to be reviewed; reviewing the labeled data to be reviewed, and using the reviewed labeled data to be reviewed as the final labeled data, and exporting all the final labeled data from the labeling platform to construct a risk event corpus.

[0006] In the second aspect, the present application provides an unbalanced risk event labeling system, which includes: a coarse screening module for obtaining original data, and performing risk event coarse screening on the original data according to the coarse screening layer of the pre-screening model to obtain candidate risk event data; a fine classification module for identifying the risk event type of the candidate risk data through the fine stratification of the pre-screening model, and using the risk event type identification result to determine the fine stratification execution model to automatically label the event sequence to obtain pre-screened data to be labeled; a label determination module for importing the pre-screened data to be labeled into the labeling platform, and determining the data type labeling specification based on the pre-screened data to be labeled; a multi-person labeling module for labeling or correcting the pre-screened data to be labeled using the labeling specification label through the multi-person labeling channel to obtain the labeled data to be reviewed; a labeling review module for reviewing the labeled data to be reviewed, and the labeled data to be reviewed that passes the review is used as the final labeled data, and all the final labeled data are exported from the labeling platform to construct a risk event corpus.

[0007] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in any one of the first aspects is implemented.

[0009] The technical solution provided in this application has at least the following technical effects:

[0010] The present application provides a method and system for labeling unbalanced risk events. The method comprises obtaining raw data, performing a coarse risk event screening on the raw data according to a coarse screening layer of a pre-screening model, and filtering out non-risk events to obtain candidate risk event data. The method further comprises identifying the risk event type of the candidate risk data using a fine stratification layer of the pre-screening model. The fine stratification execution model is then used to automatically label event sequences using the risk event type identification results, thereby obtaining pre-screened data to be labeled. This ensures that data of each risk event type is effectively labeled, thereby ensuring a balance in the risk event type data. The pre-screened data to be labeled is imported into a labeling platform, and data type labeling specifications are determined based on the pre-screened data to be labeled. The pre-screened data to be labeled is labeled or corrected according to the labeling specifications using a multi-person labeling channel with consistent labeling to obtain labeled data to be reviewed. The data is then manually labeled using a customized labeling specification corresponding to the labeled data type. This ensures that the specifications are clear and easy to grasp during the labeling process, covering various linguistic phenomena, thereby improving consistency between different annotators and ensuring labeling quality. The labeled data to be reviewed are then reviewed, and the reviewed labeled data to be reviewed is used as the final labeled data to construct a risk event corpus. This approach achieves the goal of combining computer models with manual annotation, effectively improving the reliability and efficiency of annotation. By customizing annotation specifications that match the data type, the annotation process ensures clear and easy-to-understand specifications, covering a wide range of linguistic phenomena, thereby improving consistency between different annotators and ensuring annotation quality. This allows for the annotation of pre-defined risk events and corresponding event elements from raw financial news data from multiple data sources, thereby building a large-scale risk event corpus. This solves the technical problem in existing technologies of low precision in small-category classification results when training models using labeled data, caused by the large disparity in sample data quantity for unbalanced risk events. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flowchart of a method for marking an imbalance risk event according to an embodiment of the present application is shown;

[0012] Figure 2 This is a flow chart of obtaining pre-screened data to be labeled in a method for labeling unbalanced risk events in an embodiment of the present application;

[0013] Figure 3 A schematic diagram of a BERT model in an imbalance risk event labeling method according to an embodiment of the present application;

[0014] Figure 4 This is a structural diagram of an imbalance risk event labeling system according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] This application provides a method and system for labeling unbalanced risk events to solve the technical problem in the existing technology that there is a large difference in the number of sample data for unbalanced risk events, resulting in low accuracy of small category classification results when the labeled data is used for model training.

[0016] The solution of the present invention is described in detail below with reference to specific embodiments.

[0017] Example 1

[0018] like Figure 1 As shown, an embodiment of the present application provides a method for marking an imbalance risk event, the method comprising:

[0019] S10: obtaining original data, and performing a coarse screening of risk events on the original data according to the coarse screening layer of the pre-screening model to obtain candidate risk event data;

[0020] Furthermore, the raw data is subjected to a coarse screening of risk events according to the coarse screening layer of the pre-screening model, which includes: S101: acquiring a BERT framework to construct a pre-trained language model; S102: obtaining unannotated text corpus, and using the unannotated text corpus to train the pre-trained language model to obtain the general grammatical semantics of the text; S103: obtaining risk event annotated corpus, introducing the risk event annotated corpus to train the pre-trained language model for risk event recognition, and obtaining a coarse screening model, which is used as the coarse screening layer of the pre-screening model.

[0021] Specifically, the raw data consists of financial news texts and financial institution data from various data sources. A pre-screening model is first used to pre-screen the raw data, i.e., the data to be labeled. This pre-screening by the computer model reduces the workload of manual labeling and improves labeling accuracy. The initial screening model includes a coarse screening layer and a fine screening layer. The coarse screening layer is used to screen risk events from the raw data. To effectively capture a small number of risk events that occur in real-world scenarios and reduce labeling costs for labelers, non-risky events are filtered out, thereby identifying candidate risk events as the data that needs to be labeled, i.e., candidate risk event data, for labelers to label.

[0022] The coarse screening model of the coarse screening layer is a pre-trained language model (BERT) plus fine-tuning technology route. First, the model is trained from a massive amount of unlabeled text corpus to obtain the general grammatical and semantic knowledge of the text. On this basis, the manually labeled corpus of the risk event extraction task is introduced to continue training, and finally it significantly exceeds the acceptance indicators for various risk event types.

[0023] The coarse screening model extracts candidate risk events from a large amount of news text data. This problem can be regarded as a binary classification problem, that is, judging whether a piece of news describes a risk event. The classification result (i.e. 0 or 1) of the original data is obtained through the BERT model to indicate whether the news describes a risk event. Figure 3 As shown in Figure 2, the BERT model is used to pre-screen the original data.

[0024] S20: Identifying risk event types for the candidate risk data through the refined stratification of the pre-screening model, and automatically labeling event sequences using the risk event type identification results to determine the refined stratification execution model, thereby obtaining pre-screened data to be labeled;

[0025] Furthermore, the candidate risk data is subjected to event type identification and event sequence automatic labeling through the refined stratification of the pre-screening model, which includes: S201: constructing a refined model using the RoBerta+DNN framework; S202: constructing a preset risk event type list, and obtaining training data for each risk event type in the preset risk event type list; S203: training the refined model using the training data of each risk event type, obtaining a refined model for each risk event type, and adding the refined model for each risk event type as the refined stratification to the pre-screening model.

[0026] Furthermore, the risk event type of the candidate risk data is identified through the fine stratification of the pre-screening model, and the risk event type identification result is used to determine the fine stratification execution model to automatically label the event sequence to obtain pre-screened data to be labeled, such as Figure 2 As shown, S20 includes: S2001: identifying the risk event type of the candidate risk data to determine the risk event type; S2002: performing type matching based on the risk event type and the refined analysis model of each risk event type in the refined layer to obtain a matching type refined analysis model; S2003: using the matching type refined analysis model to extract the event occurrence time, event subject, and start and end positions of the event fragments from the candidate risk data, and automatically mark them.

[0027] Specifically, after the risk event data is obtained through rough screening, the risk event type is identified through the fine stratification of the pre-screening model. The corresponding fine stratification model of the risk event type is used to predict and extract the time of occurrence of the event, the event subject, and the start and end positions of the event fragments of the candidate risk data, and automatically label them.

[0028] The precision model, constructed using the RoBerta+DNN framework, performs the classic sequence labeling task of predicting the time of occurrence, event subject, and start and end positions of event segments. Due to the sporadic nature of risk events and the varying probabilities of different risk events, the actual labeled data exhibits significant imbalances in the distribution of the quantity of risk event types. See Table 1 below for details, which shows the statistical results of the labeled data for different risk event types.

[0029]

[0030] Table 1

[0031] If data from all event types is mixed for model training, the model will perform extremely poorly on a smaller number of types. To address this data characteristic, the present embodiment sets a target for various risk event types that occur in financial risk control, totaling 68 risk event types. See Table 2 below for details of the risk event types. Separate extraction models, namely precision models, are trained for each of the 68 risk event types, significantly improving the model's stability and extraction accuracy.

[0032]

[0033] Table 2

[0034] Based on the types of different risk events, this paper defines a total of 68 risk event types that are of primary concern to the business side of the risk domain. Examples of some of these types are shown in Table 2. Among these categories, common risk event types in news data include loss events and management events, which typically account for 70% of the total labeled event categories. By labeling data in a regular, sequential manner, a large amount of data can be obtained, which is beneficial for model training. However, events such as security incidents and product issues appear less frequently in the text, accounting for only about 10%, making it difficult for the model to correctly identify these types of events. Therefore, it is necessary to effectively label this less common type of data.

[0035] By training separate extraction models for different risk event types to automatically label risk events, we can define keywords, key phrases, and templates for a small number of risk events to extract events of that type, avoiding the problem of data imbalance and the problem of poor accuracy in labeling results for a small number of types of data.

[0036] S30: importing the pre-screened data to be annotated into the annotation platform, and determining the data type annotation specification based on the pre-screened data to be annotated;

[0037] Furthermore, the data type annotation specification is determined based on the pre-screened data to be annotated, S30 includes: S3001: obtaining a preset risk event type; S3002: obtaining a risk event annotation element, wherein the risk event annotation element includes a risk event type, a trigger word, a risk event subject, and a time; S3003: based on the risk event annotation element, respectively determining the annotation specification for each risk event type in the preset risk event type; S3004: identifying the data type based on the pre-screened data to be annotated, determining the risk event type for annotation specification matching.

[0038] Furthermore, based on the risk event annotation elements, the annotation specifications of each risk event type in the preset risk event types are determined separately, including: identifying the preset risk event types as verb events and noun events; when the preset risk event type is a verb event, determining the trigger word using the preset verb structure; when the preset risk event type is a noun event, performing a validity judgment using a noun phrase, and determining the trigger word according to the validity judgment result.

[0039] Specifically, annotation specifications and corresponding tags are predefined based on the needs and the type of data to be annotated. An event, as a form of information, is traditionally defined as a type of event that occurs at a specific time and place, involving several actors and exhibiting certain action characteristics. The elements that make up an event include trigger words, event type, event elements, and time. Based on this, annotation specifications are defined for each type and corresponding annotation tags are assigned. For example, the event "Company B experienced a net loss in the first quarter of 2019" is shown in Table 3 below.

[0040]

[0041] Table 3

[0042] It is necessary to mark the risk event type, trigger words, risk event subject, time, etc. according to the set marking specifications.

[0043] Event triggers are used to indicate the occurrence of an event, primarily including verbal events and noun-based events. The core word of a verb-based event is the verb, which includes the event's core action, agent, and patient. A verb-based event must include a verb expressing the action, while other components may be absent. Common verb-based event structures are shown in Table 4. Event triggers in subject-predicate structures are typically intransitive, while those in verb-object structures are mostly transitive.

[0044]

[0045]

[0046] Table 4

[0047] The core words of noun events are nouns that represent concepts, phenomena, etc., such as earthquake and traffic ticket. The judgment standard is to determine whether a noun phrase can "occur" or "establish". If it can, it can be considered a noun event. The structure of noun events is shown in Table 5 below:

[0048] structure Example trigger words Subject-verb-object structure Company A received a fine ticket Verb-object structure There is a record of punishment in the file Punishment

[0049] Table 5

[0050] Risk Event Subject Notation: Define the subject of the risk event as the company, including the full company name, abbreviation, and names of its subsidiaries. If a company name appears but is unrelated to the risk event, no company name is required.

[0051] Instead of only considering the attributes and relationships of event entities, we consider the subjects occurring in the event, the trigger words of the event type, etc.

[0052] Time stamp: The time stamp for risk events is defined as follows:

[0053] Limit the time to the month, that is, the time format is year-month. For example, December 2020 should be marked as 202012, which is a six-digit number. For time periods, use the end of the time period to represent the time when the event occurred, such as the end of 2020, which should be marked as 202012. For the current time, but not mentioned in the news article, mark it as the time the news was released, such as "recently" or "currently".

[0054] Risk event type labeling: Label the 68 risk event types shown in Table 2 above.

[0055] S40: Annotating or revising the pre-screened data to be annotated according to annotation specifications through a multi-person annotation channel with consistent annotation to obtain annotated data to be reviewed;

[0056] Furthermore, the pre-screened data to be annotated is annotated or corrected according to the annotation specifications through a multi-person annotation channel with annotation consistency, which includes: S401: randomly extracting k corpora from the text data, where k is a positive integer; S402: obtaining a text annotation task, where the text annotation task includes w labels, where w is a positive integer; S403: selecting m annotation channels from the multi-person annotation channel, where m is a positive integer not less than 2, and sending the text annotation task to the m annotation channels, and annotating the k corpora with risk event types and risk event elements according to the text annotation tasks; S404: performing consistency detection on the annotation results of the m annotation channels; S405: when the consistency detection requirements are met, opening the multi-person annotation channel.

[0057] Furthermore, the consistency check of the annotation results of the m annotation channels is performed, S404 includes: S40401: using the formula: Calculate the evaluation values ​​of m annotation results respectively, where P i +=∑ j P ij , P +i =∑ i P ij , P0 represents the consistency ratio of the annotation channels, P e represents the theoretical consistency ratio of m annotation channels, P ii Represents the value on the diagonal, n ij Represents the number of samples in the i-th and j-th columns, and n represents the total number of samples; S40402: When the evaluation value reaches the preset threshold, the consistency detection requirement is met; S40403: When the evaluation value does not reach the preset threshold, the annotation specification and / or text annotation task is adjusted, and the evaluation value calculation is repeated until the evaluation value meets the preset threshold.

[0058] Specifically, after pre-screening by the model, the data to be labeled, i.e. the pre-screened data to be labeled, is imported into the labeling platform and manually labeled through a multi-person labeling channel. The labeling platform has an optional design: the specific task information is on the left; the labeling mode switching tab, labeling management, clear labeling button, file name, page number switching area, labeling main interface, and document switching area are displayed in the middle; the document labeling result list is displayed on the right.

[0059] In text annotation mode, users can annotate information by sliding. The annotation method is to first select the text area and then add tags. Add tags such as risk event type, trigger words, and subject to a news article.

[0060] Optionally, the annotation platform has the following functions and features: a visual operation interface that supports the annotation of attributes such as risk events, risk event subjects, and occurrence time, with simple operation; progress control that automatically assigns tasks to indicate the progress of data annotation, which makes it convenient for annotation personnel to check the progress on the one hand; on the other hand, it ensures annotation efficiency; data import and export functions; quality inspection function: by randomly distributing some labeled data, the reliability of the annotation personnel is detected.

[0061] Before manually labeling multiple channels, a consistency check is required. Once the consistency check passes, labeling can begin. In the specific process, administrators first develop labeling specifications and annotation point requirements for risk event tasks based on labeling needs. Each labeling task is then assigned to multiple labelers. Secondly, after trial labeling, and before formal labeling, data labelers conduct a consistency check for labeling quality. After multiple rounds of iteration, once the labeling consistency of multiple labelers meets the requirements, large-scale formal labeling can begin. Finally, after labeling is complete, the labeled data can be handed over to model trainers to train the algorithms and models required for the task. Here, the quality of the labeled data is primarily verified by auditors.

[0062] The quality of the labeled data is measured based on its own characteristics and business needs. Since there is no publicly released test corpus for risk event detection at home and abroad, the embodiment of this application uses manually labeled corpus for evaluation.

[0063] Assume that there are w category labels for the text annotation task (taking risk event annotation as an example), which are C _1 ,C _2 ,…,C _w , k pieces of corpus are randomly extracted from the text data and handed over to at least two annotators, that is, at least two annotation channels for annotation. The embodiment of the present application takes two annotation channels as an example. The annotation process is as follows: two annotators independently annotate the text data, annotate event types, event elements, etc., as shown in Table 6 below, which is the data annotated by the two annotation channels; calculate the consistency of the annotation results of the two annotation channels, and calculate the Kappa value of the two groups of annotation results, that is, the evaluation value.

[0064]

[0065] Table 6

[0066] If the Kappa value for each task is above 0.7, the annotation quality is high. If the Kappa value is low, it is necessary to check whether the annotation task is too difficult, whether the annotation standards are feasible, and whether the annotators understand the annotation standards. After analyzing the reasons, adjustments can be made, or the annotation channels (i.e., the annotators) can be adjusted until the preset threshold is reached to meet the consistency test, and then formal annotation can be started.

[0067] According to the received tasks, manual labelers use the labeling platform and the labels set in the labeling specifications to correct and label the pre-screened data to be labeled that has been pre-labeled by the pre-screening model.

[0068] S50: reviewing the unreviewed annotated data, taking the unreviewed annotated data that passes the review as final annotated data, and exporting all the final annotated data from the annotation platform to construct a risk event corpus.

[0069] Specifically, the data awaiting review, manually annotated through the multi-person annotation channel, undergoes a quality check. This quality check includes three levels of inspection: first, self-inspection by the data annotators; second, 100% quality inspection by auditors of the completed data; and third, sampling and review by administrators of the completed data. The annotated data that passes quality inspection is stored as the final annotated data in the annotation platform's corresponding annotation database, thereby constructing an annotated corpus. This approach effectively improves the reliability and efficiency of annotation by combining computer models with manual annotation. By utilizing customized annotation specifications tailored to the data type, the annotation process maintains clear and easy-to-understand standards, covering a wide range of linguistic phenomena, thereby improving consistency across annotators and ensuring annotation quality. This approach achieves the goal of annotating predefined risk events and their corresponding event elements from raw financial news data from multiple data sources, constructing a large-scale risk event corpus and ensuring the balanced classification of various risk event types within the annotated data. This approach addresses the existing technical issue of large variations in the number of sample data for unbalanced risk events, which results in low accuracy for small-category classification when training models using annotated data.

[0070] Furthermore, the method also includes: S601: setting preset training data requirements; S602: when the amount of the final labeled data reaches the preset training data requirements, constructing a training corpus; S603: using the training corpus to retrain the pre-screening model, and updating the pre-screening model according to the training results.

[0071] Specifically, the preset training data requirement is the data volume of the set training corpus. For example, if the preset training data requirement is 2000, then after the final number of labeled data reaches 2000, all data will be exported, and a training corpus will be constructed to retrain the pre-screening model. By training the model with a labeled training corpus, the model can help labelers improve labeling efficiency. Through continuous optimization of the model, the workload of labelers will become less and less, and they only need to make corrections to the automatic labeling results of the machine, reducing the workload. The model is continuously optimized using labeled data. Whenever the preset number of training data requirements is reached, the labeled data will be exported to optimize the model and continue training, thereby improving the reliability of the model and constructing a high-precision training corpus to make up for the current lack of reliable and researchable training corpora.

[0072] In summary, the embodiments of the present application have the following beneficial effects:

[0073] 1. Customize corresponding annotation specifications based on risk control needs and the characteristics of risk events. This includes defining 68 types of events that risk control personnel are concerned about. Systematically define the subject (rather than entities such as company names or personal names), time, and type of each type of event. This allows for customized annotation specifications that match the data type. This ensures that the specifications are clear and easy to grasp during the annotation process, covers various language phenomena, and ensures a balance between all types of annotated data.

[0074] 2. Since risk control events account for a small proportion of all events (such as entertainment, sports, and other news), labeling all collected news would waste a lot of labelers' time. Therefore, a human-machine collaborative approach is proposed for pre-screening. After labeling every 2,000 data items, a coarse screening and fine classification model is trained simultaneously. Labelers can make corrections based on the machine's labeling results, improving labeling efficiency and reducing their workload.

[0075] 3. Due to the sporadic nature of risk events and the differences in the probability of occurrence between different events, the actual labeled data has serious quantitative imbalances between different risk event types. A risk type labeling system is proposed, including a set of pre-screening methods based on event type, subject, time, etc., and a precision model is trained independently for each risk event type to ensure the quantitative requirements of small-type event labeling data and avoid data imbalance among different event types.

[0076] 4. Ensure the reliability of annotations by performing consistency checks on manual annotations, use annotation review and quality inspection to refine industry annotation tasks and annotation quality assessments, and develop quality assessment methods based on differences in annotation consistency to ensure consistency requirements for annotation results and improve the reliability of annotation results.

[0077] Example 2

[0078] Based on the same inventive concept as the imbalance risk event labeling method in the aforementioned embodiment, the embodiment of the present application provides an imbalance risk event labeling system, such as Figure 4 As shown, the system includes:

[0079] A coarse screening module is used to obtain raw data, and perform risk event coarse screening on the raw data according to the coarse screening layer of the pre-screening model to obtain candidate risk event data;

[0080] A fine classification module is used to identify the risk event type of the candidate risk data through the fine stratification of the pre-screening model, and use the risk event type identification result to determine the fine stratification execution model to automatically label the event sequence to obtain pre-screened data to be labeled;

[0081] A label determination module is used to import the pre-screened data to be annotated into the annotation platform and determine the data type annotation specification based on the pre-screened data to be annotated;

[0082] A multi-person annotation module is used to annotate or modify the pre-screened data to be annotated using annotation specification tags through a multi-person annotation channel to obtain annotated data to be reviewed;

[0083] The annotation review module is used to review the annotation data to be reviewed, and the annotation data to be reviewed that passes the review is used as the final annotation data. The final annotation data is exported from the annotation platform to build a risk event corpus.

[0084] Furthermore, the system further comprises:

[0085] Preset training unit, used to set preset training data requirements;

[0086] A training corpus exporting unit, configured to construct a training corpus when the amount of the final labeled data reaches the preset training data requirement;

[0087] A model training unit is used to retrain the pre-screening model using the training corpus and update the pre-screening model according to the training results.

[0088] Furthermore, the system further comprises:

[0089] The coarse screening model building unit is used to collect BERT frameworks to build pre-trained language models;

[0090] A coarse screening model training unit is used to obtain unlabeled text corpus, and use the unlabeled text corpus to train the pre-trained language model to obtain the general grammatical semantics of the text;

[0091] The coarse screening model obtaining unit is used to obtain risk event annotated corpus, introduce the risk event annotated corpus to train the pre-trained language model for risk event recognition, obtain a coarse screening model, and use the coarse screening model as the coarse screening layer of the pre-screening model.

[0092] Furthermore, the system further comprises:

[0093] The precision model building unit is used to build a precision model using the RoBerta+DNN framework;

[0094] A risk type training data acquisition unit is used to construct a preset risk event type list and obtain training data for each risk event type in the preset risk event type list;

[0095] The precision model training unit is used to train the precision model using the training data of each risk event type, obtain the precision model of each risk event type, and add the precision model of each risk event type as the precision layer to the pre-screening model.

[0096] Furthermore, the system further comprises:

[0097] a risk type identification unit, configured to identify the risk event type of the candidate risk data and determine the risk event type;

[0098] a type model matching unit, configured to perform type matching based on the risk event type and the fine-division model of each risk event type in the fine-division layer to obtain a matching type fine-division model;

[0099] The automatic labeling unit is used to use the matching type precision model to extract the event occurrence time, event subject, and event segment start and end positions of the candidate risk data, and automatically label them.

[0100] Furthermore, the system further comprises:

[0101] Corpus extraction unit, used to randomly extract k pieces of corpus from text data, where k is a positive integer;

[0102] A labeling task obtaining unit, configured to obtain a text labeling task, wherein the text labeling task includes w labels, where w is a positive integer;

[0103] a labeling task sending unit, configured to select m labeling channels from the multi-person labeling channel, where m is a positive integer not less than 2, send the text labeling task to the m labeling channels, and label the k pieces of corpus with risk event types and risk event elements according to the text labeling tasks;

[0104] A consistency detection unit, used to perform consistency detection on the annotation results of the m annotation channels;

[0105] The multi-person annotation channel confirmation unit is used to open the multi-person annotation channel when the consistency detection requirement is met.

[0106] Furthermore, the consistency detection unit further includes:

[0107] Evaluation value calculation unit, used to calculate the value through the formula: Calculate the evaluation values ​​of m annotation results respectively, where P i+ =∑ j P ij , P +i =∑ i P ij , P0 represents the consistency ratio of the annotation channels, P e represents the theoretical consistency ratio of m annotation channels, P ii Represents the value on the diagonal, n ij represents the number of samples in the i-th and j-th columns, and n represents the total number of samples;

[0108] A consistency judgment unit, configured to satisfy the consistency detection requirement when the evaluation value reaches a preset threshold;

[0109] The task adjustment unit is used to adjust the annotation specification and / or the text annotation task when the evaluation value does not reach the preset threshold, and repeat the evaluation value calculation until the evaluation value meets the preset threshold.

[0110] An imbalance risk event labeling system provided in an embodiment of the present application can implement any process of an imbalance risk event labeling method in embodiment 1. Please refer to the details of embodiment 1 and will not be repeated here.

[0111] Example 3

[0112] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, any method step of the imbalance risk event labeling method described in the first embodiment is implemented.

[0113] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] This specification and drawings are merely illustrative of the present application and may be modified and combined in various ways without departing from the spirit and scope of the present application. To the extent such modifications and variations fall within the scope of the present claims and their equivalents, the present application is intended to include such modifications and variations.

Claims

1. A method for labeling imbalance risk events, characterized in that: The method comprises: Obtaining raw data, and performing a coarse screening of risk events on the raw data according to the coarse screening layer of the pre-screening model to obtain candidate risk data; Identify the risk event type of the candidate risk data through the refined stratification of the pre-screening model, and use the risk event type identification result to determine the refined stratification execution model to automatically label the event sequence to obtain pre-screened data to be labeled; Importing the pre-screened data to be annotated into the annotation platform, and determining the data type annotation specification based on the pre-screened data to be annotated; Annotate or revise the pre-screened data to be annotated according to the annotation specification through a multi-person annotation channel with consistent annotation to obtain the annotated data to be reviewed; Reviewing the annotated data to be reviewed, using the approved annotated data as final annotated data, and exporting all the final annotated data from the annotation platform to construct a risk event corpus; The coarse screening of risk events on the original data according to the coarse screening layer of the pre-screening model includes: Acquire the BERT framework to build a pre-trained language model; Obtaining unlabeled text corpus, and using the unlabeled text corpus to train the pre-trained language model to obtain universal grammatical semantics of the text; Obtain risk event annotated corpus, introduce the risk event annotated corpus to perform risk event recognition training on the pre-trained language model to obtain a coarse screening model, and use the coarse screening model as the coarse screening layer of the pre-screening model; The candidate risk data is identified for event type through the refined stratification of the pre-screening model, and event sequence is automatically labeled, which previously includes: Use the RoBerta+DNN framework to build a precision analysis model; Construct a preset risk event type list and obtain training data for each risk event type in the preset risk event type list; The precision model is trained using the training data of each risk event type to obtain a precision model for each risk event type, and the precision model for each risk event type is added to the pre-screening model as the precision layer.

2. The method according to claim 1, wherein The method further comprises: Set preset training data requirements; When the amount of the final labeled data reaches the preset training data requirement, constructing a training corpus; The pre-screening model is retrained using the training corpus, and the pre-screening model is updated according to the training results.

3. The method according to claim 1, wherein Identifying the risk event type of the candidate risk data through the refined stratification of the pre-screening model, and automatically labeling the event sequence using the risk event type identification result to determine the refined stratification execution model to obtain pre-screened data to be labeled, including: Identifying the risk event type of the candidate risk data to determine the risk event type; Perform type matching based on the risk event type and the precision classification model of each risk event type in the precision classification layer to obtain a matching type precision classification model; The matching type precision model is used to extract the event occurrence time, event subject, and event segment start and end positions of the candidate risk data, and automatically annotate them.

4. The method according to claim 1, wherein The pre-screened data to be annotated is annotated or corrected according to the annotation specification through a multi-person annotation channel with consistent annotation, which includes: Randomly extract k pieces of text from the text data, where k is a positive integer; Obtain a text annotation task, wherein the text annotation task includes w labels, where w is a positive integer; Select m annotation channels from the multi-person annotation channels, where m is a positive integer not less than 2, send the text annotation task to the m annotation channels, and annotate the k pieces of corpus with risk event types and risk event elements according to the text annotation tasks; Perform consistency check on the annotation results of m annotation channels; When the consistency detection requirements are met, the multi-person annotation channel is opened.

5. The method according to claim 4, wherein The consistency check of the labeling results of the m labeling channels includes: By formula: , calculate the evaluation values ​​of m annotation results respectively, where, , , , , , Indicates the consistency ratio of the annotation channels, Indicates the theoretical consistency ratio of m annotation channels, and indicates the value on the diagonal. Indicates the Hedi The number of samples in the column, represents the total number of samples; When the evaluation value reaches a preset threshold, the consistency detection requirement is met; When the evaluation value does not reach the preset threshold, the annotation specification and / or the text annotation task is adjusted, and the evaluation value calculation is repeated until the evaluation value meets the preset threshold.

6. The method according to claim 1, wherein Determining data type annotation specifications based on the pre-screened data to be annotated includes: Obtain the preset risk event type; Obtaining risk event annotation elements, wherein the risk event annotation elements include risk event type, trigger word, risk event subject, and time; Based on the risk event annotation elements, respectively determine the annotation specifications for each risk event type in the preset risk event types; The data type is identified based on the pre-screened data to be annotated, and the risk event type is determined to match the annotation specifications.

7. The method according to claim 6, wherein Based on the risk event annotation element, a annotation specification is determined for each risk event type in the preset risk event type, including: Identify verb events and noun events for the preset risk event types; When the preset risk event type is a verb event, the trigger word is determined using a preset verb structure; When the preset risk event type is a noun event, a noun phrase is used to perform a determination of the event's establishment, and the trigger word is determined based on the determination result.

8. An imbalance risk event labeling system, characterized by: The system comprises: A coarse screening module is used to obtain raw data, and perform coarse screening of risk events on the raw data according to the coarse screening layer of the pre-screening model to obtain candidate risk data; A fine classification module is used to identify the risk event type of the candidate risk data through the fine stratification of the pre-screening model, and use the risk event type identification result to determine the fine stratification execution model to automatically label the event sequence to obtain pre-screened data to be labeled; A label determination module is used to import the pre-screened data to be annotated into the annotation platform and determine the data type annotation specification based on the pre-screened data to be annotated; A multi-person annotation module is used to annotate or modify the pre-screened data to be annotated using annotation specification tags through a multi-person annotation channel to obtain annotated data to be reviewed; An annotation review module is used to review the annotation data to be reviewed, and the annotation data to be reviewed that passes the review is used as the final annotation data. The final annotation data is exported from the annotation platform to build a risk event corpus; The coarse screening model building unit is used to collect BERT frameworks to build pre-trained language models; A coarse screening model training unit is used to obtain unlabeled text corpus, and use the unlabeled text corpus to train the pre-trained language model to obtain the general grammatical semantics of the text; a coarse screening model obtaining unit, configured to obtain risk event annotated corpus, introduce the risk event annotated corpus to perform risk event recognition training on the pre-trained language model, obtain a coarse screening model, and use the coarse screening model as a coarse screening layer of the pre-screening model; The precision model building unit is used to build a precision model using the RoBerta+DNN framework; A risk type training data acquisition unit is used to construct a preset risk event type list and obtain training data for each risk event type in the preset risk event type list; The precision model training unit is used to train the precision model using the training data of each risk event type, obtain the precision model of each risk event type, and add the precision model of each risk event type as the precision layer to the pre-screening model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images