Method, system, device and medium for generating training data set based on labeled text

Abstract

This application provides a method, system, device and medium for generating a training data set based on labeled text. By obtaining multiple texts to be labeled, each original long text is disassembled into multiple short texts that are decomposed and deduplicated and cleaned. Processing; after processing, store it in the database to be assigned to a unique database id; use the forward maximum matching sentence algorithm to obtain the corresponding matching information in the database; perform entity / association annotation on the short text of the split sentence to generate a unique annotation id , and obtain the mapping relationship between the corresponding database id and the label id according to the short texts of the split sentences; according to the matching information and the mapping information, the short texts of the split sentences are spliced ​​into long texts containing entity / association annotations for training set data. This application can greatly reduce the cost of manual labeling for enterprises, ensure the consistency of repeated text labeling, and at the same time reduce the interference caused by inconsistent corpus during model algorithm training, and improve the accuracy of model learning.

Description

technical field [0001] This application relates to the technical field of training set data generation, in particular to a method, system, device and medium for generating training data sets based on labeled text. Background technique [0002] The lack of training data is an eternal problem in the field of natural language processing NLP. Lack of labeled data, a lot of noise in samples, and data deviation are all common phenomena. In the field of natural language processing, especially vertical fields (such as medicine), This phenomenon is more obvious. The current industry labeling mainly has the following problems: [0003] 1) For enterprises, the cost of large-scale manual data labeling is extremely high; 2) For labelers, medical data requires labelers to have professional medical knowledge and basic linguistic knowledge; 3) Manual labeling workload is heavy, boring, And there are a lot of repeated texts in the labeling process, and the labelers cannot remember the accur...

AI Technical Summary

Problems solved by technology

[0003] 1) For enterprises, the cost of large-scale manual data labeling is extremely high; 2) For labelers, medical data requires labelers to have professional medical knowledge and basic linguistic knowledge; 3) Manual labeling workload is heavy, boring, And there are a lot of repeated texts in the labeling process, and the labelers cannot remember the accurate labeling method of each repeated text, resulting in inconsistent labeling before and after; 4) In the process of large-scale multi-person collaborative labeling, different labelers have different understandings of the same sentence, which will lead to A large number of labeling deviations cannot guarantee the consistency of labeling, which will seriously affect the training effect of subsequent model algorithms

Images