Semantic and label difference fused named entity identification field self-adaption method

Abstract

The invention provides a method for selecting positive sample data in source domain data to extend training data of a target domain by fusing semantic difference and label difference of sentences in the source domain and the target domain, so as to achieve the purpose of enhancing named entity recognition performance of the target domain. Based on a conventional Bi-LSTM+CRF model, in order to fusesemantic differences and label differences of sentences in a source domain and a target domain, semantic difference and label difference are introduced through state representation and reward settingin reinforcement learning; therefore, the trained decision network can select sentences having positive influence on the named entity recognition performance of the target domain in the data of the source domain, expand the training data of the target domain, solve the problem of insufficient training data of the target domain, and improve the named entity recognition performance of the target domain at the same time.

Description

technical field [0001] The invention relates to the field of Internet technology, in particular to a fusion of semantic differences and label differences between domains, and domain migration on named entity recognition tasks. Background technique [0002] In recent years, deep learning and machine learning have made great progress in computer vision and natural language processing. In terms of computer vision, people use deep neural networks to classify images, such as using convolutional neural networks to recognize handwritten digits, and in this regard, the accuracy rate exceeds that of human self-recognition; in terms of natural language processing, deep learning is more It is applied in various life scenarios, such as using neural networks to analyze users' browsing records and consumption behaviors, recommending products that users may like, and using a large number of parallel corpora to train translation systems, so that machines can achieve high-level translation c...

AI Technical Summary

Problems solved by technology

[0007] In the field migration of Chinese named entity recognition, there are two problems: one is that there are great differences in the semantics of sentences in the corpus, and the other is that there are differences in the label sets of sentences in the corpus, which are caused by different labeling rules. of

[0016] 1. The traditional way of judging whether the current source domain sentence is conducive to the training of the target domain naming recognition model based on the semantic similarity between the source domain and the target domain does not take into account the impact of entity label differences

[0017] 2. When using the label transfer relationship between the source domain and the target domain for migration, the situation that the semantic vectors of the sentences in the source domain and the target domain are too different is not fully considered

Images