Cross-language named entity recognition model training method and device, and storage medium

Abstract

The application discloses a cross-language named entity recognition model training method and device and a storage medium. An original cross-language named entity recognition model is constructed, and the model comprises a source branch module, a target branch module and an optimization module; the source branch module is trained according to pre-acquired supervision data of a source language, and the target branch module is trained according to pre-acquired weak supervision data of a target language; based on a self-distillation mechanism, the source branch module and the target branch module are optimized according to unlabeled data of the target language, and a final cross-language named entity recognition model is obtained. The embodiment of the application can train the model by taking the supervision data of the source language, the weak supervision data of the target language and the unlabeled data of the target language as training samples, and is suitable for low-resource languages, does not need to establish explicit constraints and is suitable for named entity recognition of non-homologous languages.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a cross-language named entity recognition model training method and device and a storage medium. BACKGROUND

[0002] Named Entity Recognition (NER) is the basis of many Natural Language Processing (NLP) tasks, aiming to identify the boundaries of named entities and classify them into predefined categories, such as person names, place names or organization names. For example, in the sentence "Xiaoming comes from China", NER can identify that the first two characters "Xiaoming" refer to a person, while the last two characters "China" refer to a place name. As a basic research task of Natural Language Processing (NLP), NER has a large number of applications in various industrial products. For example, in commercial web search engines such as Microsoft Bing, NER is crucial for query understanding, web information extraction and intelligent question answering. For voice assistants such as Siri, Alexa and Cortana, NER is a key component of Spoken Language Understanding (SLU).

[0003] In the prior art, deep neural networks are widely used in NER and have achieved good performance. However, deep neural network models usually require a large amount of training data, so for languages with a large amount of high-quality annotated corpus, neural NER models are very successful. However, most languages, especially low-resource languages, do not have enough annotated data to train fully supervised models, and the problem of data scarcity limits the application of neural networks in low-resource fields with less annotated data. In the scenario of no target language artificial annotated corpus, cross-language NER technology can effectively improve the performance of target language NER, which aims to transfer knowledge from the source language (usually a high-resource language with rich annotated data) to the target language (usually a low-resource language with little or even no annotated data). Cross-language named entity recognition without target language annotated corpus has attracted great attention in the research community in recent years, and cross-language NER methods can be roughly divided into data transfer, model transfer and knowledge distillation (KD) based methods. The data transfer method is based on bilingual dictionary or alignment information in bilingual parallel corpus to construct a bilingual mapping space, and then directly uses the label information of the source language to perform named entity recognition of the target language. However, this method is not feasible in most low-resource languages, and the acquisition of high-quality bilingual dictionaries or parallel corpora even requires higher cost. The model transfer method is based on multi-task learning, modeling the named entity of each language as a task, and all tasks share the same encoding layer. These methods can transfer knowledge through a shared encoder without using bilingual resources. However, without explicit constraints, it cannot be guaranteed that the encoder extracts language-independent universal language features. In addition, when the size of the annotated resources is imbalanced, the encoder may be biased towards the resource-dominant language. The knowledge distillation method can alleviate the above problems. It uses a trained source language model to generate pseudo-labels for the target language corpus, which not only does not require any artificial features and bilingual dictionary information, but also makes full use of the unlabeled corpus of the target language. However, different languages usually have different basic sequence structures. This method works well when the target language is a homologous language of the source language, but it is not friendly to heterologous target languages. When the same target language is guided by different source languages, the effect of named entity recognition varies greatly. SUMMARY

[0004] The embodiment of the application aims to provide a cross-language named entity recognition model training method and device and a storage medium. By constructing a model including a source branch module, a target branch module and an optimization module and training each module to obtain a cross-language named entity recognition model, it is suitable for low-resource languages and does not require explicit constraints and is suitable for non-homologous language named entity recognition.

[0005] To achieve the above object, the embodiment of the present application provides a cross-language named entity recognition model training method, comprising:

[0006] constructing and training a cross-language named entity recognition model;

[0007] The cross-language named entity recognition model comprises a source branch module, a target branch module and an optimization module; the source branch module is used for training according to supervised data of a source language, the target branch module is used for training according to weakly supervised data of a target language, and the optimization module is used for optimizing the source branch module and the target branch module according to unannotated data of the target language based on a self-distillation mechanism.

[0008] As an improvement of the above scheme, the source branch module is specifically trained in the following manner:

[0009] In each round of training of the source branch module, the pre-acquired source language sequence is embedded into the mBERT src model, the label probability distribution of each word in the source language sequence is calculated by using a first MLP classifier and a softmax function, and the mBERT src model is adjusted in parameters to enter a new round of training until the model converges.

[0010] As an improvement of the above scheme, the weakly supervised data comprises first weakly supervised data and second weakly supervised data.

[0011] The target branch module is specifically trained in the following manner:

[0012] The pre-acquired first weakly supervised data of the target language is shielded from entities at a preset shielding probability, and a model is trained based on an MLM target.

[0013] Based on the model, the target language sequence hidden representation of the first weakly supervised data is sent into a second MLP classifier to predict whether each word in the target language sequence is a boundary of an entity, and a model is obtained.

[0014] A dictionary-based distant supervision method is used to generate second weakly supervised data from the first weakly supervised data, wherein the second weakly supervised data is weakly supervised data with a specific entity type.

[0015] Based on the model, in each round of training, the target language sequence of the second weakly supervised data is used as a training sample for model training, and a model is obtained. ​​The model; wherein, after each training iteration of the target branch module, the model obtained in the current round is used to predict the second weakly supervised data again When the probability of the predicted label distribution of an entity is greater than a preset probability threshold and the entity does not exist in the second weakly supervised data, the second weakly supervised data is updated for the next training iteration.

[0016] As an improvement of the above scheme, the optimization module specifically optimizes the source branch and the target branch in the following manner:

[0017] In each round of training of the optimization module, the pre-acquired unlabeled target language sequence is respectively sent into the source branch module and the target branch module to obtain the source hidden state, the source label probability distribution, the target hidden state and the target label probability distribution.

[0018] The source hidden state and the target hidden state are fused based on a preset weight coefficient, and a fused entity label probability distribution is calculated.

[0019] The mean square error of the entity label probability distribution and the source label probability distribution is calculated as a first mean square error.

[0020] The mean square error of the entity label probability distribution and the target label probability distribution is calculated as a second mean square error.

[0021] When the sum of the first mean square error and the second mean square error is greater than a preset error threshold, the source branch module and the target branch module are adjusted in parameters and then enter the next round of training of the optimization module.

[0022] When the sum of the first mean square error and the second mean square error is less than or equal to the error threshold, the training is ended.

[0023] As an improvement of the above scheme, the weakly supervised data of the target language includes first weakly supervised data and second weakly supervised data, and the weakly supervised data is acquired in the following manner:

[0024] Unlabeled data of the target language is extracted from Wikipedia, and each anchor point is taken as an entity to construct the first weakly supervised data.

[0025] The weakly supervised data of the target language is generated from the first weakly supervised data based on a dictionary-based distant supervision method.

[0026] To achieve the above object, the embodiment of the present application further provides a cross-language named entity recognition model training device, comprising:

[0027] A model construction module is configured to construct a cross-language named entity recognition model.

[0028] a model training module configured to train the cross-lingual named entity recognition model;

[0029] The cross-lingual named entity recognition model comprises a source branch module, a target branch module and an optimization module; the source branch module is configured to be trained according to supervised data in a source language, the target branch module is configured to be trained according to weakly supervised data in a target language, and the optimization module is configured to optimize the source branch module and the target branch module according to unannotated data in the target language based on a self-distillation mechanism.

[0030] As an improvement of the above scheme, the source branch module is specifically trained in the following manner:

[0031] In each round of training of the source branch module, the pre-acquired source language sequence is embedded into the mBERT src model, and the label probability distribution of each word in the source language sequence is calculated by using a first MLP classifier and a softmax function, and the mBERT src model is adjusted in parameters to enter a new round of training until the model converges.

[0032] As an improvement of the above scheme, the weakly supervised data comprises first weakly supervised data and second weakly supervised data.

[0033] The target branch module is specifically trained in the following manner:

[0034] The pre-acquired first weakly supervised data in the target language is shielded from entities at a preset shielding probability, and a model is trained based on an MLM target.

[0035] Based on the model, the hidden representation of the target language sequence of the first weakly supervised data is sent into a second MLP classifier to predict whether each word in the target language sequence is a boundary of an entity, and a model is obtained.

[0036] A dictionary-based distant supervision method is used to generate second weakly supervised data from the first weakly supervised data, wherein the second weakly supervised data is weakly supervised data with a specific entity type.

[0037] Based on the model, in each round of training, the target language sequence of the second weakly supervised data is used as a training sample for model training, and a model is obtained; wherein after each training iteration of the model, the model obtained in the current round is used.The model predicts the second weakly supervised data again, and when there is an entity whose predicted label distribution probability is greater than a preset probability threshold and does not exist in the second weakly supervised data, the second weakly supervised data is updated for the next training iteration.

[0038] As an improvement of the above scheme, the optimization module specifically optimizes the source branch and the target branch by the following manner:

[0039] In each round of training of the optimization module, the pre-acquired unlabeled target language sequence is respectively sent into the source branch module and the target branch module to obtain a source hidden state, a source label probability distribution, a target hidden state and a target label probability distribution;

[0040] The source hidden state and the target hidden state are fused based on a preset weight coefficient, and a fused entity label probability distribution is calculated;

[0041] The mean square error of the entity label probability distribution and the source label probability distribution is calculated as a first mean square error;

[0042] The mean square error of the entity label probability distribution and the target label probability distribution is calculated as a second mean square error;

[0043] When the sum of the first mean square error and the second mean square error is greater than a preset error threshold, the source branch module and the target branch module are adjusted in parameters and then enter the next round of training of the optimization module;

[0044] When the sum of the first mean square error and the second mean square error is less than or equal to the error threshold, the training is ended.

[0045] To achieve the above object, the embodiment of the present application further provides a computer readable storage medium, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, and the processor executes the computer program to implement the cross-language named entity recognition model training method according to any one of the above embodiments.

[0046] Compared with the prior art, the cross-language named entity recognition model training method, device and storage medium disclosed by the embodiment of the application first construct an original cross-language named entity recognition model, wherein the cross-language named entity recognition model comprises a source branch module, a target branch module and an optimization module, then train the source branch module according to the pre-acquired supervised data of the source language, train the target branch module according to the pre-acquired weakly supervised data of the target language, and finally optimize the source branch module and the target branch module based on a self-distillation mechanism according to the unlabeled data of the target language to obtain the trained cross-language named entity recognition model. The embodiment of the application can construct a model comprising a source branch module, a target branch module and an optimization module, and use the supervised data of the source language, the weakly supervised data of the target language and the unlabeled data of the target language as training samples to train the model, which is suitable for low-resource languages, does not need to establish explicit constraints and is suitable for named entity recognition of non-homologous languages. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of a cross-language named entity recognition model training method provided by an embodiment of the application;

[0048] Figure 2 is a schematic diagram of a pre-training process of a source branch provided by an embodiment of the application;

[0049] Figure 3 is a schematic diagram of a pre-training process of a target language branch provided by an embodiment of the application;

[0050] Figure 4 is a schematic diagram of a bilateral branch optimization provided by an embodiment of the application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the application.

[0052] Referring to Figure 1 is a flowchart of a cross-language named entity recognition model training method provided by an embodiment of the application, and the cross-language named entity recognition model training method comprises steps S1-S2:

[0053] S1, construct a cross-language named entity recognition model; wherein the cross-language named entity recognition model comprises a source branch module, a target branch module and an optimization module;

[0054] S2, training the cross-lingual named entity recognition model; wherein the source branch module is configured to be trained according to supervised data of a source language, the target branch module is configured to be trained according to weakly supervised data of a target language, and the optimization module is configured to optimize the source branch module and the target branch module according to unannotated data of the target language based on a self-distillation mechanism.

[0055] Specifically, the cross-lingual named entity recognition model provided by the embodiment of the present application is the architecture of SD-BBN, which is composed of three modules: (1) a source branch that learns source language features from supervised data of a source language; (2) a target branch that obtains specific language knowledge from weakly supervised data of a target language; and (3) an optimization module that optimizes the outputs of the two branches by using a self-distillation (SD) mechanism. Training the SD-BBN includes three steps: pre-training the source branch, pre-training the target branch, and optimizing the two branches using the optimization module. The base models of the two branches are multilingual BERT.

[0056] Compared with the prior art, the cross-lingual named entity recognition model training method, device and storage medium disclosed by the embodiment of the present application first construct an original cross-lingual named entity recognition model, wherein the cross-lingual named entity recognition model includes a source branch module, a target branch module and an optimization module, then train the source branch module according to the pre-acquired supervised data of the source language, train the target branch module according to the pre-acquired weakly supervised data of the target language, and finally optimize the source branch module and the target branch module according to unannotated data of the target language based on a self-distillation mechanism to obtain a trained cross-lingual named entity recognition model. The embodiment of the present application can construct a model including a source branch module, a target branch module and an optimization module, and use supervised data of a source language, weakly supervised data of a target language and unannotated data of the target language as training samples to train the model, which is suitable for low-resource languages, does not need to establish explicit constraints and is suitable for named entity recognition of non-homologous languages.

[0057] In an embodiment, the source branch module is specifically trained by the following method:

[0058] In each round of training of the source branch module, the pre-acquired source language sequence is embedded into the mBERT src model, the label probability distribution of each word in the source language sequence is calculated by using a first MLP classifier and a softmax function, and the mBERT src model is adjusted in parameters to enter a new round of training until the model converges.

[0059] Specifically, referring to Figure 2The schematic diagram of the pre-training process of the source branch is shown, and the target of the pre-training source branch of the SD-BBN is to learn the entity features of the source language to support cross-language NER. This step is a standard NER training step, and the mBERT model is fine-tuned using supervised data of the source language to perform the NER task.

[0060] For example, given a source language sequence with L words Using mBERT to embed it into a hidden vector Wherein Then, using a multi-layer perceptron (MLP) classifier plus a softmax function, the label probability distribution of is calculated:

[0061]

[0062] In each round of training of the source branch module, the loss function of the corresponding model needs to be calculated to determine whether the model converges by the loss function. When the model converges, the training is exited, and when the model does not converge, the model is adjusted and enters a new round of training.

[0063] In an embodiment, the weakly supervised data includes first weakly supervised data and second weakly supervised data.

[0064] The target branch module is specifically trained by the following method:

[0065] The first weakly supervised data of the target language is shielded with a preset shielding probability, and based on the MLM target, a model is trained.

[0066] Based on the model, the target language sequence hidden representation of the first weakly supervised data is sent to a second MLP classifier to predict whether each word of the target language sequence is the boundary of an entity, and a model is obtained.

[0067] The second weakly supervised data is generated according to the first weakly supervised data, wherein the second weakly supervised data is weakly supervised data with a specific entity type.

[0068] Based on the model, in each round of training, the target language sequence of the second weakly supervised data is used as a training sample for model training, and a model is obtained; wherein after each training iteration of the target branch module, the The model predicts the second weakly supervised data again. If the predicted label distribution probability of an entity is greater than a preset probability threshold and does not exist in the second weakly supervised data, the second weakly supervised data is updated for use in the next training iteration.

[0069] Specifically, see Figure 3 , Figure 3 This is a schematic diagram of the pre-training process of the target language branch provided in this embodiment of the invention. The purpose of pre-training the target branch of SD-BBN is to learn specific knowledge of the target language to support cross-language NER. First, a weakly supervised dataset is constructed for each target language. Based on this dataset, the target branch is pre-trained in three steps: (1) selectively masked language modeling; (2) entity boundary detection (EBD); and (3) weakly supervised NER.

[0070] (1) Selective masking language modeling

[0071] The first step in pre-training the target branch is to inject rich entity knowledge into the word representations. Therefore, the MLM objective is used on the first weakly supervised data D. g Pre-train the mBERT model. Randomly mask α% of entities; for example, α is set to 15. Then, a fine-tuned language model is obtained.

[0072] (2) Entity boundary detection

[0073] The second pre-training step is to enable the target branch to distinguish between entity words and non-entity words. This step is performed on the first weakly supervised data D. g right Perform a fine-tuning task for entity boundary detection. The goal of this task is to predict the start and end positions using two term-level classifiers. For example, given a first weakly supervised dataset D... g The target language sequence w = {w0, w1, ..., w1} has L words. L}, hidden representation The data is fed into two MLP classifiers to predict w. i Is it the first or last boundary of the entity? After this step, we will get a...

[0074] (4) Weakly supervised NER

[0075] To further learn entity knowledge, the pre-training step of the third objective branch targets NER. Fine-tuning is required. The goal of this task is to fine-tune the dictionary-labeled dataset D.tgt Train a NER model. This step is similar to the pre-training step of the source branch, except that the dataset used to train the target language NER model is a weakly supervised dataset.

[0076] Since the dataset D tgt is constructed based on a dictionary with limited entities, this can cause some problems such as entity missing. To solve this problem, after each training iteration, the current training model is used to predict D tgt again. Then, if the predicted probability of an entity (all the words in the entity) is greater than a predetermined threshold, but does not exist in the dictionary, then this entity is considered a new entity and is added to the dictionary. The threshold in this patent is only set to 0.5 according to experience. The dictionary will be updated to D tgt for the next training iteration. The above steps will be repeated until the model converges. Finally, we get a language model

[0077] In an implementation, the optimization module specifically optimizes the source branch and the target branch by the following way:

[0078] In each round of training of the optimization module, the pre-acquired unlabeled target language sequence is respectively sent into the source branch module and the target branch module to obtain source hidden states, source label probability distribution, target hidden states and target label probability distribution;

[0079] The source hidden states and the target hidden states are fused based on a preset weight coefficient, and a fused entity label probability distribution is calculated;

[0080] The mean square error of the entity label probability distribution and the source label probability distribution is calculated as a first mean square error;

[0081] The mean square error of the entity label probability distribution and the target label probability distribution is calculated as a second mean square error;

[0082] When the sum of the first mean square error and the second mean square error is greater than a preset error threshold, the source branch module and the target branch module are adjusted in parameters and then enter the next round of training of the optimization module;

[0083] When the sum of the first mean square error and the second mean square error is less than or equal to the error threshold, the training is ended.

[0084] Specifically, the above pre-training process makes the source language and target language branches contain entity knowledge of the source language and the target language respectively. Referring to Figure 4 , Figure 4is a schematic diagram of the bilateral branch optimization provided by the embodiment of the present application, in which the model is further optimized in the optimization module by using unlabelled target language data for self-distillation to optimize the source and target branches. The samples are first sent into the source language and target language branches, and then the optimization module fuses the outputs of the two branches to generate pseudo labels for the samples. The pseudo labels have information from the two branches and are regarded as real labels to update the two branches.

[0085] For example, given a target language sequence with L words It is sent into two branches to get its source and target hidden states. And Wherein, h src = mBERT src (w tgt ) and Then the outputs of the two branches are fused by a weight α:

[0086]

[0087] Wherein And α are randomly initialized learnable parameters. Then, an MLP classifier is used to calculate the entity label probability distribution of .

[0088]

[0089] Is used as a real label to update the parameters of the source and target branches. According to the standard self-distillation process, the model uses the pseudo soft label (entity label probability distribution) as supervision to train the two branches, and minimizes the mean square error between the pseudo label prediction and the branch prediction.

[0090]

[0091]

[0092]

[0093] Wherein, And Respectively represent the output probabilities of the two branches.

[0094] In an embodiment, the weakly supervised data of the target language includes first weakly supervised data and second weakly supervised data, and the weakly supervised data is obtained by the following method:

[0095] Extracting unlabelled data of the target language from Wikipedia and taking each anchor point as an entity to construct the first weakly supervised data;

[0096] The dictionary-based distant supervision method generates weakly supervised data of the target language according to the first weakly supervised data.

[0097] Specifically, the target language weakly supervised NER data is constructed as follows:

[0098] A weakly supervised data set is established for each target language. First, unlabeled data is extracted from a large-scale Wikipedia corpus. Then, each anchor in the data set is regarded as an entity. Each anchor in the data set is regarded as a general type of entity, and then a weakly supervised NER data set D g (first weakly supervised data) is constructed. In addition, in order to capture more fine-grained entity knowledge, a dictionary-based distant supervision method is used to automatically generate a weakly supervised NER data set D g (second weakly supervised data) with specific entity types derived from D tgt .

[0099] In constructing the specific entity dictionary, first, an entity type or "other" is assigned to each English Wikipedia entry. We leverage the AMR corpus, in which each entity name mention is manually labeled as one of 139 types and linked to Wikipedia where possible. A total of a batch of seed entity mentions, as well as their AMR entity types, Wikipedia titles, YAGO entity types and DBpedia properties are obtained in this way. For each pair of AMR entity type and YAGO entity type, we calculate the pointwise mutual information (PMI) between all mentions of AMR entity types and YAGO entity types in the AMR corpus. Therefore, each name is assigned a list of YAGO entity types, which is sorted according to their PMI scores with AMR types. In this way, three levels of entity type patterns with different granularities can be generated, and only the top-level entity type pattern (containing 3 entity types: person name, place name and organization name) is considered. Then, the entity properties in DBpedia are used as features for assigning types. For example, an entity with a birth date may be a person name, and an entity with a population attribute may be a place name. Using all the entity properties of DBpedia as features (60,231 in total), a maximum entropy model is trained to assign entity types, and entity type assignment is performed on all English Wikipedia pages. Next, the labels of each English Wikipedia page are propagated to all entity mentions in all languages of the entire Wikipedia through monolingual redirection links and cross-language links.

[0100] Compared with the prior art, the embodiment of the present application has the following advantages:

[0101] 1. A cross-language named entity recognition framework combining unsupervised and supervised

[0102] In the process of constructing the target language named entity recognition, the application utilizes the knowledge of the supervised source language cross-language named entity recognition model and the unsupervised target language model, maintains the language knowledge of the target language beneficial to the named entity recognition in the process of learning the cross-language named entity feature representation, combines the advantages of the source language and the target language, and can further improve the named entity recognition effect of the target language.

[0103] 2. The knowledge distillation technology is applied to cross-language research, and the current situation of language resource scarcity is alleviated

[0104] By utilizing the knowledge distillation technology, in the process of constructing the target language named entity recognition, the application does not need to use any labeled corpus of the target language, only needs the labeled corpus of the source language and the unsupervised corpus of the target language, which greatly alleviates the current situation of low-resource language resource scarcity, and makes it possible to realize a relatively high-performance cross-language named entity recognition model of the low-resource language.

[0105] 3. A large amount of unsupervised corpus of the target language is utilized to fully mine the language features of the target language

[0106] In the past research, the model is constructed in limited target language corpus (bilingual alignment resources and less labeled resources), and the knowledge distillation technology and the language model training technology used in the application have relatively low requirements for the target language corpus, and do not need any supervised knowledge. Such corpus is often relatively easy to obtain and contains rich language knowledge. The application grasps this feature and fully utilizes the unsupervised corpus of the target language to obtain language knowledge beneficial to the named entity recognition of the target language.

[0107] 4. The knowledge distillation is popularized to other cross-language tasks, and the development of cross-language research is promoted

[0108] The distillation technology is not only for the named entity recognition task, but can be widely applied to many cross-language researches. Therefore, the application aims to apply the knowledge distillation technology to the cross-language named entity recognition task, fills the blank of low-resource language resources in this respect, and also provides certain ideas and help for other cross-language research.

[0109] The embodiment of the application also provides a cross-language named entity recognition model training device, which comprises:

[0110] A model construction module is used to construct a cross-language named entity recognition model.

[0111] A model training module is used to train the cross-language named entity recognition model.

[0112] The cross-language named entity recognition model comprises a source branch module, a target branch module and an optimization module; the source branch module is used for training according to supervised data of a source language, the target branch module is used for training according to weakly supervised data of a target language, and the optimization module is used for optimizing the source branch module and the target branch module according to unannotated data of the target language based on a self-distillation mechanism.

[0113] In an implementation, the source branch module is specifically trained by the following manner:

[0114] In each round of training of the source branch module, a pre-acquired source language sequence is embedded into an mBERT src model, a first MLP classifier and a softmax function are used to calculate a label probability distribution of each word in the source language sequence, and the mBERT src model is adjusted in parameters to enter a new round of training until the model converges.

[0115] In an implementation, the weakly supervised data comprises first weakly supervised data and second weakly supervised data.

[0116] The target branch module is specifically trained by the following manner:

[0117] The first weakly supervised data of the target language is pre-acquired and shielded from entities at a preset shielding probability, and an MLM target is used to train a model;

[0118] Based on the model, a target language sequence hidden representation of the first weakly supervised data is sent into a second MLP classifier to predict whether each word in the target language sequence is a boundary of an entity, thereby obtaining a model;

[0119] A dictionary-based remote supervision method is used to generate second weakly supervised data from the first weakly supervised data, wherein the second weakly supervised data is weakly supervised data with a specific entity type.

[0120] Based on the model, in each round of training, a target language sequence of the second weakly supervised data is used as a training sample for model training, thereby obtaining a model; wherein after each training iteration of the model, the model obtained in the current round is used to predict the second weakly supervised data again, when a predicted label distribution probability of an entity is greater than a preset probability threshold and the entity does not exist in the second weakly supervised data, the second weakly supervised data is updated for the next training iteration.

[0121] In an embodiment, the optimization module optimizes the source branch and the target branch specifically by the following ways:

[0122] In each round of training of the optimization module, the pre-acquired unlabeled target language sequence is respectively sent into the source branch module and the target branch module to obtain the source hidden state, the source label probability distribution, the target hidden state and the target label probability distribution;

[0123] The source hidden state and the target hidden state are fused based on a preset weight coefficient, and the fused entity label probability distribution is calculated;

[0124] The mean square error of the entity label probability distribution and the source label probability distribution is calculated as a first mean square error;

[0125] The mean square error of the entity label probability distribution and the target label probability distribution is calculated as a second mean square error;

[0126] When the sum of the first mean square error and the second mean square error is greater than a preset error threshold, the source branch module and the target branch module are adjusted in parameters and then enter the next round of training of the optimization module;

[0127] When the sum of the first mean square error and the second mean square error is less than or equal to the error threshold, the training is ended.

[0128] It is worth noting that the working process of the cross-language named entity recognition model training device can refer to the working process of the cross-language named entity recognition model training method in the above embodiments, which will not be repeated here.

[0129] The cross-language named entity recognition model training device provided by the embodiment of the application can obtain a cross-language named entity recognition model by constructing a model including a source branch module, a target branch module and an optimization module and training each module, which is suitable for low-resource languages, does not need to establish explicit constraints and is suitable for named entity recognition of non-homologous languages.

[0130] The embodiment of the application further provides a cross-language named entity recognition model training device, which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the steps in the above cross-language named entity recognition model training method embodiments when executing the computer program, such as the steps S1-S2 in the above embodiment. Figure 1 Or, the processor implements the functions of each module in the above device embodiments when executing the computer program.

[0131] For example, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the cross-language named entity recognition model training device. For example, the computer program can be divided into a plurality of modules, and each module has the following specific functions:

[0132] a model construction module, configured to construct a cross-language named entity recognition model;

[0133] a model training module, configured to train the cross-language named entity recognition model;

[0134] The cross-language named entity recognition model comprises a source branch module, a target branch module and an optimization module; the source branch module is configured to be trained according to supervised data of a source language, the target branch module is configured to be trained according to weakly supervised data of a target language, and the optimization module is configured to optimize the source branch module and the target branch module according to unannotated data of the target language based on a self-distillation mechanism.

[0135] The specific working process of each module can refer to the working process of the cross-language named entity recognition model training apparatus described in the above embodiments, which will not be described here.

[0136] The cross-language named entity recognition model training device can be a desktop computer, a notebook computer, a palm computer and a cloud server, etc. The cross-language named entity recognition model training device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the schematic diagram is only an example of the cross-language named entity recognition model training device, and does not constitute a limitation on the cross-language named entity recognition model training device, and can include more or fewer components than the diagram, or combine certain components, or different components, for example, the cross-language named entity recognition model training device can also include an input / output device, a network access device, a bus, etc.

[0137] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is a control center of the cross-language named entity recognition model training device, and is connected with various parts of the cross-language named entity recognition model training device through various interfaces and lines.

[0138] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the cross-language named entity recognition model training device by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required by a function, and the like; and the data storage area can store data created according to the use of the mobile phone, and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory device.

[0139] If the cross-language named entity recognition model training device integrated module is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. When the processor executes the computer program, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0140] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A method for training a cross-lingual named entity recognition model, the method comprising: The application comprises the following steps: constructing and training a cross-language named entity recognition model; wherein the cross-language named entity recognition model comprises a source branch module, a target branch module and an optimization module; the source branch module is used for training according to supervised data of a source language, the target branch module is used for training according to weakly supervised data of a target language, and the optimization module is used for optimizing the source branch module and the target branch module according to unannotated data of the target language based on a self-distillation mechanism; the weakly supervised data comprises first weakly supervised data and second weakly supervised data; the target branch module is specifically trained in the following manner: The pre-acquired first weakly supervised data in the target language is masked to the entity at a preset masking probability, and mBERT1 is trained based on an MLM target tgt model; based on the mBERT1 tgt model, the target language sequence hidden representations of the first weakly supervised data are fed into a second MLP classifier to predict whether each token of the target language sequence is a boundary of an entity, resulting in an mBERT2 tgt model; a dictionary-based distant supervision method is used to generate second weakly supervised data from the first weakly supervised data, wherein the second weakly supervised data is weakly supervised data with a specific entity type; based on the mBERT2 tgt model, the target language sequence of the second weakly supervised data is used as a training sample for model training in each round of training to obtain an mBERT3 tgt model; wherein after each training iteration of the target branch module, the mBERT3 tgt model obtained in the current round is used to predict the second weakly supervised data again, and when the predicted label distribution probability of an entity is greater than a preset probability threshold and the entity does not exist in the second weakly supervised data, the second weakly supervised data is updated for the next training iteration. 2.The cross-lingual named entity recognition model training method of claim 1, wherein, the source branch module is specifically trained in the following manner: In each round of training of the source branch module, the pre-acquired source language sequence is embedded into the mBERT src hidden vector of the model, the label probability distribution of each word in the source language sequence is calculated by using the first MLP classifier and the softmax function, and the mBERT src model is adjusted in parameters to enter a new round of training until the model converges. 3.The method of claim 2, wherein, the optimization module specifically optimizes the source branch and the target branch in the following manner: in each round of training of the optimization module, the pre-acquired unannotated target language sequence is respectively input into the source branch module and the target branch module to obtain source hidden states, source label probability distribution, target hidden states and target label probability distribution; the source hidden states and the target hidden states are fused based on a preset weight coefficient, and the fused entity label probability distribution is calculated; the mean square error of the entity label probability distribution and the source label probability distribution is calculated as a first mean square error; the mean square error of the entity label probability distribution and the target label probability distribution is calculated as a second mean square error; when the sum of the first mean square error and the second mean square error is greater than a preset error threshold, the source branch module and the target branch module are adjusted in parameters and then enter the next round of training of the optimization module; when the sum of the first mean square error and the second mean square error is less than or equal to the error threshold, the training is ended. 4.The method of claim 1, wherein, The weakly supervised data of the target language comprises first weakly supervised data and second weakly supervised data, and the weakly supervised data is acquired in the following manner: unlabeled data of the target language is extracted from Wikipedia, and each anchor point is taken as an entity to construct the first weakly supervised data; the target language weakly supervised data is generated from the first weakly supervised data based on a dictionary-based distant supervision method.

5. A cross-language named entity recognition model training device, characterized in that, The application comprises the following steps: a model construction module is used for constructing a cross-language named entity recognition model; a model training module is used for training the cross-language named entity recognition model; wherein the cross-language named entity recognition model comprises a source branch module, a target branch module and an optimization module; the source branch module is used for training according to supervised data of a source language, the target branch module is used for training according to weakly supervised data of a target language, and the optimization module is used for optimizing the source branch module and the target branch module according to unannotated data of the target language based on a self-distillation mechanism; the weakly supervised data comprises first weakly supervised data and second weakly supervised data; the target branch module is specifically trained in the following manner: The pre-acquired first weakly supervised data in the target language is masked to the entity at a preset masking probability, and mBERT1 is trained based on an MLM target tgt model; based on the mBERT1 tgt model, the target language sequence hidden representations of the first weakly supervised data are fed into a second MLP classifier to predict whether each token of the target language sequence is a boundary of an entity, resulting in an mBERT2 tgt model; The dictionary-based distant supervision method generates second weakly supervised data according to the first weakly supervised data, wherein the second weakly supervised data is weakly supervised data with a specific entity type; based on the mBERT2 tgt model, the target language sequence of the second weakly supervised data is used as a training sample for model training in each round of training to obtain an mBERT3 tgt model;wherein, after each training iteration of the target branch module, the mBERT3 tgt model obtained in the current round is used to predict the second weakly supervised data again, and when the predicted label distribution probability of an entity is greater than a preset probability threshold and the entity does not exist in the second weakly supervised data, the second weakly supervised data is updated for the next training iteration. 6.The cross-lingual named entity recognition model training apparatus of claim 5, wherein, The source branch module is specifically trained in the following manner: In each round of training of the source branch module, the pre-acquired source language sequence is embedded into the mBERT src hidden vector of the model, the label probability distribution of each word in the source language sequence is calculated by using the first MLP classifier and the softmax function, and the mBERT src model is adjusted in parameters to enter a new round of training until the model converges. 7.The cross-lingual named entity recognition model training apparatus of claim 6, wherein, The optimization module optimizes the source branch and the target branch in the following manner: In each round of training of the optimization module, the pre-acquired unlabeled target language sequence is respectively fed into the source branch module and the target branch module to obtain source hidden states, source label probability distribution, target hidden states and target label probability distribution; The source hidden states and the target hidden states are fused based on a preset weight coefficient, and an entity label probability distribution after fusion is calculated; The mean square error of the entity label probability distribution and the source label probability distribution is calculated as a first mean square error; The mean square error of the entity label probability distribution and the target label probability distribution is calculated as a second mean square error; When the sum of the first mean square error and the second mean square error is greater than a preset error threshold, the source branch module and the target branch module are adjusted in parameters and then enter the next round of training of the optimization module; When the sum of the first mean square error and the second mean square error is less than or equal to the error threshold, the training is ended.

8. A computer-readable storage medium, characterized in that, The computer program is configured to be executed by the processor, and the processor implements the cross-language named entity recognition model training method according to any one of claims 1-4 when executing the computer program.

Images