Vietnamese entity recognition method and device based on improved multi-head self-attention mechanism

Abstract

The present application relates to a Vietnamese entity recognition method and device based on an improved multi-head self-attention mechanism, and belongs to the technical field of natural language processing. In order to solve the problem that Vietnamese entity recognition does not consider dependency syntax knowledge, resulting in poor effect, and in view of the influence of dependency knowledge on the entity recognition task, the present application proposes a Vietnamese entity recognition method based on an improved multi-head self-attention mechanism, which includes four parts: Vietnamese entity data annotation and data preprocessing, Vietnamese text feature representation with integrated dependency syntax knowledge, Vietnamese entity recognition model construction based on an improved multi-head self-attention mechanism, and Vietnamese text entity recognition. According to the four functional modules, a Vietnamese entity recognition device based on an improved multi-head self-attention mechanism is made, which can perform entity recognition on the input sentence, and effectively improves the Vietnamese entity recognition effect.

Description

TECHNICAL FIELD

[0001] The application relates to a Vietnamese entity recognition method and device based on an improved multi-head self-attention mechanism and belongs to the technical field of natural language processing. BACKGROUND

[0002] Entity recognition aims to identify predefined entity types in a text. At present, research on Vietnamese entity recognition is mostly divided into three categories: 1. a method based on a Vietnamese pre-training language model, which aims to make the model learn part of the language knowledge in advance to provide entity recognition effect through a large amount of monolingual corpus; 2. a method based on joint training, which aims to enable the joint training tasks to learn useful knowledge from each other and reduce error transmission to improve the effect of the entity recognition task; and 3. a method of fusing multiple features, which aims to enrich semantic information by fusing multiple feature information to improve the entity recognition effect. Regardless of the above methods, although they have achieved good results, they do not consider the influence of dependency knowledge on the entity recognition task. Whether a word is an entity, which class of entity it belongs to, and its adjacent words and its dependent words and words dependent on it have a great influence. Integrating dependency syntax knowledge into the model enables the model to automatically learn this part of language knowledge and focus only on the information that needs to be focused on, thereby improving the entity recognition effect. SUMMARY

[0003] The application provides a Vietnamese entity recognition method and device based on an improved multi-head self-attention mechanism to improve Vietnamese entity recognition and enhance the representation of Vietnamese texts with integrated dependency syntax information.

[0004] The technical solution of the application is as follows: in a first aspect, a Vietnamese entity recognition method based on an improved multi-head self-attention mechanism, the specific steps of the Vietnamese entity recognition based on the improved multi-head self-attention mechanism are as follows:

[0005] Step 1, Vietnamese entity data labeling and data preprocessing: in view of the problem of lacking data sets, a crawler technology and manual combination are used to generate Han-Viet cross-language entity linking candidate entities and construct a Han-Viet cross-language entity disambiguation task data set, PhoNLP is used to automatically label data for an existing data set VLSP2016, and the data set is constructed by performing data cleaning and the like on the network crawled data to remove duplicates, filter special characters and the like.

[0006] Step 2, Vietnamese text feature representation with integrated dependency syntax knowledge: the Vietnamese words input into the model are vectorized, including the vector representation of each Vietnamese word obtained through the Vietnamese pre-training language model PhoBERT and the vector representation of the part of speech, chunk label and dependency relationship label of each word obtained through table lookup.

[0007] Step3, Vietnamese entity recognition model based on improved multi-head self-attention mechanism is used for recognition: the model effect is improved by improving the multi-head self-attention mechanism and integrating the dependency syntax knowledge, the vector obtained by encoding and integrating the dependency relationship information is subjected to feature extraction by the embedding layer output of BiLSTM, and then the adjacency matrix composed of the word itself, the adjacent words before and after the word, the dependent word and the dependent word is used, and the result obtained by the adjacency matrix through the mask self-attention mechanism is linked with the feature extraction result obtained by BiLSTM, and then sent to the decoding layer, the entities in the Vietnamese sentence are predicted by the decoder CRF, and the final Vietnamese entity recognition result is obtained.

[0008] As a preferred scheme of the present application, the specific steps of Step1 are:

[0009] Step1.1, the Vietnamese entity recognition dataset VLSP2016 is annotated in word granularity, including 5 columns of information, wherein the first column is the word itself, the second column is the part-of-speech tag, the third column is the chunk tag, the fourth column is the entity tag, and the fifth column is the nested entity tag. Four kinds of entity categories, Person, Location, Organization and Miscellaneous, are annotated, and the "BIO" form is used for annotation, including the labels "B-PER", "I-PER", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-MISC", "I-MISC" and "O". The label information and sentence length in the data level are statistically analyzed, and the dependency syntax label is added on the basis of the data set.

[0010] Step1.2, the latest Chinese article page file is obtained through the Wikipedia public mirror data download address, and the Chinese knowledge base is obtained by parsing through Python language. The obtained Chinese knowledge base contains Chinese entities, and the knowledge base is stored in a TXT file, each line contains a Chinese entity and the corresponding Wikipedia entity serial number. The acquisition of Vietnamese-Chinese aligned entities is realized through network crawler technology, and the crawler code is written by using Python language Requests library and the Vietnamese title in the HTML source code returned by Wikipedia is located through xpath to obtain Vietnamese-Chinese bilingual aligned entities.

[0011] Step1.3, the Vietnamese-Chinese cross-language entity linking dataset is constructed by removing the Vietnamese-Chinese data crawled from the network, filtering special characters and other processing, and the Vietnamese-Chinese cross-language text is represented based on character-level vectorization.

[0012] As a preferred scheme of the present application, the specific steps of Step1.3 are:

[0013] Step1.3.1, The data obtained by the network crawler contains some illegal characters, garbled characters and non-Chinese language data, so the regular matching technology and manual method are used to clean the data of the Chinese knowledge base, Chinese-Vietnamese aligned entities and Chinese entity alias set obtained by the crawler.

[0014] Step1.3.2, WikiANN is a data set for English Wikipedia knowledge base, and the annotation information is only English Wikipedia title, which is not consistent with the Chinese Wikipedia title required by the application. Therefore, the crawler technology is used to crawl the Vietnamese entities corresponding to the Chinese entities and the first sentence of the Chinese entity page describing the entity by using the multilingual links of Wikipedia. Since the WikiANN data set contains a large amount of repeated data, a large amount of data with less than 4 phonetic units and a large amount of garbled data, these data will have a negative impact on the model, so manual data cleaning is required to remove the garbled data and repeated data in the data set.

[0015] As a preferred scheme of the application, the specific steps of Step2 are:

[0016] Step2.1, word vector representation: given a sentence composed of n words w1;w2;...;w n as input, the embedding representation word of the i-th word w i in the Vietnamese input is obtained by PhoBERT, and the specific formula is as follows:

[0017] word i =PhoBERT base (w 1：n ,i)

[0018] Because PhoBERT uses BPE to sub-word segmentation, the word embedding representation is actually obtained by adding the representation of multiple sub-words divided by a word;

[0019] Step2.2, multi-feature vector representation: in order to obtain more rich semantic information from the embedding layer representation, the part-of-speech label embedding p i , chunk label embedding c i and dependency relation label embedding d i are spliced after the above word embedding, and the specific formula is as follows:

[0020] p i =Lookup pos (word i )

[0021] d i =Lookup dep (word i )

[0022] c i = Lookup chunk (word i )

[0023] Step2.3, Vietnamese word vector, part-of-speech label vector, chunk label and dependency relation label vector splicing: based on Step2.1 and Step2.2, Vietnamese word, part-of-speech, chunk and dependency relation vectors are obtained, in order to train the model, Vietnamese word, part-of-speech, chunk and dependency relation vectors are spliced to obtain the final embedding layer representation e i , the specific formula is as follows:

[0024]

[0025] Wherein Indicates the matrix splicing operation, the part-of-speech label embedding, the chunk label embedding and the dependency relation label embedding are generated by random initialization, and are trained and optimized together with the model training.

[0026] Step2.4, syntax graph construction: in order to make each word only pay attention to the information of the word that needs to be paid attention to, reduce the noise information caused by Self-Attention paying attention to the global. The adjacency matrix is constructed through the relationship between each word and the dependent word and the adjacent word, and the mask matrix D mask Used for Mask Self-Attention is obtained. The specific formula is as follows:

[0027]

[0028] As a preferred scheme of the present application, the specific steps of Step3 are:

[0029] Step3.1, the final embedding layer representation e 1：n Obtained in Step2.3 is sent into BiLSTM to obtain the feature representation h i After extracting feature information, the specific formula is as follows:

[0030]

[0031] Step3.2, after obtaining the feature representation h 1：n Using Mask Self-Attention to obtain hidden representation Wherein the mask matrix is D mask Above-mentioned. The specific formula is as follows:

[0032]

[0033] Step3.3, in order not to lose the feature information of the word itself, the feature information of the word itself and the hidden information obtained after MaskSelf-Attention are effectively fused, the model is fused by adding h i The vector obtained by adding h1 is normalized and then sent to FFNN to obtain the final representation r of the encoding layer i The specific formula is as follows:

[0034]

[0035] In a second aspect, the Vietnamese entity recognition device based on the improved multi-head self-attention mechanism comprises the following modules:

[0036] The Vietnamese entity data labeling and data preprocessing module is used to complete the following functions: in view of the problem of lacking data sets, the crawler technology and manual work are combined to generate Han-Viet cross-language entity linking candidate entities and construct Han-Viet cross-language entity disambiguation task data sets, the existing data set VLSP2016 is automatically labeled by using PhoNLP, and the data set is constructed by removing, filtering special characters and data cleaning and the like processing of the network crawled data.

[0037] The Vietnamese text feature representation module integrating dependency syntax knowledge is used to complete the following functions: the input model of each Vietnamese word is vectorized and represented, which comprises the vector representation of each Vietnamese word obtained by the Vietnamese pre-training language model PhoBERT and the vector representation of the part of speech, block label and dependency relationship label of each word obtained by table lookup.

[0038] The Vietnamese entity recognition model based on the improved multi-head self-attention mechanism and the Vietnamese text entity recognition module are used to complete the following functions: the improved multi-head self-attention mechanism is used to integrate the dependency syntax knowledge to improve the model effect, the vector obtained by encoding and integrating the dependency relationship information is used to extract features of the embedding layer output by BiLSTM, then the adjacency matrix composed of the word itself, the adjacent words before and after the word, the dependency word and the dependent word is used, the result obtained by the mask self-attention mechanism through the adjacency matrix is linked with the feature extraction result obtained by BiLSTM, and then the result is sent to the decoding layer, the entities in the input Vietnamese sentence are predicted by the decoder CRF, and the final Vietnamese entity recognition result is obtained.

[0039] The beneficial effects of the present application are:

[0040] 1、The Vietnamese entity recognition model is improved by using the dependency syntax information, and the model can achieve better effect on Vietnamese entity recognition.

[0041] 2、The Vietnamese entity recognition effect is effectively improved based on the improved multi-head self-attention mechanism, and the adjacency matrix composed of only the current word, the adjacent word of the current word and the word having the dependency relationship is used as the mask matrix of the mask self-attention mechanism as a constraint, so that the encoded feature vector not only has its own characteristic information but also has rich guiding information of the dependency word, and the noise problem caused by the original self-attention mechanism paying attention to all word information is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The Vietnamese dependency relationship example in the application is shown in the figure;

[0043] Figure 2 The entity label statistics chart in the VLSP2016 data set in the application is shown in the figure;

[0044] Figure 3 The Vietnamese entity recognition framework based on the improved multi-head self-attention mechanism in the application is shown in the figure;

[0045] Figure 4 The overall flowchart of the application is shown in the figure;

[0046] Figure 5 The heat map of different encoding layer experiments in the application is shown in the figure. DETAILED DESCRIPTION

[0047] Embodiment 1: As shown in the figure, in a first aspect, the Vietnamese entity recognition method based on the improved multi-head self-attention mechanism, the specific steps of the Vietnamese entity recognition method based on the improved multi-head self-attention mechanism are as follows: Figures 1-5

[0048] Step 1, Vietnamese entity data labeling and data preprocessing: in view of the problem of lacking data set, the crawler technology and manual combination are adopted to generate Han-Viet cross-language entity linking candidate entity and construct Han-Viet cross-language entity disambiguation task data set, and PhoNLP is used to automatically label the existing data set VLSP2016, and the data set is constructed by de-duplicating, filtering special characters and data cleaning and other treatments on the network crawled data.

[0049] ​Step 2: Vietnamese text feature representation incorporating dependency syntactic knowledge: Each Vietnamese word input into the model is vectorized, including the vector representation of each Vietnamese word obtained through the Vietnamese pre-trained language model PhoBERT, as well as the vector representation of each word's part of speech, chunk label, and dependency relationship label obtained through table lookup.

[0050] Step 3. Use the Vietnamese entity recognition model based on the improved multi-head self-attention mechanism for recognition: The model effect is improved by incorporating dependency syntactic knowledge through the improved multi-head self-attention mechanism. The vector obtained by encoding and incorporating dependency information is output through BiLSTM to perform feature extraction on the embedding layer. Then, the word itself, the word's preceding and following adjacent words, the dependent words, and the dependent words are used to form an adjacency matrix. The result obtained after the masked self-attention mechanism is residually linked with the feature extraction result obtained by BiLSTM and sent to the decoding layer. The decoder CRF is used to predict the entities in the input Vietnamese sentence to obtain the final Vietnamese entity recognition result.

[0051] As a preferred embodiment of the present invention, the specific steps of Step 1 are:

[0052] Step 1.1: The Vietnamese entity recognition dataset (VLSP2016) is annotated at the word level and contains five columns of information: the first column is the word itself, the second column is the part-of-speech tag, the third column is the chunk tag, the fourth column is the entity tag, and the fifth column is the nested entity tag. Four entity categories are annotated: Person, Location, Organization, and Miscellaneous. This annotation uses the "BIO" format, with the following labels: "B-PER," "I-PER," "B-LOC," "I-LOC," "B-ORG," "I-ORG," "B-MISC," "I-MISC," and "O." We statistically analyzed the label information and sentence length at this level, and added dependency syntactic tags to this dataset.

[0053] Step 1.2: Obtain the latest Chinese article page files from the Wikipedia public mirror data download link and parse them using Python to obtain a Chinese knowledge base. The resulting Chinese knowledge base contains Chinese entities and is stored as a TXT file, with each line containing a Chinese entity and its corresponding Wikipedia entity number. Acquiring the Chinese-Vietnamese aligned entities is accomplished through web crawling technology. The crawler code is written using the Python Requests library and uses XPath to locate Vietnamese titles in the HTML source code returned by Wikipedia to obtain the Chinese-Vietnamese bilingual aligned entities.

[0054] Step 1.3: The Chinese-Vietnamese data crawled from the web were deduplicated and special characters were filtered to construct a Chinese-Vietnamese cross-language entity link dataset, and the Chinese-Vietnamese cross-language texts were represented based on character-level vectorization.

[0055] As a preferred embodiment of the present invention, the specific steps of Step 1.3 are:

[0056] Step 1.3.1. The data obtained through web crawlers contain some illegal characters, garbled codes and non-Chinese language data. Therefore, the Chinese knowledge base, Chinese-Vietnamese aligned entities and Chinese entity alias sets obtained through the crawlers are cleaned using regular matching technology and manual methods.

[0057] Step 1.3.2: WikiANN is a dataset for the Wikipedia English knowledge base. The annotation information only includes English Wikipedia titles, which do not match the Chinese Wikipedia titles required by this method. Therefore, we used crawler technology to extract the Chinese entities corresponding to Vietnamese entities and the first sentence describing the entity on the Chinese entity page using Wikipedia's multilingual links. Because the WikiANN dataset contains a large amount of duplicate data, data with fewer than four syllables, and a large amount of garbled data, which negatively impact the model, manual data cleaning is required to remove the garbled and duplicate data.

[0058] As a preferred embodiment of the present invention, the specific steps of Step 2 are:

[0059] Step 2.1, word vector representation: given n words w1; w2; ...; w n The sentence composed of is taken as input, and PhoBERT is used to obtain the i-th word w in the Vietnamese input i The embedding representation of word is as follows:

[0060] word i =PhoBERT base (w 1:n ,i)

[0061] Because PhoBERT uses BPE to segment sentences into subwords, the word embedding representation is actually obtained by adding together the representations of multiple subwords segmented from a word.

[0062] Step 2.2, multi-feature vector representation: In order to obtain richer semantic information in the embedding layer representation, the word part-of-speech tag embedding p is concatenated after obtaining the above word embedding. i , chunk tag embedding c i and dependency label embedding d i , the specific formula is as follows:

[0063] p i = Lookup pos (word i )

[0064] d i = Lookup dep (word i )

[0065] c i = Lookup chunk (word i )

[0066] Step2.3, Vietnamese word vector, part-of-speech label vector, chunk label and dependency relation label vector splicing: based on Step2.1 and Step2.2, Vietnamese word, part-of-speech, chunk and dependency relation vectors are obtained, in order to train the model, Vietnamese word, part-of-speech, chunk and dependency relation vectors are spliced to obtain the final embedding layer representation e i , the specific formula is as follows:

[0067]

[0068] Wherein represents the matrix splicing operation, the part-of-speech label embedding, the chunk label embedding and the dependency relation label embedding are generated by random initialization, and are trained and optimized together with the model training.

[0069] Step2.4, syntax graph construction: in order to make each word only pay attention to the information of the word that needs to be paid attention to, and reduce the noise information caused by Self-Attention paying attention to the global. The adjacency matrix is constructed through the relationship between each word and the dependent word and the adjacent word, and the mask matrix D mask used for Mask Self-Attention is obtained. The specific formula is as follows:

[0070]

[0071] As a preferred scheme of the present application, the specific steps of Step3 are:

[0072] Step3.1, after obtaining the final embedding layer representation e 1：n in Step2.3, it is sent into BiLSTM to obtain the feature representation h i after extracting feature information, the specific formula is as follows:

[0073]

[0074] Step3.2, after obtaining the feature representation h1：n Post-use Mask Self-Attention to obtain hidden representation Wherein the mask matrix is D mask The specific formula is as follows:

[0075]

[0076] Step 3.3, in order not to lose the feature information of the word itself, the feature information of the word itself and the hidden information obtained after Mask Self-Attention can be effectively fused, the model obtains the hidden information h i And the vector obtained by adding h1 is normalized and then sent to FFNN to obtain the final representation r of the encoding layer i The specific formula is as follows:

[0077]

[0078] In order to verify the effect of the Vietnamese entity recognition model based on the improved multi-head self-attention mechanism proposed in the application, comparative experiments and ablation experiments are designed.

[0079] In order to illustrate the effect of the application, the following comparative experiments are carried out, and the experimental data used are as follows. The Vietnamese entity data set used is VLSP2016, which contains 16,861 sentences in the training set and 2,831 sentences in the test set. And since this data set does not have a validation set, the application randomly selects 2,000 sentences from the training set as a validation set for parameter adjustment. The results of statistical analysis of the sentence length of the training set, test set and validation set after segmentation are shown in Table 1:

[0080] Table 1 Sentence length distribution statistics

[0081]

[0082] In order to illustrate the effect of the application, the following comparative experiments are carried out. In order to check the performance of the proposed model, the evaluation indexes are selected as precision (Precision), recall (Recall) and F1 value (F1-Measure), which are used to measure whether the model can correctly recognize Vietnamese entities.

[0083]

[0084]

[0085]

[0086] Wherein N pre represents the number of predicted entities, N trueN represents the number of entities predicted correctly real N represents the number of actual entities.

[0087] Experiment one: Comparison of different neural network models

[0088] Table 2 is the experimental results of the model of the present application and the best Vietnamese entity recognition model in recent years on the VLSP2016 data set. From the comparison of the experimental results of BiLSTM_CNN_CRF and BiLSTM in the following table, it can be seen that combining more feature information (part of speech, chunk and rule information) can effectively improve the Vietnamese entity recognition effect. The results of VELEC+BiLSTM_Attn and VNER show that the attention mechanism can improve the Vietnamese entity recognition effect to a certain extent. Through the experimental results of PhoBERT_base and VELEC+BiLSTM_Attn, it can be concluded that the pre-training language model plays a great role in Vietnamese entity recognition effect.

[0089] Table 2 Experimental results of different models

[0090]

[0091]

[0092] By comparing the experimental results of other Vietnamese entity recognition models, it can be seen that the model proposed in the present application has a great improvement in F1 value, and the joint model PhoNLP proposed by Nguyen in 2021 is improved by 2.85% in F1 value, reaching the SOAT of Vietnamese entity recognition task. This again proves that after the fusion of dependency knowledge, the model can pay more attention to the information it needs to pay attention to, thereby reducing the influence of useless information on the model.

[0093] Table 3 P, R, F1 and NUM values of each label

[0094]

[0095] Table 3 is the P, R and F1 value results of the model of the present application under each entity label in the test set. It can be seen that the F1 value of the model of the present application in the training set is more than 98% in the LOC and PER labels with more entity labels, but the F1 value in the MISC and ORG with less entity labels is only 87.23% and 90.6% respectively. From the experimental results of this table, it can be analyzed that the model can learn limited information in the label with less training label, thereby leading to poor effect of the label.

[0096] Experiment two: Ablation experiment

[0097] Table 4 is the result of ablation experiment, from the table, it can be seen that after eliminating the Deps mask self-attention proposed in the application, the experimental F1 value decreases by 0.23%, after eliminating the dependency label information and the part-of-speech label information, the experimental F1 value decreases by 0.38% and 0.74% respectively, which further proves the effectiveness of the experiment proposed in the application. But after eliminating the Chunk information and all feature information, the model effect decreases sharply, through analysis, it is found that the dependency analysis knowledge used in the application is obtained by automatic labeling, and the UAS and LAS of the automatic labeler are only “85.47%” and “79.11%” respectively, resulting in certain errors in the labeled dependency relationship and dependency label. The Chunk information can help the model digest these incorrect labels well, especially the dependency relationship errors occurring between the chunks. Therefore, under this condition, not using the Chunk information will make the method of the application lose effect.

[0098] Table 4 is the result of ablation experiment, from the table, it can be seen that after eliminating the Deps mask self-attention proposed in the application, the experimental F1 value decreases by 0.23%, after eliminating the dependency label information and the part-of-speech label information, the experimental F1 value decreases by 0.38% and 0.74% respectively, which further proves the effectiveness of the experiment proposed in the application. But after eliminating the Chunk information and all feature information, the model effect decreases sharply, through analysis, it is found that the dependency analysis knowledge used in the application is obtained by automatic labeling, and the UAS and LAS of the automatic labeler are only “85.47%” and “79.11%” respectively, resulting in certain errors in the labeled dependency relationship and dependency label. The Chunk information can help the model digest these incorrect labels well, especially the dependency relationship errors occurring between the chunks. Therefore, under this condition, not using the Chunk information will make the method of the application lose effect.

[0099]

[0100] Experiment three different encoding layer experiment:

[0101] Table 5 is the comparison of the experimental results of different encoders, it can be seen that when using the Transformer as the encoder, the F1 value of the model decreases by 3.42% compared with using the BiLSTM as the encoder, the reason is that the Transformer itself does not consider the position relationship between words, while the entity recognition itself is a task that requires position relationship very much.

[0102] Table 5 is the comparison of the experimental results of different encoders, it can be seen that when using the Transformer as the encoder, the F1 value of the model decreases by 3.42% compared with using the BiLSTM as the encoder, the reason is that the Transformer itself does not consider the position relationship between words, while the entity recognition itself is a task that requires position relationship very much.

[0103]

[0104] The model using BiLSTM and self-attention as the encoder reduces the F1 value by 0.09% compared with the model using only BiLSTM as the encoder, it can be known that the self-attention focusing on the global will bring noise information to the entity recognition task, resulting in a decrease in effect. The Transformer model using DEP mask self-attention improves the F1 value by 2.89% and 0.32% respectively compared with the ordinary Transformer model and the model proposed in the application compared with the model using BiLSTM and self-attention encoder, which further proves our guess that the traditional self-attention will obtain a lot of noise information, resulting in a decrease in model performance, and the mask matrix constructed by us can constrain the model to pay attention to useful information, effectively reducing the noise.

[0105] Figure 5 (a) "Jobs thành Apple Inc. Cupertino." This sentence is the heat map of attention after Mask Self-attention. Figure 5 (b) is the heat map of the sentence after normal self-attention. It can be seen in detail that Figure 5 In (b), each word pays different degrees of attention to many unnecessary words, causing it to lose a lot of important information to determine whether it is an entity. Figure 5 As can be seen in (a), each entity word focuses on words that can better determine whether it is an entity, but the entity words focus on themselves. The results of these two heat maps show why the method proposed in this paper can achieve better results.

[0106] Secondly, the Vietnamese entity recognition device based on the improved multi-head self-attention mechanism includes the following modules:

[0107] The Vietnamese entity data annotation and data preprocessing module is used to complete the following functions: to address the problem of lack of data sets, a combination of crawler technology and manual work is used to construct Chinese-Vietnamese cross-language entity link candidate entity generation and Chinese-Vietnamese cross-language entity disambiguation task datasets, and PhoNLP is used to automatically annotate the existing dataset VLSP2016. The crawled data is deduplicated, special characters are filtered, and data cleaning is performed on the data to construct the dataset.

[0108] The Vietnamese text feature representation module, which incorporates dependency syntactic knowledge, is used to perform the following functions: vectorize each Vietnamese word input into the model, including the vector representation of each Vietnamese word obtained through the Vietnamese pre-trained language model PhoBERT, and the vector representation of each word's part of speech, chunk label, and dependency relationship label obtained through table lookup.

[0109] A Vietnamese entity recognition model based on an improved multi-head self-attention mechanism and a Vietnamese text entity recognition module are constructed to complete the following functions: the model effect is improved by incorporating dependency syntactic knowledge through the improved multi-head self-attention mechanism, and the vector obtained by encoding and incorporating dependency information is used to extract features of the embedding layer output through BiLSTM. The adjacency matrix is ​​then constructed using the word itself, its preceding and following adjacent words, dependent words, and dependent words. The result obtained after the masked self-attention mechanism is applied to the adjacency matrix and the feature extraction result obtained by BiLSTM is residually linked and sent to the decoding layer. The entities in the input Vietnamese sentence are predicted through the decoder CRF to obtain the final Vietnamese entity recognition result.

[0110] The specific embodiments of the present application are described above in detail with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.

Claims

1. A Vietnamese entity recognition method based on an improved multi-head self-attention mechanism, characterized in that: The specific steps of the Vietnamese entity recognition method based on the improved multi-head self-attention mechanism are as follows: Step 1, Vietnamese entity data annotation and data preprocessing: adopt the combination of crawler technology and manual work to generate Han-Viet cross-language entity linking candidate entities and construct Han-Viet cross-language entity disambiguation task dataset, use PhoNLP to automatically annotate the existing dataset VLSP2016, and perform data cleaning processing such as de-duplication, filtering special characters, and data cleaning to construct the dataset; Step 2, Vietnamese text feature representation integrated with dependency syntax knowledge: vectorize each word in the input Vietnamese model, including the vector representation of each Vietnamese word obtained through the Vietnamese pre-trained language model PhoBERT and the vector representation of each word's part of speech, chunk label, and dependency relationship label obtained through table lookup; Step 3, use the Vietnamese entity recognition model based on the improved multi-head self-attention mechanism for recognition: improve the multi-head self-attention mechanism to integrate dependency syntax knowledge to improve the model's performance, extract features from the embedding layer output using BiLSTM, use the word itself, its adjacent words, dependent words, and dependent words to form an adjacency matrix, and then perform mask self-attention mechanism on the adjacency matrix to obtain the result, and then perform residual connection on the feature extraction result obtained by BiLSTM, and then input it into the decoding layer, use the decoder CRF to predict the entities in the input Vietnamese sentence, and obtain the final Vietnamese entity recognition result; The specific steps of Step 2 are as follows: Step2.1, word vector representation: Given a sentence consisting of n words w1;w2;:::;wn as input, the embedding representation of the i-th word wi in the Vietnamese input is obtained by PhoBERT, denoted as word, which is specified as follows: n Step2.1, word vector representation: Given a sentence consisting of n words w1;w2;:::;wn as input, the embedding representation of the i-th word wi in the Vietnamese input is obtained by PhoBERT, denoted as word, which is specified as follows: i Step2.1, word vector representation: Given a sentence consisting of n words w1;w2; ； Because PhoBERT uses BPE to split the sentence into subwords, the word embedding representation is actually obtained by adding the representations of multiple subwords split by a word; Step 2.2, Multi-feature vector representation: In order to embed the layer representation to obtain more rich semantic information, after obtaining the above word embedding, the part-of-speech label embedding p i , chunk label embedding c i and dependency relation label embedding d i are spliced, and the specific formula is as follows: ； Step 2.3, Vietnamese word vector, part-of-speech tag vector, chunk tag, and dependency relation tag vector splicing: based on Step 2.1 and Step 2.2, the Vietnamese word, part-of-speech, chunk, and dependency relation vectors are obtained. In order to train the model, the Vietnamese word, part-of-speech, chunk, and dependency relation vectors are spliced to obtain the final embedding layer representation e i The specific formula is as follows ； Where "∘" represents matrix concatenation, and the part-of-speech label embedding, chunk label embedding, and dependency relationship label embedding are generated by random initialization and are trained and optimized together with the model; Step 2.4, Syntax graph construction: In order to make each word only focus on the information of the word that needs to be focused on, and reduce the noise information brought by Self-Attention focusing on the global; through the relationship between each word and the dependent word and the adjacent words, an adjacency matrix is constructed, and a mask matrix D used for Mask Self-Attention is obtained mask ; The specific formula is as follows: ； The specific steps of Step 3 are as follows: Step3.1, obtain the final embedding layer representation e by Step2 1:n After that, it is sent into BiLSTM to obtain the feature representation h after extracting the feature information i The specific formula is as follows: ； Step 3.2, get the feature representation h 1:n using Mask Self-Attention to get the hidden representation where the mask matrix is D mask The specific formula is as follows: ； Step3.3、To not lose the feature information of the word itself, make the feature information of the word itself and the hidden information obtained after MaskSelf-Attention effectively fused, the model normalizes the vector after the addition of ℎ i and ℎ l and then sends it to FFNN to get the final representation r of the encoding layer i The specific formula is as follows: 。 2. The Vietnamese named entity recognition method based on the improved multi-head self-attention mechanism according to claim 1, characterized in that: The specific steps of Step 1 are as follows: Step 1.1, the Vietnamese entity recognition dataset VLSP2016 is annotated at the word level, containing 5 columns of information, the first column is the word itself, the second column is the part-of-speech label, the third column is the chunk label, the fourth column is the entity label, and the fifth column is the nested entity label; four entity categories, Person, Location, Organization, and Miscellaneous, are annotated using the "BIO" format, including "B-PER", "I-PER", "B-LOC", "I-LOC", "B-ORG", "I-ORG", "B-MISC", "I-MISC", and "O", and the label information and sentence length in the dataset are statistically analyzed, and dependency syntax labels are added based on the dataset; Step1.2, get the latest Chinese article page file through the Wikipedia public mirror data download address, and get the Chinese knowledge base through Python language parsing; the finally obtained Chinese knowledge base contains Chinese entities, and the knowledge base is stored in a TXT file, each line contains a Chinese entity and a corresponding Wikipedia entity serial number, and the Chinese-Vietnamese aligned entity is obtained through network crawler technology, a crawler code is written by using a Python language Requests library, and a Vietnamese title in HTML source code returned by Wikipedia is located through xpath to obtain the Chinese-Vietnamese bilingual aligned entity; Step1.3, the Chinese-Vietnamese cross-language entity link dataset is constructed by removing duplicate data and filtering special characters of the network crawled Chinese-Vietnamese data, and the Chinese-Vietnamese cross-language text is represented based on a character level vector.

3. The Vietnamese named entity recognition method based on the improved multi-head self-attention mechanism according to claim 2, characterized in that: The specific steps of Step1.3 are as follows: Step1.3.1, the data obtained through the network crawler contains some illegal characters, garbled characters and non-Chinese language data, therefore, the Chinese knowledge base, Chinese-Vietnamese aligned entity and Chinese entity alias set obtained through the crawler are cleaned by using a regular matching technology and a manual method; Step1.3.2, WikiANN is a data set for the English knowledge base of Wikipedia, wherein the annotated information is only the English Wikipedia title, which is not consistent with the Vietnamese Wikipedia title required, therefore, the Vietnamese entity corresponding to the Chinese entity and the first sentence of the Chinese entity page describing the entity are crawled by using the multilingual link of Wikipedia through the crawler technology; since the WikiANN data set contains a large amount of repeated data, a large amount of data with less than 4 phonetic nodes and a large amount of garbled data, these data will have a negative impact on the model, therefore, the data set needs to be cleaned by using a manual method to remove the garbled data and repeated data.

4. The Vietnamese named entity recognition apparatus based on the improved multi-head self-attention mechanism, characterized in that: A module for performing the method of any one of claims 1-3.

Images