Knowledge embedding method based on vector representation

Through the knowledge embedding method of vector representation, the problem of inconsistent knowledge representation in natural language processing is solved. By constructing Trie tree and AC tree algorithms to generate multi-dimensional background knowledge embedding vectors, the common sense errors and knowledge of neural network models during prediction are solved. Conflict issues improve the accuracy of entity identification.

CN119003786AInactive Publication Date: 2024-11-22CHINA SHIP DEV & DESIGN CENT
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411093243.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In existing natural language processing technologies, the lack of unified representation of the type, representation method and length of knowledge, resulting in common sense errors or knowledge conflicts in neural network models when predicting.

Method used

Using a knowledge embedding method based on vector representation, we use the Trie tree and AC tree algorithm to find background knowledge words, divide the dimensions according to the length of the word, generate multi-dimensional background knowledge embedding vectors, and splice the word embedding vectors and background knowledge embedding vectors, and enter entity recognition. The model is used for identification.

Benefits of technology

It realizes unified representation of knowledge of different types, representation methods and lengths, provides background knowledge for neural network models, avoids prediction errors and knowledge conflicts, and improves the accuracy of entity recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003786A_ABST
    Figure CN119003786A_ABST
Patent Text Reader

Abstract

The invention discloses a knowledge embedding method based on vector representation. The method comprises the following steps: 1) constructing a Trie tree according to words of all background knowledge bases V; 2) according to the Trie tree and a given input sentence, searching background knowledge words in the input sentence; 3) carrying out dimension division on the background knowledge words according to lengths, and then converting the background knowledge words into word embedding vectors; 4) introducing part-of-speech tags to the background knowledge words to represent the part-of-speech of the words, and generating multi-dimensional background knowledge embedded vectors; 5) splicing the character embedding vector and the multi-dimensional background knowledge embedding vector to obtain an input vector; and 6) inputting the input vector into an entity recognition model, and recognizing an entity existing in the natural text by using the entity recognition model. According to the method, knowledge of different types, representation modes and lengths is represented as unified vector embedding neural network input, background knowledge is provided for a neural network model, and the situation that neural network prediction has common sense errors or prediction and knowledge conflicts is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to natural language processing technology, and in particular to a knowledge embedding method based on vector representation. Background Art

[0002] At present, there is a lack of a unified representation for the type, representation, and length of knowledge in natural language. Neural networks learn knowledge from training sets, but the input of the neural network model lacks common sense knowledge, and the input of the model needs to determine the dimension and form. Therefore, it is urgent to design a new knowledge embedding method to represent knowledge of different types, representations, and lengths as a unified vector and embed it into the neural network input, provide background knowledge for the neural network model, avoid common sense errors in neural network predictions or conflicts between predictions and knowledge, and make up for the shortcomings of existing technologies. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a knowledge embedding method based on vector representation in view of the defects in the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is: a knowledge embedding method based on vector representation, comprising the following steps:

[0005] 1) Build a Trie tree based on all the words in the background knowledge base V;

[0006] 2) Based on the Trie tree and the given input sentence, use the AC tree algorithm to find the background knowledge words in the input sentence;

[0007] 3) Divide the background knowledge words in the input sentence into dimensions according to their length, and then convert the words of the same dimension into word embedding vectors;

[0008] 4) Introduce part-of-speech tags for background knowledge words in the input sentence to indicate the part of speech to which the words belong, and generate a multi-dimensional background knowledge embedding vector;

[0009] 5) Concatenate the word embedding vector with the multi-dimensional background knowledge embedding vector to obtain the input vector;

[0010] 6) Inputting the input vector into an entity recognition model, and using the entity recognition model to recognize entities existing in natural text.

[0011] According to the above scheme, in step 2), the AC tree algorithm is used to construct the Trie tree for the input sentence S and the background knowledge base V to obtain the set C of all background knowledge base words in the sentence S.

[0012] According to the above scheme, the dimension division according to length is divided according to the length of words. Words of different lengths correspond to embedded vector information of different dimensions, and the dimension of the multi-dimensional background knowledge embedding vector is 50 dimensions.

[0013] According to the above scheme, in step 4), the details are as follows:

[0014] 4.1) Divide the background knowledge words in the input sentence according to their length;

[0015] 4.2) Tagging the background knowledge words with part-of-speech tags in turn. The part-of-speech tags of the background knowledge base include:

[0016] Noun, B-ming; Verb, B-dong; Numeral, B-shu; Preposition, B-jie; Conjunction, B-lian; Tone, B-yu; Auxiliary, B-zhu; Idiom, B-cheng;

[0017] The part-of-speech tag only considers the first character tag B and irrelevant tag O, excluding the I tag;

[0018] 4.3) Based on the part-of-speech tags of the input sentences and background knowledge words, the background knowledge embedding vectors of each dimension will be obtained, and finally a multi-dimensional background knowledge embedding vector will be generated.

[0019] According to the above scheme, in step 5), the splicing is as follows:

[0020]

[0021] Among them, X input represents the input vector, X char represents the word embedding vector, X background-i represents the background knowledge vector of the i-th dimension, Represents a vector concatenation operation.

[0022] The beneficial effects produced by the present invention are:

[0023] The present invention represents knowledge of different types, representation methods and lengths as a unified vector and embeds it into the neural network input, providing background knowledge for the neural network model and avoiding common sense errors in neural network predictions or conflicts between predictions and knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0025] Figure 1 is a method flow chart of an embodiment of the present invention;

[0026] Figure 21 is a distribution diagram of entities of each category in the CoNLL2003 training set and test set of an embodiment of the present invention;

[0027] Figure 3 : is a distribution diagram of entities of each category in the MSRA training set and the test set of an embodiment of the present invention;

[0028] Figure 4 It is a distribution diagram of entities of various categories in the OntoNotes 4.0 training set and test set of an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0030] like Figure 1 As shown, a knowledge embedding method based on vector representation includes the following steps:

[0031] 1) Build a Trie tree based on all the words in the background knowledge base V;

[0032] 2) Based on the Trie tree and the given input sentence, use the AC tree algorithm to find the background knowledge words in the input sentence;

[0033] 3) Divide the background knowledge words in the input sentence into dimensions according to their length, and then convert the words of the same dimension into word embedding vectors of that dimension;

[0034] The word embedding vector uses a 300-dimensional pre-trained Word2Vec vector, corresponding to word-level information;

[0035] 4) Introduce part-of-speech tags for background knowledge words in the input sentence to indicate the part of speech to which the words belong, and generate a multi-dimensional background knowledge embedding vector;

[0036] The details are as follows:

[0037] 4.1) Divide the background knowledge words in the input sentence according to their length;

[0038] 4.2) The background knowledge words are labeled with part-of-speech tags in sequence. The part-of-speech tags of the background knowledge base are shown in Table 1, including:

[0039] Noun, B-ming; Verb, B-dong; Numeral, B-shu; Preposition, B-jie; Conjunction, B-lian; Tone, B-yu; Auxiliary, B-zhu; Idiom, B-cheng;

[0040] The part-of-speech tags only consider the first-character tags B and the irrelevant tag O, excluding the I tag;

[0041] Table 1 Part-of-speech tags of the background knowledge base

[0042] noun verb numeral preposition conjunction Tone particle idiom B-ming B-dong B-shu B-jie B-lian B-yu B-zhu B-cheng

[0043] 4.3) According to the input sentence and the part-of-speech tags of the background knowledge words, the background knowledge embedding vectors of each dimension will be obtained, and finally a multi-dimensional background knowledge embedding vector will be generated.

[0044] For example: For the sentence "Mitsubishi-built shipyard, Nagasaki Shipyard", the words with a length of 2 in the AC search results include: "Mitsubishi", "shipbuilding", "ship captain", "Nagasaki". Taking the character "三" as the first character, the background knowledge embedding vector with a dimension of 2 is "B-ming O B-dong B-ming B-ming O B-dong O"; the first "B-ming" tag means that the word "Mitsubishi" with a length of 2 starting with the character "三" is a word in the background knowledge base, and its part of speech is a noun. The generation of the remaining tags is类推以此类推.

[0045] 5) Concatenate the character embedding vector and the multi-dimensional background knowledge embedding vector to obtain the input layer vector;

[0046] When the input sentence in the present invention is "Mitsubishi-built shipyard, Nagasaki Shipyard", the corresponding input layer embedding vector.

[0047]

[0048] Among them, X input represents the input vector, X char represents the character vector, X background-i represents the i-th dimension background knowledge vector, represents the vector concatenation operation.

[0049] The embedding vector corresponding to each background knowledge word includes two parts, namely the character embedding vector and the 50-dimensional background knowledge embedding vector.

[0050] Among them, the multi-dimensional background knowledge embedding vector is composed of 50-dimensional knowledge encodings, which belong to additional dimensions. Among them, the i-th dimension corresponds to the entity name and its part-of-speech information with a length of i. Taking the character "长" as an example, the dimension of the character "长" is 1, and the label is "O", indicating that the character "长" is not a word in the background knowledge base. The label "O" will be mapped to 1 background knowledge embedding vector, with a default length of 250. Similarly, the label of the second dimension of the character "长" is "B-ming", indicating that the word "长崎" is a noun in the background knowledge base, and it will also be mapped to 1 background knowledge embedding vector. And so on, the labels of the 3rd to 50th dimensions of the character "长" will each be mapped to background knowledge embedding vectors with fixed dimensions. Finally, the 50-dimensional background knowledge embedding vectors are concatenated to form the background knowledge embedding vector of "长".

[0051] 6) Input the input vector into the entity recognition model, and use the entity recognition model to identify the entities existing in the natural text. The input vector includes the feature information of the input sentence and the word information of the background knowledge base.

[0052] The knowledge embedding method provided by the present invention can convert knowledge of different types, methods, and lengths into the same vector representation, and can be applied to the input of any neural network model for entity recognition.

[0053] Next, the positive effects of the present invention will be further described in combination with specific experimental data.

[0054] 1. The comparison table 2 of the method provided by the present invention with the prior art is as follows.

[0055] Table 2 Comparison results of the present method and the prior art

[0056] Model MSRA NER Resume NER BERT 94.95 95.53 BERT+Knowledge Embedding 95.03 96.78

[0057] 2. The effects of the existing literature and the method provided by the present invention are compared as follows:

[0058] CoNLL2003: This dataset is an English dataset containing 4 entity types, including LOC, ORG, PER, and MISC.

[0059] MSRA: This dataset is a Chinese dataset containing 3 entity types, including personal names, place names, and organization names.

[0060] OntoNotes 4.0: This dataset is a Chinese dataset from the news field, which annotates 18 named entity types. As shown in Table 3, the distribution of the training set and the test set.

[0061] Table 3 Distribution of the training set and the test set

[0062] Dataset Number of training sets Number of test sets zh_onto4 15537 4346 zh_msra 41726 4362 conll03 14029 3449

[0063] Figure 2 It is the distribution diagram of entities of each category in the CoNLL2003 training set and test set, where Figure 2 (a) is the distribution diagram of entities of each category in the CoNLL2003 training set. Figure 2 (b) is the distribution diagram of entities of each category in the CoNLL2003 test set.

[0064] Figure 3 It is the distribution diagram of entities of each category in MSRA training set and test set. Figure 3 (a) is the distribution diagram of entities of each category in the MSRA training set. Figure 3 (b) is the distribution diagram of entities of each category in the MSRA test set.

[0065] Figure 4 This is the distribution diagram of entities in each category of OntoNotes 4.0 training set and test set. Figure 4 (a) is the distribution diagram of entities of each category in the OntoNotes 4.0 training set. Figure 4 (b) is the distribution diagram of entities in each category of the OntoNotes 4.0 test set.

[0066] On the baseline model:

[0067] BERT-Tagger: Devlin et al. (Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805, 2018) proposed the BERT model, which regards named entity recognition as a tagging task.

[0068] BERT-MRC: Li et al. (Li X, Feng J, Meng Y, et al. A unified MRC framework for named entity recognition [J]. arXiv preprint arXiv:1910.11476, 2019) regarded the named entity recognition task as a reading comprehension task.

[0069] BERT-FLAT: Li et al. (Li X, Yan H, Qiu X, et al. FLAT: Chinese NER using flat-lattice transformer [J]. arXiv preprint arXiv: 2004.11795, 2020) designed the FLAT model, which converts the lattice structure into a set of fragments and introduces specific positional encodings.

[0070] LEBERT: Liu et al. (Liu W, Fu X, Zhang Y, et al. Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter[C] / / Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the11th International Joint Conference on Natural Language Processing(Volume 1:Long Papers).2021) proposed the LEBERT model, which directly injects dictionary information between the Transformer layers in BERT through a dictionary adapter.

[0071] The comparison and analysis of the experimental results are shown in Table 4.

[0072] Table 4 Comparison results of the present invention with other models

[0073]

[0074] By using the method of the present invention, the F1 scores of the three data sets were improved by 0.26, 0.15, and 0.95 respectively.

[0075] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A knowledge embedding method based on vector representation, characterized in that: The following steps are involved: 1) Build a Trie tree based on all the words in the background knowledge base V; 2) Based on the Trie tree and the given input sentence, use the AC tree algorithm to find the background knowledge words in the input sentence; 3) Divide the background knowledge words in the input sentence into dimensions according to their length, and then convert the words of the same dimension into word embedding vectors; 4) For the background knowledge words in the input sentence, part-of-speech tags are introduced to indicate the part of speech to which the words belong, and a multi-dimensional background knowledge embedding vector is generated; 5) concatenating the word embedding vector with the multi-dimensional background knowledge embedding vector to obtain the input vector; 6) Inputting the input vector into an entity recognition model, and using the entity recognition model to recognize entities existing in natural text.

2. The knowledge embedding method based on vector representation according to claim 1 is characterized in that: In the step 2), the Trie tree constructed for the input sentence S and the background knowledge base V is used to obtain the set C of all background knowledge base words in the sentence S using the AC tree algorithm.

3. The knowledge embedding method based on vector representation according to claim 1 is characterized in that: In the step 4), the dimension division according to length is divided according to the length of the words, and words of different lengths correspond to embedded vector information of different dimensions. The dimension of the multi-dimensional background knowledge embedding vector is 50 dimensions.

4. The knowledge embedding method based on vector representation according to claim 1 is characterized in that: In the step 4), the details are as follows: 4.1) Divide the background knowledge words in the input sentence according to their length; 4.2) Tagging the background knowledge words with part-of-speech tags in turn. The part-of-speech tags of the background knowledge base include: Noun, B-ming; Verb, B-dong; Numeral, B-shu; Preposition, B-jie; Conjunction, B-lian; Tone, B-yu; Auxiliary, B-zhu; Idiom, B-cheng; The part-of-speech tag only considers the first character tag B and irrelevant tag O, excluding the I tag; 4.3) Based on the part-of-speech tags of the input sentences and background knowledge words, the background knowledge embedding vectors of each dimension will be obtained, and finally a multi-dimensional background knowledge embedding vector will be generated.

5. The knowledge embedding method based on vector representation according to claim 1 is characterized in that: In the step 5), the splicing is as follows: Among them, X input represents the input vector, X char represents the word embedding vector, X background-i represents the background knowledge vector of the i-th dimension, Represents a vector concatenation operation.

6. An electronic device, characterized in that: include: one or more processors; as well as a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Knowledge embedding method and system, medium, equipment and data processing terminal

    CN114912611A

  • Named entity recognition preprocessing method based on knowledge embedding

    CN116681073A