A method, system, medium, device and data processing terminal for knowledge embedding

Abstract

The present invention belongs to the technical field of knowledge embedding in natural language and discloses a method, system, medium, device, and data processing terminal for knowledge embedding. A Trie tree is constructed based on the words in all background knowledge bases. Given an input sentence, the background knowledge words present in the input sentence are searched using the AC tree algorithm; words of different dimensions are converted into word embedding vectors; a multi-dimensional background knowledge embedding vector is generated; and the input vector obtained by concatenating the word embedding vector and the multi-dimensional background knowledge embedding vector is input into an entity recognition model. The knowledge embedding method provided by the present invention converts knowledge of different types, methods, and lengths into the same vector representation and can be applied to the input of any neural network model. The present invention achieved F1 values ​​of 95.03 and 96.78 on the open source named entity recognition datasets MSRA and Resume, respectively. The knowledge embedding improved the model by 1 to 2 points. The F1 value was also improved on the CoNLL2003, MSRA, and OntoNotes 4.0 datasets.

Description

Technical Field

[0001] The present invention belongs to the technical field of knowledge embedding in natural language, and in particular relates to a method, system, medium, device and data processing terminal for knowledge embedding. Background Art

[0002] Currently, there's no unified representation for the types, representations, and length of knowledge in natural language. Neural networks learn knowledge from training data, but due to a lack of common sense, they often make common sense errors. Consequently, neural network model inputs lack common sense knowledge, and model inputs require a specific dimension and format. Therefore, there's an urgent need to design a new knowledge embedding method and system to address the shortcomings of existing technologies.

[0003] Through the above analysis, the problems and defects of the existing technology are as follows:

[0004] (1) There is a lack of a unified representation for the type, representation, and length of knowledge in natural language.

[0005] (2) Neural networks learn knowledge from training sets, but often make common sense mistakes. Therefore, the input of the neural network model lacks common sense knowledge, and the input of the model needs to have a certain dimension and form.

[0006] The difficulty of solving the above problems and defects is:

[0007] 1) Naturally existing knowledge varies in type, representation, and length. 2) Neural network model inputs must be vectors of uniform fixed dimensions. 3) When embedding words, part-of-speech information must also be embedded, and knowledge may be nested.

[0008] The significance of solving the above problems and defects is:

[0009] Knowledge of different types, representation methods, and lengths is represented as a unified vector and embedded into the neural network input to provide background knowledge for the neural network model and avoid common sense errors in neural network predictions or conflicts between predictions and knowledge. Summary of the Invention

[0010] In response to the problems existing in the prior art, the present invention provides a knowledge embedding method, system, medium, device and data processing terminal.

[0011] The present invention is implemented as follows: a method for knowledge embedding includes: converting knowledge of different types, methods, and lengths in the natural world into the same vector representation, providing background knowledge for the entity recognition model, solving the problem of insufficient domain knowledge of the entity recognition model, and enabling the entity recognition model to more accurately identify entities existing in natural text.

[0012] Specifically, the knowledge embedding method includes the following steps: Step 1, construct a Trie tree based on all the words in the background knowledge base, given an input sentence, and then use the AC tree algorithm to find the background knowledge words in the input sentence;

[0013] Step 2: The background knowledge words in the input sentence are divided into different dimensions according to their length, and then the words of different dimensions are converted into word embedding vectors;

[0014] Step 3: Introduce part-of-speech tags to represent the part of speech of a word and generate a multi-dimensional background knowledge embedding vector;

[0015] Step 4: Input the input vector obtained by concatenating the word embedding vector and the multi-dimensional background knowledge embedding vector into the entity recognition model, and use the entity recognition model to identify entities existing in natural text. The input vector contains the feature information of the input sentence and the word information of the background knowledge base.

[0016] Furthermore, the step 1 uses the AC tree algorithm to search and obtain all the background knowledge bases including:

[0017] For the input sentence, the AC tree search algorithm is used to obtain all the words belonging to the background knowledge base; the words are divided according to their length, and words of different lengths correspond to embedding vector information of different dimensions; part-of-speech tags are introduced to generate a multi-dimensional background knowledge embedding vector; among them, the part-of-speech tags of the input layer only consider the first character label B and the irrelevant label O, and do not include the I label.

[0018] Furthermore, the part-of-speech tags of the background knowledge base include:

[0019] ①Noun, B-ming; ②Verb, B-dong; ③Number, B-shu; ④Preposition, B-jie; ⑤Conjunction, B-lian; ⑥Mood, B-yu; ⑦Auxiliary word, B-zhu; ⑧Idiom, B-cheng.

[0020] Furthermore, the AC tree algorithm is used to find all background knowledge base words in sentence S given a background knowledge base V and a sentence S.

[0021] Furthermore, the expression of the input layer embedding vector is:

[0022]

[0023] Among them, X input represents the input vector, X char represents the word vector, X background-i represents the background knowledge vector of the i-th dimension, Represents a concatenation operation.

[0024] Furthermore, the word embedding vector in step 2 uses a 300-dimensional pre-trained Word2Vec vector, corresponding to word-level information; the multi-dimensional background knowledge embedding vector is generated based on the AC tree search algorithm, corresponding to word-level information.

[0025] Another object of the present invention is to provide a knowledge embedding system comprising: a word acquisition module for constructing a Trie tree based on the words of all background knowledge bases, and given an input sentence, searching for background knowledge words present in the input sentence using an AC tree algorithm;

[0026] The word embedding vector generation module is used to distinguish the dimensions of the background knowledge words in the input sentence according to their length, and then convert the words of different dimensions into word embedding vectors;

[0027] A multi-dimensional background knowledge embedding vector generation module is used to introduce part-of-speech tags to represent the part of speech of a word and generate a multi-dimensional background knowledge embedding vector;

[0028] The input vector generation module is used to input the input vector obtained by splicing the word embedding vector and the multi-dimensional background knowledge embedding vector into the entity recognition model. The input vector contains the feature information of the input sentence and the word information of the background knowledge base.

[0029] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0030] Use the AC tree algorithm to search and obtain all words belonging to the background knowledge base; generate embedding vector information of different dimensions for words of different lengths; introduce part-of-speech tags to generate multi-dimensional background knowledge embedding vectors; splice the word embedding vectors with the multi-dimensional background knowledge embedding vectors to obtain the input vector.

[0031] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor performs the following steps:

[0032] Use the AC tree algorithm to search and obtain all words belonging to the background knowledge base; generate embedding vector information of different dimensions for words of different lengths; introduce part-of-speech tags to generate multi-dimensional background knowledge embedding vectors; splice the word embedding vectors with the multi-dimensional background knowledge embedding vectors to obtain the input vector.

[0033] Another object of the present invention is to provide an information data processing terminal, which is used to implement the knowledge embedding system.

[0034] Combining all of the above technical solutions, the advantages and positive effects of the present invention are as follows: The knowledge embedding method provided by the present invention converts knowledge of different types, formats, and lengths into a single vector representation, which can be applied to any neural network model input. When the knowledge embedding method of the present invention was applied to a named entity recognition model, F1 scores of 95.03 and 96.78 were achieved on the open-source named entity recognition datasets MSRA and Resume, respectively. Knowledge embedding improved the model by 1 to 2 points, and F1 scores were also improved on the CoNLL2003, MSRA, and OntoNotes4.0 datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0036] Figure 1 This is a flow chart of the knowledge embedding method provided by an embodiment of the present invention.

[0037] Figure 2 This is a system structure diagram of knowledge embedding provided by an embodiment of the present invention;

[0038] FIG3 is a distribution diagram of entities of various categories in the CoNLL2003 training set and test set provided by an embodiment of the present invention, wherein FIG3(a) is a distribution diagram of entities of various categories in the CoNLL2003 training set, and FIG3(b) is a distribution diagram of entities of various categories in the CoNLL2003 test set.

[0039] Figure 4 shows the distribution of entities in each category of the MSRA training set and test set provided by an embodiment of the present invention. Figure 4(a) shows the distribution of entities in each category of the MSRA training set, and Figure 4(b) shows the distribution of entities in each category of the MSRA test set.

[0040] Figure 5 shows the entity distribution diagrams for each category in the OntoNotes 4.0 training set and test set provided by an embodiment of the present invention. Figure 5(a) shows the entity distribution diagram for each category in the OntoNotes 4.0 training set, and Figure 5(b) shows the entity distribution diagram for each category in the OntoNotes 4.0 test set.

[0041] In the figure: 1. Word acquisition module; 2. Word embedding vector generation module; 3. Multi-dimensional background knowledge embedding vector generation module; 4. Input vector generation module.

[0042] Figure 6 Schematic diagram of the process of generating a multi-dimensional background knowledge embedding vector provided by an embodiment of the present invention.

[0043] Figure 7 Schematic diagram of the MBK input layer provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] In response to the problems existing in the prior art, the present invention provides a method, system, medium, device and data processing terminal for knowledge embedding. The present invention is described in detail below with reference to the accompanying drawings.

[0046] The knowledge embedding method provided by the present invention includes: converting knowledge of different types, methods, and lengths in the natural world into the same vector representation, providing background knowledge for the entity recognition model, solving the problem of insufficient domain knowledge of the entity recognition model, and enabling the entity recognition model to more accurately identify entities existing in natural text.

[0047] Specifically, if Figure 1 As shown, the knowledge embedding method provided by the embodiment of the present invention includes the following steps:

[0048] S101, build a Trie tree based on all the words in the background knowledge base, given an input sentence, and then use the AC tree algorithm to find the background knowledge words in the input sentence;

[0049] S102, distinguishing the latitudes of the background knowledge words in the input sentence according to their lengths, and then converting the words of different latitudes into word embedding vectors;

[0050] S103, introduce part-of-speech tags to represent the part of speech of a word and generate a multi-dimensional background knowledge embedding vector;

[0051] S104: Inputting an input vector obtained by concatenating the word embedding vector and the multi-dimensional background knowledge embedding vector into an entity recognition model, and using the entity recognition model to recognize entities existing in natural text, the input vector includes feature information of the input sentence and word information of the background knowledge base.

[0052] like Figure 2 As shown, the knowledge embedding system provided by the embodiment of the present invention includes:

[0053] The word acquisition module 1 is used to construct a Trie tree based on the words of all background knowledge bases. Given an input sentence, the AC tree algorithm is used to search for background knowledge words in the input sentence.

[0054] The word embedding vector generation module 2 is used to distinguish the dimensions of the background knowledge words in the input sentence according to their length, and then convert the words of different dimensions into word embedding vectors;

[0055] Multi-dimensional background knowledge embedding vector generation module 3, used to introduce part-of-speech tags to represent the part of speech of a word and generate a multi-dimensional background knowledge embedding vector;

[0056] The input vector generation module 4 is used to input the input vector obtained by splicing the word embedding vector and the multi-dimensional background knowledge embedding vector into the entity recognition model. The input vector contains the feature information of the input sentence and the word information of the background knowledge base.

[0057] The technical solution of the present invention is further described below in conjunction with specific embodiments.

[0058] The present invention designs a knowledge embedding method to convert knowledge of different types, modes and lengths into the same vector representation, which can be applied to any neural network model input.

[0059] (1) Complete operation implementation steps:

[0060] 1. Use the AC tree algorithm to find all the words in the background knowledge base;

[0061] 2. Words of different lengths generate embedding vector information of different dimensions;

[0062] 3. Introduce part-of-speech tags to generate multi-dimensional background knowledge embedding vectors;

[0063] 4. The word embedding vector is concatenated with the multi-dimensional background knowledge embedding vector to obtain the input vector.

[0064] (2) Model representation

[0065] Knowledge embedding methods include word embedding vectors and multi-dimensional background knowledge embedding vectors. Word embedding vectors use 300-dimensional pre-trained Word2Vec vectors, corresponding to word-level information. Multi-dimensional background knowledge embedding vectors are generated based on the AC tree search algorithm and correspond to word-level information. For the input sentence, first, the AC tree search algorithm is used to obtain all words belonging to the background knowledge base; then, the words are divided according to their length, and words of different lengths correspond to embedding vector information of different dimensions; finally, part-of-speech tags are introduced to generate a multi-dimensional background knowledge embedding vector. The specific types of part-of-speech tags are shown in Table 1. Note that the part-of-speech tags at the input layer only consider the first character tag B and the irrelevant tag O, and do not include the I tag.

[0066] Table 1 Part-of-speech tags of background knowledge base

[0067]

[0068] (3) Generation process of multi-dimensional background knowledge embedding vector based on AC tree algorithm

[0069] The generation process of the multi-dimensional background knowledge embedding vector is shown in Figure 3.

[0070] For the AC tree algorithm, given a background knowledge base V and a sentence S, all background knowledge base words in the sentence S can be found. Among the background knowledge base words found by the labeled AC tree algorithm. First, for the input sentence S and the background knowledge base V, using the AC tree algorithm, obtain the set C of all background knowledge base words in the sentence S; then, classify according to the length of the words to generate the background knowledge embedding vector of the corresponding dimension. The words with a length of 2 in the AC search results include: "Wuhan", "Mayor", "Yangtze River", "Bridge". Therefore, for the sentence "Wuhan Yangtze River Bridge", the background knowledge embedding vector of dimension 2 is "B-ming OB-ming B-ming O B-ming O", where "B-ming" is a noun label. For example, the first "B-ming" label indicates that the word "Wuhan" with the first character "Wu" and a length of 2 is a background knowledge base word and its词性 is a noun. The generation of the remaining labels follows the same pattern.

[0071] (4) Detailed explanation of the input framework

[0072] The generation process of the multi-dimensional background knowledge embedding vector of the knowledge embedding method provided by the present invention is described in detail with examples below. When the input sentence in the present invention is "Wuhan Yangtze River Bridge", the corresponding input layer embedding vector.

[0073]

[0074] Among them, X input represents the input vector, X char represents the character vector, X background-i represents the background knowledge vector of the i-th dimension, represents the concatenation operation.

[0075] In the present invention, the embedding vector corresponding to each word includes two parts, namely the word embedding vector and a 50-dimensional background knowledge embedding vector. Since the word embedding vector has been explained above, the multi-dimensional background knowledge embedding vector will be mainly introduced. The multi-dimensional background knowledge embedding vector is composed of 50-dimensional knowledge encodings, which belong to additional dimensions. Among them, the i-th dimension corresponds to the entity name with length i and its词性 information. Taking the word "武" as an example, the label of the 1st dimension of the word "武" is "O", indicating that the word "武" is not a word in the background knowledge base. The label "O" will be mapped to 1 background knowledge embedding vector, with a default length of 250. Similarly, the label of the 2nd dimension of the word "武" is "B-ming", which means that the word "武汉" is a noun in the background knowledge base and will also be mapped to 1 background knowledge embedding vector. And so on, the labels of the 3rd to 50th dimensions of the word "武" will each be mapped to background knowledge embedding vectors with fixed dimensions. Finally, the 50-dimensional background knowledge embedding vectors are concatenated to form the background knowledge embedding vector of "武".

[0076] Applying the knowledge embedding method of the present invention to the named entity recognition model, the F1 values of 95.03 and 96.78 are respectively obtained on the open-source named entity recognition datasets MSRA and Resume. The improvement brought by knowledge embedding to the model is 1-2 points, and the F1 values are improved on the CoNLL2003, MSRA, and OntoNotes 4.0 datasets.

[0077] The positive effects of the present invention will be further described below in combination with specific experimental data.

[0078] 1. The comparison table 2 of the method provided by the present invention and the prior art is as follows.

[0079] Table 2

[0080]

[0081] 2. The comparison of the effects of the existing literature and the method provided by the present invention is as follows:

[0082] CoNLL2003[1](Sang E F, De Meulder F. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition[J]. arXiv preprint cs / 0306050, 2003): This dataset is an English dataset containing 4 entity types, including LOC, ORG, PER, and MISC.

[0083] MSRA[2](Levow G A.The third international Chinese language processing bakeoff:Word segmentation and named entity recognition[C] / / Proceedings of theFifth SIGHAN Workshop on Chinese Language Processing.2006:108-117): This dataset is a Chinese dataset containing three types of entities, including names of people, places, and organizations.

[0084] OntoNotes 4.0[3] (Weischedel R, Pradhan S, Ramshaw L, et al. Ontonotes release 4.0[J]. LDC2011T03, Philadelphia, Penn.: Linguistic Data Consortium, 2011): This dataset is a Chinese dataset from the news field, which annotates 18 named entity types. The distribution of the training and test sets is shown in Table 3.

[0085] Table 3

[0086] dataset train test zh_onto4 15537 4346 zh_msra 41726 4362 conll03 14029 3449

[0087] Figure 3 shows the distribution of entities in each category of the CoNLL2003 training set and test set, where Figure 3(a) shows the distribution of entities in each category of the CoNLL2003 training set, and Figure 3(b) shows the distribution of entities in each category of the CoNLL2003 test set.

[0088] Figure 4 shows the distribution of entities in each category in the MSRA training and test sets. Figure 4(a) shows the distribution of entities in each category in the MSRA training set, and Figure 4(b) shows the distribution of entities in each category in the MSRA test set.

[0089] Figure 5 shows the entity distribution of each category in the OntoNotes 4.0 training set and test set. Figure 5(a) shows the entity distribution of each category in the OntoNotes 4.0 training set, and Figure 5(b) shows the entity distribution of each category in the OntoNotes 4.0 test set.

[0090] On the baseline model:

[0091] BERT-Tagger: Devlin et al. [4] (Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding [J]. arXiv preprint arXiv: 1810.04805, 2018) proposed the BERT model, which treats named entity recognition as a tagging task.

[0092] BERT-MRC: Li et al. [5] (Li X, Feng J, Meng Y, et al. A unified MRC framework for named entity recognition [J]. arXiv preprint arXiv:1910.11476, 2019) regarded the named entity recognition task as a reading comprehension task.

[0093] BERT-FLAT: Li et al. [6] (Li X, Yan H, Qiu X, et al. FLAT: Chinese NER using flat-lattice transformer [J]. arXiv preprint arXiv: 2004.11795, 2020) designed the FLAT model, which converts the lattice structure into a set of fragments and introduces specific positional encoding.

[0094] LEBERT: Liu et al. [7] (Liu W, Fu X, Zhang Y, et al. Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter[C] / / Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing(Volume 1: Long Papers). 2021) proposed the LEBERT model, which directly injects dictionary information between the Transformer layers in BERT through a dictionary adapter.

[0095] Comparison and analysis of experimental results are shown in Table 4.

[0096]

[0097] The F1 scores of the three datasets were improved by 0.26, 0.15, and 0.95 respectively.

[0098] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL), or wireless (e.g., infrared, wireless, microwave, etc.)) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0099] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for knowledge embedding, characterized in that: The method of knowledge embedding includes: converting knowledge of different types, modes, and lengths in the natural world into a single vector representation, providing background knowledge for an entity recognition model, and using the entity recognition model to recognize entities in natural text; The method for knowledge embedding comprises the following steps: Step 1, constructing a Trie tree based on all the words in the background knowledge base, given an input sentence, and then searching for background knowledge words in the input sentence using an AC tree algorithm; Step 2: The background knowledge words in the input sentence are divided into different dimensions according to their length, and then the words of different dimensions are converted into word embedding vectors; Step 3: Introduce part-of-speech tags to represent the part of speech of a word and generate a multi-dimensional background knowledge embedding vector; Step 4: Input the input vector obtained by concatenating the word embedding vector and the multi-dimensional background knowledge embedding vector into the entity recognition model, and use the entity recognition model to identify entities existing in the natural text. The input vector contains feature information of the input sentence and word information of the background knowledge base; The step 1 uses the AC tree algorithm to search and obtain all words belonging to the background knowledge base, including: for the input sentence, using the AC tree search algorithm to obtain all words belonging to the background knowledge base; dividing the words according to their length, with words of different lengths corresponding to embedding vector information of different dimensions; introducing part-of-speech tags to generate a multi-dimensional background knowledge embedding vector; wherein the part-of-speech tags of the input layer only consider the first character tag B and the irrelevant tag O, and do not include the I tag.

2. The method for knowledge embedding according to claim 1, wherein: The part-of-speech tags of the background knowledge base include: ① noun, B-ming; ② verb, B-dong; ③ numeral, B-shu; ④ preposition, B-jie; ⑤ conjunction, B-lian; ⑥ mood, B-yu; ⑦ auxiliary word, B-zhu; ⑧ idiom, B-cheng.

3. The method for knowledge embedding according to claim 1, wherein: The AC tree algorithm is used to find all background knowledge base words in a given background knowledge base V and a sentence S.

4. The method for knowledge embedding according to claim 1, wherein: The expression of the input layer embedding vector is: Among them, X input represents the input vector, X char represents the word vector, X background-i represents the background knowledge vector of the i-th dimension, Represents a concatenation operation.

5. The method for knowledge embedding according to claim 1, wherein: In the step 2, the word embedding vector uses a 300-dimensional pre-trained Word2Vec vector, corresponding to word-level information; the multi-dimensional background knowledge embedding vector is generated based on the AC tree search algorithm, corresponding to word-level information.

6. A knowledge embedding system using the knowledge embedding method according to any one of claims 1 to 5, characterized in that: The knowledge embedding system includes: a word acquisition module, which is used to construct a Trie tree based on the words of all background knowledge bases, and then search for background knowledge words in the input sentence using the AC tree algorithm given an input sentence; The word embedding vector generation module is used to distinguish the dimensions of the background knowledge words in the input sentence according to their length, and then convert the words of different dimensions into word embedding vectors; A multi-dimensional background knowledge embedding vector generation module is used to introduce part-of-speech tags to represent the part of speech of a word and generate a multi-dimensional background knowledge embedding vector; The input vector generation module is used to input the input vector obtained by splicing the word embedding vector and the multi-dimensional background knowledge embedding vector into the entity recognition model. The input vector contains the feature information of the input sentence and the word information of the background knowledge base.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the knowledge embedding method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the knowledge embedding method according to any one of claims 1 to 5.

9. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the knowledge embedding system as claimed in claim 6.

Images