Word vector representation method for Chinese text data based on BIE position word list

Abstract

The present invention relates to a Chinese text data word vector representation method based on a BIE position word list, which relates to the fields of deep learning and named entity recognition, and includes the following steps: S1: generating a full word set and a strongly related word set, and constructing a BIE position word list; S2: constructing a position-independent word vector using the original representation of the word vector; S3: condensing the word vector representation in the word set based on a word frequency weighted average pooling algorithm; S4: weighting the BIE position word vector of the word and concatenating it with the original word vector to generate a word vector containing vocabulary position information. The present invention can highlight the strongly related vocabulary position information while incorporating the full word position information of the vocabulary into the word vector. The word vector representation dimension is expanded, so that the Chinese entity recognition result has higher accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and named entity recognition, and relates to a Chinese text data word vector representation method based on a BIE position word list. Background Art

[0002] Named Entity Recognition (NER) is a basic task in the field of natural language processing and a subtask of information retrieval, relation extraction, question-answering systems, etc. Unlike the natural word segmentation of English text, the number of characters in a word in Chinese text is not fixed, and there is no word segmentation identifier. This makes it difficult to learn vocabulary boundary information in the Chinese named entity recognition task. Therefore, it is necessary to incorporate vocabulary position information into the Chinese character vector representation to improve the accuracy of Chinese named entity recognition.

[0003] At present, the widely used vocabulary enhancement method is the vocabulary enhancement method based on vocabulary lists and word vectors. There are two main enhancement paradigms: dynamic extraction framework and adaptive Embedding. The former is compatible with vocabulary input by designing a dynamic extraction framework, but this dynamic extraction framework often needs to modify the structure of the model, modify the sequence structure into a directed acyclic graph (DAG) structure, and then dynamically aggregate vocabulary information during the model training process. This method leads to complex model structure, poor parallel capability, high training and deployment costs, and the model is not portable. The latter integrates vocabulary information into character encoding in the Embedding stage by constructing adaptive Embedding, which is independent of the model framework. However, the effect of vocabulary information integration is strongly related to the Embedding strategy. The WC-LSTM structure has the problem of vocabulary information loss, and the Multi-digraph structure has the problem of relying on dictionary labels and is difficult to be widely applicable. Summary of the invention

[0004] In view of this, the purpose of the present invention is to utilize the BIE position word list to represent the vocabulary position information, and combine the word vector representation of the BIE full word set and the strongly related word set, aiming to not affect the model structure, and integrate the full position information of the vocabulary into the word vector. At the same time, the full position information of the vocabulary and the strongly related vocabulary position information can be integrated into the word vector in the Embedding stage, thereby solving the problem that traditional word vectors are difficult to represent vocabulary position information to a certain extent, and helping the model to improve its ability to solve the problem of entity nesting, expand the word vector representation dimension, and make the entity recognition results have higher accuracy.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for characterizing Chinese text data word vectors based on a BIE position word list comprises the following steps:

[0007] S1: Generate a full word set and a strongly related word set, and construct a BIE position word list;

[0008] S2: Use the original representation of the word vector to construct a position-independent word vector;

[0009] S3: Condensing the word vector representation in the word set based on the word frequency weighted average pooling algorithm;

[0010] S4: The BIE position word vector of the word is weighted and concatenated with the original word vector to generate a word vector containing vocabulary position information.

[0011] Further, the step S1 specifically includes the following steps:

[0012] S11: Assume that there is a Chinese text dataset D = {S, Y}, where S = {s1, s2, ..., s n} is a Chinese text set, Y is a label set; for each text s i ∈S, generate the full word set W according to the full dictionary matching algorithm and TF-IDF algorithm a ={w a1 , w a2 ,…,w an} and the strongly related word set W b ={w b1 , w b2 ,…,w bn}; where w = {[k1, st1, ed1], [k2, st2, ed2], …, [k m , st m ,ed m ]}, indicating that word w matches text k i st i To ed i Location;

[0013] S12: In order to express the position information of the character in the word, according to the position of the character in the word, the word set W is used a , W b , for the text s in the dataset D = {c1, c2, ..., c m},c i For each word in the text, generate a list of BIE position words for the text s division Among them ai =[B ai , I ai , E ai ], B ai , I ai , E ai Respectively represent the i-th character c in the text si The word sets generated for the beginning (begin), inside (inside), and end (end) are represented as:

[0014]

[0015]

[0016]

[0017] Similarly, generate

[0018] Further, in step S11, a word set W is generated a With W b The rule is: use the full dictionary matching algorithm to obtain all implicit words in the text as the full quantifier set W a ; Use the TF-IDF algorithm to sort the words in the training text by importance, retain 50% of the words as the strongly related dictionary, and match them with the text segmentation results to generate a strongly related word set W b .

[0019] Further, step S2 specifically includes:

[0020] S21: In order to fully represent the original word vector so that it contains multi-dimensional text information and avoid the time overhead of training word vectors, a pre-trained character embedding table is used. c Generate the original representation of the word vector of the text s in the dataset

[0021] S22: According to the full quantifier set W a and strongly related word set W b The correspondence between words and characters in the text s generates a position-independent word vector for each word w in the text s:

[0022]

[0023] Position independence is reflected in the fact that for different texts s, s′, the word vectors of the same word are the same, which prevents the same entity from being disturbed by the context of different pre-training datasets and introducing additional errors.

[0024] Further, step S3 specifically includes:

[0025] For the word c in the text s in the dataset i The corresponding BIE position word list uses the word frequency weighted average pooling algorithm to condense the word vector representation in the word set; in order to highlight the importance of high-frequency words, the word frequency is used as the importance weight to generate the full amount of BIE position word vectors respectively Strongly correlated with BIE position word vector For word set B, the contraction formula is:

[0026]

[0027]

[0028] z(w) represents the frequency of word w appearing in the text of the dataset.

[0029] Further, step S4 specifically includes the following steps:

[0030] S41: character c in text s in the dataset i , the condensed full BIE position word vector and the strongly correlated BIE position word vector are fused according to the weight T; the weight T is used to reduce the importance of redundant words in the full word set; the fused BIE position word vector e is obtained s (B, I, E):

[0031]

[0032] S42: character c in text s in the dataset i , the fused BIE position word vector e s (B, I, E) is concatenated with the original word vector to obtain the final word vector x containing vocabulary information c :

[0033] x c =[x c ;e s (B, I, E)].

[0034] Furthermore, the strongly related word set is obtained according to the TF-IDF algorithm, that is, the word c i The strongly related BIE position word list has only one dimension that may have a value and its length is 1, so the weight is set w∈bie bi .

[0035] The beneficial effect of the present invention is that since the full word set cannot highlight the importance of strongly related words, the method of fusing the full BIE position word vector with the strongly related BIE position word vector is adopted to highlight the importance of the strongly related word set while incorporating the full word set information. At the same time, according to the different positions of the characters in the words, it is divided into three BIE dimensions, and the vocabulary position information is incorporated into the character vector. To a certain extent, it solves the problem that traditional word vectors are difficult to represent vocabulary boundary information, and helps the model improve its ability to solve entity nesting problems, so that the named entity recognition results have higher accuracy. Finally, since the vocabulary information is only fused at the Embedding layer, it does not affect the structure of the entity recognition model and can be widely used.

[0036] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0038] Figure 1 A flow chart of a method for characterizing Chinese text data word vectors based on a BIE position word list provided in Example 1 of the present invention;

[0039] Figure 2 The full amount of characters and the strongly related BIE multi-dimensional position word list provided in the first embodiment of the present invention;

[0040] Figure 3 This is a diagram of the architecture for fusing the original word vector and the BIE position word vector provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0041] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and the features in the embodiments can be combined with each other without conflict.

[0042] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0043] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0044] See also Figure 1 to Figure 3 .

[0045] Embodiment 1

[0046] refer to Figure 1 , Figure 1 A flowchart of a method for characterizing Chinese text data word vectors based on a BIE position word list provided in an embodiment of the present invention specifically includes:

[0047] Chinese text word vectors are difficult to express the position and boundary information of Chinese words, which brings great challenges to Chinese named entity recognition. Therefore, in this embodiment, the discussion is mainly focused on the Chinese text dataset.

[0048] How to express the position information of a character in a corresponding word in a character vector is the key to the present invention. In the Chinese entity recognition task, learning vocabulary boundaries can help the model distinguish entity boundaries, so the three position dimensions of BIE are used to express the position information of a character in a word. At the same time, in order to take into account both the full word set and the strongly related word set, the weight T is used for weighted fusion.

[0049] 101: Generate a full word set and a strongly related word set, and construct a BIE position word list;

[0050] Step 1: Assume that there is a Chinese text dataset D = {S, Y}, where S = {s1, s2, ..., s n} is a set of Chinese texts, and Y is a set of labels. For each text s i ∈S, generate the full word set W according to the full dictionary matching algorithm and TF-IDF algorithm a ={w a1 , w a2 ,…,w an} and the strongly related word set W b ={w b1 , w b2 ,…,w bn}. Where w = {[k1, st1, ed1], [k2, st2, ed2], …, [km , st m ,ed m ]}, indicating that word w matches text k i st i To ed i Location.

[0051] Generate word set W a With W b The rule is: use the full dictionary matching algorithm to obtain all implicit words in the text as the full quantifier set W a . Use the TF-IDF algorithm to sort the words in the training text by importance, retain 50% of the words as a strongly related dictionary, and match them with the text segmentation results to generate a strongly related word set W b .

[0052] Step 2: Construct a BIE position list, refer to Figure 2 , Figure 2 is the total amount of characters and the strongly related BIE multi-dimensional position word list. In order to express the position information of characters in words, according to the position of characters in words, the word set W is used a , W b , for the text s in the dataset D = {c1, c2, ..., c m},c i For each word in the text, generate a list of BIE positions for the text s and Among them ai =[B ai , I ai , E ai ], B ai , I ai , E ai Respectively represent the i-th character c in the text s i The word set generated for the beginning (begin), inside (inside), and end (end). It is represented as:

[0053]

[0054]

[0055]

[0056] Similarly, generate

[0057] 102: Use the original representation of the word vector to construct a position-independent word vector;

[0058] Step 3: In order to fully represent the original word vector so that it contains multi-dimensional text information and avoid the time overhead of training word vectors, a pre-trained character embedding table is used. c

[0059] Generate the original representation of the word vector of the text s in the dataset

[0060] Step 4: According to the full quantifier set W a and strongly related word set W b The correspondence between words and characters in the text s generates a position-independent word vector for each word w in the text s:

[0061]

[0062] 103: Condensing word vector representations in word sets based on word frequency weighted average pooling algorithm;

[0063] Step 5: Calculate the word c in the text s in the dataset i The corresponding BIE position word list uses the word frequency weighted average pooling algorithm to condense the word vector representation in the word set. In order to highlight the importance of high-frequency words, the word frequency is used as the importance weight to generate the full amount of BIE position word vectors. Strongly correlated with BIE position word vector For word set B, the contraction formula is:

[0064]

[0065]

[0066] z(w) represents the frequency of word w appearing in the text of the dataset.

[0067] 104: Finally, the BIE position word vector of the word is weighted and concatenated with the original word vector to generate a word vector containing vocabulary position information;

[0068] Step 6: Reference Figure 3 , Figure 3 This is a diagram of the fusion architecture of the original word vector and the BIE position word vector of the present invention. i , the condensed full BIE position word vector and the strongly related BIE position word vector are weighted w∈bie bi , and fused. The weight T is used to reduce the importance of redundant words in the full word set. The fused BIE position word vector e is obtained s (B, I, E):

[0069]

[0070] Step 7: Calculate the word c in the text s in the dataset i , the fused BIE position word vector e s (B, I, E) is concatenated with the original word vector to obtain the final word vector x containing vocabulary information c :

[0071] x c =[x c ;e s (B, I, E)]

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A word vector representation method for Chinese text data based on BIE position word list, characterized by: The following steps are involved: S1: Generate a full word set and a strongly related word set, and construct a BIE position word list; S2: Use the original representation of the word vector to construct a position-independent word vector; S3: Condensing the word vector representation in the word set based on the word frequency weighted average pooling algorithm; S4: weight the BIE position word vector of the word and concatenate it with the original word vector to generate a word vector containing the lexical position information; The step S1 specifically includes the following steps: S11: Assume that there is a Chinese text dataset D = {S, Y}, where S = {s1, s2, ..., s n } is a Chinese text set, Y is a label set; for each text si∈S, a full word set W is generated based on the full dictionary matching algorithm and TF-IDF algorithm a ={w a1 , w a2 ,…,w an } and the strongly related word set W b ={w b1 , w b2 ,…,w bn }; where w = {[k1, st1, ed1], [k2, st2, ed2], …, [k m , st m ,ed m ]}, indicating that word w matches text k i st i To ed i Location; S12: Based on the position of the character in the word, use the word set W a , W b , for the text s={c1,c2,...,c m }, c i For each word in the text, generate a list of BIE position words for the text s and Among them ai =[B ai , I ai , E ai ], B ai , I ai , E ai Respectively represent the i-th character c in the text s i The word sets generated for the beginning, inside, and end are represented as: Similarly, generate In step S11, a word set W is generated a With W b The rule is: use the full dictionary matching algorithm to obtain all implicit words in the text as the full quantifier set W a ; Use the TF-IDF algorithm to sort the words in the training text by importance, retain 50% of the words as a strongly related dictionary, and match them with the text segmentation results to generate a strongly related word set W b ; Step S2 specifically includes: S21: Use pre-trained character embedding table c Generate the original representation of the word vector of the text s in the dataset S22: According to the full quantifier set W a and strongly related word set W b The correspondence between words and characters in the text s generates a position-independent word vector for each word w in the text s: Position independence is reflected in the fact that the word vectors of the same word are the same for different texts s, s′, which avoids the same entity being disturbed by the context of different pre-training datasets and introducing additional errors; Step S3 specifically includes: For the word c in the text s in the dataset i The corresponding BIE position word list uses the word frequency weighted average pooling algorithm to condense the word vector representation in the word set; using word frequency as the importance weight, the full amount of BIE position word vectors are generated respectively Strongly correlated with BIE position word vector For word set B, the contraction formula is: z(w) represents the frequency of word w appearing in the data set text; Step S4 specifically includes the following steps: S41: character c in text s in the dataset i , the condensed full BIE position word vector and the strongly correlated BIE position word vector are fused according to the weight T; the weight T is used to reduce the importance of redundant words in the full word set; the fused BIE position word vector e is obtained s (B, I, E): S42: character c in text s in the dataset i , the fused BIE position word vector e s (B, I, E) is concatenated with the original word vector to obtain the final word vector x containing vocabulary information c : x c =[x c ;e s (B,I,E)]。 2. The Chinese text data word vector representation method based on the BIE position word list according to claim 1 is characterized by: The strongly related word set is obtained according to the TF-IDF algorithm, that is, the word c i The strongly related BIE position word list has only one dimension that may have a value and its length is 1, so the weight is set w∈bie bi .

Images