An entity extraction method based on proprietary dictionary word embedding

Abstract

The application discloses an entity extraction method based on a special dictionary word embedding, and belongs to the technical field of text processing. Firstly, a target document set is split into characters, enhanced character vectors are generated by fusing the information of the current character and the next character through a sliding window model, a word set is obtained by matching each character in a special dictionary, the first K related words are screened out by using an improved TF-IDF scoring method, the word embedding of the screened words is spliced with the enhanced character vector to obtain a final character vector, the implicit features are extracted by inputting the final character vector into a single-layer BiLSTM and are mapped to a label space, and finally, the entity extraction result is obtained by introducing label constraints through a CRF layer. The application enriches the context semantic input of the characters, effectively filters irrelevant noise, reduces the dependence on general segmentation, solves the problems of semantic defects, error transmission and noise interference in the prior art, and is suitable for the Chinese entity extraction task in a special field.

Description

Technical Field

[0001] This invention belongs to the field of text processing technology, specifically relating to an entity extraction method based on word embedding from a proprietary dictionary. Background Technology

[0002] In recent years, the iteration of internet technology and the deepening of related research have led to a rapid expansion of textual information resources. Faced with massive amounts of unstructured text, people urgently need to extract useful content from it, and entity extraction, as a basic information unit, is a key entry point for information mining. Domain-specific Chinese entity extraction plays a leading role in semantic understanding, data mining, knowledge graph construction, and other scenarios, and has become an important cornerstone of the field of text processing.

[0003] The existing technologies mainly consist of the following three technical solutions:

[0004] 1. Character-level approach: Current mainstream entity extraction systems still use character-level BiLSTM-CRF as the backbone. This approach first segments the sentence into characters, then uses pre-trained models like BERT to generate character vectors on the corpus and inputs them into the BiLSTM to extract semantic latent states. Finally, the CRF layer completes entity extraction. The drawback of this approach is that it segments the sentence into individual characters, making it unable to handle polysemy and losing contextual semantics, resulting in significant information loss.

[0005] 2. Word-level approach: Current neural network models based on word features also employ the BiLSTM-CRF architecture. The difference from character-level models lies in that sentences are first segmented into words, then word vectors are trained on the corpus and used as input to the BiLSTM. After the network extracts latent features, the CRF performs entity extraction. The disadvantages of this approach are its reliance on prior word segmentation, complex boundaries of domain-specific terminology, susceptibility to missegmentation by general-purpose tools, and direct propagation of errors downstream.

[0006] 3. Character-Dictionary Level Approach: Current character-dictionary-based neural network models also employ the BiLSTM-CRF architecture, combining character-level and word-level approaches. The sentence is first segmented into characters, then matched against a dictionary. The matched words are then merged and used as input to the BiLSTM. After the network extracts latent features, the CRF performs sequence labeling. The drawback of this approach is that it indiscriminately injects all matched words into the character representation, introducing a large number of words that are irrelevant to the article's topic, context, and semantics, resulting in the character signal being overwhelmed by noise. Summary of the Invention

[0007] To address the problems mentioned in the background art, this invention provides an entity extraction method based on word embedding of a proprietary dictionary, which solves the problems of semantic information loss in existing character-level models, error propagation caused by prior word segmentation in word-level models, and the introduction of a large amount of irrelevant noise by indiscriminately injecting words into character-dictionary level models.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] An entity extraction method based on proprietary dictionary word embedding includes the following steps:

[0010] S1: Character representation; split the target document set into individual documents, then split each document into multiple sentences, and finally split each sentence into multiple characters;

[0011] S2: Sliding window enhanced representation; The sliding window model is used to fuse information from the current character and its adjacent characters to generate an enhanced character vector;

[0012] S3: Specialized dictionary matching; For each character obtained from the splitting, perform character matching in the preset specialized domain dictionary to obtain the set of all words containing that character;

[0013] S4: Dictionary-related information fusion; Each word in the word set is scored using an improved dual-scoring TF-IDF scoring method, and the top K words are retained according to the score ranking. The word embeddings of the retained words are concatenated with the enhanced character vector to obtain the final character vector.

[0014] S5: Neural network modeling; The final character vector is input into a single-layer BiLSTM neural network to extract deep hidden features and map them to the label space to obtain the label mapping probability score;

[0015] S6: CRF label annotation; Input the label mapping probability score into the CRF layer, introduce the constraint relationship between labels, calculate the label sequence score, and select the label sequence with the highest score as the entity extraction result.

[0016] Compared with the prior art, the beneficial effects of the present invention are:

[0017] 1. Richer information in the input model: Compared with the existing character-dictionary model, which only inputs the character itself in the character embedding part, this solution integrates contextual semantic information into the character embedding part based on the sliding window.

[0018] 2. Less noise in the input model: Compared with the existing character-dictionary model that indiscriminately introduces all matching words, the sliding window and improved TF-IDF score proposed in this scheme can filter words related to the article topic and words with similar meanings to the characters, and filter words that are too different from the current characters, thus solving the problem of irrelevant noise being introduced into the existing character-dictionary model.

[0019] 3. Reduce reliance on general word segmentation: Inject proprietary dictionary vocabulary information on top of character features to compensate for the lack of semantic information on word combinations in character-level models, and reduce reliance on general word segmentation through character-dictionary matching. Attached Figure Description

[0020] Figure 1 This is the overall architecture diagram of the entity extraction method in this application. Detailed Implementation

[0021] To facilitate understanding of the technical content of this invention by those skilled in the art, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific examples described herein are merely illustrative and not intended to limit the scope of the invention.

[0022] like Figure 1 As shown, an entity extraction method based on word embedding from a proprietary dictionary is implemented as follows:

[0023] S1: Character representation;

[0024] First, prepare a domain-specific dictionary as a proprietary knowledge base. Then, split the given document set: the given document set is represented as: D={d1,d2,…d…} n}, where d n Let d represent the nth document. Further, we can break down document d into sentences, represented as: d = {s1, s2, ..., s}. k}, where s k This represents the k-th sentence. Further, the sentence s is split into characters, represented as: s={c1,c2,…c…} i}, where c i This represents the i-th character.

[0025] For example, the sentence "optical fiber transmission" can be split into the character sequence {'light', 'fiber', 'transmission', 'transmission'}.

[0026] S2: Enhanced representation of sliding windows;

[0027] A single character represents limited information and lacks contextual information. To enhance character embedding representation, a method of fusing the preceding and following characters is adopted to improve the contextual information representation of a character. A sliding window model is proposed, which uses word embeddings composed of contextual characters to jointly represent a single character, as shown below:

[0028] w ic ={c i ,c i+1};

[0029] ;

[0030] Where w ic Indicates sliding window matching c i and c i+1 The vocabulary is formed by [;], which indicates vector concatenation, and e c Indicates character embedding, e w Indicates word embedding.

[0031] For example, for the character 'light' (c i ='light'), and the next character is 'fiber' (c i+1 ='fiber'), then the sliding window vocabulary w ic ={'light','fiber'}='fiber optic', enhancing the character vector X i The embedded e of the character 'light' c ('light') and the word 'fiber optic' embedding e w (Fiber optic cables) are spliced ​​together.

[0032] S3 proprietary dictionary matching;

[0033] For each character c in the splitting i The method uses character matching to match all words containing a given character in a proprietary dictionary, resulting in a set of all words that match that character.

[0034] ;

[0035] Where Retr represents the character matching method, c i w represents the i-th character. n Represents the matching dictionary set The nth word in the dictionary. For example, for the character 'light', the set of words matched in a communication-specific dictionary. It may include {'optical core', 'optical fiber', 'optical switch', 'optical module', 'optical transceiver', 'light-emitting diode', etc.}.

[0036] S4: Integration of dictionary-related information;

[0037] Because the dictionary contains words from the entire proprietary corpus, word matching... The set may also contain words unrelated to the current document d, and these irrelevant noise words will affect the model's performance. Therefore, an association algorithm is used to filter words relevant to the current document, such as the TF-IDF algorithm. Existing TF-IDF scores filter words related to document d. Although these words are related to the overall document d, they may be irrelevant to the context of the sentence. If they are directly introduced into the model, irrelevant noise will be introduced. Therefore, this solution improves the TF-IDF calculation method by adding semantic similarity scores and static similarity scores to the matched word set. Scores are calculated using the improved TF-IDF algorithm, the scores are sorted, and finally the Top K words are selected and incorporated into the model.

[0038] The improved TF-IDF score consists of two parts, Score1 and Score2. The specific calculation process is as follows:

[0039] Calculate Score1 i (Document relevance score):

[0040] ;

[0041] Where n i w represents the i-th word in the set of words to be matched. i The total number of times N appears in document d d This represents the number of words in document d.

[0042] ;

[0043] Where |D| represents the total number of documents in the document set, and N i This represents the total number of documents containing the i-th word. The final score for the i-th word is obtained. i The score is:

[0044] Score1 i =TF i ×IDF i ;

[0045] Calculate Score2 i (Contextual similarity score): Score2 i Based on semantic cosine similarity score ScoreCos i Compared with the static similarity score of words, ScoreSta i Weighted average yields:

[0046] Score2 i =ScoreCos i +∂ScoreSta i , where ∂ is the weighting coefficient.

[0047] Semantic cosine similarity score (ScoreCos) i :

[0048] ;

[0049] in The set of words that represent the matching. The i-th word w i The corresponding word vectors, The word representing a sliding window is w ic The corresponding word vectors.

[0050] Word static similarity score ScoreSta i :

[0051] ;

[0052] Where |w i ∩w ic | represents the i-th word w i With sliding window vocabulary w ic The length of the common characters, ||w i |-|w ic || represents the i-th word w i With sliding window vocabulary w ic The absolute value of the length difference.

[0053] Calculate the final score of the improved TF-IDF:

[0054] MOD-TF i -IDF i =Score1 i +Score2 i ;

[0055] MOD-TF i -IDF i The improved TF-IDF score represents the i-th word.

[0056] According to the improved MOD-TF i -IDF i The score is the set of matching words. MOD-TF is calculated for each word. i -IDF i The words are scored and sorted, with higher scores indicating greater relevance to the sentence. The top K words with the highest scores are retained, while the rest are discarded as irrelevant noise. The retained words are then input into a word embedding layer, and the resulting vectors are concatenated.

[0057] ;

[0058] Where e m Represents word embedding layer, e K This represents the embedding representation of the final top K filtered words.

[0059] Furthermore, the word vectors are concatenated with the enhanced character vectors from the sliding window to obtain the final character vector x incorporating the information. i :

[0060] ;

[0061] Where X i This represents an enhanced representation of a sliding window. For example, for the character 'light', assuming K=3, the top 3 words retained are 'optical fiber', 'optical core', and 'optical cable', then e K ('light') is obtained by concatenating the word embeddings of these three words, and the final character vector x1 is obtained by combining the enhanced character vector X1 with e. K ('light') is obtained by piecing together.

[0062] S5: Neural Network Modeling;

[0063] The character vector x of the filtered related word information obtained in the previous step i The input is fed into a single-layer Bi-LSTM neural network to extract deep, hidden features. The forward computation process of Bi-LSTM is as follows:

[0064] ;

[0065] The backward computation process is similar to the forward computation process, only the direction is different. The forward and backward latent vectors of the i-th character are concatenated to obtain the final latent feature representation of the i-th character:

[0066] ;

[0067] Mapping the latent vectors to the label space yields the mapping probability score P of the labels. i,y , where P i,y W represents the score for the i-th character labeled as y. e b represents the weight of the label mapping layer. e The bias value representing the label mapping layer:

[0068] P i,y =W e ・h i +b e .

[0069] S6: CRF labeling;

[0070] Based on the probability fraction P obtained above i,y The algorithm further calculates the score generated for each type of label, selects the highest score as the label, and employs the classic Conditional Random Field (CRF) method to introduce constraints between labels. Specifically, given the input sequence X={x1,x2,…x... n The corresponding labels are y={y1,y2,…y}. n The formula for calculating the tag score is:

[0071] ;

[0072] Where A is the transition matrix. Indicates label y i To tag y i+1 The transition probability, This is the output of the Bi-LSTM, indicating that the character at position i is predicted as y. i The probability of the label.

[0073] After applying softmax normalization to the results, the overall annotation probability of the sentence is calculated:

[0074] ;

[0075] Where ỹ represents a possible labeled sequence, Y X It represents the set of all possible labels in a label sequence.

[0076] Finally, the loss function of the model is calculated using the maximum likelihood method, and the label with the highest score is selected as the final output of the model. .

Claims

1. A method for entity extraction based on word embedding from a proprietary dictionary, characterized in that, Includes the following steps: S1: Character representation; split the target document set into individual documents, then split each document into multiple sentences, and finally split each sentence into multiple characters; S2: Sliding window enhanced representation; The sliding window model is used to fuse information from the current character and its adjacent characters to generate an enhanced character vector; S3: Specialized dictionary matching; For each character obtained from the splitting, perform character matching in the preset specialized domain dictionary to obtain the set of all words containing that character; S4: Dictionary-related information fusion; Each word in the word set is scored using an improved dual-scoring TF-IDF scoring method, and the top K words are retained according to the score ranking. The word embeddings of the retained words are concatenated with the enhanced character vector to obtain the final character vector. S5: Neural network modeling; The final character vector is input into a single-layer BiLSTM neural network to extract deep hidden features and map them to the label space to obtain the label mapping probability score; S6: CRF label annotation; Input the label mapping probability score into the CRF layer, introduce the constraint relationship between labels, calculate the label sequence score, and select the label sequence with the highest score as the entity extraction result.

2. The entity extraction method based on proprietary dictionary word embedding according to claim 1, characterized in that, In S1, the document set is represented as D={d1,d2,…d… n }, where d n This represents the nth document; each document d is split into sentences and represented as d={s1,s2,…s} k }, where s k This represents the k-th sentence; each sentence s is represented by its characters as s={c1,c2,…c i }, where c i This represents the i-th character.

3. The entity extraction method based on proprietary dictionary word embedding according to claim 2, characterized in that, In S2, the sliding window model uses the current character c to merge. i With the next character c i+1 Generate sliding window vocabulary in the following way: w ic ={c i ,c i+1 }; Enhanced character vector X i Through character embedding Word embedding with sliding window vocabulary The result obtained by piecing together is: ； Where [;] represents vector concatenation operation.

4. The entity extraction method based on proprietary dictionary word embedding according to claim 3, characterized in that, In S3, the character matching method Retr(c) is used. i Get the character c containing the i-th character. i The complete set of words: ； Where w n Represents a set of words The nth word in the text.

5. The entity extraction method based on proprietary dictionary word embedding according to claim 4, characterized in that, In S4, the improved TF-IDF scoring method includes a first part score, Score1. i Score1 i TF by word frequency i Inverse Document Frequency (IDF) i Multiplying them together, we get: ； n i w represents the i-th word i The total number of times N appears in the current document d. d This represents the number of words in the current document d; |D| represents the total number of documents in the document set, N i This indicates that the i-th word w is included. i The total number of documents.

6. The entity extraction method based on proprietary dictionary word embedding according to claim 5, characterized in that, In S4, the improved TF-IDF scoring method also includes a second part, Score2. i Score2 i Based on semantic cosine similarity score ScoreCos i Compared with the static similarity score of words, ScoreSta i The weighted result is Score2. i =ScoreCos i +∂ScoreSta i , where ∂ is the weighting coefficient; Semantic cosine similarity score (ScoreCos) i The calculation formula is: ； in w represents the i-th word i The corresponding word vectors, The word w indicates a sliding window ic The corresponding word vectors.

7. The entity extraction method based on proprietary dictionary word embedding according to claim 6, characterized in that, Word static similarity score ScoreSta i The calculation formula is: ； Where |w i ∩w ic | represents the i-th word w i With sliding window vocabulary w ic The length of the common characters, ||w i |-|w ic || represents the i-th word w i With sliding window vocabulary w ic The absolute value of the length difference.

8. The entity extraction method based on proprietary dictionary word embedding according to claim 7, characterized in that, In S4, the improved TF-IDF final score: MOD-TF i -IDF i =Score1 i +Score2 i ; According to MOD-TF i -IDF i Scoring on word set W ci The words in the list are sorted in descending order, and the top K words with the highest scores are retained. The words of the retained words are then concatenated to obtain the following: ； Where e m Represents the word embedding layer; the final character vector x i By enhancing character vector X i With e k (c i The result is obtained by splicing together the following: 。 9. The entity extraction method based on proprietary dictionary word embedding according to claim 8, characterized in that, In S5, the final character vector x i Input a single-layer BiLSTM neural network, perform forward and backward computations respectively, and concatenate the forward and backward hidden vectors of the i-th character to obtain the final hidden feature representation h. i ;The hidden feature h i Mapping to the label space yields the label mapping probability scores: P i,y =W e ・h i +b e ; Among them W e b represents the weight of the label mapping layer. e P represents the bias of the label mapping layer. i,y This represents the score for the i-th character labeled as y.

10. The entity extraction method based on proprietary dictionary word embedding according to claim 9, characterized in that, In S6, given the input sequence X={x1,x2,…x} n } and the corresponding label sequence y={y1,y2,…y n The formula for calculating the score of the label sequence is: ； Where A is the transition matrix. Indicates label y i To tag y i+1 The transition probability; The scores are normalized using softmax to obtain the overall label probability p(y|X) of the sentence. The loss function of the model is calculated using the maximum likelihood method, and the label sequence with the highest score is selected as the final entity extraction result.

Images