Text flow entity synonym real-time mining method and device

By obtaining entity mentions in text streams in real time, using pre-trained language model and Transformer's multi-head self-attention mechanism, the entity synonym collection is updated in real time, solving the problem that entity synonym mining in the prior art is difficult to capture context semantics and global semantics. Achieve efficient entity synonym mining.

CN119067104APending Publication Date: 2024-12-03ANHUI POLYTECHNIC UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410944222.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing entity synonym mining methods are difficult to capture the context semantics mentioned by entities, there are error propagation problems, and it is difficult to learn the global semantics of entity synonyms collections.

Method used

By obtaining entity mentions in text streams, using pre-trained language models to extract entity embeddings from multiple contexts, combining Transformer's multi-head self-attention mechanism, calculate the probability scores of entity mentions, and update the entity synonyms collection and embedding sequence in real time. Cache to alleviate the problem of error accumulation.

Benefits of technology

Real-time mining of entity synonyms is realized, multi-angle context semantics are captured, error accumulation problem is alleviated, and it is more flexible and efficient than traditional methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067104A_ABST
    Figure CN119067104A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of synonym recognition, in particular to a text stream entity synonym real-time mining method and device.According to the text stream entity synonym real-time mining method and device, entity mentions are extracted and embedded from multiple contexts of entity mentions through a pre-training language model, and multi-angle entity mention context semantics can be captured; according to the method, a multi-sub attention mechanism of Transform is utilized, set features are learned from extracted entity mention embedding, global semantics of a set can be better captured, and the problem of error accumulation can be relieved to a certain extent; according to the method, the text stream can be read, the entity synonym set can be mined from the text stream in real time, and compared with off-line operation of a traditional method, the method is more flexible and efficient.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of synonym mining, and in particular to a method and device for real-time mining of synonyms of text stream entities. Background Art

[0002] Entity synonyms are a concept in the fields of information retrieval, natural language processing, and knowledge graphs. They mainly represent different references to a specific entity. For example, the entity Huawei Technologies Co., Ltd. can be referred to by names such as "HUAWEI", "Huawei", and "Huawei Company", and the relationship between them is entity synonyms.

[0003] Entity synonym sets play an important role in entity-based system applications. For example, in the field of biology, these sets are crucial for understanding biological entity nouns, in question-answering systems, these sets help the system better understand user questions, and in search engines, these sets can enhance search results. Although there are already established synonym databases, the synonyms in these databases often contain many common synonyms rather than entity synonyms. In addition, such databases cannot be updated in a timely manner with the latest technical or scientific entity nouns. Therefore, a method and device for real-time mining of entity synonyms from development text streams is needed. Existing methods and devices are mainly divided into two categories, one is a statistical-based method, and the other is a deep learning-based method.

[0004] For statistical-based methods, three main techniques are used: 1) Based on contextual similarity: This type of technology mainly uses the assumption that the contexts of entity synonyms are usually semantically similar to mine entity synonyms. 2) Based on co-occurrence analysis: This type of technology assumes that entity synonyms usually appear in the same context and judges the entity synonym relationship by analyzing their co-occurrence frequency. 3) Based on pattern matching: This type of technology judges the entity synonym relationship between X and Y by identifying fixed patterns similar to "X is also called Y". For deep learning-based methods, the word embedding in the sentence is mainly calculated based on the language model, and entity synonyms are mined by establishing a model to match their semantic relationships. However, these existing methods mainly face three problems: 1) It is difficult to capture the contextual semantics of entity mentions, 2) Error propagation problem and 3) It is difficult to learn the global semantics of the entity synonym set. Summary of the invention

[0005] In view of this, the purpose of the present invention is to propose a method and device for real-time mining of text stream entity synonyms to solve the technical problems in the prior art.

[0006] Based on the above purpose, the present invention provides a method for real-time mining of synonyms of text stream entities, comprising:

[0007] Get the text stream For the text stream Each text t in i , recognize the text t i Entity mentions in ε={ε 1 ,ε 2 ,…,ε ∣ε∣}, each entity mentions ε i As key, the current text t i As a value, stored in the entity mention context cache hash table middle;

[0008] For any entity mention ε j , if the entity mentions ε j Cache hash table in entity mention context The context accumulated in The quantity is greater than the quantity threshold c s , referring to the entity ε j Add it to the entity mention set V and extract the entity mention ε j The entire context embedding sequence cache Clear the entity mention context cache hash table The entity mentioned ε j Context cache;

[0009] If the entity synonym pool For a non-empty collection, for an entity synonym collection pool Each entity synonym set S in k , both use the pre-trained entity synonym recognition model to calculate entity mention ε j Add to entity synonym set S k The probability score p add , get the maximum likelihood score maxp add , if the maximum probability score maxp add If the probability threshold λ is greater than λ, the entity is mentioned to ε j Add to the maximum likelihood score maxp add The corresponding entity synonym set S k And update the corresponding context embedding sequence cache to E pse ; otherwise, refer the entity to ε j As a single-element set {ε j}Add to the entity synonym collection pool and mention the entity ε j Context embedding sequence cache Add to the embedded sequence buffer pool L s middle;

[0010] Output entity synonym collection pool And the corresponding embedded sequence buffer pool L s .

[0011] As an optional implementation of the present invention, the method further includes:

[0012] If the entity synonym pool If it is an empty set, then the entity is mentioned to ε j As a single-element set {ε j}Add to and mention the entity ε j The entire context embedding sequence cache Add to the embedded sequence buffer pool L s middle.

[0013] As an optional implementation of the present invention, the training process of the entity synonym recognition model is:

[0014] Extract entity synonym seed data in the corpus from the knowledge base;

[0015] Constructing positive samples and negative samples based on the seed data;

[0016] Calculate the corresponding entity mention embedding in the context of each entity mention in the positive / negative samples respectively;

[0017] Calculating set features and pseudo-set features based on the entity mention embedding, and calculating quality scores of the set features and the pseudo-set features respectively;

[0018] Calculate the difference between the quality score of the set feature and the quality score of the pseudo-set feature, and convert the difference into the final predicted probability score p through the sigmoid function add , based on the probability score p add The binary cross entropy loss is calculated with the sample labels and the loss is minimized by the AdamW optimizer until the model converges.

[0019] As an optional implementation manner of the present invention, the entity synonym seed data includes: an entity synonym set and context information corresponding to each entity mention in the set.

[0020] As an optional implementation manner of the present invention, constructing positive samples and negative samples based on the seed data includes:

[0021] For each entity synonym set in the obtained entity synonym seed data, one entity mention is removed from the entity synonym set in turn, so that the remaining subset and the removed entity mention constitute a <set, candidate word> pair as a positive sample.

[0022] For each entity synonym set in the obtained entity synonym seed data, randomly select an entity mention from another entity synonym set in the entity synonym seed data, so that the current entity synonym set and the selected entity mention constitute a <set, candidate word> pair as a negative sample.

[0023] As an optional implementation of the present invention, the step of calculating the entity mention embedding corresponding to the context corresponding to each entity mention in the positive sample / negative sample includes:

[0024] For any pair of <set, candidate word> in the positive / negative samples, mark the position of the corresponding entity mention in the context corresponding to each entity mention in the set, input the context into the language model to obtain the embedding of each entity mention in the context, extract the embedding corresponding to the corresponding entity mention according to the position of the marked entity mention, and average pool all the entity mention embeddings in the context to obtain the final entity mention embedding.

[0025] As an optional implementation of the present invention, the difference is converted into a final predicted probability score p by a sigmoid function. add ,include:

[0026] p add =Sigmoid.SetScorer(H syn )-SetScorer(H pse ) / ;

[0027] Among them, all embeddings E corresponding to all entity mentions in the set syn , using a multi-layer Transformer encoder to extract the set feature H from the multi-head self-attention mechanism syn ;

[0028] The concatenation of the entity mention embeddings in the set and the entity mention embeddings of the candidate words pse , using a multi-layer Transformer encoder to extract pseudo-set features H from the multi-head self-attention mechanism pse ;

[0029] SetScorer is a set scorer, which is implemented based on a multi-layer perceptron.

[0030] As an optional implementation of the present invention, when obtaining the text stream Previously, the method also included:

[0031] Initialize entity mention set V and entity synonym set pool Embedded sequence buffer pool L sand entity mention context cache hash table

[0032] As a second aspect of the present invention, a text stream entity synonym real-time mining device is also provided, comprising:

[0033] Get module, used to get text stream For the text stream Each text t in i , recognize the text t i Entity mentions in ε={ε 1 ,ε 2 ,…,ε ∣ε∣}, each entity mentions ε i As key, the current text t i As a value, stored in the entity mention context cache hash table middle;

[0034] Identification and extraction module, for any entity mention ε j , if the entity mentions ε j Cache hash table in entity mention context The number of contexts accumulated in Greater than the quantity threshold c s , referring to the entity ε j Add it to the entity mention set V and extract the entity mention ε j The entire context embedding sequence cache Clear the entity mention context cache hash table The entity mentioned ε j Context cache;

[0035] Calculation and classification module, used for entity synonym set pool For a non-empty collection, for an entity synonym collection pool Each entity synonym set S in k , both use the pre-trained entity synonym recognition model to calculate entity mention ε j Add to entity synonym set S k The probability score p add , get the maximum probability score maxp add , if the maximum probability score maxp add If the probability threshold λ is greater than λ, the entity is mentioned to ε j Add to the maximum likelihood score maxp add The corresponding entity synonym set S k And update the corresponding context embedding sequence cache to E pse ; otherwise, refer the entity to ε jAs a single-element set {ε j}Add to the entity synonym collection pool and mention the entity ε j Context embedding sequence cache Add to the embedded sequence buffer pool L s middle;

[0036] Output module, suitable for outputting entity synonym collection pool And the corresponding embedded sequence buffer pool L s .

[0037] Beneficial effects of the present invention: In the embodiment of the present invention, firstly, entity mentions in the text are identified, and if the entity mentions ε j Cache hash table in entity mention context The context accumulated in The quantity is greater than the quantity threshold c s , referring to the entity ε j Add to the entity mention set V, if the entity synonym set pool For a non-empty collection, for an entity synonym collection pool Each entity synonym set S in k , both count entity mentions ε j Add to entity synonym set S k The probability score p add , get the maximum likelihood score maxp add , if the maximum probability score maxp add If the probability threshold λ is greater than λ, the entity is mentioned to ε j Add to the maximum likelihood score maxp add The corresponding entity synonym set S k Otherwise, refer the entity to ε j As a single-element set {ε j}Add to the entity synonym collection pool In the end, the entity synonym pool is output The present invention uses a pre-trained language model to extract entity mention embeddings from multiple contexts of entity mentions, and can capture the multi-angle entity mention context semantics; the present invention uses the multi-head sub-attention mechanism of Transformer to learn set features from the extracted entity mention embeddings, which can better capture the global semantics of the set and alleviate the error accumulation problem to a certain extent; the present invention can read text streams and mine entity synonym sets from text streams in real time, which is more flexible and efficient than the offline operation of traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flowchart of a method for real-time mining of synonyms of text stream entities in an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a calculation flow of an entity synonym recognition model according to an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of entity mention embedding extraction according to an embodiment of the present invention;

[0042] Figure 4 The figure is a flow chart of a method for real-time mining of synonyms of text stream entities in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0044] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0045] As an embodiment of the present invention, a method for real-time mining of synonyms of text stream entities is provided, comprising:

[0046] Get the text stream For the text stream Each text t in i , recognize the text t i Entity mentions in ε={ε 1 ,ε 2 ,…,ε∣ε∣}, each entity mentions ε i As key, the current text t i As a value, stored in the entity mention context cache hash table middle;

[0047] For any entity mention ε j , if the entity mentions ε j Cache hash table in entity mention context The context accumulated in The quantity is greater than the quantity threshold c s , referring to the entity ε j Add it to the entity mention set V and extract the entity mention ε j The entire context embedding sequence cache Clear the entity mention context cache hash table The entity mentioned ε j Context cache;

[0048] If the entity synonym pool For a non-empty collection, for an entity synonym collection pool Each entity synonym set S in k , both use the pre-trained entity synonym recognition model to calculate entity mention ε j Add to entity synonym set S k The probability score p add , get the maximum probability score maxp add , if the maximum probability score maxp add If the probability threshold λ is greater than λ, the entity is mentioned to ε j Add to the maximum likelihood score maxp add The corresponding entity synonym set S k And update the corresponding context embedding sequence cache to E pse ; otherwise, refer the entity to ε j As a single-element set {ε j}Add to the entity synonym collection pool and mention the entity ε j Context embedding sequence cache Add to the embedded sequence buffer pool L s middle;

[0049] Output entity synonym collection pool And the corresponding embedded sequence buffer pool L s .

[0050] In the embodiment of the present invention, firstly, entity mentions in the text are identified. If the entity mentions ε jCache hash table in entity mention context The context accumulated in The quantity is greater than the quantity threshold c s , referring to the entity ε j Add to the entity mention set V, if the entity synonym set pool For a non-empty collection, for an entity synonym collection pool Each entity synonym set S in k , both count entity mentions ε j Add to entity synonym set S k The probability score p add , get the maximum likelihood score maxp add , if the maximum probability score maxp add If the probability threshold λ is greater than λ, the entity is mentioned to ε j Add to the maximum likelihood score maxp add The corresponding entity synonym set S k Otherwise, refer the entity to ε j As a single-element set {ε j}Add to the entity synonym collection pool In the end, the entity synonym pool is output

[0051] The embodiments of the present invention have the following advantages:

[0052] (1) The present invention uses a pre-trained language model to extract entity mention embeddings from multiple contexts of entity mentions, which can capture the multi-angle context semantics of entity mentions.

[0053] (2) The present invention utilizes the multi-head sub-attention mechanism of Transformer to learn set features from the extracted entity mention embeddings, which can better capture the global semantics of the set and alleviate the error accumulation problem to a certain extent.

[0054] (3) The present invention can read text streams and mine entity synonym sets from the text streams in real time. Compared with the offline operation of traditional methods, the present invention is more flexible and efficient.

[0055] The embodiment of the present invention provides a method for real-time mining of synonyms of text stream entities. Figure 1 As shown, including:

[0056] S100, initialize entity mention set V, entity synonym set pool Embedded sequence buffer pool L s and entity mention context cache hash table

[0057] S200, get text stream For the text stream Each text t in i , recognize the text t i Entity mentions in ε={ε 1 ,ε 2 ,…,ε ∣ε∣}, each entity mentions ε i As key, the current text t i As a value, stored in the entity mention context cache hash table middle;

[0058] S300, for any entity mention ε j , if the entity mentions ε j Cache hash table in entity mention context The context accumulated in The quantity is greater than the quantity threshold c s , referring to the entity ε j Add it to the entity mention set V and extract the entity mention ε j The entire context embedding sequence cache Clear the entity mention context cache hash table The entity mentioned ε j Context cache;

[0059] S400, if entity synonym collection pool If it is an empty set, then the entity is mentioned to ε j As a single-element set {ε j}Add to and mention the entity ε j The entire context embedding sequence cache Add to the embedded sequence buffer pool L s middle.

[0060] S500, if entity synonym collection pool For a non-empty collection, for an entity synonym collection pool Each entity synonym set S in k , both use the pre-trained entity synonym recognition model to calculate entity mention ε j Add to entity synonym set S k The probability score p add , get the maximum likelihood score maxp add , if the maximum probability score maxp add If the probability threshold λ is greater than λ, the entity is mentioned to ε j Add to the maximum likelihood score maxp add The corresponding entity synonym set S kAnd update the corresponding context embedding sequence cache to E pse ; otherwise, refer the entity to ε j As a single-element set {ε j}Add to the entity synonym collection pool and mention the entity ε j Context embedding sequence cache Add to the embedded sequence buffer pool L s middle;

[0061] S600, output entity synonym collection pool And the corresponding embedded sequence buffer pool L s .

[0062] As an optional implementation, Figure 2 As shown, the training process of the entity synonym recognition model is:

[0063] S510, extracting entity synonym seed data in the corpus from the knowledge base;

[0064] The entity synonym seed data includes: an entity synonym set and context information corresponding to each entity mention in the set.

[0065] S520, constructing positive samples and negative samples based on the seed data;

[0066] As an optional implementation, S520 includes:

[0067] S521. For each entity synonym set in the obtained entity synonym seed data, remove one entity mention from the entity synonym set in turn, so that the remaining subset and the removed entity mention constitute a <set, candidate word> pair as a positive sample.

[0068] S522. For each entity synonym set in the obtained entity synonym seed data, randomly select an entity mention from another entity synonym set in the entity synonym seed data, so that the current entity synonym set and the selected entity mention constitute a <set, candidate word> pair as a negative sample.

[0069] S530, respectively calculating the entity mention embedding corresponding to the context of each entity mention in the positive sample / negative sample;

[0070] As an optional implementation, Figure 3As shown, S530 includes: for any pair of <set, candidate word> in the positive sample / negative sample, marking the position of the corresponding entity mention in the context corresponding to each entity mention in the set, inputting the context into a language model (such as BERT, BART and DeBERTa, etc.) to obtain the embedding of each entity mention in the context, extracting the embedding corresponding to the corresponding entity mention according to the position of the marked entity mention, and averaging all the entity mention embeddings in the context to obtain the final entity mention embedding.

[0071] S540, respectively calculating set features and pseudo-set features based on the entity mention embedding, and respectively calculating quality scores of the set features and the pseudo-set features;

[0072] S550, calculating the difference between the quality score of the set feature and the quality score of the pseudo-set feature, and converting the difference into the final predicted possibility score p through the sigmoid function add , based on the probability score p add The binary cross entropy loss is calculated with the sample label (positive sample is 1, negative sample is 0), and the loss is minimized by the AdamW optimizer until the model converges.

[0073] As an optional implementation, the difference is converted into a final predicted probability score p by a sigmoid function. add ,include:

[0074] p add =Sigmoid.SetScorer(H syn )-SetScorer(H pse ) / ;

[0075] Among them, all embeddings E corresponding to all entity mentions in the set syn , using a multi-layer Transformer encoder to extract the set feature H from the multi-head self-attention mechanism syn ;

[0076] The concatenation of the entity mention embeddings in the set and the entity mention embeddings of the candidate words pse , using a multi-layer Transformer encoder to extract pseudo-set features H from the multi-head self-attention mechanism pse ;

[0077] SetScorer is a set scorer, which is implemented based on a multi-layer perceptron.

[0078] Optionally, entity linking technology is used to extract entity synonym seed data in the corpus from the knowledge base.

[0079] Entity linking technology refers to identifying and extracting entity information in text by associating entity names in text with entities in knowledge base. The application of this technology enables accurate extraction of entity synonym seed data from corpus in step S510; knowledge base refers to a database containing rich entity information and its related synonyms. For example, Chinese knowledge bases such as Baidu Encyclopedia contain a large amount of entity and synonym information. Through these knowledge bases, entity recognition and synonym mining can be effectively performed to provide accurate basic data for subsequent data set construction and model training; entity synonym is a concept in the fields of information retrieval, natural language processing and knowledge graph. It mainly represents different references to a specific entity. For example, for the entity Huawei Technologies Co., Ltd., we can refer to it with names such as "HUAWEI", "Huawei" and "Huawei Company".

[0080] Entity mentions refer to entity names or references that appear in text. These entity mentions can be names of people, places, organizations, products, etc. When performing entity linking and synonym mining, you first need to identify and extract these entity mentions; the seed data contains the entity synonym set and the context information corresponding to each entity mention in the set.

[0081] For each entity mention in the set, multiple embeddings can be extracted from the corresponding multiple contexts. A candidate word also corresponds to multiple contexts, and a candidate word can extract multiple embeddings from the corresponding multiple contexts.

[0082] Corresponding to the text stream entity synonym real-time mining method, the present invention also provides a text stream entity synonym real-time mining device, such as Figure 4 As shown, including:

[0083] Acquisition module 10, used to acquire text stream For the text stream Each text t in i , recognize the text t i Entity mentions in ε={ε 1 ,ε 2 ,…,ε ∣ε∣}, each entity mentions ε i As key, the current text t i As a value, stored in the entity mention context cache hash table middle;

[0084] Identification and extraction module 20, for any entity mention ε j , if the entity mentions ε j Cache hash table in entity mention context The number of contexts accumulated in Greater than the quantity threshold c s , referring to the entity ε j Add it to the entity mention set V and extract the entity mention ε j The entire context embedding sequence cache Clear the entity mention context cache hash table The entity mentioned ε j Context cache;

[0085] The calculation and classification module 30 is used for the entity synonym set pool For a non-empty collection, for an entity synonym collection pool Each entity synonym set S in k , both use the pre-trained entity synonym recognition model to calculate entity mention ε j Add to entity synonym set S k The probability score p add , get the maximum probability score maxp add , if the maximum probability score maxp add If the probability threshold λ is greater than λ, the entity is mentioned to ε j Add to the maximum likelihood score maxp add The corresponding entity synonym set S k And update the corresponding context embedding sequence cache to E pse ; otherwise, refer the entity to ε j As a single-element set {ε j}Add to the entity synonym collection pool and mention the entity ε j Context embedding sequence cache Add to the embedded sequence buffer pool L s middle;

[0086] Output module 40, adapted to output entity synonym set pool And the corresponding embedded sequence buffer pool L s .

[0087] In the embodiment of the present invention, firstly, entity mentions in the text are identified. If the entity mentions ε j Cache hash table in entity mention context The context accumulated in The quantity is greater than the quantity threshold c s , referring to the entity ε j Add to the entity mention set V, if the entity synonym set pool For a non-empty collection, for an entity synonym collection pool Each entity synonym set S in k , both count entity mentions εj Add to entity synonym set S k The probability score p add , get the maximum likelihood score maxp add , if the maximum probability score maxp add If the probability threshold λ is greater than λ, the entity is mentioned to ε j Add to the maximum likelihood score maxp add The corresponding entity synonym set S k Otherwise, refer the entity to ε j As a single-element set {ε j}Add to the entity synonym collection pool In the end, the entity synonym pool is output

[0088] In order to facilitate the understanding of the present invention, the present invention also provides a specific embodiment.

[0089] Example

[0090] Optionally, the entity linking technology uses Freebase (https: / / developers.google.com / freebase) as the knowledge base and DBpedia Spotlight (https: / / github.com / dbpedia-spotlight) as the entity linker; the corpus data uses the open data of Wikipedia; in the benchmark data set constructed based on the seed data, the ratio of positive and negative samples is 7:3; 60% of the benchmark data set is used as a training set; 20% is used as a validation set; and 20% is used as a test set.

[0091] Optionally, the entity mention embedding extraction module obtains the pre-trained model weights from HuggingFace (https: / / huggingface.co / ); the calculation module of the set features and pseudo-set features is constructed using the Transformer model provided by the Pytorch library (https: / / pytorch.org / ); SetScorer is a set scorer, which is implemented based on a multi-layer perceptron.

[0092] Optionally, the entity synonym recognition model is trained by an AdamW optimizer with a learning rate of 5e-5; the model is evaluated using multiple indicators including adjusted mutual information (AMI), Fulks and Malos index (FMI), recall, precision and F1 score (F1), and the model weight with the best performance is finally saved.

[0093] Optionally, the text stream is real-time text data from a social platform.

[0094] Assuming there is a text stream as follows:

[0095] t 1 :Huawei released its latest smartphone;

[0096] t 2 :Many people choose Huawei's electronic products;

[0097] t 3 :Huawei is actively expanding its overseas markets;

[0098] t 4 :Xiaomi has achieved remarkable results in the international market;

[0099] t 5 :Xiaomi's new product launch attracted widespread attention;

[0100] t 6 :Brown rice is as healthy as millet;

[0101] t 7 : Traditional millet is an important material for making whole grain foods;

[0102] t 8 :Millet is used as a staple food in many places;

[0103] t 9 :Huawei's R&D team has made breakthrough progress in 5G technology.

[0104] t 10 Xiaomi has launched a powerful smartwatch that has been widely praised by users.

[0105] t 11 Farmers harvested a large amount of millet in the fall and prepared to make healthy food.

[0106] The specific processing flow of the online entity synonym set mining module is as follows:

[0107] 1) If Figure 1 As described in s = 2 and threshold λ = 0.5, using BERT as the pre-trained model for the entity mention embedding extraction module. The online entity synonym set mining module first traverses the text stream Identify entity mentions in text using named entity recognition tools;

[0108] 2) Processing 1 Get, "<ε 1 > Huawei < / ε 1 >Launched the latest smartphone. ”;

[0109] 3) Storage ε 1 The entity mention context cache hash table,

[0110] 4) The size is 1, which does not meet the threshold c s ;

[0111] 5)t 1 End of entity mention traversal in ;

[0112] 6) Processing 2 Get, "Many people choose <ε 1 > Huawei < / ε 1 > electronic products. ";

[0113] 7) Storage ε 1 The entity mention context cache hash table,

[0114] 8) The size is 2, reaching the threshold c s , add "Huawei" to the entity mention set V;

[0115] 9) 1 : "[START]Huawei released its latest smartphone.",t 2 "Many people chose electronic products from [START] Huawei ." Input into the entity mention embedding extraction module to get the entity mention embedding of the entity mention "Huawei"

[0116] 10) Clear

[0117] 11) If it is empty, add {"Huawei Company"} In the 华为公司 Join L s At this time, L s =[E 华为公司 ], V = [“Huawei”];

[0118] 12)t 2 End of entity mention traversal in ;

[0119] 13) Processing 3 Get, "<ε 1 > Huawei < / ε 1 >We are actively expanding overseas markets. ”;

[0120] 14) Storage ε 1 The entity mention context cache hash table,

[0121] 15) The size is 1, which does not meet the threshold c s ;

[0122] 16) Processing 4 Get, "<ε 1 >Xiaomi < / ε 1 >Achieved remarkable results in the international market. ";

[0123] 17) Storage ε 1 The entity mention context cache hash table,

[0124] 18) The size is 1, which does not meet the threshold c s ;

[0125] 19) Processing 5 Get, "<ε 1 > Xiaomi < / ε 1 >The launch of the new product attracted wide attention. ”;

[0126] 20) Storage ε 1 The entity mention context cache hash table,

[0127] twenty one) The size is 1, which does not meet the threshold c s ;

[0128] 22) Processing 6 Get, "<ε 1 >Brown Rice < / ε 1 > and <ε 2 > Xiaomi < / ε 2 >same health;. ”;

[0129] 23) Storage ε 1 The entity mention context cache hash table,

[0130] twenty four) The size is 1, which does not meet the threshold c s ;

[0131] 25) Storage ε 2 The entity mention context cache hash table,

[0132] 26) The size is 2, reaching the threshold c s , add "Xiaomi" to the entity mention set V;

[0133] 27) Add t 5 : "[START]Xiaomi 's new product launch has attracted wide attention.", t 6 : "Brown rice is as healthy as [START]Xiaomi ." into the entity mention embedding extraction module to obtain the entity mention embedding of the entity mention "Xiaomi"

[0134] 28) Clear

[0135] 29) is not empty, traverse Take E 华为公司 as input to the set encoding module, as input to the pseudo-set encoding module, and respectively obtain and Through the set scoring module, the probability p of adding "Xiaomi" to the set {"Huawei Company"} can be calculated add1 = 0.2;

[0136] 30) After the traversal ends, the maximum value p add1 is less than the threshold λ, add {"Xiaomi"} to in, add E 小米 to L s in. At this time, L s = [E 华为公司 , E 小米 , V = ["Huawei Company", "Xiaomi"];

[0137] 31) After the traversal of the current entity mention ends, continue to read the text stream;

[0138] 32) Process t 7 to get, "Traditional <ε 1 >foxtail millet< / ε 1 >is an important material for making coarse grain foods;.";

[0139] 33) Store the entity mention context cache hash table of ε 1 ,

[0140] 34) The size is 1, not meeting the threshold c s ;

[0141] 35) Process t8 Get, "<ε 1 >Corn < / ε 1 >It is used as a staple food in many places;. ”;

[0142] 36) Storage ε 1 The entity mention context cache hash table,

[0143] 37) The size is 1, which does not meet the threshold c s ;

[0144] 38) Processing 9 Get, "<ε 1 > Huawei < / ε 1 >'s R&D team is in <ε 2 >5G < / ε 2 >Technological breakthroughs have been made. ";

[0145] 39) Storage ε 1 The entity mention context cache hash table,

[0146] 40) The size is 2, which satisfies the threshold c s , add "Huawei" to the entity mention set V;

[0147] 41) t 3 :"[START]Huawei is actively expanding its overseas markets.",t 9 : "[START]Huawei 's R&D team has made breakthrough progress in 5G technology." Input into the entity mention embedding extraction module to get the entity mention embedding of "Huawei"

[0148] 42) Clear

[0149] 43) Not empty, traverse E 华为公司 As Input set encoding module, As Input the pseudo-set encoding module and get and The set scoring module can calculate the probability of "Huawei" being added to the set {"Huawei Company"}

[0150]

[0151] 44) E 小米 As Input set encoding module, As Input the pseudo-set encoding module and get and The probability of "Huawei" being added to the set {"Xiaomi"} can be calculated through the set scoring module.

[0152] 45) End of traversal, maximum value If it is greater than the threshold λ, update the set {“Huawei”} to {“Huawei”, “Huawei”}, and update L s [1] is 0E 华为公司 E 华为 1. At this time, {"Xiaomi"}], L s =[0E 华为公司 E 华为 1.E 小米 ], V = ["Huawei", "Xiaomi", "Huawei"];

[0153] 46) Storage ε 2 The entity mention context cache hash table,

[0154] 47) The size is 1, which does not meet the threshold c s ;

[0155] 48) Processing 10 Get, "<ε 1 >Xiaomi < / ε 1 >Launched a powerful smart watch that was well received by users. ”;

[0156] 49) Storage ε 1 The entity mention context cache hash table,

[0157] 50) The size is 2, which satisfies the threshold c s , ,Add “Xiaomi” to the entity mention set V;

[0158] 51) 4 :"[START]Xiaomi has achieved remarkable results in the international market.",t 10 : "[START]Xiaomi launched a powerful smartwatch that has been widely praised by users." Input into the entity mention embedding extraction module to get the entity mention embedding of the entity mention "Xiaomi"

[0159] 52) is not empty, traverse Take 0E 华为公司 E 华为 1 as the input set encoding module, as the input pseudo-set encoding module, and respectively obtain and Through the set scoring module, the probability that "Xiaomi" is added to the set {"Huawei Technologies Co., Ltd.", "Huawei"} can be calculated

[0160] 53) Take E 小米 as E syn2 the input set encoding module, as the input pseudo-set encoding module, and respectively obtain and Through the set scoring module, the probability that "Xiaomi" is added to the set {"Xiaomi"} can be calculated

[0161] 54) The traversal ends, and the maximum value is greater than the threshold λ, update the set {"Xiaomi"} to {"Xiaomi", "Xiaomi"}, and update L s [2] to 0E 小米 E Xiaomi 1. At this time, {"Xiaomi", "Xiaomi"}], L s =[0E 华为公司 E 华为 1, 0E 小米 E Xiaomi 1], V = ["Huawei Technologies Co., Ltd.", "Xiaomi", "Huawei", "Xiaomi"];

[0162] 55) The traversal of the current entity mention ends, and continue to read the text stream;

[0163] 56) Process t 11 to get, "The farmers harvested a large amount of <ε 1 >foxtail millet< / ε 1 > in autumn and prepared to make healthy food.";

[0164] 57) Store the entity mention context cache hash table of ε 1

[0165] 58) ​The size is 2, which satisfies the threshold c s , add "grain" to the entity mention set V;

[0166] 59) 7 : "Traditionally [START] millet is an important material for making whole grain foods.", t 11 : "Farmers harvested a lot of [START] millet in the fall and prepared to make healthy food." Input into the entity mention embedding extraction module to get the entity mention embedding of the entity mention "millet"

[0167] 60) Clear

[0168] 61) Not empty, traverse 0E 华为公司 E 华为 1 as Input set encoding module, As Input the pseudo-set encoding module and get and The set scoring module can calculate the probability of "Guzi" being added to the set {"Huawei Company", "Huawei"}

[0169] 62) E 小米 As Input set encoding module, As Input the pseudo-set encoding module and get and The set scoring module can calculate the probability of "millet" being added to the set {"Xiaomi", "Xiaomi"}

[0170] 63) End of traversal, maximum value is less than the threshold λ, add {"grain"} In the 谷子 Join L s At this time, {"Xiaomi", "Xiaomi"}, {"Millet"}], L s =[0E 华为公司 E 华为 1,0E 小米 E Xiaomi 1.E 谷子 ], V = ["Huawei", "Xiaomi", "Huawei", "Xiaomi", "Guzi"];

[0171] 64) The text stream reading is completed;

[0172] The set of entity synonyms mined from the text stream is [{"Huawei Technologies Co., Ltd.", "Huawei"}, {"Xiaomi", "Xiaomi"}, {"Foxtail millet"}], and it should be particularly noted that the "Xiaomi" extracted in step 51) has multiple embedded semantics, representing the semantics of a company and food respectively. In step 52), since the present invention has the ability to capture the global semantics of the set, when the pseudo-set encoding module processes the embedding 0E 小米 E Xiaomi 1, it tends to represent the semantics of a company, and the score obtained for adding to the set {"Xiaomi"} is relatively high. In step 61), since "Xiaomi" has been added to the set {"Xiaomi"} representing the semantics of a company, when calculating the score for adding "Foxtail millet" to the set {"Xiaomi", "Xiaomi"}, the obtained score is relatively low. This shows the improvement of the present invention for the problem of error accumulation.

[0173] Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present invention (including the claims) is limited to these examples; under the concept of the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the present invention as described above, which are not provided in detail for the sake of brevity.

[0174] The present invention aims to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A real-time text stream entity synonym mining method, characterized in that: include: Get the text stream For the text stream Each text t in i , recognize the text t i The entity mentions in ε={ε1,ε2,…,ε ∣ε∣ }, each entity mentions ε i As key, the current text t i As a value, stored in the entity mention context cache hash table middle; For any entity mention ε j , if the entity mentions ε j Cache hash table in entity mention context The context accumulated in The quantity is greater than the quantity threshold c s , referring to the entity ε j Add it to the entity mention set V and extract the entity mention ε j The entire context embedding sequence cache Clear the entity mention context cache hash table The entity mentioned ε j Context cache; If the entity synonym pool For a non-empty collection, for an entity synonym collection pool Each entity synonym set S in k , both use the pre-trained entity synonym recognition model to calculate entity mention ε j Add to entity synonym set S k The probability score p add , get the maximum likelihood score maxp add , if the maximum probability score maxp add If the probability threshold λ is greater than λ, the entity is mentioned to ε j Add to the maximum likelihood score maxp add The corresponding entity synonym set S k And update the corresponding context embedding sequence cache to E pse ; otherwise, refer the entity to ε j As a single-element set {ε j }Add to the entity synonym collection pool and mention the entity ε j Context embedding sequence cache Add to the embedded sequence buffer pool L s middle; Output entity synonym collection pool And the corresponding embedded sequence buffer pool L s .

2. The method for real-time mining of synonyms of text stream entities according to claim 1, characterized in that: The method further comprises: If the entity synonym pool If it is an empty set, then the entity is mentioned to ε j As a single-element set {ε j }Add to and mention the entity ε j The entire context embedding sequence cache Add to the embedded sequence buffer pool L s middle.

3. The method for real-time mining of synonyms of text stream entities according to claim 1, characterized in that: The training process of the entity synonym recognition model is as follows: Extract entity synonym seed data in the corpus from the knowledge base; Constructing positive samples and negative samples based on the seed data; Calculate the corresponding entity mention embedding in the context of each entity mention in the positive / negative samples respectively; Calculating set features and pseudo-set features based on the entity mention embedding, and calculating quality scores of the set features and the pseudo-set features respectively; Calculate the difference between the quality score of the set feature and the quality score of the pseudo-set feature, and convert the difference into the final predicted probability score p through the sigmoid function add , based on the probability score p add The binary cross entropy loss is calculated with the sample labels and the loss is minimized by the AdamW optimizer until the model converges.

4. The method for real-time mining of synonyms of text stream entities according to claim 3, characterized in that: The entity synonym seed data includes: an entity synonym set and context information corresponding to each entity mention in the set.

5. The method for real-time mining of synonyms of text stream entities according to claim 3, characterized in that: The constructing positive samples and negative samples based on the seed data includes: For each entity synonym set in the obtained entity synonym seed data, one entity mention is removed from the entity synonym set in turn, so that the remaining subset and the removed entity mention constitute a <set, candidate word> pair as a positive sample. For each entity synonym set in the obtained entity synonym seed data, randomly select an entity mention from another entity synonym set in the entity synonym seed data, so that the current entity synonym set and the selected entity mention constitute a <set, candidate word> pair as a negative sample.

6. The method for real-time mining of synonyms of text stream entities according to claim 4, characterized in that: The calculating the entity mention embedding corresponding to the context of each entity mention in the positive sample / negative sample includes: For any pair of <set, candidate word> in the positive / negative samples, mark the position of the corresponding entity mention in the context corresponding to each entity mention in the set, input the context into the language model to obtain the embedding of each entity mention in the context, extract the embedding corresponding to the corresponding entity mention according to the position of the marked entity mention, and average pool all the entity mention embeddings in the context to obtain the final entity mention embedding.

7. The method for real-time mining of synonyms of text stream entities according to claim 6, characterized in that: The difference is converted into the final predicted probability score p through the sigmoid function add ,include: p add =Sigmoid.SetScorer(H syn )-SetScorer(H pse ) / ; Among them, all embeddings E corresponding to all entity mentions in the set syn , using a multi-layer Transformer encoder to extract the set feature H from the multi-head self-attention mechanism syn ; The concatenation of the entity mention embeddings in the set and the entity mention embeddings of the candidate words pse , using a multi-layer Transformer encoder to extract pseudo-set features H from the multi-head self-attention mechanism pse ; SetScorer is a set scorer, which is implemented based on a multi-layer perceptron.

8. The method for real-time mining of synonyms of text stream entities according to claim 1, characterized in that: Getting the text stream Previously, the method also included: Initialize entity mention set V and entity synonym set pool Embedded sequence buffer pool L s and entity mention context cache hash table 9. A real-time text stream entity synonym mining device, characterized in that: include: Get module, used to get text stream For the text stream Each text t in i , recognize the text t i The entity mentions in ε={ε1,ε2,…,ε ∣ε∣ }, each entity mentions ε i As key, the current text t i As a value, stored in the entity mention context cache hash table middle; Identification and extraction module, for any entity mention ε j , if the entity mentions ε j Cache hash table in entity mention context The number of contexts accumulated in Greater than the quantity threshold c s , referring to the entity ε j Add it to the entity mention set V and extract the entity mention ε j The entire context embedding sequence cache Clear the entity mention context cache hash table The entity mentioned ε j Context cache; Calculation and classification module, used for entity synonym set pool For a non-empty collection, for an entity synonym collection pool Each entity synonym set S in k , both use the pre-trained entity synonym recognition model to calculate entity mention ε j Add to entity synonym set S k The probability score p add , get the maximum likelihood score maxp add , if the maximum probability score maxp add If the probability threshold λ is greater than λ, the entity is mentioned to ε j Add to the maximum likelihood score maxp add The corresponding entity synonym set S k And update the corresponding context embedding sequence cache to E pse ; otherwise, refer the entity to ε j As a single-element set {ε j }Add to the entity synonym collection pool and mention the entity ε j Context embedding sequence cache Add to the embedded sequence buffer pool L s middle; Output module, suitable for outputting entity synonym collection pool And the corresponding embedded sequence buffer pool L s .