Text entity disambiguation method, device, electronic device and storage medium

Abstract

The present invention discloses a method, device, electronic device and storage medium for text entity disambiguation. The method comprises: identifying all literals in an input short text, generating candidate entities corresponding to each literal using a set corpus or knowledge base; calculating the matching score of the candidate entity corresponding to each literal using the text similarity between each literal and the corresponding candidate entity, and the correlation between each literal context information and the corresponding candidate entity; calculating the prior probability of each potential entity literal after text segmentation using the candidate entity with the highest matching score; performing secondary text segmentation on the input short text using each prior probability to obtain each literal entity; and determining the disambiguation entity corresponding to the target literal entity based on the consistency value between the candidate entity corresponding to the selected literal entity and the candidate entity corresponding to the target literal entity. The technical solution of the present invention can eliminate the ambiguity of entities in high-information-density text, thereby improving the speed and accuracy of entity disambiguation.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph technology, and in particular to a text entity disambiguation method, device, electronic device and storage medium. Background Art

[0002] Entity disambiguation has become a key technology in real-world applications, such as text information extraction and question-answering systems. However, due to the characteristics of natural language, entities in text are often ambiguous and can refer to related entities in multiple knowledge graphs. Therefore, it is crucial to accurately associate entities in text with entities in knowledge graphs and disambiguate entities in text.

[0003] In the existing technology, entity disambiguation is usually completed by local or global methods through supervised learning or unsupervised learning. However, if supervised learning is used, a large amount of training data is required. Since the training data of existing supervised learning methods usually only have English datasets and are not applicable to

[0004] High-information-density texts have a high density of information, so a high-quality annotated dataset is required. However, this is very costly. Unsupervised learning, typically based on a global approach, is often used. However, due to the strong semantic consistency of high-information-density texts, the distinction between entities is low when using a global approach for disambiguation, resulting in reduced accuracy. Therefore, how to disambiguate entities in high-information-density texts and improve the speed and accuracy of text entity disambiguation is a pressing issue. Summary of the Invention

[0005] The present invention provides a text entity disambiguation method, device, electronic device and storage medium, which can eliminate the ambiguity of entities in high-information-density text and improve the speed and accuracy of text entity disambiguation.

[0006] According to one aspect of the present invention, a text entity disambiguation method is provided, comprising:

[0007] Identify all literals in the input short text and generate candidate entities corresponding to each literal using a set corpus or knowledge base;

[0008] The matching score of the candidate entity corresponding to each literal is calculated by using the text similarity between each literal and the corresponding candidate entity, and the relevance between the context information of each literal and the corresponding candidate entity;

[0009] The candidate entity with the highest matching score is used to calculate the prior probability of each potential entity literal after text segmentation;

[0010] Use the prior probabilities to perform secondary text segmentation on the input short text to obtain the literal entities;

[0011] The disambiguation entity corresponding to the target literal entity is determined according to the consistency values ​​of the candidate entities corresponding to the to-be-selected literal entities in each literal entity and the candidate entities corresponding to the target literal entity.

[0012] According to another aspect of the present invention, a text entity disambiguation device is provided, comprising:

[0013] The candidate entity generation module is used to identify all literals in the input short text and generate candidate entities corresponding to each literal using a set corpus or knowledge base;

[0014] A matching score calculation module is used to calculate the matching score of the candidate entity corresponding to each literal quantity by using the text similarity between each literal quantity and the corresponding candidate entity, and the relevance between the context information of each literal quantity and the corresponding candidate entity;

[0015] A priori probability calculation module is used to calculate the priori probability of each potential entity literal after text segmentation using the candidate entity with the highest matching score;

[0016] The literal entity determination module is used to perform secondary text segmentation on the input short text using various prior probabilities to obtain various literal entities;

[0017] The disambiguation entity determination module is used to determine the disambiguation entity corresponding to the target literal entity based on the consistency values ​​of the candidate entities corresponding to the to-be-selected literal entities in each literal entity and the candidate entities corresponding to the target literal entity.

[0018] According to another aspect of the present invention, an electronic device is provided, comprising:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the text entity disambiguation method described in any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text entity disambiguation method described in any embodiment of the present invention when executed.

[0023] The technical solution of the embodiment of the present invention generates candidate entities corresponding to each literal based on all literals in the input short text, further calculates the matching score of the candidate entity corresponding to each literal using the text similarity between each literal and the corresponding candidate entity, and the correlation between each literal context information and the corresponding candidate entity, and then uses the candidate entity with the highest matching score to calculate the prior probability of each potential entity literal after text segmentation, and uses each prior probability to perform secondary text segmentation on the input short text to obtain each literal entity; finally, according to the consistency value between the candidate entity corresponding to the to-be-selected literal entity in each literal entity and the candidate entity corresponding to the target literal entity, the disambiguation entity corresponding to the target literal entity is determined, which solves the problem of low entity disambiguation speed and accuracy in high-information-density text in the prior art, can eliminate the ambiguity of entities in high-information-density text, and improves the speed and accuracy of entity disambiguation in high-information-density text.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 This is a flowchart of a text entity disambiguation method provided according to the first embodiment of the present invention;

[0027] Figure 2a This is a flowchart of a text entity disambiguation method provided according to the second embodiment of the present invention;

[0028] Figure 2b This is a structural diagram of a text entity disambiguation process provided in Example 2 of the present invention;

[0029] Figure 2c 1 is a flow chart of a text entity disambiguation method provided according to the second embodiment of the present invention;

[0030] Figure 3 2 is a schematic diagram of the structure of a text entity disambiguation device provided according to the third embodiment of the present invention;

[0031] Figure 4 It is a structural diagram of an electronic device for implementing the text entity disambiguation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "target", "original", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Example 1

[0035] Figure 1 A flowchart of a text entity disambiguation method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of disambiguating entities in high-information-density text. The method can be executed by a text entity disambiguation device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0036] S110 , identifying all literals in the input short text, and generating candidate entities corresponding to each literal using a set corpus or knowledge base.

[0037] The input short text may refer to a text sequence in an input high-information-density text, for example, a sentence in a high-information-density text; a high-information-density text may refer to a text with less distinction between text contents, such as a news text.

[0038] A literal can refer to a text description fragment displayed in the input short text. For example, it can be all connected characters in the input short text. It is worth noting that literals may overlap, for example, a long literal may contain one or more short literals.

[0039] An entity can refer to a word with actual meaning, usually a noun. A candidate entity can refer to an entity that may have a literal meaning.

[0040] The pre-set corpus can refer to a pre-obtained disambiguation page from an encyclopedia website or a table of literal-entity pairs pointed to by hyperlinks in text within the encyclopedia website, i.e., an entity-literal mapping table. The knowledge base can be a pre-set database containing a dictionary of entity literals. Using this corpus or knowledge base, each literal can be matched to a corresponding candidate entity.

[0041] Therefore, by identifying the literals in the input short text, all the literals in the input short text can be obtained, and then, using the set corpus or knowledge base, preliminary matching candidate entities can be generated for each literal, providing an effective basis for subsequent operations.

[0042] S120 , calculating a matching score of the candidate entity corresponding to each literal quantity by using the text similarity between each literal quantity and the corresponding candidate entity, and the relevance between the context information of each literal quantity and the corresponding candidate entity.

[0043] Text similarity can refer to the similarity between a literal and the relevant information of the corresponding candidate entity; relevance can refer to the degree of association between the relevant information of the candidate entity and the context of the literal. The match score can be a numerical value describing the degree of match between the candidate entity and the literal. A higher match score indicates that the candidate entity is more able to express the meaning of the corresponding literal and eliminate ambiguity of the corresponding literal.

[0044] In an optional embodiment, the text similarity between each literal and the corresponding candidate entity, and the relevance between each literal context information and the corresponding candidate entity, are used to calculate the matching score of the candidate entity corresponding to each literal, including: using the formula: φ(m,e)=f(m,e)·g(m,e) to calculate the matching score of the candidate entity corresponding to each literal; wherein m is the literal, e is the candidate entity, f(m,e) is determined by the text similarity between the literal and the corresponding candidate entity; g(m,e) is determined by the relevance between the literal context information and the corresponding candidate entity; represents the prior probability of the literal-candidate entity pair, Indicates the character similarity between the literal and the candidate entity; E m is the candidate entity set composed of candidate entities, and β is the distribution parameter set according to the source of the candidate entity set.

[0045] Among them, f(m,e) is a score that is independent of the context, sim(m,e) represents the character similarity between the literal and the candidate entity. The higher the similarity, the higher the matching score will be. s Indicates the cost of deleting candidate entities and inserting literals, usually ws ∈[0,1], in w s = 1, the similarity score is the standard edit distance string similarity; in w s = 0, the similarity score has nothing to do with string comparison. It is worth noting that if the candidate entities are obtained from the knowledge base, β can be set to a uniform distribution.

[0046] Among them, g(m,e) is a score related to the context, which can be achieved through unsupervised learning methods, g(m,e)=sim t (m,e)·(1-ε+ε·sim c (m,e));sim c (m,e)=cos(v c (m),v c (e)) is the scoring function for topic consistency, which reflects the topic inertia; The concept vector representing the context, CT(m) is the context word set of the literal m; D(w,m) is the distance function used to weaken the influence of words that are far away from the literal; v c (w) represents the conceptual calculation method of vocabulary, which can usually be expressed as n(w,e) represents the number of times the word w appears in the article describing the candidate entity e, r(e,c) represents the weight of the candidate entity e and the concept c, and for a knowledge base without concept weights, it can be taken as 1. t (m,e) represents the text similarity scoring function, KP(e) represents the keyword set of candidate entity e, v w (·) represents the global word vector. The parameter ε is used to adjust the influence of the topic inertia on the scoring results. The larger the parameter ε is, the smaller the influence of the topic coherence on the scoring results is.

[0047] It is worth noting that the keyword set KP(e) for the candidate entity e can be composed of three parts: concept, attribute and web page text; the concept can refer to the topic corresponding to the current candidate entity, such as celebrity or technology; the attribute can refer to the attribute and attribute value with the highest frequency of occurrence in multiple articles of the candidate entity; the web page text can be the web page text with the greatest relevance to the candidate entity. For example, it can be calculated according to the formula Calculate, where D e is the webpage text describing the candidate entity e; avgdl is the average length of all webpage texts in the encyclopedia website; k1 and b can refer to adjustment parameters. For example, k1 can be set to 2 and b can be set to 0.75; It can refer to the frequency of word w in the web page text, para∈D eIt can refer to the paragraphs in the webpage text, n(w,para) can refer to the number of times word w appears in para, offset(para) can refer to the number of times para appears in D e The offset value in is used to reduce the impact of words that appear later in the web page text.

[0048] Therefore, by using the context-independent score and the context-dependent score, the disambiguation task of each literal is treated as a separate task, and the matching score of each candidate entity for each literal can be calculated, providing an effective basis for subsequent operations.

[0049] S130 , using the candidate entity with the highest matching score to calculate the prior probability of each potential entity literal after text segmentation.

[0050] Among them, text segmentation can refer to the literal segmentation of the input short text after calculating the matching score of each candidate entity. Potential entity literals can refer to literals that are tagged as nouns and may meet the semantic requirements of the text. They can usually be determined based on the matching scores of the candidate entities corresponding to the literals. Specifically, if there are candidate entities with a matching score of zero among the candidate entities corresponding to each literal, the literals corresponding to the candidate entities with a matching score of zero are eliminated, and the remaining literals are used as potential entity literals.

[0051] The prior probability can refer to the probability that the potential entity literal corresponding to the candidate entity with the highest matching score is split into a literal, calculated based on the popularity of the candidate entity with the highest matching score, that is, the number of views of the candidate entity page in the encyclopedia website or the number of linked entities of the candidate entity page. Generally, the higher the prior probability, the greater the probability that the corresponding potential entity literal is regarded as the true literal.

[0052] Therefore, by utilizing the popularity of the candidate entity with the highest matching score, the semantic compatibility between the text fragment and the context can be taken into consideration, and then by calculating the prior probability of each potential entity literal, it is possible to promptly discover the situation where literals that should be connected together are split into two or more literals, providing an effective basis for the subsequent accurate identification of literal entities in the input short text.

[0053] S140 , using each prior probability to perform secondary text segmentation on the input short text to obtain each literal entity.

[0054] The literal entity may refer to an actual text segmentation literal result that meets the semantic requirements in the input short text.

[0055] Specifically, the prior probability with the largest value can be screened out from the prior probabilities of multiple potential entity literals with relatively large character similarity, and the corresponding function of the original text segmentation formula can be replaced with the maximum value prior probability. In this way, literals that do not meet the semantic requirements can be filtered out, and the literal entities that actually need to be entity linked in the input short text can be obtained.

[0056] S150 , determining a disambiguation entity corresponding to the target literal entity based on consistency values ​​between candidate entities corresponding to the to-be-selected literal entities in each literal entity and candidate entities corresponding to the target literal entity.

[0057] The target literal entity may refer to a literal entity that is undergoing entity disambiguation; the candidate literal entity may refer to a literal entity that has not undergone entity disambiguation or has undergone entity disambiguation. The consistency value may refer to the correlation or similarity value between the candidate entity corresponding to the candidate literal entity and the candidate entity corresponding to the target literal entity. The consistency value can be used to filter out the candidate entity that best matches the target literal entity from multiple candidate entities corresponding to the target literal entity. The disambiguated entity may refer to the candidate entity that is most capable of disambiguating the target literal entity.

[0058] Therefore, by calculating the consistency value between the candidate entity corresponding to the to-be-selected literal entity and the candidate entity corresponding to the target literal entity, the candidate entity corresponding to the target literal entity with the maximum consistency value is screened out as the disambiguation entity corresponding to the target literal entity, so as to complete the disambiguation task of the target literal entity in the input short text through the disambiguation entity.

[0059] The technical solution of the embodiment of the present invention generates candidate entities corresponding to each literal based on all literals in the input short text, further calculates the matching score of the candidate entity corresponding to each literal using the text similarity between each literal and the corresponding candidate entity, and the correlation between each literal context information and the corresponding candidate entity, and then uses the candidate entity with the highest matching score to calculate the prior probability of each potential entity literal after text segmentation, and uses each prior probability to perform secondary text segmentation on the input short text to obtain each literal entity; finally, according to the consistency value between the candidate entity corresponding to the to-be-selected literal entity in each literal entity and the candidate entity corresponding to the target literal entity, the disambiguation entity corresponding to the target literal entity is determined, which solves the problem of low entity disambiguation speed and accuracy in high-information-density text in the prior art, can eliminate the ambiguity of entities in high-information-density text, and improves the speed and accuracy of entity disambiguation in high-information-density text.

[0060] Example 2

[0061] Figure 2aThis is a flowchart of a text entity disambiguation method provided in Example 2 of the present invention. This embodiment is refined based on the above embodiment. In this embodiment, the operation of determining the disambiguation entity corresponding to the target literal entity based on the consistency value between the candidate entity corresponding to the selected literal entity in each literal entity and the candidate entity corresponding to the target literal entity is refined. Specifically, it may include: using the formula: Calculate the entity disambiguation probability value; where coh(e i ,e j ) represents the consistency value of the candidate entity corresponding to the candidate literal entity to be selected and the candidate entity corresponding to the target literal entity; Γ is the candidate entity set composed of each candidate entity corresponding to the candidate literal entity and each candidate entity corresponding to the target literal entity; N represents the number of literal entities in the input short text; the candidate entity corresponding to the target literal entity in the maximum entity disambiguation probability value is used as the disambiguated entity corresponding to the target literal entity. Figure 2a As shown, the method includes:

[0062] S210: Identify all literals in the input short text, and generate candidate entities corresponding to each literal using a set corpus or knowledge base.

[0063] S220 , calculating a matching score of the candidate entity corresponding to each literal quantity by using the text similarity between each literal quantity and the corresponding candidate entity, and the relevance between the context information of each literal quantity and the corresponding candidate entity.

[0064] S230, using the original text cutting formula: Perform text segmentation on the input short text to obtain potential entity literals.

[0065] Among them, P(t i ) is the cutting literal t i The probability of freq(t i ) is the literal t i T is the number of literals in the input short text.

[0066] S240. Use formula: Calculate the prior probability of each potential entity literal after text segmentation.

[0067] Among them, e k is the candidate entity with the highest matching score; pop(e) represents the popularity of the candidate entity; λ is the prior coefficient.

[0068] The popularity of a candidate entity can usually be measured by the number of page views of the candidate entity in the encyclopedia website or the number of entities linked to the candidate entity page. In the embodiment of the present invention, the number of page views of the candidate entity page can be used as the popularity of the candidate entity. j pop(e j ) can refer to the sum of the popularity of all candidate entities corresponding to each potential entity literal, and j represents the number of candidate entities corresponding to the potential entity literal.

[0069] It is worth noting that P(t i ) can be pre-generated offline, which can greatly improve the processing speed of the system.

[0070] S250: Replace the literal probability in the original text segmentation formula with the target prior probability in each prior probability to generate a target text segmentation formula.

[0071] The target prior probability may refer to the largest prior probability among the prior probabilities of multiple potential entity literals with relatively high character similarity. The target text segmentation formula may refer to the text segmentation formula after the literal probabilities in the original text segmentation formula are replaced with the target prior probability.

[0072] S260: Use the target text segmentation formula to perform secondary text segmentation on the input short text to obtain each literal entity.

[0073] S270. Use formula: Calculate entity disambiguation probability values.

[0074] Among them, coh(e i ,e j ) represents the consistency value between the candidate entity corresponding to the to-be-selected literal entity and the candidate entity corresponding to the target literal entity; Γ is the candidate entity set consisting of the candidate entities corresponding to the to-be-selected literal entity and the candidate entities corresponding to the target literal entity; N represents the number of literal entities in the input short text.

[0075] The entity disambiguation probability value may represent the probability that the candidate entity is used to perform entity disambiguation on the target literal entity.

[0076] In practical applications, in order to ensure computational efficiency, the search space can be kept below 1000. For example, if the input short text contains 3 literal entities, the candidate entities ranked in the top 10 matching scores in each literal entity are taken, so that the scale of the search space is 10*10*10; among them, if the third literal entity contains only one candidate entity, the candidate entities ranked in the top 31 matching scores in the first two literal entities are taken, so that the scale of the search space is 31*31*1.

[0077] In an optional implementation, the method may further include: using the formula:

[0078] coh(e i ,e j )=γ·rel(e i ,e j )+(1-γ)·sim(e i ,e j ) calculates the consistency value of the candidate entity corresponding to the selected literal entity and the candidate entity corresponding to the target literal entity; where rel(e i ,e j ) is the correlation function between the candidate entity corresponding to the to-be-selected literal entity and the candidate entity corresponding to the target literal entity; sim(e i ,e j ) is the similarity function between the candidate entity corresponding to the to-be-selected literal entity and the candidate entity corresponding to the target literal entity; γ is a parameter for measuring the correlation function and the similarity function.

[0079] in, E1 and E2 may be corresponding entity sets with pages linking to candidate entities e1 and e2, which are counted from corresponding encyclopedia pages. E may refer to the set of all candidate entities.

[0080] rel(e1,e2)=max(rel'(e1,e2),rel'(e2,e1)), rel'(e1,e2) represents the consistency relationship between two candidate entities. R(e1,e2) can represent the set of relationships between candidate entities e1 and e2 in the knowledge graph, H(r,e) can refer to the set of head candidate entities that have relationship r with tail candidate entity e, and T(e,r) can refer to the set of tail candidate entities that have relationship r with head candidate entity e. When there are many candidate entities that have relationship r with the head candidate entity or the tail candidate entity, relationship r can be considered a weak relationship. Conversely, if there are few candidate entities that have relationship r with the head candidate entity or the tail candidate entity, relationship r is a strong relationship.

[0081] S280. The candidate entity corresponding to the target literal entity in the maximum entity disambiguation probability value is used as the disambiguation entity corresponding to the target literal entity.

[0082] The technical solution of the embodiment of the present invention generates candidate entities corresponding to each literal based on all literals in the input short text, further calculates the matching score of the candidate entity corresponding to each literal using the text similarity between each literal and the corresponding candidate entity, and the correlation between each literal context information and the corresponding candidate entity, and then uses the original text segmentation formula to perform text segmentation on the input short text to obtain potential entity literals, then calculates the prior probability of each potential entity literal after text segmentation, and replaces the literal probability in the original text segmentation formula with the target prior probability in each prior probability to generate a target text segmentation formula, and then uses the target text segmentation formula to perform secondary text segmentation on the input short text to obtain each literal entity; finally, uses the candidate entity corresponding to the target literal entity in the maximum entity disambiguation probability value among the calculated entity disambiguation probability values ​​as the disambiguated entity corresponding to the target literal entity, which solves the problem of low entity disambiguation speed and accuracy in high information density texts in the prior art, can eliminate the ambiguity of entities in high information density texts, and improves the speed and accuracy of entity disambiguation in high information density texts.

[0083] Figure 2bIt is a structural diagram of a text entity disambiguation process provided according to the second embodiment of the present invention; wherein, it is mainly divided into two parts: online calculation and offline preprocessing, and the online calculation is divided into three parts: segmented parsing, entity recognition and global entity disambiguation. In the offline preprocessing part, the vocabulary can be conceptualized and entity keywords can be extracted according to the entity concepts, entity attributes and web page texts in the knowledge base and the network corpus, and the entity keywords, conceptualized vocabulary and entity literal mapping table and entity vector can be packaged to provide a basis for segmented parsing in the online calculation; the popularity of candidate entities in the text corpus in the network corpus can be measured to generate the prior probability of each potential entity literal to provide a basis for entity recognition; the correlation information and similarity information between entities can also be used to perform consistency calculation to obtain the maximum entity disambiguation probability value to provide a basis for global entity disambiguation. In the online calculation, first, the input short text is detected by literal detection to identify all literals in the input short text. Then, the entity literal mapping table and entity vector in the offline preprocessing part are used to generate candidate entities corresponding to each literal, and the entity keywords and conceptualized vocabulary in the offline preprocessing part are used to calculate the matching score of the candidate entities corresponding to each literal to achieve local disambiguation; then, the original text segmentation formula is used to segment the input short text to obtain potential entity literals, and then the prior probability of each potential entity literal in the offline preprocessing part is used to improve the original text segmentation formula to obtain the target text segmentation formula. Furthermore, the target text segmentation formula is used to perform secondary text segmentation on the input short text to obtain each literal entity to achieve entity recognition; finally, the candidate entity corresponding to the target literal entity in the maximum entity disambiguation probability value obtained in the offline preprocessing part is used as the disambiguated entity corresponding to the target literal entity to complete global entity disambiguation.

[0084] Figure 2c1 is a flow chart of a text entity disambiguation method provided according to the second embodiment of the present invention; for example, taking the input short text "ABCD participated in EFG last year" as an example, where A, B, C, D, E, F, and G can all represent a Chinese character, first, literal detection is performed to identify all literals in the input short text, ABC, BCD, ABCD, EF, and EFG; then, the candidate entities corresponding to each literal are generated using a set corpus or knowledge base, abc1, abc2, abcd1, abcd2, efg1, and efg2; further, the text similarity between each literal and the corresponding candidate entity, and the relevance of each literal context information and the corresponding candidate entity are used to calculate the matching scores of the candidate entities corresponding to each literal, abc1=0.4, abc2=0.3, abcd1=0.8, abcd2=0.6, efg1=0.3, and efg2=0.7, to complete local disambiguation; further, the original text is cut into pieces, The formula performs text segmentation on the input short text to obtain potential entity literals ABC, ABCD, EF, and EFG (at this time, since BCD and EF have no corresponding candidate entities or the matching scores of the corresponding candidate entities are 0, the literals BCD and EF are eliminated). Then, the prior probability of each potential entity literal after text segmentation is calculated, and the literal probability in the original text segmentation formula is replaced with the maximum prior probability among the prior probabilities to generate the target text segmentation formula. The target text segmentation formula is then used to perform secondary text segmentation on the input short text and perform entity recognition to obtain each literal entity (ABCD and EFG). Then, global entity disambiguation is performed, and the maximum entity disambiguation probability value between the candidate entities abcd1 and abcd2 corresponding to ABCD and the candidate entities efg1 and efg2 corresponding to EFG (taking the entity disambiguation probability of abcd1 and efg1 as the maximum value as an example) corresponding to ABCD is used as the disambiguated entity of ABCD.

[0085] Example 3

[0086] Figure 3 This is a structural diagram of a text entity disambiguation device provided in Example 3 of the present invention. Figure 3 As shown, the apparatus includes: a candidate entity generation module 310, a matching score calculation module 320, a priori probability calculation module 330, a literal entity determination module 340 and a disambiguation entity determination module 350;

[0087] The candidate entity generation module 310 is used to identify all literals in the input short text and generate candidate entities corresponding to each literal using a set corpus or knowledge base;

[0088] A matching score calculation module 320 is configured to calculate a matching score for a candidate entity corresponding to each literal quantity by using the text similarity between each literal quantity and the corresponding candidate entity, and the relevance between the context information of each literal quantity and the corresponding candidate entity;

[0089] A priori probability calculation module 330 is used to calculate the priori probability of each potential entity literal after text segmentation using the candidate entity with the highest matching score;

[0090] A literal entity determination module 340 is configured to perform secondary text segmentation on the input short text using the prior probabilities to obtain literal entities.

[0091] The disambiguation entity determination module 350 is configured to determine the disambiguation entity corresponding to the target literal entity based on the consistency values ​​between the candidate entities corresponding to the selected literal entities in each literal entity and the candidate entities corresponding to the target literal entity.

[0092] The technical solution of the embodiment of the present invention generates candidate entities corresponding to each literal based on all literals in the input short text, further calculates the matching score of the candidate entity corresponding to each literal using the text similarity between each literal and the corresponding candidate entity, and the correlation between each literal context information and the corresponding candidate entity, and then uses the candidate entity with the highest matching score to calculate the prior probability of each potential entity literal after text segmentation, and uses each prior probability to perform secondary text segmentation on the input short text to obtain each literal entity; finally, according to the consistency value between the candidate entity corresponding to the to-be-selected literal entity in each literal entity and the candidate entity corresponding to the target literal entity, the disambiguation entity corresponding to the target literal entity is determined, which solves the problem of low entity disambiguation speed and accuracy in high-information-density text in the prior art, can eliminate the ambiguity of entities in high-information-density text, and improves the speed and accuracy of entity disambiguation in high-information-density text.

[0093] Optionally, the matching score calculation module 320 may be specifically configured to calculate the matching score of the candidate entity corresponding to each literal using the formula: φ(m,e)=f(m,e)·g(m,e); wherein m is the literal, e is the candidate entity, f(m,e) is determined by the textual similarity between the literal and the corresponding candidate entity, and g(m,e) is determined by the relevance between the literal context information and the corresponding candidate entity.

[0094] represents the prior probability of the literal-candidate entity pair, Indicates the character similarity between the literal and the candidate entity; E m is the candidate entity set composed of candidate entities, and β is the distribution parameter set according to the source of the candidate entity set.

[0095] Optionally, the text entity disambiguation device may further include a text segmentation module for using the original text segmentation formula before calculating the prior probability of each potential entity literal after text segmentation using the candidate entity with the highest matching score: Perform text segmentation on the input short text to obtain potential entity literals; where P(t i ) is the cutting literal t i The probability of freq(t i ) is the literal t i frequency.

[0096] Optionally, the prior probability calculation module 330 may be specifically configured to: utilize the formula: Calculate the prior probability of each potential entity literal after text segmentation; where e k is the candidate entity with the highest matching score; pop(e) represents the popularity of the candidate entity; λ is the prior coefficient.

[0097] Optionally, the literal entity determination module 340 can be specifically used to: replace the literal probability in the original text segmentation formula with the target prior probability in each prior probability to generate a target text segmentation formula; use the target text segmentation formula to perform secondary text segmentation on the input short text to obtain each literal entity.

[0098] Optionally, the disambiguation entity determination module 350 may be specifically configured to: utilize the formula: Calculate the entity disambiguation probability value; where coh(e i ,e j ) represents the consistency value between the candidate entity corresponding to the to-be-selected literal entity and the candidate entity corresponding to the target literal entity; Γ is the candidate entity set consisting of the candidate entities corresponding to the to-be-selected literal entity and the candidate entities corresponding to the target literal entity; N represents the number of literal entities in the input short text; the candidate entity corresponding to the target literal entity in the maximum entity disambiguation probability value is taken as the disambiguated entity corresponding to the target literal entity.

[0099] Optionally, the text entity disambiguation device may further include a consistency numerical calculation module for using the formula: coh(e i ,e j )=γ·rel(e i ,e j )+(1-γ)·sim(e i ,e j ) calculates the consistency value of the candidate entity corresponding to the selected literal entity and the candidate entity corresponding to the target literal entity; where rel(e i ,e j) is the correlation function between the candidate entity corresponding to the to-be-selected literal entity and the candidate entity corresponding to the target literal entity; sim(e i ,e j ) is the similarity function between the candidate entity corresponding to the to-be-selected literal entity and the candidate entity corresponding to the target literal entity; γ is a parameter for measuring the correlation function and the similarity function.

[0100] The text entity disambiguation device provided in the embodiment of the present invention can execute the text entity disambiguation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0101] Example 4

[0102] Figure 4 A schematic diagram of the structure of an electronic device 410 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0103] like Figure 4 As shown, the electronic device 410 includes at least one processor 420, and a memory connected to the at least one processor 420 in communication, such as a read-only memory (ROM) 430, a random access memory (RAM) 440, etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 420 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 430 or the computer program loaded from the storage unit 490 to the random access memory (RAM) 440. Various programs and data required for the operation of the electronic device 410 can also be stored in the RAM 440. The processor 420, ROM 430 and RAM 440 are connected to each other via a bus 450. An input / output (I / O) interface 460 is also connected to the bus 450.

[0104] Multiple components in the electronic device 410 are connected to the I / O interface 460, including an input unit 470, such as a keyboard, a mouse, etc.; an output unit 480, such as various types of displays, speakers, etc.; a storage unit 490, such as a magnetic disk, an optical disk, etc.; and a communication unit 4100, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 4100 allows the electronic device 410 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0105] The processor 420 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the processor 420 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 420 executes the various methods and processes described above, such as the text entity disambiguation method.

[0106] The method includes:

[0107] Identify all literals in the input short text and generate candidate entities corresponding to each literal using a set corpus or knowledge base;

[0108] The matching score of the candidate entity corresponding to each literal is calculated by using the text similarity between each literal and the corresponding candidate entity, and the relevance between the context information of each literal and the corresponding candidate entity;

[0109] The candidate entity with the highest matching score is used to calculate the prior probability of each potential entity literal after text segmentation;

[0110] Use the prior probabilities to perform secondary text segmentation on the input short text to obtain the literal entities;

[0111] The disambiguation entity corresponding to the target literal entity is determined according to the consistency values ​​of the candidate entities corresponding to the to-be-selected literal entities in each literal entity and the candidate entities corresponding to the target literal entity.

[0112] In some embodiments, the text entity disambiguation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 490. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 430 and / or the communication unit 4100. When the computer program is loaded into the RAM 440 and executed by the processor 420, one or more steps of the text entity disambiguation method described above may be performed. Alternatively, in other embodiments, the processor 420 may be configured to execute the text entity disambiguation method in any other appropriate manner (e.g., by means of firmware).

[0113] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0115] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0117] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0118] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0119] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0120] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A text entity disambiguation method, characterized in that: include: Identify all literals in the input short text and generate candidate entities corresponding to each literal using a set corpus or knowledge base; The matching score of the candidate entity corresponding to each literal is calculated by using the text similarity between each literal and the corresponding candidate entity, and the relevance between the context information of each literal and the corresponding candidate entity; The candidate entity with the highest matching score is used to calculate the prior probability of each potential entity literal after text segmentation; Use the prior probabilities to perform secondary text segmentation on the input short text to obtain the literal entities; The disambiguation entity corresponding to the target literal entity is determined according to the consistency values ​​of the candidate entities corresponding to the to-be-selected literal entities in each literal entity and the candidate entities corresponding to the target literal entity.

2. The method according to claim 1, characterized in that The method of calculating the matching score of the candidate entity corresponding to each literal quantity by using the text similarity between each literal quantity and the corresponding candidate entity, and the relevance between the context information of each literal quantity and the corresponding candidate entity, includes: The matching score of the candidate entity corresponding to each literal is calculated using the formula: φ(m,e) = f(m,e) g(m,e); where m is the literal, e is the candidate entity, f(m,e) is determined by the textual similarity between the literal and the corresponding candidate entity, and g(m,e) is determined by the relevance between the literal context information and the corresponding candidate entity. represents the prior probability of the literal-candidate entity pair, Indicates the character similarity between the literal and the candidate entity; E m is the candidate entity set composed of candidate entities, and β is the distribution parameter set according to the source of the candidate entity set.

3. The method according to claim 1, characterized in that Before using the candidate entity with the highest matching score to calculate the prior probability of each potential entity literal after text segmentation, it also includes: Using the original text cutting formula: Perform text segmentation on the input short text to obtain potential entity literals; where P(t i ) is the cutting literal t i The probability of freq(t i ) is the literal t i frequency.

4. The method according to claim 1, wherein The method of calculating the prior probability of each potential entity literal after text segmentation using the candidate entity with the highest matching score includes: Using the formula: Calculate the prior probability of each potential entity literal after text segmentation; where e k is the candidate entity with the highest matching score; pop(e) represents the popularity of the candidate entity; λ is the prior coefficient.

5. The method according to claim 3, characterized in that The method of using the prior probabilities to perform secondary text segmentation on the input short text to obtain the literal entities includes: Replace the literal probability in the original text segmentation formula with the target prior probability in each prior probability to generate the target text segmentation formula; The target text segmentation formula is used to perform secondary text segmentation on the input short text to obtain each literal entity.

6. The method according to claim 1, characterized in that The step of determining the disambiguation entity corresponding to the target literal entity according to the consistency values ​​between the candidate entities corresponding to the selected literal entities in each literal entity and the candidate entities corresponding to the target literal entity includes: Using the formula: Calculate the entity disambiguation probability value; where coh(e i ,e j ) represents the consistency value between the candidate entity corresponding to the to-be-selected literal entity and the candidate entity corresponding to the target literal entity; Γ is the candidate entity set consisting of each candidate entity corresponding to the to-be-selected literal entity and each candidate entity corresponding to the target literal entity; N represents the number of literal entities in the input short text; The candidate entity corresponding to the target literal entity in the maximum entity disambiguation probability value is used as the disambiguated entity corresponding to the target literal entity.

7. The method according to claim 6, characterized in that The method further comprises: Use the formula: coh(e i ,e j )=γ·rel(e i ,e j )+(1-γ)·sim(e i ,e j ) calculates the consistency value of the candidate entity corresponding to the selected literal entity and the candidate entity corresponding to the target literal entity; where rel(e i ,e j ) is the correlation function between the candidate entity corresponding to the to-be-selected literal entity and the candidate entity corresponding to the target literal entity; sim(e i ,e j ) is the similarity function between the candidate entity corresponding to the to-be-selected literal entity and the candidate entity corresponding to the target literal entity; γ is a parameter for measuring the correlation function and the similarity function.

8. A text entity disambiguation device, characterized in that: include: The candidate entity generation module is used to identify all literals in the input short text and generate candidate entities corresponding to each literal using a set corpus or knowledge base; A matching score calculation module is used to calculate the matching score of the candidate entity corresponding to each literal quantity by using the text similarity between each literal quantity and the corresponding candidate entity, and the relevance between the context information of each literal quantity and the corresponding candidate entity; A priori probability calculation module is used to calculate the priori probability of each potential entity literal after text segmentation using the candidate entity with the highest matching score; The literal entity determination module is used to perform secondary text segmentation on the input short text using various prior probabilities to obtain various literal entities; The disambiguation entity determination module is used to determine the disambiguation entity corresponding to the target literal entity based on the consistency values ​​of the candidate entities corresponding to the to-be-selected literal entities in each literal entity and the candidate entities corresponding to the target literal entity.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the text entity disambiguation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text entity disambiguation method according to any one of claims 1 to 7 when executed.

Images