A short text theme hierarchical mining method based on word co-occurrence graph and anchor word extraction

Abstract

The application discloses a short text theme hierarchical mining method based on a word co-occurrence graph and anchor word extraction, which comprises the following steps: S1, a theme hierarchical mining framework first realizes theme inference and anchor word extraction based on a word co-occurrence graph; then, frequent anchor word phrases are mined by applying an association rule; S2, a probability sorting function is used to quantize the coverage, phrase and purity of the anchor word phrases, the anchor word phrases are sorted, and the most representative theme phrase is found. The application constructs a method combining a word co-occurrence graph and anchor word extraction, uses the characteristic that anchor words have good interpretability, converts theme phrase mining into finding anchor words in themes, and makes the mined theme phrases have better interpretability.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology applications, and in particular to a short text topic-level mining method based on word co-occurrence graphs and anchor word extraction. Background Technology

[0002] Phrases are the representation of topics, and traditional phrase definitions require that phrases must consist of consecutive words. Current techniques for mining text topics, using traditional phrase definitions and document-word methods, cannot discover subtopics within the text, thus hindering the construction of a topic hierarchy.

[0003] 1) Using the traditional definition of phrases, in a collection of short texts, it may be impossible to distinguish different phrases that represent the same topic, leading to incorrect topic classification;

[0004] 2) Short text sets are sparsity. Existing technologies use document-word distribution to represent topics, which cannot mine subtopics or form a topic hierarchy. Summary of the Invention

[0005] The purpose of this invention is to provide a short text topic-level mining method based on word co-occurrence graphs and anchor word extraction. In the process of constructing the topic-level tree, it solves the problem of key phrase loss caused by phrase morphology and structure during topic mining by redefining the concept of phrases. In the anchor word extraction process, it uses a linear combination of anchor word distributions to represent non-anchor word distributions, thus solving the problem of polysemy in topic mining.

[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0007] A short text topic hierarchy mining method based on word co-occurrence graphs and anchor word extraction includes the following steps:

[0008] S1. The topic-level mining framework first achieves topic inference and anchor word extraction based on word co-occurrence graphs; then, it applies association rules to mine frequent anchor word phrases.

[0009] S2. A probability ranking function is used to quantify the coverage, phraseification, and purity of anchor phrases, and the anchor phrases are ranked to find the most representative topic phrases.

[0010] Furthermore, step S1 includes the following process:

[0011] I. Constructing a word co-occurrence graph generation model

[0012] Each topic contains k subtopics, denoted as z∈[1,k]; if l ij Indicates w i and w j The number of edges formed is then... Therefore, to cluster a topic with k subtopics, we only need to use a word co-occurrence graph generation model to estimate the word co-occurrence graph G.

[0013] II. Inference of Model Parameters for Word Co-occurrence Graphs

[0014] Assume random variable The service Poisson distribution is shown in equations (1) and (2);

[0015]

[0016]

[0017] Based on the expected value of the Poisson distribution, Therefore, the expected total number of edges belonging to the theme is: If used Let G represent all edges in graph G. Then, according to the cumulative properties of the Poisson distribution, we have:

[0018]

[0019] Given the model parameters, the probabilities of all edges in the word co-occurrence graph are as shown in formula (3);

[0020]

[0021] in, The information to be observed is W, which is the set of words; therefore, the parameters to be learned are the words w belonging to topic z. i or w j The distribution of the edge, and the edge given the topic z. Expectations;

[0022] The parameters in formula (3) are estimated by means of maximum likelihood estimation and the expectation-maximization algorithm is used to update the formulas as shown in formulas (4) and (5);

[0023] E-Step:

[0024] M-step:

[0025] In formula (4) It is a hidden variable, representing the word w. i or w j Edges belonging to the same theme z The expected quantity, of which, For l ij The ratio is proportional to the Poisson distribution parameter.

[0026] Formula (5) calculates the estimated value δ of the maximum likelihood parameter.z and in, Indicator w i The ratio of the total number of edges belonging to topic z to the total number of edges belonging to topic z; the EM algorithm terminates when it reaches a local maximum of the log-likelihood; since each iteration requires updating... δ z and Therefore, the time complexity of each iteration of the EM algorithm is O(|L|k), where |L| is the number of edges in the graph;

[0027] III. Subtopic Extraction

[0028] To infer the subtopics of a topic, we need to extract subgraphs from graph G. Then in G z Using the same generative model, repeat steps 1 and 2 until the entire topic hierarchy is constructed.

[0029] IV. Theme Phrase Mining

[0030] Anchor words have good topic explanatory power, and their inference is based on word co-occurrence maps as a statistical basis. To improve the quality of the mined phrases, the parameters learned in step 2 are used. δ z and Further extract anchor words from topic z; assume that each topic contains at least one anchor, and s represents the known set of anchor words for topic z: s = {s1, s2, ..., s} n};use Indicates the known word w i The remaining w in the edges belonging to theme z j The probability of; at this point, the reducible parameter The word w in j ,calculate

[0031] Any anchor word s i ∈s can represent topic z. The key to anchor word extraction is to combine anchor words in a linear manner to represent non-anchor words. This process is called reconstruction.

[0032] If C represents the reconstructed coefficient matrix, then the elements C in the matrix are... iz Indicates the known word w i If the probability that a non-anchor word belongs to topic z, then C iz The calculation is shown in formula (6);

[0033]

[0034] for The word w in i , The calculation formula is shown in equation (7);

[0035]

[0036] in, This indicates that the anchor word set excluding s i The probability of anchor words other than those mentioned above; known The anchor word s representing the topic z can be obtained using formula (7);

[0037] V. Anchor Keyword-Based Association Rule Mining

[0038] Thematic phrase mining involves two steps:

[0039] 1) Apply the FP-growth association rule mining algorithm to obtain a set of frequent patterns. In this process, integrity criteria are used to prune frequent itemsets during the iteration process to accelerate the mining process;

[0040] 2) Use the anchor words obtained in step 4 to delete... Topic phrases without anchor words are obtained

[0041] Furthermore, the generation process of the word co-occurrence graph G is as follows:

[0042] 1) Generate word w i ~Multi(α);

[0043] 2) Generate word w j ~Multi(α);

[0044] 3) Generate edges (w) belonging to topic z using the distribution z ~ Bernoulli(γ). i w j This forms an edge in graph G;

[0045] 4) Repeat steps 1)-3) until no new edges are generated, to obtain graph G;

[0046] If remember Assuming that the generation of each edge in graph G is independent, then the successful generation of an edge (w) belonging to topic z i w j The probability of )

[0047] If the event "Generate edges (w) belonging to topic z" is executed... i w j )”δ z Next, when δ z When it is large, It approximately follows a Poisson distribution.

[0048] Furthermore, step S2 includes the following process:

[0049] VI. Anchor Phrase Order

[0050] set up This indicates that the topic z contains the anchor phrase P. s The number of short texts, This indicates that all topics contain the anchor phrase P. s The number of short texts, f z (P s ) indicates anchor phrase P s The number of times the anchor phrase appears in topic z is given, and the probability of the anchor phrase appearing given the topic is shown in formula (8);

[0051]

[0052] In addition, to quantify the interdependence and comparability of phrases of different lengths, independent probability and mixed probability are introduced; the independent probability is the probability of independently observing each term in the anchor phrase given the topic, denoted as q1;

[0053]

[0054] Mixed probability is when multiple sibling topics z are known, and the observed anchor phrase P is... s The probability of this is denoted as q2;

[0055]

[0056] Therefore, the probability of the anchor phrase appearing p(P) s |z) Coverage standards can be directly quantified; the ratio of occurrence probability to independent probability. The logarithm is a measurable criterion for phraseification; the logarithm of the ratio of occurrence probability to mixed probability. A standard for the purity of a measurable phrase;

[0057] Finally, the relative entropy KL fusion of the three criteria of coverage, phraseification, and purity is used to obtain the sorting function represented by formula (11);

[0058]

[0059] Where λ is the weight that controls the phraseification standard.

[0060] In summary, the present invention has the following beneficial effects:

[0061] This invention designs a short text representation method based on word co-occurrence graphs, models the mined topics as a probability distribution, and uses the EM (Expectation-Maximization) algorithm to infer the parameters of the model, forming the basis for realizing topic clustering at different granularities.

[0062] This invention enhances the interpretability of topics and accelerates the mining process. It also designs a topic hierarchy mining framework that integrates anchor words, applies association rules to mine frequent anchor phrases, and uses a probability ranking function to quantify the coverage, phraseification, and purity of anchor phrases. The anchor phrases are then ranked to find the most representative topic phrases, thus realizing a method for recursively constructing topic hierarchy. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the word co-occurrence graph generation model described in this invention.

[0064] Figure 2 This is a flowchart of the topic-level mining process described in this invention. Detailed Implementation

[0065] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to the figures and specific embodiments.

[0066] like Figure 1 and Figure 2 As shown, the present invention proposes a short text topic hierarchy mining method based on word co-occurrence graphs and anchor word extraction, which includes the following steps:

[0067] S1. The topic-level mining framework first achieves topic inference and anchor word extraction based on word co-occurrence graphs; then, it applies association rules to mine frequent anchor word phrases.

[0068] S2. A probability ranking function is used to quantify the coverage, phraseification, and purity of anchor phrases, and the anchor phrases are ranked to find the most representative topic phrases.

[0069] Furthermore, step S1 includes the following process:

[0070] I. Constructing a word co-occurrence graph generation model

[0071] Each topic contains k subtopics, denoted as z∈[1,k]; if l ij Indicates w i and w j The number of edges formed is then... Therefore, to cluster a topic with k subtopics, we only need to use a word co-occurrence graph generation model to estimate the word co-occurrence graph G.

[0072] II. Inference of Model Parameters for Word Co-occurrence Graphs

[0073] Assume random variable It follows a Poisson distribution, as shown in formulas (1) and (2);

[0074]

[0075]

[0076] Based on the expected value of the Poisson distribution, Therefore, the expected total number of edges belonging to the theme is: If used Let G represent all edges in graph G. Then, according to the cumulative properties of the Poisson distribution, we have:

[0077]

[0078] Given the model parameters, the probability of observing all edges in the graph is shown in formula (3);

[0079]

[0080] in, The information to be observed is W, which is the set of words; therefore, the parameters to be learned are the words w belonging to topic z. i or w j The distribution of the edge, and the edge given the topic z. Expectations;

[0081] The parameters in formula (3) are estimated by means of maximum likelihood estimation and the expectation-maximization algorithm is used to update the formulas as shown in formulas (4) and (5);

[0082] E-Step:

[0083] M-step:

[0084] In formula (4) It is a hidden variable, representing the word w. i or w j Edges belonging to the same theme z The expected quantity, of which, For l ij The ratio is proportional to the Poisson distribution parameter.

[0085] Formula (5) calculates the estimated value δ of the maximum likelihood parameter. z and in, Indicator w i The ratio of the total number of edges belonging to topic z to the total number of edges belonging to topic z; the EM algorithm terminates when it reaches a local maximum of the log-likelihood; since each iteration requires updating... δ z and Therefore, the time complexity of each iteration of the EM algorithm is O(|L|k), where |L| is the number of edges in the graph;

[0086] III. Subtopic Extraction

[0087] To infer the subtopics of a topic, we need to extract subgraphs from graph G. Then in G z Using the same generative model, repeat steps 1 and 2 until the entire topic hierarchy is constructed.

[0088] IV. Theme Phrase Mining

[0089] Anchor words have good topic explanatory power, and their inference is based on word co-occurrence maps as a statistical basis. To improve the quality of the mined phrases, the parameters learned in step 2 are used. δ z and Further extract anchor words from topic z; assume that each topic contains at least one anchor, and s represents the known set of anchor words for topic z: s = {s1, s2, ..., s} n};use Indicates the known word w i The remaining w in the edges belonging to theme z j The probability of; at this point, the reducible parameter The word w in j ,calculate

[0090] Any anchor word s i ∈s can represent topic z. The key to anchor word extraction is to combine anchor words in a linear manner to represent non-anchor words. This process is called reconstruction.

[0091] If C represents the reconstructed coefficient matrix, then the elements C in the matrix are... iz Indicates the known word w i If the probability that a non-anchor word belongs to topic z, then C iz The calculation is shown in formula (6);

[0092]

[0093] for The word w in i , The calculation formula is shown in equation (7);

[0094]

[0095] in, This indicates that the anchor word set excluding s i The probability of anchor words other than those mentioned above; known The anchor word s representing the topic z can be obtained using formula (7);

[0096] V. Anchor Keyword-Based Association Rule Mining

[0097] Thematic phrase mining involves two steps:

[0098] 1) Apply the FP-growth association rule mining algorithm to obtain a set of frequent patterns. In this process, integrity criteria are used to prune frequent itemsets during the iteration process to accelerate the mining process;

[0099] 2) Use the anchor words obtained in step 4 to delete... Topic phrases without anchor words are obtained

[0100] Furthermore, the generation process of the word co-occurrence graph G is as follows:

[0101] 1) Generate word w i ~Multi(α);

[0102] 2) Generate word w j ~Multi(α);

[0103] 3) Generate edges (w) belonging to topic z using the distribution z ~ Bernoulli(γ). i w j This forms an edge in graph G;

[0104] 4) Repeat steps 1)-3) until no new edges are generated, to obtain graph G;

[0105] If remember Assuming that the generation of each edge in graph G is independent, then the successful generation of an edge (w) belonging to topic z i w j The probability of )

[0106] If the event "Generate edges (w) belonging to topic z" is executed... i w j )”δ z Next, when δ z When it is large, It approximately follows a Poisson distribution.

[0107] Furthermore, step S2 includes the following process:

[0108] VI. Anchor Phrase Order

[0109] set up This indicates that the topic z contains the anchor phrase P. s The number of short texts, This indicates that all topics contain the anchor phrase P.s The number of short texts, f z (P s ) indicates anchor phrase P s The number of times the anchor phrase appears in topic z is given, and the probability of the anchor phrase appearing given the topic is shown in formula (8);

[0110]

[0111] In addition, to quantify the interdependence and comparability of phrases of different lengths, independent probability and mixed probability are introduced; the independent probability is the probability of independently observing each term in the anchor phrase given the topic, denoted as q1;

[0112]

[0113] Mixed probability is when multiple sibling topics z are known, and the observed anchor phrase P is... s The probability of this is denoted as q2;

[0114]

[0115] Therefore, the probability of the anchor phrase appearing p(P) s |z) Coverage standards can be directly quantified; the ratio of occurrence probability to independent probability. The logarithm is a measurable criterion for phraseification; the logarithm of the ratio of occurrence probability to mixed probability. A standard for the purity of a measurable phrase;

[0116] Finally, the relative entropy KL fusion of the three criteria of coverage, phraseification, and purity is used to obtain the sorting function represented by formula (11);

[0117]

[0118] Where λ is the weight that controls the phraseification standard.

[0119] This invention provides a topic-level mining framework based on anchor words co-occurrence graphs. In the process of constructing the topic-level tree, it redefines the concept of phrases to solve the problem of key phrase loss caused by phrase morphology and structure during topic mining. In the anchor extraction process, it uses a linear combination of anchor word distributions to represent non-anchor word distributions, thus solving the problem of polysemy in topic mining.

[0120] This invention establishes a word co-occurrence graph from a probabilistic perspective, and estimates the expected value δ of edges belonging to a certain topic z in the word co-occurrence graph G. z Word w i The number of edges belonging to topic z These two parameters infer the topic distribution in the short text set and estimate the anchor words for each topic based on this. This invention defines a probabilistic ranking function that quantifies the coverage, phraseification, purity, and completeness of anchor word phrases, and uses this function to rank the phrases, finding the best phrase representing the topic based on the ranking results.

[0121] The short text topic-level mining method using anchor word extraction employed in this invention essentially extracts key phrases from text and establishes semantic links between different key phrases. The extraction of key phrases and other information is not only applicable to any enterprise involved in natural language processing applications (such as text classification, text clustering, and text summarization), but is also one of the important methods for developing search engines in vertical industries.

[0122] In this document, the terms "upper," "lower," "front," "back," "left," "right," "top," "bottom," "inner," "outer," "vertical," and "horizontal," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only used for the clarity of expressing the technical solution and for the convenience of description, and therefore should not be construed as limiting the present invention.

[0123] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0124] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A short text topic-level mining method based on word co-occurrence graphs and anchor word extraction, characterized in that, Includes the following steps: S1. The topic-level mining framework first achieves topic inference and anchor word extraction based on word co-occurrence graphs; then, it applies association rules to mine frequent anchor word phrases. S2. A probability ranking function is used to quantify the coverage, phraseification, and purity standards of anchor phrases, and the anchor phrases are ranked to find the most representative topic phrases. Step S2 includes the following process: VI. Anchor Phrase Order set up Indicates belonging to the topic Contains anchor phrases The number of short texts, This indicates that all topics contain anchor phrases. The number of short texts, Anchor phrase On the topic The number of times it appears in Given the number of subtopics, the probability of anchor phrases appearing for known topics is shown in formula (8). (8) Furthermore, to quantify the interdependence and comparability of phrases of different lengths, independent probability and mixed probability are introduced; independent probability is the probability of independently observing each term in the anchor phrase given the topic, denoted as . ; (9) Mixed probability is when multiple sibling topics are known. Observe anchor phrases The probability of is denoted as ; For belonging to the theme The word; (10) Therefore, the probability of anchor phrases appearing Coverage standards can be directly quantified; the ratio of occurrence probability to independent probability. The logarithm is a measurable criterion for phraseification; the logarithm of the ratio of occurrence probability to mixed probability. A standard for the purity of a measurable phrase; Finally, the relative entropy KL fusion of the three criteria of coverage, phraseification, and purity is used to obtain the sorting function represented by formula (11); (11) Where λ is the weight that controls the phraseification standard.

2. The short text topic hierarchy mining method based on word co-occurrence graph and anchor word extraction according to claim 1, characterized in that, Step S1 includes the following process: I. Constructing a word co-occurrence graph generation model Each topic contains Sub-theme, record If used express and The number of edges formed is then... Therefore, if we want to cluster a group containing... For each sub-topic, the word co-occurrence graph generation model only needs to be used to estimate the word co-occurrence graph. In ; II. Inference of Model Parameters for Word Co-occurrence Graphs Assume random variable It follows a Poisson distribution, as shown in formulas (1) and (2); (1) ， ， ， ， (2) Based on the expected value of the Poisson distribution, Therefore, the expected total number of edges belonging to the theme is: If used Representation diagram Given all edges in the array, according to the cumulative properties of the Poisson distribution, we have: ; Given the model parameters, the probabilities of all edges in the word co-occurrence graph are as shown in formula (3); (3) in, It is observation information. It is a set of words; therefore, the parameters that need to be learned are... Belongs to the topic words or The distribution and themes Given the edge Expectations; The parameters in formula (3) are estimated by means of maximum likelihood estimation and the expectation-maximization algorithm is used to update the formulas as shown in formulas (4) and (5); E-Step: (4) M-step: , (5) In formula (4) It is a hidden variable, representing words. or Belonging to the same theme edge The expected quantity, of which, right The ratio is proportional to the Poisson distribution parameter. ; Formula (5) calculates the estimated value of the maximum likelihood parameter. and ,in, Indicator Belongs to the topic The total number of edges and belonging to the topic The ratio of the total number of edges; the EM algorithm terminates when it reaches a local maximum of the log-likelihood; since each iteration requires updating , and Therefore, the time complexity of each iteration of the EM algorithm is O(n log n). , It is the number of edges in the graph; III. Subtopic Extraction To infer the subtopics of a topic, it is necessary to extract the graph. subgraph Then Using the same generative model, repeat steps 1 and 2 until the entire topic hierarchy is constructed. IV. Theme Phrase Mining Anchor words have good topic explanatory power, and their inference is based on word co-occurrence maps as a statistical basis; to improve the quality of the mined phrases, the parameters learned in step 2 are used. , and Further extract themes Anchor words in the text; assume that each topic contains at least one anchor. Indicates the topic Known anchor word set ;use Indicates a known word This belongs to the topic. The remaining edges The probability of; at this point, the reducible parameter words in ,calculate ; Any anchor word Can represent the topic The key to anchor word extraction is to combine anchor words in a linear manner to represent non-anchor words; this process is called reconstruction. If used This represents the reconstructed coefficient matrix, where the elements are... Indicates a known word Non-anchor words belong to the topic The probability of then The calculation is shown in formula (6); (6) for words in , The calculation formula is shown in equation (7); (7) in, Indicates that, except for the set of anchor words The probability of anchor words other than those mentioned above; known Formula (7) can be used to obtain the expression representing the theme. anchor words ; V. Anchor Keyword-Based Association Rule Mining Thematic phrase mining involves two steps: 1) Apply the FP-growth association rule mining algorithm to obtain a set of frequent patterns. In this process, integrity criteria are used to prune frequent itemsets during the iteration process to accelerate the mining process; 2) Use the anchor words obtained in step 4 to delete... Topic phrases without anchor words are obtained .

3. The short text topic hierarchy mining method based on word co-occurrence graph and anchor word extraction according to claim 1, characterized in that, The word co-occurrence graph The generation process is as follows: 1) Generated words ; 2) Word generation ; 3) Based on distribution Generate topics edge , forming a diagram One of the edges in; 4) Repeat steps 1)-3) until no new edges are generated, resulting in the graph. ; If remember , Assuming the graph If the generation of each edge is independent, then a successful generation belonging to the topic is achieved. edge probability ; If the execution event "Generation" belongs to the topic edge " Next, then When it is large, It approximately follows a Poisson distribution. .

Images