A topic mining method based on text fragments for virtual digital human interaction

Abstract

The present invention discloses a topic mining method based on text fragments for virtual digital human interaction, comprising: processing text data to obtain a word string sequence; constructing a text fragment sequence based on the word string sequence; counting the word co-occurrence patterns of the text fragments in the text fragment sequence to construct a word-word co-occurrence matrix; normalizing the word-word co-occurrence matrix to obtain a word-word correlation matrix S; and decomposing the word-word correlation matrix S into the product of a word-topic matrix #imgabs0# and its transpose. The topic mining method of the present application shifts the focus from documents to text fragments in a local context, thereby providing more semantically relevant topic information while reducing erroneous semantic information. In addition, using text fragments instead of documents allows for a unified approach to processing a wide variety of texts, whether long or short text datasets or individual documents (e.g., a novel).

Description

Technical Field

[0001] The present invention relates to the field of computers, and more specifically, to virtual reality / augmented reality virtual digital human technology and a topic mining method based on text fragments. Background Art

[0002] With the rapid development of technologies like artificial intelligence, big data, and the Internet of Things (IoT), the digitization process is accelerating across industries. Different business scenarios are placing higher demands on online user services. AI assistants must not only provide a more intelligent interactive experience but also enhance their understanding, adaptability, and expressiveness of industries, applications, and users through self-learning and adaptive learning. Human-computer interaction has evolved from text-based customer service to voice-based customer service and then to multimodal human-computer interaction systems, often referred to as digital humans. This has significantly improved communication efficiency and service quality.

[0003] Powered by AI technology, digital human products now possess powerful expressive, recognition, and comprehension capabilities, ensuring optimal service performance. They can "hear clearly, understand, and express" in a wide range of scenarios. Hearing clearly means accurately converting human voices into text. Hearing clearly also requires understanding. In terms of expressive communication capabilities, digital humans are synthesized using personalized immersive text-to-speech (TTS), stylish speech synthesis technology, and virtual humans with image, emotion, and movement. They feature a variety of synthesized speech tones and multiple anthropomorphic avatars, enabling multimodal and multi-layered expression of conversational content. This paper identifies texts of varying lengths for virtual digital human interactions and, through the design of a topic model, automatically discovers topics within conversations, achieving true "understanding."

[0004] Current topic models are almost all built on a three-tiered "document-topic-term" framework. Specifically, a document is composed of multiple topics with varying weights, and a topic is a cluster of semantically similar terms. Essentially, these topic models analyze global term-document relationships (i.e., which terms tend to appear in the same document, and which terms tend to appear in different documents) before abstracting the underlying set of topics within that document collection. Therefore, for a topic model to perform well, a document collection must contain a sufficient number of documents, and each document must contain a sufficient number of terms. This is not a problem for long text collections (such as news and academic papers); therefore, traditional topic models such as LDA and NMF can learn highly readable topics on long text datasets. However, for short text datasets (such as tweets, short messages, headlines, questions, and text snippets), due to the shortness of the documents, word co-occurrence statistics are insufficient, making traditional topic models less effective. To address this challenge, researchers have recently proposed a number of new short text topic models using various techniques, such as Twitter-LDA and GPU-DMM.

[0005] Although there are many effective topic models, they are either designed for long text datasets or short text datasets, and not for both long and short text datasets. In addition, if you want to mine topics in a document (such as a book), current methods cannot be used directly. A simple idea is to split a book into a collection of many pseudo-documents (for example, if one page is considered a document, then there are as many pseudo-documents as there are pages, and thus a book becomes a collection of pseudo-documents), so that traditional topic models can work on it. However, although this method is simple, it often does not produce satisfactory results (which will be demonstrated in the subsequent experimental section). Summary of the Invention

[0006] The present invention provides a topic mining method based on symmetric non-negative matrix decomposition of text fragments, comprising: processing text data to obtain a word string sequence; constructing a text fragment sequence based on the word string sequence; counting the word co-occurrence patterns of the text fragments in the text fragment sequence to construct a word-word co-occurrence matrix; normalizing the word-word co-occurrence matrix to obtain a word-word correlation matrix S, wherein the word-word correlation matrix S represents the word meaning correlation between two words; decomposing the word-word correlation matrix S into a word-topic matrix The product of M and its transpose, where M represents the number of words, K represents the number of topics, each row of the word-topic matrix U represents each word in the topic space, and each column of the word-topic matrix U is a topic, and satisfies: Among them, O(U) represents the amount of information loss, and F represents the F-norm of the matrix.

[0007] In some embodiments, constructing a text segment sequence based on the word string sequence includes: starting from the first word, and using the first word as a pointer, with a window with a first size as a radius as a text segment; moving the pointer downward according to a step size to form a second text segment; and continuing to move the pointer downward according to the step size until the end of the word string sequence.

[0008] In some embodiments, Among them, V is an intermediate auxiliary variable, which is a copy of U. α is a non-negative hyperparameter used to control the closeness of matrix V and matrix U. The larger the value of α, the closer the two matrices are to being equal.

[0009] In some embodiments, Among them, γ is a non-negative hyperparameter used to weigh the importance of different factors. U1 and U2 are obtained by splitting the word-topic matrix U, and the scale of U1 is smaller than that of U2.

[0010] The topic mining method of this application shifts the focus from documents to text fragments in local context, which can bring more semantically relevant topic information while also reducing erroneous semantic information. In addition, using text fragments instead of documents can process a variety of texts in a unified manner, whether it is a long text dataset, a short text dataset, or a single document (such as a novel). BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram illustrating processing a text sequence from Wikipedia into a word string sequence according to some embodiments is shown.

[0012] Figure 2 An example of a sliding window sliding from the first word to the last word according to some embodiments is shown.

[0013] Figures 3a to 3d Shown Figure 2 The examples in are how to construct word-word co-occurrence networks.

[0014] Figure 4 shows an example of a topic model based on symmetric non-negative matrix factorization, i.e., S≈UU T .

[0015] Figure 5 The topic model based on symmetric non-negative matrix factorization of matrix blocks is shown, namely

[0016] Figure 6A parallelized implementation of symmetric non-negative matrix factorization based on matrix partitioning under random sampling according to some embodiments is shown.

[0017] Figure 7 A schematic diagram of topic mining on a single document using the SBTM-R method according to some embodiments is shown.

[0018] Figure 8 The PMI-Score values of different topic models on different novel partitions according to some embodiments are shown.

[0019] Figure 9 The figure shows the loss curves of the SBTM-S algorithm on long and short text datasets according to some embodiments. DETAILED DESCRIPTION

[0020] The following examples may enable those skilled in the art to more fully understand the present invention, but are not intended to limit the present invention in any way.

[0021] In the field of natural language processing, the great success of word embedding algorithms (skip-gram / CBOW) and the self-attention mechanism of transformers has shown that rich and deep semantic information can be learned from word-word co-occurrence patterns in local contexts. This application proposes a topic mining method based on symmetric non-negative matrix factorization under text fragments (shingles) rather than documents. This application applies text fragments (shingles) to text topic modeling and abbreviates this method as Shingle-based Topic Modeling (SBTM).

[0022] Essentially, the SBTM model is based on word co-occurrence patterns within text snippets (shingles) with local context, rather than global semantic information at the document level. This has two advantages: (1) shifting the focus from documents to text snippets with local context can bring more semantically relevant topic information while reducing erroneous semantic information; (2) using text snippets instead of documents allows a unified approach to handle a wide variety of texts, whether they are long text datasets, short text datasets, or single documents (such as a novel).

[0023] 1.1 Unified processing of text data

[0024] Traditional topic models are all built on the "document-topic-word" three-layer model, which is obviously based on a document collection. This means that there must be enough documents and the word co-occurrence pattern under each document must also be sufficient. Only when these conditions are met can the above model obtain high-quality topics. This topic learning model limits the application of most existing topic models to single documents (such as a book or a doctoral thesis) and collections containing a few documents. In order to enable the topic model to process a variety of text data sets, this application introduces text fragments (shingles) and then uses the word co-occurrence information in the local semantic environment to mine topics. The advantage of this is that no matter what type of text data set (long text, short text, or single document), it can be uniformly represented as a collection of text fragments (shingles), laying a unified foundation for the model to adapt to different application scenarios. In addition, text fragments are generally short, and the semantic quality of co-occurrence between words is higher, which helps the topic model obtain higher quality topics.

[0025] Specifically, for any text corpus, the first step is to aggregate it into a text sequence. For example, if it's a collection of documents, each document can be concatenated according to its category label [Explanation: The general principle of concatenating documents is: documents with similar semantics should be concatenated as much as possible] to form a long text sequence. If it's a book, it's naturally a long text sequence. Next, use natural language processing tools (such as NLTK) to process the text sequence into a sequence of word strings. Figure 1 The following is a schematic flow chart illustrating how a text sequence from Wikipedia is processed into a word sequence according to some embodiments: If the text is in English, NLTK is used to restore the words to their original form (if the text is in Chinese, the Chinese text is segmented). Punctuation marks, non-noun words, etc. are then filtered out to construct a word sequence.

[0026] 1.2 Construction of word-word co-occurrence network

[0027] Given a text string sequence, a sliding window technique (with the window size parameter denoted as winSize and the sliding step parameter denoted as stepSize) is used to construct a sequence of text fragments. This involves starting with the first word and using that word as a pointer (denoted as focus), with a window with a radius of winSize forming a text fragment. Then, by moving the pointer focus downward by the step size winSize, a second text fragment is naturally formed, and so on, until the end of the word string sequence. In other words, each time the window slides, a text fragment is generated. At the same time, the word co-occurrence patterns within the text fragment are counted and accumulated to construct a word-word co-occurrence network. This word co-occurrence network records the frequency of co-occurrence patterns between words. A larger value indicates a closer semantic relationship between the pair of words, and therefore a greater likelihood that they appear in the same topic. Figure 2 An example of a sliding window sliding from the first word to the last word according to some embodiments is shown (Note: represents a placeholder and can be treated as a meaningless word, winSize=3 and stepSize=3); followed by Figures 3a to 3d Shown Figure 2 How to construct a word-word co-occurrence network in the example, that is: the first step is to Figure 2 In the first sliding window, “ai”, “machine” and “intelligence” appear at the same time, then the association between them and the association between themselves is established, and the weight is set to 1, such as Figure 3a As shown. Similarly, we can Figure 2 The second and third sliding windows in Figure 3b and Figure 3c The co-occurrence network in . Figure 3d The first three pieces are added together Figures 3a-3c The co-occurrence network of Figure 1 A word co-occurrence network based on text fragments for the entire text word sequence in . The higher the word-word co-occurrence frequency in this network, the closer their semantic relationship.

[0028] It is worth noting that winSize in this application is actually half of the window size. Figure 2 The parameters set in (winSize=3 and stepSize=3) are for illustration only. In actual applications, they can be flexibly adjusted according to specific circumstances (i.e., these parameters can also be variable during the window sliding process). In this way, the previous short text topic models TNMF, BTM, and WNTM can be regarded as examples of the above method when constructing word co-occurrence networks.

[0029] Through the following algorithm, a text dataset can be encoded into a word-word co-occurrence matrix W (the value of each element in the matrix is the word-word co-occurrence frequency).

[0030]

[0031] However, due to the different sizes of different data sets, the values of different encoding matrices W will vary greatly in magnitude. Therefore, the matrix W can be further processed as follows:

[0032] S=D -1 / 2 WD -1 / 2 , (1)

[0033] Where D = diag(W1), and 1 is a column vector of all ones. Formula (1) is a very popular way to normalize data. Its biggest advantage is that it does not change the inherent graph structure of the data. Here, the matrix S can be viewed as a semantic correlation matrix between words.

[0034] 1.3 Topic Mining Based on Symmetric Non-negative Matrix Factorization

[0035] Drawing on the idea of NMF learning topics, this paper uses the method of symmetric non-negative matrix factorization to mine topics. Figure 4 shows that a word-word affinity matrix S is decomposed into a term-topic matrix according to some embodiments The product of M and its transpose, where M represents the number of words and K represents the number of topics. Since the S matrix represents the semantic relevance between two words, and each row of the U matrix can be seen as the representation of each word in the topic space, non-negative symmetric matrix factorization is essentially looking for a semantic (topic) representation of words so that the semantic relevance between words can match the true relevance in S. Therefore, the topic model based on symmetric non-negative matrix factorization is an optimization problem that minimizes the loss between the true word relevance matrix S and the word-topic representation matrix U and its transpose, as follows:

[0036]

[0037] Each column of the word-topic matrix U can be considered a topic, and the larger the weight of each word under the topic, the greater the relevance of the word to the corresponding topic. Equation (2) above is a topic mining method based on word-word co-occurrence networks in text fragments; for simplicity, it is abbreviated as SBTM.

[0038] It is worth mentioning that the SBTM method (e.g., Formula (2)) only requires the semantic relevance between words, which is fundamentally different from the traditional topic model based on "document-topic-word". The advantage of this design is that it can establish a unified model to adapt to a variety of text datasets.

[0039] In summary: (1) Although TNMF also uses the technology of symmetric non-negative matrix factorization to learn topics, this method is mainly used for short text datasets. In addition, the algorithm designed in TNMF is very unstable in practical applications because it involves the calculation of pseudo-inverse matrix. However, the SBTM model is suitable for a variety of text collections. In addition, this application designs two fast and efficient solutions, one of which is based on random sampling technology and has the ability to process large data. (2) The other two typical probabilistic topic models based on word-word co-occurrence, namely BTM and WNTM, are proposed to solve the problem of short text topic mining; however, they can also be applied to long text datasets. It is worth noting that whether these two models can obtain high-quality topics on a single long text is still unknown. (3) In short, this application constructs a word-word co-occurrence network in a local semantic environment from the perspective of text fragments (shingles), and establishes a unified topic model that can handle various texts.

[0040] 2. Topic Mining Based on Relaxed SBTM (SBTM-R)

[0041] The original optimization problem (2) can be equivalently transformed into the following form:

[0042]

[0043] Where V is an intermediate auxiliary variable (it can also be regarded as a copy of U). In order to make the above optimization (3) simpler, it can be relaxed as follows:

[0044]

[0045] Where α is a non-negative hyperparameter that controls the closeness between the matrix V and the matrix U. The larger its value, the closer the two matrices are to being equal. Equation (4) above is the topic mining method based on the relaxed SBTM, which is abbreviated here as SBTM-R.

[0046] Objective function O r (U, V) Taking the derivative of variables U and V respectively, we can get:

[0047]

[0048] and

[0049]

[0050] Based on the above two derivative formulas, the update rules of variables U and V can be designed as follows:

[0051]

[0052] and

[0053]

[0054] where ρ mk and They represent the iteration step size of the (mk)th variable in matrix U and matrix V respectively.

[0055] If the order and Then we have:

[0056]

[0057] and

[0058]

[0059] At this point, with the iterative formulas (9) and (10), the solution of the SBTM-R is described by the following algorithm:

[0060]

[0061] If the update formulas (9) and (10) are iterated alternately, the objective function O r The value of (U, V) will become smaller and smaller until it converges to a local minimum.

[0062] Under the premise that the initialization matrix is non-negative, the above two update formulas (9) and (10) will automatically maintain the non-negative characteristics during the alternating iteration process and make the value of the objective function continuously decrease. In addition, the objective function is non-negative and has a lower bound. Therefore, r (U,V) will converge to a local minimum.

[0063] Although SBTM-R can theoretically guarantee convergence and is very efficient and fast in many practical applications, its time complexity is O(KM 2 ), which means that when the dictionary length of the text corpus is very large, the algorithm will be very slow. Next, this application proposes a fast and efficient solution: Scalable Symmetric Non-Negative Matrix Factorization (SBTM-S). The advantage of this solution is its ability to handle large-scale text datasets.

[0064] 3. Topic Mining Based on Scalable SBTM (SBTM-S)

[0065] The SBTM-R algorithm uses all the S information in each iteration, which inevitably leads to a high time complexity. To solve this difficult problem, the word-topic matrix U can be split into two sub-matrices U1 and U2, one of which is smaller in size. Specifically, the original optimization problem (2) can be equivalent to the following mathematical form (see also Figure 5 ):

[0066]

[0067]

[0068] Where S is symmetrical, that is: S 12 =S 21 , and S 11 / S 22 is also a symmetric submatrix. Therefore, in order to capture the information of each element as quickly and efficiently as possible, the optimization problem (11) can be approximated as follows:

[0069]

[0070] Where γ is a non-negative hyperparameter used to weigh the importance of different factors. Note that: Theoretically, this will result in a certain amount of information loss due to being discarded in the objective function. However, this lost information can be recovered to a certain extent through the following random process. Equation (12) is the topic mining method based on the relaxed SBTM, which is abbreviated here as SBTM-S.

[0071] In each iteration of the algorithm, a subset Q is randomly extracted from the index set M = {1, 2, ..., M}, and then the matrix is selected according to this index subset Q and Where |Q|<<|M|. Since each iterative update will randomly capture some element information in S, after a certain number of iterations, the information in the matrix S will be fully utilized. In addition, in each iteration, the above random process can be repeated independently in parallel several times (for example, P times), and then the average of their results is taken as the final output of the algorithm, such as Figure 6 shown.

[0072] Similar to the solution process of optimization problem (4), the iterative formula of optimization problem (12) can be obtained:

[0073]

[0074] and

[0075]

[0076] Based on the above two formulas, the solution of SBTM-S is described in the following algorithm.

[0077]

[0078] It is worth noting that the time complexity of this algorithm is O(KMQ) (generally, K, Q << M), which is much smaller than the time complexity of SBTM-R. Because the SBTM-S algorithm can be executed in parallel, it has the potential to process large amounts of data.

[0079] 4. Experimental Verification and Analysis

[0080] 4.1 Experimental Setup

[0081] 4.1.1 Public Datasets

[0082] In the long-text topic modeling experiments, this section used four public datasets: SogouNews, 20Newsgroups, Reuters, and RCV1. In the short-text topic modeling experiments, this section also used four public datasets: Snippet, SinaNews, StackOverflow, and BaiduQA. Table 1 below shows the statistical characteristics of these public datasets. For the single-document topic modeling experiments, we selected Jin Yong's novel, The Legend of the Condor Heroes, to commemorate his contributions to world literature.

[0083] Table 1

[0084]

[0085]

[0086] For each text corpus, if it is in English, first remove non-Latin characters and numbers, convert all uppercase letters to lowercase, and stem each word; if it is in Chinese, first split the Chinese text into a series of Chinese word strings, and retain noun words; secondly, discard words with a document frequency of less than 5; and filter out documents with a length of less than 2 words. It is worth noting that for a single document Chinese novel, only step 1 is required. Finally, a text file can be obtained, with each line corresponding to a document, and each line is a sequence of word strings (words are separated by spaces). Table 1 above summarizes some statistical features of the dataset used in the experiment, where #doc represents the number of documents in the dataset, #cls represents the number of class labels, #voc represents the size of the dictionary, and #len represents the average length of the document.

[0087] 4.1.2 Comparison Method

[0088] The topic models compared on the long text dataset are: (1) NMF, (2) LDA, (3) WNTM, and (4) KGNMF. These selected methods are the most representative long text topic models.

[0089] The topic models compared on the short text dataset are: (1) BTM, (2) WNTM, (3) SPTM, (4) DMM, (5) GPUDMM, (6) LFDMM, (7) GKLDA and (8) KGNMF. These methods are selected as representative short text topic models.

[0090] The topic models compared on a single document are: (1) NMF, (2) LDA, (3) BTM, and (4) WNTM. This part of the experiment specifically selected a novel to compare topic models under different learning modes. Specifically, we want to explore whether a three-layer topic model based on "document-topic-word" (such as NMF / LDA) can work on a single document; at the same time, we also want to explore whether the method based on word co-occurrence network proposed in this application performs better than BTM / WNTM in the task of topic mining.

[0091] 4.1.3 Evaluation indicators

[0092] Currently, there are many metrics for evaluating topic quality, such as perplexity, topic coherence, and word embedding-based metrics. In this section, we use a widely used metric, PMI-score (pointwise mutual information). Specifically, given a topic and its top-N topic words Ω = {w1, w2, ..., w N}, then the PMI-score of the topic is defined as:

[0093]

[0094] where w i and w j Respectively represent the i-th and j-th words under the given current topic; p(w i ,w j ) refers to w i and w j The probability of appearing in the same document; p(w i )(or p(w j )) refers to the word w i (or w j ) appears in the document collection. It is worth noting that the numerator of formula (15) contains a smoothing term of 1 to prevent the occurrence of log0 in the calculation of the index value. For the K topics generated by a model, their average value is calculated as a measure of the quality of the topics mined by the model. The larger the value, the higher the topic quality of the corresponding topic model.

[0095] In addition, given the number of topics K, people often expect a topic model to learn more diverse topics, which is conducive to their full understanding of the text collection. Therefore, in order to further evaluate the diversity of topics generated by different topic models, this section designs the following diversity indicator formula:

[0096]

[0097] where Ω (k) It represents the set of top-N words in the kth topic. Obviously, the larger the value of this indicator, the better the topic diversity of the corresponding topic model.

[0098] 4.1.4 Parameter Configuration

[0099] For all compared methods, their code is available for download online or provided by the corresponding authors. This section conducts parameter tuning and other experiments on selected public datasets, ultimately determining the best-performing parameter configuration. Regarding knowledge-guided topic models, this section utilizes word embedding vectors (dimension 100) pre-trained on millions of Wikipedia documents.

[0100] For the SBTM model, to ensure algorithm convergence, the maximum number of convergences (maxIter) can be set to 100; the number of independent repeated random samplings (P) is set to 5 (on the large dataset RCV1, the random sampling sample size Q is configured to 2500; on other datasets, Q is configured to 200). For other hyperparameters, after a series of experiments, (winSize, stepSize) was finally determined, and α and γ were (25, 25), 0.5, and 2, respectively. It is worth mentioning that the topic model proposed in this application is not very sensitive to the sliding window parameters (winSize, stepSize). That is, the SBTM method can achieve good topic effects within a wide range (3 to 60).

[0101] To ensure fair comparison between different topic models, other common parameters, such as the number of topics K, are unified to 60 and 100. PMI-score and Div-score are calculated based on the top 10 and 20 keywords of each topic in the document collection.

[0102] 4.2 Experimental Results and Discussion

[0103] Topic model experiment on document collection. Table 2 and Table 4 record the PMI-Score values of different topic models on long and short text datasets. Obviously, whether on long text datasets or short text datasets, the index value of the topic mining method SBTMs proposed in this application is much higher than that of other methods (it is worth noting that in order to quantify the significance of the experimental results, this section further conducted a t-test test, and then found that the p-value of the significance difference test between the SBTMs method and any other topic model is less than 0.01.), which confirms that the method proposed in this application is capable of learning high-quality topics on text datasets. Among them, Table 2 shows the PMI-Score values of different topic models on long text datasets; Table 3 shows the Div-Score values of different topic models on long text datasets; Table 4 shows the PMI-Score values of different topic models on short text datasets; Table 5 shows the Div-Score values of different topic models on short text datasets.

[0104] Table 2

[0105]

[0106] Table 3

[0107]

[0108] Table 4

[0109]

[0110]

[0111] Table 5

[0112]

[0113] In addition, Tables 3 and 5 further demonstrate the Div-score values of the topic diversity generated by different topic models on long and short text datasets. First, for topic models on long text datasets, SBTMs exhibits larger Div-score values under different parameter settings, indicating that the diversity of topics generated by the SBTMs method is greater than that of other methods. Second, even on short text datasets, the SBTMs method proposed in this application can obtain better or comparable topic diversity than other short text topic models.

[0114] In summary, SBTMs can not only mine high-quality topics on long and short text datasets, but also the semantic diversity of these learned topics is very good when the number of topics K is given.

[0115] Topic model experiment on a single document. Regarding the experiment of different topic models on the novel "The Legend of the Condor Heroes" (a single document), this section takes the SBTM-R method as an example and shows the topic mining results of this method when the number of topics K is 60, as shown in the following example: Figure 7 It is worth noting that the results of topic mining by other methods are shown in the same form as Figure 7 Consistent. Here, due to limited space, only some of the 60 topics of the SBTM-R method are shown. Among them, the four-character words in the green box are different topics, and the blue box shows the topic number and the top 10 topic words with the highest weight. The weight on the arrow represents the proportion of the corresponding topic in the entire topic set. The larger the value, the more words the novel uses to narrate the corresponding storyline. It should be emphasized here that "The Legend of the Condor Heroes" is a chapter-based novel. Each chapter is a chapter, and the whole book has 40 chapters. The author (Mr. Jin Yong) uses a "four-character word" to summarize the plot of each chapter. For example, Figure 7 The "Master Yideng" in the novel is a summary of the content of the 30th chapter. In this chapter, the core characters are Master Yideng and his disciples. Therefore, the first 10 keywords generated by the SBTM-R method include: Yideng, scholar, farmer, woodcutter, fisherman, etc. Among them, "Yideng" means "Master Yideng", and the last four words are the identities of his four disciples. Therefore, these keywords can easily make readers think of the core plot of the novel: "Master Yideng". Other four-character words are similar to this. In addition, from Figure 7It can be seen that the weights of "Bending the Bow and Shooting the Condor Heroes" and "The Battle of Mount Huashan" are relatively large, indicating that the two storylines are very exciting, which is consistent with the experience of readers / audiences.

[0116] To quantitatively evaluate the results of different topic models on novels (single documents), this section first invited two experts familiar with Jin Yong's novels to manually interpret the readability of topics for a set of K = 60 topics. If the first 20 words of a topic easily reminded the experts of the corresponding characters or plot in the novel, the topic was labeled "readable"; otherwise, it was labeled "unreadable." Finally, the number of readable topics generated by each model was recorded, as shown in Table 6. Table 6 shows the number of readable topics generated by different topic models on novels and the consistency of the expert interpretation results. The last column shows the consistency index (Cohen's κ) between the two experts' interpretations; this value is clearly high, indicating that the manual interpretation results are very reliable. In addition, the two classic topic models, NMF and LDA, did not learn any readable topics for novels, confirming that topic models based on the "document-topic-word" model are often ineffective for single-document datasets. In the topic model based on word co-occurrence network, the SBTMs method mines more readable topics than BTM and WNTM, which means that the topics learned by SBTMs are more readable than those of BTM and WNTM.

[0117] Table 6

[0118]

[0119] In addition, this section also calculates the Div-Score values of the topic sets generated by different topic models on novels, as shown in Table 7. Table 7 shows the Div-Score values of the topic sets generated by different topic models on novels. Obviously, the Div-score value of the SBTMs method is still the largest, which shows that: given the number of topics K, the method proposed in this application can mine more semantic structures.

[0120] Table 7

[0121]

[0122] In order to further calculate the PMI-score values of different topic models, the novel can be divided into several text segments, each of which has a length of l dEach text segment can be treated as a pseudo-document; thus, a novel becomes a collection of pseudo-documents, on which the PMI-score can be calculated. The topic readability in Table 6 shows that NMF and LDA fail in topic mining for novels; therefore, only the topic quality of the other four methods (based on PMI-scores) is evaluated here. Figure 8 It shows that different topic models have different parameter configurations and different novel divisions (l d =10, 20, 40, 60, 100, 140, 200). Clearly, the experimental results in the figure show that the PMI-scores of SBTMs are much higher than those of BTM and WNTM, which also means that the quality of the topics learned by SBTMs on novels is higher than that of BTM and WNTM.

[0123] Table 8 summarizes some of the topics learned by different topic models for the novel. Only the top 10 keywords for each topic are shown. The first two methods (NMF and LDA) are based on the "document-topic-word" topic model, while the last four methods (BTM, WNTM, SBTM-R, and SBTM-S) are based on word co-occurrence topic models. It is clear that the keywords generated by NMF and LDA lack semantic focus, making it difficult for readers to discern which plot point each topic describes. The other four methods all generate readable topics, and it is clear that SBTM-R and SBTM-S demonstrate comparable expressiveness in describing the novel's plot, significantly outperforming the other two methods (BTM and WNTM). For example, the first topic of both SBTM-R and SBTM-S depicts a fascinating novel plot: The Battle of Mount Hua (represented by keywords such as: East Evil, West Poison, South Emperor, North Beggar, (Central) Divine Power, Wang Chongyang, Zhou Botong, Mount Hua, and Battle of Sword). Obviously, their keywords are of higher quality and more distinctive than the keywords presented by WNTM (corresponding to the third topic). However, BTM failed to discover this topic.

[0124] In general, multiple topic model experiments designed on novels further revealed that SBTMs can also mine a set of topics with high semantic quality and rich diversity in a single document.

[0125] Table 8

[0126]

[0127]

[0128] 4.3 Algorithm Convergence and Efficiency Research

[0129] The convergence of the SBTM-R algorithm has been theoretically guaranteed above. Next, this section will study the convergence of the SBTM-S algorithm through experiments. Figure 9 The following plots show the loss curves of the SBTM-S algorithm on long and short text datasets, with the x-axis representing the number of iterations and the y-axis representing the target loss function. Clearly, all curves converge within 100 iterations. It is worth noting that, although the SBTM-S algorithm is a stochastic algorithm, it converges smoothly to a local minimum in most cases. However, when applied to the large-scale text dataset RCV1 (with a vocabulary size of 35,675 words, as shown in Table 1), the SBTM-S algorithm's curves exhibit local fluctuations. This is because the parameters set in this section, the number of independent repeated random sampling (P) of 5 and the sample size of 2500 per sampling, are not large enough. Consequently, the SBTM-S algorithm cannot achieve a strictly decreasing curve on a dataset of RCV1's size.

[0130] To compare the efficiency of different topic mining methods, this section specifically records the time required for them to complete topic learning on two large-scale text datasets, RCV1 (long text dataset, table on the left) and BaiduQA (short text dataset, table on the right). Table 9 shows the time (in minutes) required for different topic models to complete topic learning on the datasets RCV1 and BaiduQA. Note: These experiments were all performed on the same server, and the machine configuration is as follows: CPU E7-4850v2@2.30GHz (12 cores) 512GB RAM.

[0131] Table 9

[0132]

[0133] Table 9 clearly shows that the SBTM-S method proposed in this chapter takes the least amount of time to learn on both long and short text datasets, confirming the high efficiency of this randomized algorithm. Furthermore, the SBTM-R method also demonstrates impressive learning efficiency, ranking second only to the SBTM-S method. Interestingly, most topic mining methods based on non-negative matrix factorization (such as NMF, KGNMF, and SBTMs) learn faster than topic models based on LDA (such as LDA, WNTM, and GKLDA). This is primarily because matrix operations are more easily parallelized, while the statistical inference in LDA-based methods is mostly serial and multiply nested, resulting in slower learning speeds.

[0134] It should be understood by those skilled in the art that the above embodiments are merely exemplary embodiments and that various changes, substitutions, and alterations may be made without departing from the spirit and scope of the present invention.

Claims

1. A topic mining method based on text fragments, characterized in that: include: Process the text data to obtain a word string sequence; Constructing a text segment sequence based on the word string sequence; Counting word co-occurrence patterns of text segments in the text segment sequence to construct a word-word co-occurrence matrix; Normalizing the word-word co-occurrence matrix to obtain a word-word correlation matrix S, where the word-word correlation matrix S represents the semantic correlation between two words; Decompose the word-word affinity matrix S into a word-topic matrix The product of M and its transpose, where M represents the number of words, K represents the number of topics, each row of the word-topic matrix U represents each word in the topic space, and each column of the word-topic matrix U is a topic, and satisfies: Among them, O(U) represents the amount of information loss, and F represents the F-norm of the matrix; in, Among them, V is an intermediate auxiliary variable, which is a copy of U. α is a non-negative hyperparameter used to control the closeness of matrix V and matrix U. The larger the value of α, the closer the two matrices are to being equal.

2. The topic mining method according to claim 1, characterized in that: Constructing a text segment sequence based on the word string sequence includes: starting from the first word, and using the first word as a pointer, and a window with a first size as a radius is a text segment; moving the pointer downward according to the step size to form a second text segment; continuing to move the pointer downward according to the step size until the end of the word string sequence.

3. The topic mining method according to claim 1, characterized in that: Among them, γ is a non-negative hyperparameter used to weigh the importance of different factors. U1 and U2 are obtained by splitting the word-topic matrix U, and the scale of U1 is smaller than that of U2.

Images