A deep text clustering method and device based on adaptive structure learning

Abstract

The application discloses a deep text clustering method based on adaptive structure learning, and comprises the following steps: step one, constructing a K-neighbor graph; step two, generating an adaptive structure graph through an adaptive threshold strategy; step three, using a threshold decay strategy to dynamically adjust the topological range of the graph convolution kernel, thereby learning an adaptive structured semantic representation of the text; step four, learning the semantic representation of the text by using a self-encoder, and integrating the semantic representation into the structured semantic representation learned in step three layer by layer to learn a fused and enhanced semantic representation; and step five, continuously optimizing the encoder parameters to obtain a final text clustering result. The structural information is fused into the text semantic representation, and the structural information and the semantic information are used to jointly supervise the clustering process, thereby effectively solving the problems of text representation difference and insufficient supervision in unsupervised text clustering, improving the accuracy of the clustering result, and obtaining a clustering result more suitable for downstream tasks.

Description

Technical Field

[0001] This invention relates to the fields of information extraction and text processing, and in particular to a deep text clustering method and apparatus based on adaptive structure learning, belonging to the fields of data mining and natural language processing technology. Background Technology

[0002] With the development of deep neural networks, deep text clustering has gradually become a popular research area in recent years. Deep text clustering refers to the task of using neural networks to learn semantic representations of text and, based on this, grouping text documents with similar representations into a single class. It can be applied to various fields such as text analysis, business applications, web search, recommendation systems, and biomedicine. Existing semantic representation learning in deep text clustering mainly consists of learning semantic representations of the text's own content and learning structural semantic representations of the text dataset. Since these two representations are not independent but complementary, deep text clustering models that integrate semantic representations of both text content and structural semantic representations of the text dataset often achieve better clustering results than models that only learn a single representation. Currently, in deep text clustering models that integrate semantic representations of both text content and structural semantic representations of the text dataset, the initial structural features of the text dataset are twofold: first, using the K-nearest neighbor algorithm to calculate the K nearest neighbor samples for each text document to form a K-nearest neighbor graph; second, utilizing the inherent graph structure of the text dataset itself, such as relationships like co-authors and co-references to form a structural graph. However, in current deep text clustering models, the value of K in the K-nearest neighbor graph used by the text is fixed. If K is too small, it contains too little structural information; if K is too large, it introduces excessive noise data. Both factors severely affect the final clustering result. Furthermore, the graph structure inherent in the text dataset itself also contains noise data; two documents connected by an edge may not belong to the same class. For example, in the common citation graph structure of the real text dataset Citeseer, 26.45% of the connected nodes do not belong to the same class. Therefore, how to learn an adaptive structural graph from a text dataset is a problem worthy of further research.

[0003] Furthermore, current deep text clustering models that integrate semantic representations of the text's content and structural semantic representations of the text dataset primarily fall into two categories regarding the learning of structural semantic representations: graph convolutional neural networks (GCNNs), which learn local features of each sample and its neighbors using GCNNs; and attention-based graph attention neural networks (GANNs), which learn the different effects of each neighbor on the sample's structural representation using an attention mechanism. However, neither of these methods considers the role of multi-hop neighbors in the graph structure—that is, the influence of a neighbor's neighbor on the sample representation learning. In practical applications, similar texts may not belong to the same cluster, and some texts, although geographically distant, may still be classified into the same cluster. This leads to poor clustering performance in existing technologies. Summary of the Invention

[0004] This invention proposes a deep text clustering method and apparatus based on adaptive structure learning to overcome the shortcomings of existing technologies.

[0005] The technical solution of this invention is: a deep text clustering method based on adaptive structure learning, the method comprising the following steps:

[0006] Step 1: Construct a K-nearest neighbor graph from the original text data using the K-nearest neighbor method;

[0007] Step 2: Filter out adjacent nodes with low similarity using an adaptive threshold strategy to generate an adaptive graph structure;

[0008] Step 3: Input the graph with an adaptive structure into a neural network with an adaptive topology structure, and use a threshold decay strategy to allow the graph convolution kernel to dynamically adjust the topology range, thereby learning an adaptive structured semantic representation of the text;

[0009] Step 4: Input the text's own information into the autoencoder to learn the text's own semantic representation, and then integrate the text's own semantic representation layer by layer into the adaptive structured semantic representation learned in Step 3, learning a fusion-enhanced semantic representation with both text's own semantic representation and structured semantic representation;

[0010] Step 5: Use softmax to cluster the fused and enhanced semantic representation obtained in Step 4 to divide the cluster distribution. Use a dual self-supervised mechanism to calculate the loss between the adaptive structured semantic representation and the text's own semantic representation. Continuously back-adjust and optimize the encoder parameters to obtain the final semantic representation and text clustering results.

[0011] Specifically, step one involves: firstly, preprocessing the text data, using at least the following methods: bag-of-words model, TF, TF-IDF, or Word2Vec; then, utilizing the principle that texts and their neighbors have similar features, constructing a K-nearest neighbor graph with K neighbors for each text using the KNN algorithm.

[0012] Furthermore, the adaptive threshold strategy involves setting a threshold for the number of neighbors that is automatically adjusted for each text in the K-nearest neighbor graph obtained in step one.

[0013] Preferably, the threshold for automatically adjusting the number of neighbors is set by the following method:

[0014] Initialize the threshold of the first filter to 0.006, and set the threshold of the m-th filter accordingly. Its threshold size is the threshold of m-1 filters. Half of it, the formula is as follows:

[0015]

[0016] Specifically, the adaptive structured semantic representation of the text is calculated using the following formula:

[0017]

[0018] in, The number of layers in the neural network. , For the first Multinomial convolution kernel, It is the first Layer polynomial parameters, where M represents the number of convolutional kernels, and b is the learnable bias matrix. For the first The number of vertices in each layer is used as a vector of all 1s, and A represents the similarity matrix of the automatically adjusted graph structure information obtained in step two. For the first An adaptive structured semantic representation of layered text, where σ is the activation function.

[0019] Specifically, the fused and enhanced semantic representation is iteratively calculated using the following formula:

[0020]

[0021]

[0022] in, Indicates the first Layer's own semantic representation, Indicates the first Layer-adaptive structured semantic representation; Indicates the first Adaptive structured semantic representation of layered text Indicates the first Layer fusion-enhanced semantic representation, where σ is the activation function and b is a learnable bias matrix. for The number of vertices in a layer is represented as a vector of all 1s. For the first Multinomial convolution kernel.

[0023] Specifically, the dual self-supervised mechanism includes: dividing the clusters into distributions R, obtaining the text data distribution Q from the text semantic representation learned by the autoencoder using the student distribution, calculating the distribution P based on this, calculating the difference loss between distributions P and Q and the difference loss between P and R using the KL divergence formula, learning high-confidence assignments to fine-tune the model parameters by minimizing the difference loss between distributions P and Q and the difference loss between P and R, and combining this with the reconstruction loss of the autoencoder to jointly constitute the model's loss function, and obtaining the clustering results through R.

[0024] Specifically, the method for calculating the difference loss between the P and Q distributions is as follows:

[0025]

[0026] Where the subscript i represents the i-th sample, the subscript j represents the j-th cluster center, p represents a value in P, q represents a value in Q, and L ae This represents the difference loss between the P and Q distributions.

[0027] Specifically, the method for calculating the difference loss between the P and R distributions is as follows:

[0028]

[0029] Where the subscript i represents the i-th sample, the subscript j represents the j-th cluster center, p represents a value in P, r represents a value in R, and L tagcn This represents the difference loss between the P and R distributions.

[0030] A deep text clustering device based on adaptive structure learning is disclosed. The device includes a processor and a memory, wherein the memory stores computer program instructions suitable for execution by the processor, and the computer program instructions, when executed by the processor, cause the processor to perform the deep text clustering method based on adaptive structure learning as described above.

[0031] Beneficial effects

[0032] The beneficial effects of this invention are: compared with the prior art, the technical solution of this invention,

[0033] 1) This scheme optimizes the clustering process by using adaptive structural information. It can obtain structural information that meets the clustering requirements by automatically adjusting the threshold. Experimental results show that the model can achieve good performance. It can mine and fuse adaptive structural information into deep text clustering, thereby obtaining a more fused and enhanced semantic representation that can better express text features, as well as better clustering results, and better serving downstream tasks.

[0034] 2) By simultaneously utilizing structural and semantic information to jointly supervise the clustering process, the problems of text representation differences and insufficient supervision in unsupervised text clustering are effectively solved, thereby improving the accuracy of clustering results and providing new ideas for subsequent research on deep text clustering;

[0035] 3) Given the important role of text clustering in the field of natural language processing, deep text clustering that incorporates adaptive structural information can achieve better clustering results and provide more favorable support for different application scenarios, which has great theoretical significance and practical value.

[0036] 4) The deep text clustering method based on adaptive structure learning in this invention can learn an adaptive structure graph for a text dataset and takes into account the role of multi-hop neighbors in learning text structure representation. It solves the problem of poor clustering effect in existing technologies and improves the clustering effect by automatically selecting multi-hop neighbors in the text clustering process. Attached Figure Description

[0037] Figure 1 A technical roadmap provided for at least one embodiment of this disclosure;

[0038] Figure 2 A schematic diagram of an adaptive thresholding method provided in at least one embodiment of this disclosure;

[0039] Figure 3 A functional block diagram provided for at least one embodiment of this disclosure;

[0040] Figure 4 A schematic block diagram of an apparatus provided for at least one embodiment of the present disclosure;

[0041] Figure 5 This is a schematic block diagram of the storage medium of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings.

[0043] Example 1: As shown in the attached document Figures 1-3 As shown, a deep text clustering method based on adaptive structure learning includes the following steps: Step 1: Construct a K-nearest neighbor graph from the original text data using the K-nearest neighbor method; Step 2: Generate an adaptive structure graph using an adaptive threshold strategy; Step 3: Input the adaptive structure graph into a neural network with an adaptive topology structure, and use a threshold decay strategy to dynamically adjust the topology range of the graph convolution kernel, thereby learning an adaptive structured semantic representation of the text; Step 4: Use an autoencoder to learn the semantic representation of the text, and integrate it layer by layer into the structured semantic representation learned in Step 3, learning a semantic representation with fusion enhancement of textual semantic representation and structured semantic representation. The feature representation output in this step is used in the subsequent clustering process; Step 5: Calculate the reconstruction loss using the original text and the reconstructed text, calculate the loss between the graph structured semantic representation and the text semantic representation using a dual self-supervised mechanism, continuously back-adjust and optimize the encoder parameters to obtain the final semantic representation, and use the student distribution to cluster the semantic representation obtained in the above steps to obtain the final text clustering result.

[0044] Further, step one specifically involves preprocessing the text data. This step should utilize at least the following methods: Bag-of-Words model, TF (Term Frequency), TF-IDF (Term Frequency-Inverse Text Frequency Index), or Word2Vec. Then, leveraging the principle that texts share similar features with their neighbors, the preprocessed text data is used to construct a K-nearest neighbor graph with K neighbors for each text element using the KNN algorithm. This construction of the K-nearest neighbor graph achieves the goal of obtaining structural information about the text data, facilitating subsequent calculations.

[0045] Furthermore, the adaptive threshold strategy involves setting an automatically adjusted threshold for the number of neighbors for each text in the K-nearest neighbor graph obtained in step one. This allows for the automatic filtering out of samples with low clustering effectiveness, resulting in a graph with an automatically adjusted adaptive structure. Figure 3 In the middle, x i This represents the original data sample. The threshold for the first filter is initialized, and the threshold for the m-th filter is... Its threshold size is the threshold of m-1 filters. Half of it, the formula is as follows:

[0046]

[0047] The purpose of setting a threshold is to automatically select suitable neighbor samples, i.e., structural information, which essentially reduces or increases the number of neighbors for each sample. This results in an undirected graph with adaptive structural information.

[0048] Furthermore, in step three, the automatically adjusted graph structure information obtained in step two is input into a neural network with an adaptive topology structure. A threshold decay strategy is used to allow the graph convolution kernel to dynamically adjust the topology range, thereby learning an adaptive structured semantic representation of the text.

[0049] This invention constructs an automatic learning module for structural representation to mine adaptive graph structural information. It utilizes a topological adaptive graph convolutional network to dynamically adjust the graph convolution kernel, thereby obtaining a dynamic structural representation based on the adaptive graph structure. The technical principle of this part is as follows:

[0050]

[0051]

[0052] in It is a multinomial convolution kernel. , For the first Multinomial convolution kernel, It is the first Layer polynomial parameters, where M represents the number of convolutional kernels, and b is the learnable bias matrix. For the first The number of vertices in each layer is used as a vector of all 1s, and A represents the similarity matrix of the automatically adjusted graph structure information obtained in step two. For the first An adaptive structured semantic representation of layered text, where σ is the activation function.

[0053] Furthermore, in step four, leveraging the superior performance of neural networks in representation extraction, an autoencoder is used to mine the hidden semantic representations of the text data, and these representations are then integrated layer by layer into the structural representations learned in step three. This process learns a semantic representation Z that is a fusion of the text's own semantic representations and structural representations. The technical principle behind this part is as follows:

[0054]

[0055]

[0056] Furthermore, step five specifically includes: dividing the fused and enhanced semantic representation obtained in step four into cluster distributions R using a softmax function; obtaining the text data distribution Q from the text semantic representation learned from the autoencoder using the Student t-distribution; calculating the distribution P based on this; calculating the difference loss between distributions P and Q and between distributions P and R using the KL divergence formula; minimizing this loss helps the model learn high-confidence assignments to fine-tune the model parameters. Based on this, combined with the reconstruction loss of the autoencoder, the final clustering result can be obtained through R, thus achieving the goal of using adaptively adjusted structural information to guide the clustering process and obtaining better text clustering results.

[0057] By utilizing both structural and semantic information to jointly supervise the clustering process, the problems of inconsistent text representations and insufficient supervision in unsupervised text clustering are effectively addressed. The dual self-supervised mechanism comprises two parts: calculating the difference loss between the high-confidence distribution P and the text semantic representation distribution Q using KL divergence, and calculating the difference loss between P and the distribution R of the fused semantic representation and adaptive structural representation. This dual supervision effectively counter-modulates the learning of the two representations, resulting in more stable and accurate clustering results. To achieve this goal, the present invention sets up an optimization function in the following form:

[0058]

[0059]

[0060] In summary, the deep text clustering method based on adaptive structure learning proposed in this invention has excellent performance.

[0061] The subscript i represents the i-th sample, the subscript j represents the j-th cluster center, p represents a value in P, r represents a value in R, and L ae L represents the difference loss between the P and Q distributions. tagcn This represents the difference loss between the P and R distributions.

[0062] Example 2: At least one embodiment of this disclosure also provides a deep text clustering device based on adaptive structure learning. Figure 4 This is a schematic block diagram of a deep text clustering apparatus based on adaptive structure learning provided in at least one embodiment of this disclosure. Figure 3As shown, the deep text clustering device based on adaptive structure learning includes a processor and a memory. The memory stores computer program instructions suitable for execution by the processor. When the processor executes these computer program instructions, it causes the processor to perform the deep text clustering method based on adaptive structure learning provided in at least one embodiment of this disclosure. For example, this deep text clustering device 1) can achieve better clustering results and better serve downstream tasks; 2) by simultaneously utilizing structural and semantic information to jointly supervise the clustering process, it effectively solves the problems of text representation differences and insufficient supervision in unsupervised text clustering; 3) introducing adaptive structural information into deep text clustering can yield better clustering results; 4) considering the role of multi-hop neighbors in text structure representation learning, it solves the problem of poor clustering performance in existing technologies, and improves the clustering effect by automatically selecting multi-hop neighbors during the text clustering process. The program instructions inside the computer can be divided according to function as follows: Figure 2 The functional modules shown.

[0063] For example, the processor may be a central processing unit (CPU), a graphics processing unit (GPU), a tensor processor (TPU), or other processing units with data processing and / or instruction execution capabilities. For instance, the processor can be implemented as a general-purpose processor, or as a microcontroller, microprocessor, digital signal processor, dedicated image processing chip, or field-programmable logic array, etc. For example, the memory may include at least one type of volatile memory and non-volatile memory, such as read-only memory (ROM), hard disk, flash memory, etc. Accordingly, the memory can be implemented as one or more computer program products, which may include various forms of computer-readable storage media on which one or more computer program instructions may be stored. The processor can execute the program instructions to perform any of the deep text clustering methods based on adaptive structure learning provided in at least one embodiment of this disclosure. The memory can also store various other applications and various data, such as various data used and / or generated by the applications.

[0064] Example 3: At least one embodiment of this disclosure also provides a storage medium (e.g., a non-transitory storage medium). Figure 5 This is a schematic block diagram of a storage medium provided in at least one embodiment of this disclosure. Figure 5As shown, the storage medium includes computer program instructions stored on the storage medium. When executed by a processor, the computer program instructions perform the deep text clustering method based on adaptive structure learning provided in at least one embodiment of this disclosure. For example, the storage medium 1) can achieve better clustering results and better serve downstream tasks; 2) by simultaneously utilizing structural and semantic information to jointly supervise the clustering process, it effectively solves the problems of text representation differences and insufficient supervision in unsupervised text clustering; 3) deep text clustering with adaptive structural information can obtain better clustering results; 4) it considers the role of multi-hop neighbors in text structure representation learning, solving the problem of poor clustering performance in existing technologies, and improving the clustering effect by automatically selecting multi-hop neighbors during the text clustering process.

[0065] For example, storage media can take many forms, including tangible storage media, carrier media, or physical transmission media. Stable storage media can include optical discs or magnetic disks, and other storage systems used in computers or similar devices that enable the system components described in the figure. Unstable storage media can include dynamic memory, such as the main memory of a computer platform. Tangible transmission media can include coaxial cables, copper cables, and optical fibers, such as the lines that form a bus within a computer system. Carrier transmission media can transmit electrical signals, electromagnetic signals, acoustic signals, or optical signals. These signals can be generated by radio frequency or infrared data communication methods. Typical storage media (e.g., computer-readable media) include hard disks, floppy disks, magnetic tapes, and any other magnetic media; CD-ROMs, DVDs, DVD-ROMs, and any other optical media; punched cards and any other physical storage media containing a punch pattern; RAM, PROMs, EPROMs, FLASH-EPROMs, and any other memory chips or magnetic tapes; carrier waves for transmitting data or instructions, cables or connection devices for transmitting carrier waves, and any other data that can be read using computer program instructions (e.g., program code) and / or by a computer.

[0066] Computer program instructions (e.g., program code) for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0067] In some examples, the functionality described in at least one embodiment of this disclosure may also be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0068] All aspects not detailed herein are well-known to those skilled in the art. Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of this invention and not intended to limit it. Although the invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this invention without departing from the spirit and scope of the invention, and all such modifications and substitutions should be covered within the scope of the claims of this invention.

Claims

1. A deep text clustering method based on adaptive structure learning, characterized in that, The method includes the following steps: Step 1: Construct a K-nearest neighbor graph from the original text data using the K-nearest neighbor method; Step 2: Filter out adjacent nodes with low similarity using an adaptive threshold strategy to generate an adaptive graph structure; Step 3: Input the graph with an adaptive structure into a neural network with an adaptive topology structure, and use a threshold decay strategy to allow the graph convolution kernel to dynamically adjust the topology range, thereby learning an adaptive structured semantic representation of the text; Step 4: Input the text's own information into the autoencoder to learn the text's own semantic representation, and then integrate the text's own semantic representation layer by layer into the adaptive structured semantic representation learned in Step 3, learning a fusion-enhanced semantic representation with both text's own semantic representation and structured semantic representation; Step 5: Use softmax to cluster the fused and enhanced semantic representation obtained in Step 4 to divide the cluster distribution. Use a dual self-supervised mechanism to calculate the loss between the adaptive structured semantic representation and the semantic representation of the text itself. Continuously back-adjust and optimize the encoder parameters to obtain the final semantic representation and text clustering results. The adaptive structured semantic representation of the text is calculated using the following formula: ， in, The number of layers in the neural network. , For the first Multinomial convolution kernel, It is the first Layer polynomial parameters, where M represents the number of convolutional kernels, and b is the learnable bias matrix. For the first The number of vertices in each layer is used as a vector of all 1s, and A represents the similarity matrix of the automatically adjusted graph structure information obtained in step two. For the first An adaptive structured semantic representation of layered text, where σ is the activation function; The dual self-supervised mechanism specifically includes: dividing the cluster distribution R, obtaining the text data distribution Q from the text semantic representation learned by the autoencoder using the student distribution, and calculating the distribution P based on this. The difference loss between distributions P and Q and between distributions P and R is calculated using the KL divergence formula. The high-confidence assignment is learned by minimizing the difference loss between distributions P and Q and between distributions P and R to fine-tune the model parameters. Combined with the reconstruction loss of the autoencoder, they together constitute the model's loss function. The clustering result is obtained through R. The method for calculating the difference loss between the P and Q distributions is as follows: ， Where the subscript i represents the i-th sample, the subscript j represents the j-th cluster center, p represents a value in P, q represents a value in Q, and L ae This represents the difference loss between the P and Q distributions; The method for calculating the difference loss between the P and R distributions is as follows: ， Where the subscript i represents the i-th sample, the subscript j represents the j-th cluster center, p represents a value in P, r represents a value in R, and L tagcn This represents the difference loss between the P and R distributions.

2. The deep text clustering method based on adaptive structure learning according to claim 1, characterized in that: Step one specifically involves: firstly, preprocessing the text data, using at least the following methods: bag-of-words model, TF, TF-IDF, or Word2Vec; then, utilizing the principle that texts and their neighbors have similar features, constructing a K-nearest neighbor graph with K neighbors for each text using the KNN algorithm.

3. The deep text clustering method based on adaptive structure learning according to claim 1, characterized in that: The adaptive threshold strategy involves setting a threshold for the number of neighbors for each text in the K-nearest neighbor graph obtained in step one.

4. The deep text clustering method based on adaptive structure learning according to claim 3, characterized in that: The threshold for automatically adjusting the number of neighbors is set using the following method: Initialize the threshold of the first filter to 0.006, and set the threshold of the m-th filter accordingly. Its threshold size is the threshold of m-1 filters. Half of it, the formula is as follows: 。 5. The deep text clustering method based on adaptive structure learning according to claim 1, characterized in that: The fused and enhanced semantic representation is calculated iteratively using the following formula: ， ， in, Indicates the first Layer's own semantic representation, Indicates the first Layer-adaptive structured semantic representation; Indicates the first Adaptive structured semantic representation of layered text Indicates the first Layer fusion-enhanced semantic representation, where σ is the activation function and b is a learnable bias matrix. for The number of vertices in a layer is represented as a vector of all 1s. For the first Multinomial convolution kernel.

6. A deep text clustering device based on adaptive structure learning, characterized in that: The apparatus includes a processor and a memory, wherein the memory stores computer program instructions suitable for execution by the processor, the computer program instructions being executed by the processor causing the processor to perform the deep text clustering method based on adaptive structure learning as described in any one of claims 1-5.

Images