A text data clustering method

Abstract

The application relates to a text data clustering method, and belongs to the technical field of big data mining, and solves the problem of poor clustering performance in the prior art. The method comprises the following steps: obtaining text data in a target time period; extracting a structured fusion vector of each piece of text data; deducing core parameters of hierarchical density clustering based on data density distribution in the target time period; and clustering based on the structured fusion vector of each piece of text data by using a hierarchical density clustering algorithm to obtain a clustering result. The method improves the clustering efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of big data mining technology, and in particular to a method and system for clustering text data. Background Technology

[0002] With the widespread adoption of various online platforms such as social media, news websites, and forums, massive amounts of text information are generated daily, containing a wealth of event clues. How to quickly and accurately extract and cluster meaningful information about trending events from this open-source data using big data processing techniques has become a key issue in the field of intelligence analysis.

[0003] Hierarchical density clustering is a commonly used clustering method. Existing optimization methods for hierarchical density clustering parameters mainly rely on grid search combined with external evaluation metrics such as silhouette coefficients for iterative adjustments. This approach suffers from high computational costs, inability to guarantee parameter consistency across adjacent time windows, and the possibility that the evaluation metrics themselves may not be suitable for the characteristics of text data. This iterative approach of "clustering first, then evaluating, and then adjusting" is not only inefficient but also difficult to adapt to the real-time parameter adjustment requirements of streaming data scenarios. Summary of the Invention

[0004] Based on the above analysis, the embodiments of the present invention aim to provide a text data clustering method to solve the problem of poor performance of existing clustering methods.

[0005] On one hand, embodiments of the present invention provide a text data clustering method, including the following steps: Retrieve text data within a target time period; Extract the structured fusion vector from each text data entry; The core parameters of hierarchical density clustering are derived based on the data density distribution within the target time period; the hierarchical density clustering algorithm is used to cluster the data based on the structured fusion vector of each text data to obtain the clustering results.

[0006] Based on the above method, the core parameters of hierarchical density clustering are derived in the following way: For each text data point within the target time period, calculate its core distance to the k-th nearest neighbor based on its structured fusion vector. ; The density boundary threshold is determined based on the core distance of all text data; Text data with a core distance smaller than the density boundary threshold are used to obtain a high-density sample set. Core parameters are determined based on a high-density sample set.

[0007] Based on the above method, a further improvement is made to determine the core parameters using a high-density sample set: A minimum spanning tree is constructed based on the reachability distance of the high-density sample set; the minimum spanning tree is pruned to obtain multiple connected subgraphs; the minimum cluster size is obtained based on the size of the minimum connected subgraphs. The minimum number of nearest neighbors is determined based on the minimum cluster size and local stability tests.

[0008] Based on the above method, the minimum nearest neighbor number is determined according to the minimum cluster size in the following way: Candidate values ​​for the minimum neighbor number are determined based on the minimum cluster size; Calculate the coefficient of variation of the core distance distribution for each candidate value; The candidate value corresponding to the smallest coefficient of variation is used as the smallest nearest neighbor number.

[0009] Based on the further improvement of the above method, the coefficient of variation is calculated using the following formula:

[0010] in, This represents the core distance from the m-th nearest neighbor to each text data point in the high-density sample set. Indicates taking the variance. This indicates taking the average.

[0011] Based on the above method, a hierarchical density clustering algorithm is used to cluster the structured fusion vector of each text data, including: The text data is initially clustered based on the hierarchical density clustering algorithm to obtain initial clusters. The initial clusters are merged in multiple stages to obtain the final clusters.

[0012] Based on further improvements to the above method, a multi-stage inter-cluster merging process is performed on the initial clusters to obtain the final clusters, including: Entity co-occurrence detection is performed on any two initial clusters to obtain the correlation degree of event elements, and candidate cluster pairs are determined based on the correlation degree of event elements. Calculate the density bridging degree and hierarchical correlation degree for each pair of candidate clusters; The candidate cluster pairs are merged based on their event element correlation, density bridging, and hierarchical structure correlation; the final cluster is then obtained.

[0013] Based on the above method, the density bridging degree of candidate cluster pairs is calculated in the following way: Determine the boundary samples of each cluster in the candidate cluster pair, and calculate the local density of each boundary sample; The density bridging degree of candidate cluster pairs is obtained based on the local density of boundary samples.

[0014] Based on the further improvement of the above method, the local density of the boundary sample is calculated using the following formula:

[0015] in, Representing boundary samples Local density, This represents the total number of boundary samples in the candidate cluster pair. This indicates that the boundary sample x is the center and the boundary sample is the boundary sample. The volume of a hypersphere with a radius equal to the distance between its k nearest neighbors and the core.

[0016] Based on the further improvement of the above method, the following formula is used to calculate the correlation degree of event elements:

[0017]

[0018]

[0019] in, and Indicates the weighting coefficient. Represents the co-occurrence rate of entities. Indicates the verb reduplication rate. and Representing the initial clusters and A collection of named entities and Representing the initial clusters and The set of event verbs, This indicates the number of elements in the set.

[0020] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. Determine the core parameters for clustering based on the actual distribution of the data first, avoiding clustering first, then evaluating, and then adjusting. This not only improves the efficiency of parameter determination, but also ensures that the determined core parameters are more in line with the characteristics of the text data to be clustered, thereby improving the accuracy of clustering.

[0021] 2. The multi-stage inter-cluster merging algorithm comprehensively utilizes three dimensions of information: entity co-occurrence, spatial density, and hierarchical structure. It effectively solves the problems of under-merging and over-merging caused by single threshold judgment, and improves the accuracy of clustering.

[0022] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0023] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 This is a flowchart of the text data clustering method in this embodiment. Detailed Implementation

[0024] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0025] A specific embodiment of the present invention discloses a text data clustering method, such as... Figure 1 As shown, it includes the following steps: S1. Obtain text data within the target time period; S2. Extract the structured fusion vector for each text data; S3. Based on the data density distribution within the target time period, deduce the core parameters of hierarchical density clustering; based on the structured fusion vector of each text data, use the hierarchical density clustering algorithm to perform clustering and obtain the clustering results.

[0026] Compared with existing technologies, the text data clustering method provided in this embodiment extracts the structured fusion vector of each text data, deduces the core parameters of hierarchical density clustering based on the data density distribution within the target time period, and then uses the hierarchical density clustering algorithm to cluster based on the structured fusion vector of each text data to obtain the clustering results. Thus, the core parameters of clustering are determined first according to the actual data distribution, avoiding the need for clustering first, then evaluating, and then adjusting. This not only improves the efficiency of parameter determination, but also ensures that the determined core parameters are more consistent with the characteristics of the text data to be clustered, thereby improving the accuracy of clustering.

[0027] During implementation, text data is acquired in real time from multiple data sources, such as open data sources like social media platforms, news portals, and forum posts. The data is updated using a sliding time window approach via APIs or web crawlers.

[0028] During implementation, the text content of these data sources can be captured in real time based on a set sliding window (e.g., sliding every 6 hours).

[0029] During implementation, the collected raw text data undergoes preprocessing, including cleaning (removing invalid content such as HTML tags, whitespace characters, emojis, and URL links), deduplication (using SimHash for approximate deduplication with a similarity threshold of 0.9), and stop word removal. Text data within the target time period is then extracted.

[0030] Furthermore, event elements are extracted from each text data point within the target time period. A structured fusion vector for each text data point is obtained based on its TF-IDF vector and the event element encoding vector. Specifically, this includes: Calculate the TF-IDF vector for each text data item; Event elements are extracted from text data based on dependency parsing, and the event element encoding vector for each text data is obtained based on the embedding vector of each event element. The TF-IDF vector and event element encoding vector of the text data are fused to obtain the structured fused vector of the text data.

[0031] In practice, for example, word segmentation tools such as LAC or Jieba can be used to segment each piece of text data within the target time period, and the TF-IDF vector of each piece of text data can be calculated based on the segmentation results. For example, TfidfVectorizer can be used to obtain the TF-IDF vector of each piece of text data.

[0032] In implementation, to overcome the predicament of traditional TF-IDF being dominated by high-frequency background words (such as "held", "stated", "today") that obscure key proper nouns, one embodiment of the present invention calculates the TF-IDF vector for each piece of text data in the following manner: For each piece of text data, entity recognition is performed to obtain the named entity of each piece of text data. Based on the recognized named entity, the text data is labeled with entity type. Based on the entity type label, the TF-IDF vector of each piece of text data is calculated.

[0033] In implementation, existing named entity recognition models can be used to identify the named entities in each piece of text data. Then, the text data is modified by adding entity type labels to the entities as new words, and then the TF-IDF vector of each piece of text is calculated.

[0034] For example, if the original word segmentation of a text is [Zhang San, went to, Beijing], and named entities + type labels are used as new words, the transformed word segmentation will be [Zhang San, went to, Beijing, PER_Zhang San, LOC_Beijing]. Feature words like PER_Zhang San and LOC_Beijing will be added to the vocabulary, allowing the model to simultaneously capture word frequency and entity type information.

[0035] By explicitly transforming named entities (such as people, organizations, and locations) into independent enhanced features, it can significantly improve the sensitivity of the core subject of an event in the vector space. This allows texts describing the same hot topic (such as "a company's press conference" or "a natural disaster in a certain place") to exhibit a stronger clustering effect in clustering calculations, regardless of changes in verbs and modifiers, due to the sharing of high-intensity entity features (such as PER_Zhang San, LOC_Beijing). At the same time, the explicitly labeled entity tags can accurately separate the vector distance between easily confused texts such as "Apple Inc." and "apple fruit," ultimately making the cluster granularity of the cluster output more closely match the granularity of the real topic, greatly reducing misclustering and loose clustering caused by semantic generalization.

[0036] To better distinguish between different expressions of the same event and similar expressions of different events, and to reduce the possibility of the same event being scattered across multiple clusters or unrelated texts being mistakenly included in the same cluster, this invention extracts event elements from each piece of text data.

[0037] During implementation, event elements are extracted from the text data based on dependency parsing. Event elements include a quintuple of events: (Who, What, When, Where, Why).

[0038] During implementation, dependency parsing (such as LTP or HanLP tools) is applied to each text d, and event quintuple elements are extracted from dependency arc relations (SBV subject-verb relation, VOB verb-object relation, ATT attributive-head relation, etc.). Who: Locate the subject noun phrase through SBV relationship, and identify the PER / ORG entities within it by combining NER; What: Locate the core verb (HED relation), extract all arguments of that verb (SBV, VOB) and combine them to form an event action phrase; When: Extract time information through TIME entity recognition and time expression normalization; Where: Extract location information through LOC entity recognition; Why: Locate the reason clause by using causal conjunctions ("because", "due to", "the reason is", etc.).

[0039] The extracted result is represented as a quintuple: T(d) = (Who, What, When, Where, Why).

[0040] In implementation, each identified element is independently encoded to obtain an embedding vector for each event element. Existing text embedding models (such as the BERT encoder) can be used to encode the event elements to obtain the embedding vector for each event element.

[0041] In implementation, entity-aware encoding is used for the Who and Where elements (special markers [ENT]...[ / ENT] are inserted before and after the entity token before being fed into the BERT encoder to obtain the output at the corresponding position). For the What element, the core verb and its argument combination are extracted and then fed into BERT to obtain the output vector at the [CLS] position. For the When element, the time entity is normalized (e.g., "last Friday" → specific date "2026-XX-XX"), and the normalization result is mapped to a time-series encoded vector of fixed dimension d_h through sinusoidal positional encoding. For the Why element, after locating the reason clause, the clause is independently fed into BERT encoding, and the average pooled vector of all token outputs is obtained.

[0042] In practice, the embedding vectors of each element are mapped to the same dimension, and then the event element encoding vector of each text data is obtained based on the embedding vector of each event element.

[0043] Specifically, the weight of each event element for the target time period is determined based on the data source type. In practice, social media sources focus more on the task (Who) and event action (What), news source data focuses more on the location (Where) and event (What), and forum source text focuses more on the reason (Who). The sum of the weights of the five event elements is 1.

[0044] The event element encoding vector of the text data is obtained by weighted fusion of the embedding vectors of the event elements.

[0045] During implementation, the weights of event elements can be updated based on the final clustering results. After obtaining the final clustering results, for each event element, the variance of that event element in the paper's data within each cluster is calculated, and the average variance of all clusters is taken to obtain the mean variance of that event element. Then, the mean of the mean variances of all event elements is calculated.

[0046] Update the weights using the following method:

[0047] in, This represents the weight coefficient of the i-th event element at the previous time (t-1). This represents the update weight of the i-th event element. This represents the mean of the average variances of all event elements. Let represent the average variance of the i-th event element.

[0048] The update weights need to be normalized so that their sum equals 1, resulting in the final update weights. Features with smaller variance have larger weights (meaning they contribute more to intra-cluster consistency), thus achieving adaptive feature weight updates driven by cluster feedback.

[0049] The TF-IDF vector and event element encoding vector of the text data are fused to obtain the structured fused vector of the text data.

[0050] Specifically, the fusion will be carried out in the following ways: Map the TF-IDF vectors of text data to the same dimensional space as the event element encoding vectors; Using the TF-IDF vector of text data as the query and the event element encoding vector as the key and value, a structured fusion vector of text data is obtained through scaling dot product attention.

[0051] During implementation, since TF-IDF vectors are high-dimensional sparse vectors, the TF-IDF vectors of the text data are first mapped to the same dimensional space as the event element encoding vectors through linear layer projection.

[0052] Then, fusion is achieved by scaling the dot product attention:

[0053] in, A structured fusion vector representing text data. The TF-IDF vector representing text data. The event element encoding vector represents the text data. This represents the scaling factor, which is the dimension of the event element encoding vector. This represents the softmax function.

[0054] Structured fusion vector It retains the global word frequency features of the text (introduced through queries) and enhances the semantic information related to the event structure (introduced through keys and values).

[0055] After obtaining the structured fusion vector of each text data, a hierarchical density clustering algorithm is used for clustering.

[0056] The core parameters of the hierarchical density clustering algorithm include: Minimum cluster size (min_cluster_size): Minimum number of nodes per cluster. Minimum nearest neighbor number min_samples: controls the strictness of noise point detection.

[0057] Unlike the existing trial-and-error approach of "grid search + silhouette coefficient evaluation", this invention determines the core parameters of hierarchical density clustering based on the data density distribution within the target time period.

[0058] Specifically, the core parameters of hierarchical density clustering are derived using the following method: For each text data point within the target time period, calculate its core distance to the k-th nearest neighbor based on its structured fusion vector. ; Density thresholds are determined based on the core distance of all text data. ; The statistical core distance is less than the density boundary threshold. The text data yields a high-density sample set. ; Core parameters are determined based on a high-density sample set.

[0059] In practice, for a given text data, the core distance of that text data is obtained by calculating its distance to its k-th nearest neighbor. In implementation, the distance between the text data and its k-th nearest neighbor's structured fusion vector is used as the core distance of the text data. During implementation, k can be selected based on the amount of text data in the target time period, for example, taking a specific value. .

[0060] The distance between any two text data is:

[0061] in, This represents the core distance of the i-th text data. This represents the core distance of the j-th text data. This represents the distance between the i-th text data and the j-th text data.

[0062] Construct a core distance distribution histogram for N text data points. The bin of the histogram can be selected. .

[0063] The density decomposition threshold can be obtained by performing bimodal analysis on the histogram. .

[0064] During implementation, the core distance can also be considered. Sort by size from largest to smallest, and use the 75th percentile as the density decomposition threshold. .

[0065] Also available to Sort the curves in ascending order and find the point where the slope of the curve changes the most (where the second derivative is the largest) as the starting point. .

[0066] Statistical core distance less than The text data yields a high-density sample set. Then, the core parameters were determined based on the high-density sample set.

[0067] Specifically, the core parameters are determined based on a high-density sample set using the following method: A minimum spanning tree is constructed based on the reachability distance of the high-density sample set; the minimum spanning tree is pruned to obtain multiple connected subgraphs; the minimum cluster size is obtained based on the size of the minimum connected subgraphs. The minimum number of nearest neighbors is determined based on the minimum cluster size and local stability tests.

[0068] In implementation, a minimum spanning tree is constructed for the high-density sample set based on the mutual reachability distance. Based on a preset first threshold, the minimum spanning tree is pruned by deleting edges with weights greater than the first threshold, thus obtaining multiple connected subgraphs.

[0069] In practice, a certain quantile (such as the 90th percentile) of the reachability distance of the high-density sample set can be used as the first threshold.

[0070] Since the reachability distance between two points is inversely proportional to the density connectivity between them, high-weight edges correspond to low-density gap regions in the feature space, serving as natural boundaries for inter-cluster partitioning; while low-weight edges correspond to internal connections within high-density core regions. Therefore, by removing high-weight edges exceeding a threshold, complete and independent high-density connected components can be effectively separated, thereby accurately estimating the minimum cluster size.

[0071] In implementation, the size of the minimum connected subgraph is used as the initial value of the minimum cluster size, mcs0. A lower bound constraint is applied to min_cluster_size: min_cluster_size = max(mcs0, N × 0.01 The final determined minimum cluster size is obtained.

[0072] The minimum nearest neighbor number is determined based on the minimum cluster size through a local stability test, specifically including: Candidate values ​​for the minimum neighbor number are determined based on the minimum cluster size; Calculate the coefficient of variation of the core distance distribution for each candidate value; The candidate value corresponding to the smallest coefficient of variation is used as the smallest nearest neighbor number.

[0073] Set the minimum nearest neighbor number min_samples to [ min_cluster_size / 3 , min_cluster_size / 2 Within the range of [ ], calculate min_samples in [ min_cluster_size / 3 , min_cluster_size / 2 Candidate values ​​within the range Coefficient of variation of the lower core distance distribution ,in, This represents the core distance from the m-th nearest neighbor to each text data point in the high-density sample set. Indicates taking the variance. This indicates taking the average.

[0074] Through stability testing, a minimum number of nearest neighbors is selected that can smooth density estimation and avoid local extreme fluctuations.

[0075] During implementation, to ensure the stability of clustering results across consecutive windows and avoid event tracking breaks caused by parameter jumps, parameter consistency checks can be performed. For each core parameter, calculate the rate of change from the previous time period to the target time period. If the rate of change exceeds the preset rate of change threshold, then smooth the core parameter.

[0076] For each core parameter p (min_cluster_size or min_samples), calculate the rate of change:

[0077] in, The core parameters representing the target time period This represents the core parameter of the previous time period before the target time period. If the rate of change exceeds the rate of change threshold (e.g., 0.3), smoothing is performed.

[0078] in, This represents the smoothing coefficient, for example, 0.7. Hierarchical density clustering is performed based on the smoothed core, thus ensuring the consistency of clustering results between adjacent time windows.

[0079] After determining the core parameters, hierarchical density clustering algorithm is used to cluster the data based on the structured fusion vector of each text data. Specifically, this includes: The text data is initially clustered based on the hierarchical density clustering algorithm to obtain initial clusters. The final cluster is obtained by multi-stage inter-clustering based on the co-occurrence of event elements and density connectivity.

[0080] During implementation, based on the determined parameters, the existing hierarchical density clustering algorithm is used for preliminary clustering to obtain initial clusters.

[0081] A three-phase merge decision is performed on the initial cluster to obtain the final cluster.

[0082] Specifically, it includes: Entity co-occurrence detection is performed on any two initial clusters to obtain the correlation degree of event elements, and candidate cluster pairs are determined based on the correlation degree of event elements. Calculate the density bridging degree and hierarchical correlation degree for each pair of candidate clusters; The candidate cluster pairs are merged based on their event element correlation, density bridging, and hierarchical structure correlation; the final cluster is then obtained.

[0083] Specifically, the correlation degree of event elements is calculated using the following method: For two initial clusters and Extract the initial clusters respectively and The set of named entities and the set of event verbs; The entity co-occurrence rate and verb overlap rate are calculated based on the named entity set and the event verb set, respectively. The initial clusters are obtained by weighted summation of entity co-occurrence rate and verb redundancy rate. and The degree of correlation between event elements.

[0084] Specifically, the formula is used. Calculate the entity co-occurrence rate. Among them, and Representing the initial clusters and A collection of named entities This indicates the number of elements in the set.

[0085] Specifically, the formula is used. Calculate the verb reduplication rate. Among them, and Representing the initial clusters and The set of event verbs, This indicates the number of elements in the set.

[0086] Relevance of Event Elements , and This represents the weighting coefficients, and the sum of the two is 1. For example, take... .

[0087] Specifically, the density bridging degree of candidate cluster pairs is calculated using the following method: Determine the boundary samples of each cluster in the candidate cluster pair, and calculate the local density of each boundary sample; The density bridging degree of candidate cluster pairs is obtained based on the local density of boundary samples.

[0088] Specifically, for candidate clusters Clusters The boundary samples in the data are distance clusters. The top 10% of samples closest to the centroid. Cluster The boundary samples in the data are distance clusters. The centroid is the nearest 10% of the samples.

[0089] Specifically, the local density of the boundary samples is calculated using the following formula:

[0090] in, This represents the local density of the boundary sample x. This represents the total number of boundary samples in the candidate cluster pair. Let x represent the volume of the hypersphere centered at the boundary sample x and with the k nearest neighbor core distance of the boundary sample x as its radius.

[0091] The physical meaning of the local density calculation formula is: taking a point... Centered on, to its first Draw a hypersphere with the nearest neighbor distance as the radius, and count the proportion of points contained inside the hypersphere, i.e., the local density.

[0092] In practice, k is determined using the aforementioned minimum nearest neighbor number.

[0093] The density of the boundary samples is compared with the noise density threshold. (In practice, noise density is estimated using the average k-nearest neighbor distance of noise points marked by HDBSCAN). Density bridging is calculated using the following formula:

[0094] in, Representing a cluster The boundary sample set, Representing a cluster The boundary sample set. This indicates the number of elements in the set.

[0095] Specifically, using the compressed hierarchical tree generated by HDBSCAN, the hierarchical association degree of candidate cluster pairs is calculated using the following formula:

[0096] in, Representing a cluster and The nearest common ancestor node of Value (density level when the node splits). The maximum value in the hierarchical tree representing hierarchical density clustering. value.

[0097] when This indicates that the two clusters split only at extremely high density levels and do not share a parent cluster. .

[0098] The event element correlation, density bridging degree, and hierarchical structure correlation of candidate cluster pairs are weighted and comprehensively evaluated as the merging decision:

[0099] in, , and These represent weighting coefficients, which sum to 1, for example... , , .

[0100] Only when Exceeding the adaptive threshold The merging process is performed in real time. This achieves a more accurate and robust inter-cluster merging effect than a single threshold judgment.

[0101] Adaptive threshold The calculation is as follows:

[0102] in, This represents the base threshold (in practice, it is reduced to 0.5). Indicates the current number of clusters. This represents the expected number of clusters (which can be calculated based on the average number of clusters over several past time periods).

[0103] when > When the ln term is positive, Increase to avoid over-merging; when < When the ln term is negative, Reduce to facilitate merger.

[0104] By utilizing three dimensions of information—event association, spatial density, and hierarchical structure—the problem of under-merging and over-merging caused by a single threshold judgment is effectively solved, thereby improving the accuracy of clustering.

[0105] For each text data in each final cluster, calculate its generalized Jaccard similarity with the cluster it belongs to, take the text data with the highest score as the center sentence, and further extract the event summary through the subject-verb-object structure as the hot events corresponding to the cluster.

[0106] Based on the occurrence time and cluster density of each event, a popularity score for popular events is calculated. ,in The number of samples within the cluster. The system automatically generates a trend chart and a time-series evolution chart of the event's popularity based on the frequency of mentions within the target time period.

[0107] In implementation, the similarity between trending events and each text within the target time period is calculated (which can be based on structured fusion vectors). If the similarity exceeds a threshold (e.g., 80%), the frequency is incremented by 1, thus obtaining... .

[0108] During implementation, the final clustering events and their sub-topics can be displayed graphically, including keyword clouds, event lists, and hierarchical relationship trees.

[0109] During implementation, periodic event monitoring reports are automatically generated, including TOP-N event summaries, central sentence extraction, and keyword statistics, improving the efficiency of analysts.

[0110] Electronic device example: One specific implementation of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the text data clustering method in the method embodiment.

[0111] Examples of readable storage media: One specific implementation of this application discloses a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the text data clustering method in the method embodiment.

[0112] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0113] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A text data clustering method, characterized in that, Includes the following steps: Retrieve text data within a target time period; Extract the structured fusion vector from each text data entry; The core parameters of hierarchical density clustering are derived based on the data density distribution within the target time period; the hierarchical density clustering algorithm is used to cluster the data based on the structured fusion vector of each text data to obtain the clustering results.

2. The text data clustering method according to claim 1, characterized in that, The core parameters of hierarchical density clustering are derived using the following method: For each text data point within the target time period, calculate its core distance to the k-th nearest neighbor based on its structured fusion vector. ; The density boundary threshold is determined based on the core distance of all text data; Text data with a core distance smaller than the density boundary threshold are used to obtain a high-density sample set. Core parameters are determined based on a high-density sample set.

3. The text data clustering method according to claim 2, characterized in that, The core parameters are determined based on a high-density sample set using the following method: A minimum spanning tree is constructed based on the reachability distance of the high-density sample set; the minimum spanning tree is pruned to obtain multiple connected subgraphs; the minimum cluster size is obtained based on the size of the minimum connected subgraphs. The minimum number of nearest neighbors is determined based on the minimum cluster size and local stability tests.

4. The text data clustering method according to claim 3, characterized in that, The minimum nearest neighbor number is determined based on the minimum cluster size using the following method: Candidate values ​​for the minimum neighbor number are determined based on the minimum cluster size; Calculate the coefficient of variation of the core distance distribution for each candidate value; The candidate value corresponding to the smallest coefficient of variation is used as the smallest nearest neighbor number.

5. The text data clustering method according to claim 4, characterized in that, The coefficient of variation is calculated using the following formula: in, This represents the core distance from the m-th nearest neighbor to each text data point in the high-density sample set. Indicates taking the variance. This indicates taking the average.

6. The text data clustering method according to claim 1, characterized in that, Clustering is performed using a hierarchical density clustering algorithm based on the structured fusion vector of each text data point, including: The text data is initially clustered based on the hierarchical density clustering algorithm to obtain initial clusters. The initial clusters are merged in multiple stages to obtain the final clusters.

7. The text data clustering method according to claim 6, characterized in that, The initial clusters are merged in multiple stages to obtain the final clusters, including: Entity co-occurrence detection is performed on any two initial clusters to obtain the correlation degree of event elements, and candidate cluster pairs are determined based on the correlation degree of event elements. Calculate the density bridging degree and hierarchical correlation degree for each pair of candidate clusters; The candidate cluster pairs are merged based on their event element correlation, density bridging, and hierarchical structure correlation; the final cluster is then obtained.

8. The text data clustering method according to claim 7, characterized in that, The density bridging degree of candidate cluster pairs is calculated using the following method: Determine the boundary samples of each cluster in the candidate cluster pair, and calculate the local density of each boundary sample; The density bridging degree of candidate cluster pairs is obtained based on the local density of boundary samples.

9. The text data clustering method according to claim 8, characterized in that, The local density of the boundary sample is calculated using the following formula: in, Representing boundary samples Local density, This represents the total number of boundary samples in the candidate cluster pair. This indicates that the boundary sample x is the center and the boundary sample is the boundary sample. The volume of a hypersphere with a radius equal to the distance between its k nearest neighbors and the core.

10. The text data clustering method according to claim 7, characterized in that, The relevance of event elements is calculated using the following formula: in, and Indicates the weighting coefficient. Represents the co-occurrence rate of entities. Indicates the verb reduplication rate. and Representing the initial clusters and A collection of named entities and Representing the initial clusters and The set of event verbs, This indicates the number of elements in the set.

Images