Text clustering system for open source security intelligence

Abstract

The invention provides a text clustering system for open source security intelligence. The text clustering system comprises a data acquisition module, a preprocessing module, a feature extraction module, a similarity calculation module, a clustering initialization module, an iterative clustering module, an evaluation module and a result output module. The system collects a target text, removes noise, calculates a word frequency and an inverse document frequency to construct a feature vector, calculates similarity after normalization processing, randomly selects an initial clustering center, iteratively updates the clustering center until the clustering center is stable, evaluates a clustering effect, outputs a clustering result and generates a topic tag. According to the invention, the analysis efficiency and accuracy of open source security intelligence can be improved, and security threats can be quickly identified and responded.

Description

Technical Field

[0001] The present invention relates to the field of open source security intelligence, and more specifically, to a text clustering system for open source security intelligence. Background Art

[0002] In the field of open source security intelligence, with the rapid development of Internet technology and the explosive growth of information, how to quickly and accurately extract and analyze security-related intelligence from massive data has become an urgent problem that needs to be solved. Existing text processing technologies often rely on manual screening and classification, which is not only inefficient, but also easily affected by subjective judgment and difficult to adapt to the needs of large-scale data processing. In addition, traditional clustering algorithms often find it difficult to achieve ideal clustering effects when processing text data with complex structures and high-dimensional features, and when dynamically updating data, the complete clustering calculation needs to be re-performed, which is not only time-consuming but also computationally expensive.

[0003] In the process of implementing the embodiments of the present invention, the inventors found that there are at least the following problems or defects in the prior art: the existing text clustering system lacks adaptability to the characteristics of specific data sources when processing open source security intelligence, resulting in uneven quality of collected data; in the preprocessing stage, the removal of noise data is not accurate enough, affecting the accuracy of subsequent feature extraction; in the feature extraction process, the importance weights of words are not fully considered, resulting in inaccurate feature vector construction; in the clustering process, the method of initializing and iteratively updating the cluster center lacks optimization and is prone to falling into local optimal solutions; in the clustering evaluation stage, there is a lack of effective evaluation indicators to measure the clustering effect; when the results are output, the generated topic labels are not accurate enough and it is difficult to accurately reflect the substantive content of the clustering results. In addition, when facing data updates, the existing system lacks an effective incremental clustering algorithm and cannot quickly adapt to the addition of new data. These problems limit the efficiency and accuracy of open source security intelligence processing. Summary of the invention

[0004] The present invention provides a text clustering system for open source security intelligence, comprising: Data collection module, used to collect target text data containing open source security intelligence; The text preprocessing module is used to remove noise data from the target text data by using regular expression matching; A feature extraction module is used to calculate the word frequency of a word in the target text data and the inverse document frequency of the word in the entire target text data set based on the preprocessed target text data, determine the word importance weight according to the product of the two, and construct a feature vector; A similarity calculation module is used to perform normalization processing on the feature vectors and calculate the similarity between the normalized feature vectors; The cluster initialization module is used to randomly select k target text data as initial cluster centers, where k is the preset number of clusters; Iterative clustering module: used to assign the remaining target text data to the corresponding clusters according to the similarity with the cluster center. In each iteration, the cluster center is recalculated by the mean calculation method. The cluster center feature vector is the average value of the feature vectors of all target text data in the cluster. The target text data is assigned and the cluster center is updated repeatedly until the cluster center is stable or the preset number of iterations is reached. The clustering evaluation module is used to calculate the average distance between the target text data and other target text data in the cluster to which it belongs and the average distance to the target text data in other clusters, determine the evaluation index value of the target text data based on the relationship between the two, calculate the average value of the evaluation index values ​​of all target text data to obtain the evaluation index value of the clustering result, and judge the clustering effect accordingly; The result output module is used to output the final clustering results. Each cluster represents a class of similar open source security intelligence texts. By counting the high-frequency words in the target text data in the cluster and using semantic analysis tools to select representative words or phrases to generate topic labels, the word order is adjusted and redundant words are removed.

[0005] Furthermore, in the data collection module, collecting target text data containing open source security intelligence includes: Construct different collection strategy rules based on the characteristics of different target text data sources, set data acquisition limits according to data source rules, use the parsing library to parse the data source structure for label recognition and content extraction, filter the target text data length, determine the specific length range based on historical data statistical analysis, and only retain the target text data within this range; The collection depth is calculated based on the total amount of relevant target text data and the ratio of target text data related to open source security intelligence. The formula is: D=√(N×P); where D is the collection depth, N is the total amount of relevant target text data in the data source, and P is the ratio of target text data related to open source security intelligence. The target text data is collected based on the collection depth to determine the scope of collection.

[0006] Furthermore, in the text preprocessing module, the use of regular expression matching to remove noise data in the target text data includes: A regular expression is constructed based on the characteristics of noise data, and advertising noise information and special symbol noise are removed based on regular expression matching.

[0007] Furthermore, in the feature extraction module, based on the preprocessed target text data, the word frequency in the target text data and the inverse document frequency in the entire target text data set are calculated, and the word importance weight is determined according to the product of the two. The feature vector is constructed including: Count the number of times words appear in the target text data to get the word frequency, determine the inverse document frequency, and calculate the word In the target text data d Value, the formula is: in, For words In the target text data In value, For words In the target text data The word frequency in is the total number of target text data in the target text dataset, To contain words The target text data number.

[0008] Furthermore, in the similarity calculation module, the feature vectors are normalized, and the similarity between the normalized feature vectors is calculated by: Normalized processing, the calculation formula is: Calculate the similarity between normalized feature vectors, assuming that the feature vector The calculation formula is: Where n is the dimension of the feature vector.

[0009] Furthermore, in the clustering initialization module, the target text data set is randomly sorted, and the first k target text data are selected as the initial centers; Calculate the average similarity between the initial center and the surrounding target text data. The formula is: in, is the initial central target text data, is the surrounding target text data set, for The characteristic vector of for middle The feature vector of the target text data, for The elements, for The elements, for If the similarity is lower than the set value, the initial center is replaced. The set value is determined according to experimental data.

[0010] Furthermore, in the iterative clustering module, after each update of the cluster center, a specific optimization algorithm is used to locally optimize the clustering results. The initial parameters of the algorithm are set based on the scale and complexity of the target text data set, the parameter change rate is determined experimentally, and the probability of accepting an inferior solution is calculated. The formula is: in, is the energy difference, that is, the difference between the objective function value of the new solution and the current solution, is the current parameter value, and with a certain probability, it accepts inferior solutions to jump out of the local optimum. The termination condition is set when the clustering result does not improve after several consecutive iterations. The number of iterations is determined by experiments.

[0011] Furthermore, in the clustering evaluation module, the target text data is calculated To the cluster to which it belongs The average distance of other target text data , to other clusters Average distance between target text data , calculate the target text data The evaluation index value is: in, The target text data To the cluster to which it belongs The average distance of other target text data within The target text data To other clusters The average distance of the internal target text data is calculated, and the average value of the evaluation index values ​​of all target text data is obtained to obtain the evaluation index value of the clustering result. A threshold is set. If the average value of the evaluation index value is lower than the threshold, it is determined that the clustering effect is not ideal. The threshold is determined based on experimental and empirical data; At the same time, calculate the internal density of each cluster, assuming that the cluster The amount of target text data is , the eigenvector is , the cluster center feature vector is , the calculation formula is: in, For clustering The amount of target text data, For clustering Internal target text data The characteristic vector of is the cluster center feature vector, for The elements, for The elements, and the clustering quality is judged by combining the evaluation index value and the internal compactness.

[0012] Furthermore, in the result output module, the threshold of high-frequency words is determined based on the number of target text data and word distribution in the cluster. The frequency distribution of words in the cluster is set as The total number of words is , the calculation formula is: in, is the frequency distribution of words within the cluster, For the cluster The frequency of words, is the total number of words, is the adjustment coefficient; By utilizing the semantic relationship of words in the semantic analysis tool, topic tags are selected based on statistical high-frequency words for expansion and association, and the generated topic tags are manually reviewed and revised based on professional knowledge and language expression standards in the field of security intelligence.

[0013] Furthermore, it also includes a data update module, which is used to determine the time interval based on the update frequency of open source security intelligence, regularly collect new data from the target text data source, use an incremental clustering algorithm to merge the new data with the clustered data and re-cluster them, use the clustered results to build a data index, and quickly locate the cluster range to which the new data belongs for classification and update; When the data is updated, the pre-screening conditions are determined based on the historical data characteristics and the current clustering requirements. The new data text is set to , the clustered data text set is The eigenvectors are and , calculate the similarity between the new data and the clustered data, the formula is: in, is the new data text, is the clustered data text set, for The characteristic vector of for The feature vector of the Chinese text, for The elements, for The elements and pre-screen them according to their similarity.

[0014] According to the above-mentioned embodiments of the present invention, at least the following beneficial effects are achieved: the text clustering system can collect target text data containing open source security intelligence from different data sources through an automated data collection module, and ensure that the collected data is both comprehensive and targeted by constructing specific collection strategy rules and data acquisition quantity restrictions. The system uses regular expression matching to remove noise data, which can improve the cleanliness of text data and the accuracy of subsequent processing. The feature extraction module determines the importance weight of words by calculating word frequency and inverse document frequency, and constructs a more accurate feature vector, which helps to improve the accuracy and efficiency of clustering. The similarity calculation module calculates the similarity after normalizing the feature vector, which can more accurately evaluate the similarity between texts, thereby achieving a better clustering effect.

[0015] The design of clustering initialization module and iterative clustering module can ensure the stability and convergence of clustering process. The cluster center is recalculated by the mean calculation method, making the clustering result more stable and reliable. The clustering evaluation module can effectively evaluate the clustering effect by calculating the average distance of target text data to other target text data in the cluster to which it belongs and the average distance to target text data in other clusters, and provide quantitative evaluation indicators for clustering results. The result output module not only outputs the final clustering results, but also generates topic labels by counting the high-frequency words of target text data in the cluster and selecting representative words or phrases using semantic analysis tools, which can more intuitively display the clustering results and facilitate user understanding and further analysis. In addition, the design of data update module enables the system to adapt to the dynamic changes of data. The incremental clustering algorithm is used to merge new data with clustered data and re-cluster them, which can improve the real-time and adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which: Figure 1 A schematic diagram of the structure of a text clustering system for open source security intelligence provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0018] Those skilled in the art know that the embodiments of the present invention can be implemented as a system, device, apparatus, method or computer program product. Therefore, the present invention can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0019] It should be noted that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0020] Reference below Figure 1 , Figure 1 The structure diagram of a text clustering system for open source security intelligence provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, a text clustering system 100 for open source security intelligence includes: The data collection module 101 is used to collect target text data containing open source security intelligence; The text preprocessing module 102 is used to remove noise data in the target text data by using regular expression matching; The feature extraction module 103 is used to calculate the word frequency of the word in the target text data and the inverse document frequency in the entire target text data set based on the preprocessed target text data, determine the word importance weight according to the product of the two, and construct a feature vector; A similarity calculation module 104 is used to perform normalization processing on the feature vectors and calculate the similarity between the normalized feature vectors; The cluster initialization module 105 is used to randomly select k target text data as initial cluster centers, where k is a preset number of clusters; Iterative clustering module 106, used to distribute the remaining target text data to the corresponding clusters according to the similarity with the cluster center. In each iteration, the cluster center is recalculated by the mean calculation method. The cluster center feature vector is the average value of the feature vectors of all target text data in the cluster. The target text data is repeatedly distributed and the cluster center is updated until the cluster center is stable or a preset number of iterations is reached; The cluster evaluation module 107 is used to calculate the average distance between the target text data and other target text data in the cluster to which it belongs and the average distance between the target text data and the target text data in other clusters, determine the evaluation index value of the target text data based on the relationship between the two, calculate the average value of the evaluation index values ​​of all target text data to obtain the evaluation index value of the clustering result, and judge the clustering effect accordingly; The result output module 108 is used to output the final clustering results. Each cluster represents a class of similar open source security intelligence texts. The high-frequency words in the target text data in the cluster are counted and representative words or phrases are selected using semantic analysis tools to generate topic tags, and the word order is adjusted and redundant words are removed.

[0021] It should be noted that the system includes a data collection module, whose function is to collect target text data containing open source security intelligence. Open source security intelligence refers to information related to network security extracted from publicly available resources. The data collection module is the first link of the system and is responsible for obtaining raw text data from a variety of data sources, which may include news websites, forums, social media, etc.

[0022] Specifically, the data collection module will work according to preset collection strategy rules, which are built based on the characteristics of different target text data sources. For example, if the data source is a news website, the collection strategy rules may include specific keywords, release time, author and other information to ensure that only text related to open source security intelligence is collected. In addition, the module will also set data acquisition limits based on data source rules, use the parsing library to parse the data source structure for label recognition and content extraction, filter the target text data length, and determine a specific length range based on historical data statistical analysis, and only retain the target text data within this range.

[0023] Preferably, the data collection module can be further refined, for example, by setting the collection depth formula D=√(N×P) to calculate the collection depth, where N is the total number of relevant target text data in the data source, and P is the proportion of target text data related to open source security intelligence. This formula can help the system determine the scope of collection to ensure that the collected data is both comprehensive and targeted.

[0024] Furthermore, the system can also determine the time interval according to the frequency of data update, regularly collect new data from the target text data source, and use an incremental clustering algorithm to merge the new data with the clustered data and re-cluster them, so as to maintain the timeliness and relevance of the data.

[0025] In some embodiments, in the data collection module, collecting target text data containing open source security intelligence includes: Construct different collection strategy rules based on the characteristics of different target text data sources, set data acquisition limits according to data source rules, use the parsing library to parse the data source structure for label recognition and content extraction, filter the target text data length, determine the specific length range based on historical data statistical analysis, and only retain the target text data within this range; The collection depth is calculated based on the total amount of relevant target text data and the ratio of target text data related to open source security intelligence. The formula is: D=√(N×P); where D is the collection depth, N is the total amount of relevant target text data in the data source, and P is the ratio of target text data related to open source security intelligence. The target text data is collected based on the collection depth to determine the scope of collection.

[0026] It should be noted that in the data collection module, the process of collecting target text data containing open source security intelligence involves building different collection strategy rules, which is to adapt to the characteristics of different data sources. Collection strategy rules refer to a series of predefined guidelines that customize the data collection process according to the different characteristics of the data source. For example, for social media, you may need to pay attention to the frequency of user posts and the emotional tendency of the content; while for news websites, you may pay more attention to the time of news release and the reliability of the source.

[0027] Specifically, the collection strategy rules will set data acquisition limits based on the data source rules. This means that the system will determine the amount of data collected based on the characteristics and importance of each data source. For example, for a high-value data source, the system may set a higher data acquisition limit to ensure that enough information can be captured. At the same time, the system will use the parsing library to parse the structure of the data source, perform tag recognition and content extraction, in order to extract useful information from the data source. The system will also screen the length of the target text data, determine a specific length range based on historical data statistical analysis, and only retain the target text data within this range, which can improve the efficiency and accuracy of data processing.

[0028] Preferably, the system will also calculate the depth of collection during the collection process, which is achieved through the formula D=√(N×P), where D represents the collection depth, N represents the total amount of relevant target text data in the data source, and P represents the proportion of target text data related to open source security intelligence. This formula helps the system determine how deep the data should be collected to ensure that it neither misses important information nor wastes resources on irrelevant data.

[0029] Furthermore, the system can dynamically adjust the collection strategy based on the update frequency of the data source and the importance of the data to adapt to the ever-changing data environment. For example, if a data source suddenly starts to publish a large amount of information related to open source security intelligence, the system can automatically increase the amount of collection from this data source.

[0030] In some embodiments, in the text preprocessing module, removing noise data in the target text data by using regular expression matching includes: A regular expression is constructed based on the characteristics of noise data, and advertising noise information and special symbol noise are removed based on regular expression matching.

[0031] It should be noted that in the text preprocessing module, the system uses regular expression matching to remove noise data from the target text data. Regular expression is a powerful text processing tool used to search and replace text that matches a certain pattern. In this context, noise data refers to text content that is irrelevant to open source security intelligence and interferes with information extraction, such as advertisements, special symbols, etc.

[0032] Specifically, the text preprocessing module builds regular expressions based on the characteristics of the noise data. This means that the system will identify common features of the noise data, such as specific advertising formats, repeated meaningless characters, etc., and then create corresponding regular expressions to match and remove these contents. For example, if advertising noise information usually contains a large number of all-capital letters and exclamation marks, the system can build a regular expression to identify and remove these patterns. Similarly, for special symbol noise, such as unnecessary punctuation or control characters, the system can also build regular expressions for cleaning.

[0033] Preferably, the system can further refine the steps of text preprocessing, for example, automatically identifying and classifying noise data through machine learning methods, and then dynamically updating the regular expression library to adapt to new noise patterns. In addition, the system can also set a threshold to determine when a text segment is considered to be noise and needs to be removed. This threshold can be set based on factors such as the proportion of noise data in the text and the repetition of noise data.

[0034] Furthermore, as an alternative, the system can also use natural language processing techniques, such as named entity recognition (NER), to identify and retain key information while removing irrelevant noise data. Such an approach can improve the accuracy of preprocessing and ensure the quality and efficiency of subsequent feature extraction.

[0035] In some embodiments, in the feature extraction module, based on the preprocessed target text data, the word frequency of the word in the target text data and the inverse document frequency in the entire target text data set are calculated, and the word importance weight is determined according to the product of the two, and the feature vector is constructed, including: Count the number of times words appear in the target text data to get the word frequency, determine the inverse document frequency, and calculate the word In the target text data d Value, the formula is: in, For words In the target text data In value, For words In the target text data The word frequency in is the total number of target text data in the target text dataset, To contain words The number of target text data; It should be noted that in the feature extraction module, the system calculates the word frequency in the target text data and the inverse document frequency in the entire target text data set based on the preprocessed target text data, determines the word importance weight based on the product of the two, and constructs a feature vector. A feature vector is a mathematical vector used to represent text features in text analysis, and word frequency (TF) and inverse document frequency (IDF) are two commonly used statistics when calculating feature vectors, which together determine the importance of a word in a text.

[0036] Specifically, the system will count the number of times a word appears in the target text data to obtain the word frequency, which is an indicator of how often a word appears in a single document. At the same time, the system will also determine the inverse document frequency, which is an indicator of how rare a word is in the entire document collection. The calculation formula for the inverse document frequency is TF-IDF, that is, the TF-IDF value of the word in the target text data d, and the calculation formula is TF-IDF(d, w) = TF(d, w) × IDF(w), where TF(d, w) is the word frequency of word w in the target text data d, N is the total number of target text data in the target text data set, and DF(w) is the number of target text data containing word w.

[0037] Preferably, the system can use different methods to refine the operation steps or provide alternatives when calculating the TF-IDF value. For example, the system can use smoothing to avoid the situation where the denominator is zero, or use different TF and IDF calculation formulas to adapt to different application scenarios.

[0038] Furthermore, the system can also introduce semantic information of words and enhance the expressiveness of feature vectors through natural language processing techniques such as word embedding. When constructing feature vectors, the system can also set a threshold to filter out words with TF-IDF values ​​below the threshold to reduce the dimension of the feature space and improve processing efficiency. These methods can help the system capture the characteristics of text data more accurately and provide more effective input for subsequent clustering analysis.

[0039] In some embodiments, in the similarity calculation module, the feature vectors are normalized, and the similarity between the normalized feature vectors is calculated by: Normalized processing, the calculation formula is: Calculate the similarity between normalized feature vectors, assuming that the feature vector The calculation formula is: Where n is the dimension of the feature vector.

[0040] It should be noted that in the similarity calculation module, the system uses normalization processing on the feature vectors to calculate the similarity between the normalized feature vectors. Normalization processing is a technique for adjusting the range of numerical data so that the data can be compared on the same scale. In the text clustering system, normalization processing helps to eliminate the impact of the dimensions between different feature values, making the similarity calculation more accurate.

[0041] Specifically, the system uses L2 normalization processing on the feature vector, and the calculation formula is: in is the original value of the eigenvector, is the mean of the eigenvectors, is the variance of the feature vector. In this way, each dimension in the feature vector is converted to a standard score, that is, the difference between the value and the mean divided by the standard deviation. When calculating the similarity between normalized feature vectors, the system uses the cosine similarity formula: in and are two eigenvectors, and are their respective elements.

[0042] Preferably, the system can further refine the operation steps or provide alternatives when performing similarity calculations. For example, the system can use different normalization methods, such as Min-Max normalization or Z-score normalization, to adapt to different data distribution characteristics. When calculating cosine similarity, the system can also introduce weight factors to weight the importance of different features to reflect the importance of features in specific application scenarios.

[0043] Furthermore, the system can also adopt other similarity measurement methods, such as Jaccard similarity or Manhattan distance, to provide more similarity calculation options. These methods can help the system process different types of text data more flexibly and improve the accuracy and reliability of clustering results.

[0044] In some embodiments, in the clustering initialization module, the target text data set is randomly sorted, and the first k target text data are selected as the initial centers; Calculate the average similarity between the initial center and the surrounding target text data. The formula is: in, is the initial central target text data, is the surrounding target text data set, for The characteristic vector of for middle The feature vector of the target text data, for The elements, for The elements, for If the similarity is lower than the set value, the initial center is replaced. The set value is determined according to experimental data.

[0045] It should be noted that in the cluster initialization module, the system randomly sorts the target text data set and selects the first k target text data as the initial centers. The initial centers refer to a group of data points selected at the beginning of the clustering algorithm. They serve as the starting points of clustering. Subsequent data points will be assigned to different clusters based on their similarity to these centers. k is a preset number of clusters, indicating that the system divides the data into k clusters.

[0046] Specifically, the system first randomly sorts all target text data to ensure that the selection of the initial center will not be affected by the order of the data. Then, the system selects the first k data points from the sorted data set as the initial clustering center. This process involves calculating the average similarity between the initial center and the surrounding target text data, and the formula is: in is the feature vector of the target text data, is the eigenvector of the initial center, is the surrounding target text data set, is the dimension of the feature vector. This formula calculates the cosine similarity between feature vectors.

[0047] Preferably, the system can further refine the operation steps or provide alternatives in the cluster initialization module. For example, the system can use the k-means++ algorithm to select the initial center, which selects the initial center through probability distribution to reduce the sensitivity of the clustering results to the selection of the initial center.

[0048] Furthermore, the system can also set a similarity threshold. If the calculated similarity is lower than this threshold, the initial center is replaced. This threshold can be determined based on experimental data and experience to ensure that the selection of the initial center can better represent the distribution of subsequent clusters. As an alternative, the system can also consider using a density-based clustering algorithm, such as DBSCAN, to determine the initial center. This method does not require the pre-specification of the number of clusters k, but determines the cluster center based on the density characteristics of the data.

[0049] In some embodiments, in the iterative clustering module, after each update of the cluster center, a specific optimization algorithm is used to locally optimize the clustering results, the initial parameters of the algorithm are set based on the scale and complexity of the target text data set, the parameter change rate is determined experimentally, and the probability of accepting an inferior solution is calculated, and the formula is: in, is the energy difference, that is, the difference between the objective function value of the new solution and the current solution, is the current parameter value, and with a certain probability, it accepts inferior solutions to jump out of the local optimum. The termination condition is set when the clustering result does not improve after several consecutive iterations. The number of iterations is determined by experiments.

[0050] It should be noted that in the iterative clustering module, after each update of the cluster center, the system uses a specific optimization algorithm to locally optimize the clustering results. The optimization algorithm here refers to a series of algorithms used to improve the clustering quality, which can fine-tune the clustering results after each iteration to improve the accuracy and stability of clustering.

[0051] Specifically, after each iteration of cluster center update, the system sets the initial parameters of the optimization algorithm according to the size and complexity of the target text data set. These parameters may include the number of iterations, learning rate, cooling rate, etc., which affect the convergence speed and final effect of the optimization algorithm. For example, the system may use simulated annealing, a probabilistic algorithm that simulates the movement of particles in the physical annealing process to find an approximate optimal solution to the problem. The system calculates the probability of accepting an inferior solution, and the formula is: in is the energy difference, that is, the difference between the objective function value of the new solution and the current solution, is the current parameter value. This parameter controls the probability of the algorithm accepting an inferior solution and jumping out of the local optimum with a certain probability of accepting an inferior solution.

[0052] Preferably, the system can further refine the operation steps or provide alternatives in the iterative clustering module. For example, the system can set a dynamically adjusted parameter change rate, which can be adjusted according to the historical performance of clustering and the current clustering effect to adapt to the clustering needs at different stages.

[0053] Furthermore, the system can also introduce a variety of optimization algorithms, such as genetic algorithms, particle swarm optimization algorithms, etc., to provide more options to optimize clustering results. The system can also set the termination condition when the clustering results no longer improve after several consecutive iterations. This number can be determined based on experimental data and experience to ensure that the algorithm achieves a good clustering effect within a reasonable time. These detailed steps and alternatives can help the system respond to different clustering challenges more flexibly and improve the efficiency and effectiveness of clustering.

[0054] In some embodiments, in the clustering evaluation module, the target text data is calculated To the cluster to which it belongs The average distance of other target text data , to other clusters Average distance between target text data , calculate the target text data The evaluation index value is: in, The target text data To the cluster to which it belongs The average distance of other target text data within The target text data To other clusters The average distance of the internal target text data is calculated, and the average value of the evaluation index values ​​of all target text data is obtained to obtain the evaluation index value of the clustering result. A threshold is set. If the average value of the evaluation index value is lower than the threshold, it is determined that the clustering effect is not ideal. The threshold is determined based on experimental and empirical data; At the same time, calculate the internal density of each cluster, assuming that the cluster The amount of target text data is , the eigenvector is , the cluster center feature vector is , the calculation formula is: in, For clustering The amount of target text data, For clustering Internal target text data The characteristic vector of is the cluster center feature vector, for The elements, for The elements, and the clustering quality is judged by combining the evaluation index value and the internal compactness.

[0055] It should be noted that in the cluster evaluation module, the system calculates the average distance between the target text data and other target text data in the cluster to which it belongs and the average distance between the target text data and the target text data in other clusters, and determines the evaluation index value of the target text data based on the relationship between the two. The evaluation index value here is a quantitative measure used to measure the effect of clustering, especially the closeness and separation of clusters.

[0056] Specifically, the system calculates the average distance of each target text data point to other data points in its cluster, as well as the average distance to data points in other clusters. This distance can be calculated using the Euclidean distance between feature vectors, i.e. in and are the eigenvectors of the two data points, is the dimension of the feature vector. The system uses these distances to calculate the evaluation index value, the formula is The in-cluster distance is the average distance from the target text data to other target text data in its cluster, and the between-cluster distance is the average distance from the target text data to the target text data in other clusters.

[0057] Preferably, the system can further refine the operation steps or provide alternatives in the clustering evaluation module. For example, the system can set a threshold, and if the average value of the evaluation index value is lower than this threshold, the clustering effect is judged to be unsatisfactory. This threshold can be determined based on experimental and empirical data.

[0058] Furthermore, the system can also calculate the internal density of each cluster, using the formula in is clustering, is the number of target text data in the cluster, and is the target text data within the cluster, is the cluster center. This formula can help the system evaluate the compactness of clusters, that is, the similarity of data points within the cluster. Combining the evaluation index value with the internal compactness, the system can comprehensively judge the clustering quality and provide a more comprehensive clustering effect evaluation.

[0059] In some embodiments, in the result output module, the threshold of statistical high-frequency words is determined based on the amount of target text data and word distribution in the cluster, and the frequency distribution of words in the cluster is set as The total number of words is , the calculation formula is: in, is the frequency distribution of words within the cluster, For the cluster The frequency of words, is the total number of words, is the adjustment coefficient; By utilizing the semantic relationship of words in the semantic analysis tool, topic tags are selected based on statistical high-frequency words for expansion and association, and the generated topic tags are manually reviewed and revised based on professional knowledge and language expression standards in the field of security intelligence.

[0060] It should be noted that in the result output module, the system determines the threshold for counting high-frequency words based on the amount of target text data and word distribution in the cluster, and uses semantic analysis tools to select representative words or phrases to generate topic tags. High-frequency words here refer to words that appear more frequently in the cluster, and the threshold is used to determine which words are considered to be the critical point of high frequency.

[0061] Specifically, the system will set a calculation formula to determine the threshold for counting high-frequency words. The formula is: in It is the first The frequency of words, is the total number of words, is the total number of words in the cluster. This formula can balance the importance of different words by adjusting the coefficient to determine which words are high-frequency. The semantic analysis tool uses the semantic relationship of words to expand and associate the selected topic tags based on the statistical high-frequency words. This involves natural language processing technology, such as word vectors and dependency syntactic analysis, to ensure that the selected words or phrases can accurately represent the theme of the cluster.

[0062] Preferably, the system can further refine the operation steps or provide alternatives in the result output module. For example, the system can introduce more natural language processing technologies, such as sentiment analysis, to evaluate the sentiment tendency of high-frequency words, so as to more comprehensively understand the cluster content. In addition, the system can set up a machine learning-based model to automatically learn from high-frequency words and predict possible topic tags, reducing the workload of manual review and correction.

[0063] Furthermore, as an alternative, the system can also use graph-based clustering algorithms, such as LDA (Latent Dirichlet Allocation), to identify topic tags, which can better handle large-scale text data and discover potential topic structures. Through these refined steps and alternatives, the system can more accurately generate topic tags that reflect the clustering content and improve the interpretability and usability of the clustering results.

[0064] In some embodiments, a data update module is further included, which is used to determine the time interval based on the update frequency of the open source security intelligence, regularly collect new data from the target text data source, use an incremental clustering algorithm to merge the new data with the clustered data and re-cluster them, use the clustered results to build a data index, and quickly locate the cluster range to which the new data belongs for classification and update; When the data is updated, the pre-screening conditions are determined based on the historical data characteristics and the current clustering requirements. The new data text is set to , the clustered data text set is The eigenvectors are and , calculate the similarity between the new data and the clustered data, the formula is: in, is the new data text, is the clustered data text set, for The characteristic vector of for The feature vector of the Chinese text, for The elements, for The elements and pre-screen them according to their similarity.

[0065] It should be noted that the system includes a data update module, which is used to determine the time interval based on the open source security intelligence update frequency and regularly collect new data from the target text data source. The data update module refers to the part of the system responsible for tracking and integrating new data to maintain the timeliness and relevance of the clustering results. The open source security intelligence update frequency refers to the speed at which new information appears in the open source security intelligence data source.

[0066] Specifically, the system will set a reasonable time interval based on the frequency of new data updates from the open source security intelligence data source. For example, if the data source releases new data every hour, the system may be set to collect data every hour or every few hours. After collecting new data, the system will use an incremental clustering algorithm to merge these new data with the clustered data and re-cluster them. The incremental clustering algorithm is a clustering method that can handle dynamic data streams. It allows the system to assign new data points to existing clusters without re-clustering the entire data set. The system uses the clustered results to build a data index and quickly locate the cluster range to which the new data belongs for classification and update, which can improve the efficiency of data processing.

[0067] Preferably, the system can further refine the operation steps or provide alternatives in the data update module. For example, the system can determine the pre-screening conditions based on the historical data characteristics and the current clustering requirements, and pre-process and screen the new data to reduce the amount of new data that needs to be clustered. The pre-screening conditions can be set based on the similarity between the new data text and the clustered data text set, and the similarity calculation formula is: in is the feature vector of the new data text, is the feature vector of a text in the clustered data text set, is the dimension of the feature vector. The system can decide whether to assign new data to the corresponding cluster or further refine the cluster based on this similarity score.

[0068] Furthermore, the system can also use an incremental clustering algorithm with adaptive learning rate to adapt to changes in data distribution over time and improve the accuracy and adaptability of clustering. Through these refined steps and alternatives, the system can process new data more efficiently and keep the clustering results dynamically updated and accurate.

[0069] The above-mentioned various embodiments of the present invention have the following beneficial effects: the text clustering system described in the present invention can provide a series of optimization measures for key links such as data collection, preprocessing, feature extraction, similarity calculation, cluster initialization, iterative clustering, evaluation and result output in the field of open source security intelligence. Through these measures, the system can improve the processing efficiency and clustering accuracy of open source security intelligence text data, thereby making security intelligence analysis faster and more accurate. In particular, in the data acquisition module, the system can construct different acquisition strategy rules according to the characteristics of the data source, and set data acquisition volume restrictions according to the data source rules, so as to ensure that the collected data is both comprehensive and targeted, while avoiding data redundancy and unnecessary processing burden.

[0070] Furthermore, the system uses regular expression matching in the text preprocessing module to remove noise data, which can improve the cleanliness of text data and the accuracy of subsequent processing. The feature extraction module determines the importance weight of words by calculating word frequency and inverse document frequency, and constructs a more accurate feature vector, which helps to improve the accuracy and efficiency of clustering. The similarity calculation module calculates the similarity after normalizing the feature vector, which can more accurately evaluate the similarity between texts, thereby achieving a better clustering effect. The design of clustering initialization and iterative clustering modules can ensure the stability and convergence of the clustering process, and recalculate the cluster center through the mean calculation method, making the clustering result more stable and reliable. The clustering evaluation module can effectively evaluate the clustering effect by calculating the average distance from the target text data to other target text data in the cluster to which it belongs and the average distance to the target text data in other clusters, and provide quantitative evaluation indicators for the clustering results. The result output module not only outputs the final clustering results, but also generates topic tags by counting the high-frequency words of the target text data in the cluster and selecting representative words or phrases using semantic analysis tools, which can more intuitively display the clustering results, which is convenient for users to understand and further analyze. The design of the data update module enables the system to adapt to the dynamic changes of data. Through the incremental clustering algorithm, the new data is merged with the clustered data and re-clustered, thus improving the real-time and adaptability of the system.

[0071] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0072] The above descriptions are only some preferred embodiments of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present invention is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention.

Claims

1. A text clustering system for open source security intelligence, characterized in that: include: Data collection module, used to collect target text data containing open source security intelligence; The text preprocessing module is used to remove noise data from the target text data by using regular expression matching; A feature extraction module is used to calculate the word frequency of a word in the target text data and the inverse document frequency of the word in the entire target text data set based on the preprocessed target text data, determine the word importance weight according to the product of the two, and construct a feature vector; A similarity calculation module is used to perform normalization processing on the feature vectors and calculate the similarity between the normalized feature vectors; The cluster initialization module is used to randomly select k target text data as initial cluster centers, where k is the preset number of clusters; Iterative clustering module: used to assign the remaining target text data to the corresponding clusters according to the similarity with the cluster center. In each iteration, the cluster center is recalculated by the mean calculation method. The cluster center feature vector is the average value of the feature vectors of all target text data in the cluster. The target text data is assigned and the cluster center is updated repeatedly until the cluster center is stable or the preset number of iterations is reached. The clustering evaluation module is used to calculate the average distance between the target text data and other target text data in the cluster to which it belongs and the average distance to the target text data in other clusters, determine the evaluation index value of the target text data based on the relationship between the two, calculate the average value of the evaluation index values ​​of all target text data to obtain the evaluation index value of the clustering result, and judge the clustering effect accordingly; The result output module is used to output the final clustering results. Each cluster represents a class of similar open source security intelligence texts. By counting the high-frequency words in the target text data in the cluster and using semantic analysis tools to select representative words or phrases to generate topic labels, the word order is adjusted and redundant words are removed.

2. The text clustering system for open source security intelligence according to claim 1, characterized in that: In the data collection module, the target text data containing open source security intelligence is collected, including: Construct different collection strategy rules based on the characteristics of different target text data sources, set data acquisition limits according to data source rules, use the parsing library to parse the data source structure for label recognition and content extraction, filter the target text data length, determine the specific length range based on historical data statistical analysis, and only retain the target text data within this range; The collection depth is calculated based on the total amount of relevant target text data and the ratio of relevant target text data of open source security intelligence. The formula is: D=√(N×P); where D is the collection depth, N is the total amount of relevant target text data in the data source, and P is the proportion of target text data related to open source security intelligence. The target text data is collected based on the collection depth to determine the scope of collection.

3. The text clustering system for open source security intelligence according to claim 1, characterized in that: In the text preprocessing module, the use of regular expression matching to remove noise data in the target text data includes: A regular expression is constructed based on the characteristics of noise data, and advertising noise information and special symbol noise are removed based on regular expression matching.

4. The text clustering system for open source security intelligence according to claim 1, characterized in that: In the feature extraction module, based on the preprocessed target text data, the word frequency in the target text data and the inverse document frequency in the entire target text data set are calculated, and the word importance weight is determined based on the product of the two. The feature vector is constructed including: Count the number of times words appear in the target text data to get the word frequency, determine the inverse document frequency, and calculate the word In the target text data d Value, the formula is: in, For words In the target text data In value, For words In the target text data The word frequency in is the total number of target text data in the target text dataset, To contain words The target text data number.

5. The text clustering system for open source security intelligence according to claim 1, characterized in that: In the similarity calculation module, the feature vectors are normalized, and the similarity between the normalized feature vectors is calculated by: Normalized processing, the calculation formula is: Calculate the similarity between normalized feature vectors, assuming that the feature vector The calculation formula is: Where n is the dimension of the feature vector.

6. The text clustering system for open source security intelligence according to claim 1, characterized in that: In the clustering initialization module, the target text data set is randomly sorted, and the first k target text data are selected as the initial center; Calculate the average similarity between the initial center and the surrounding target text data. The formula is: in, is the initial central target text data, is the surrounding target text data set, for The characteristic vector of for middle The feature vector of the target text data, for The elements, for The elements, for The number of elements of ; If the similarity is lower than a set value, the initial center is replaced, and the set value is determined according to experimental data.

7. The text clustering system for open source security intelligence according to claim 1, characterized in that: In the iterative clustering module, after each update of the cluster center, a specific optimization algorithm is used to locally optimize the clustering results. The initial parameters of the algorithm are set based on the scale and complexity of the target text data set. The parameter change rate is determined experimentally, and the probability of accepting an inferior solution is calculated. The formula is: in, is the energy difference, that is, the difference between the objective function value of the new solution and the current solution, is the current parameter value, accepts inferior solutions to jump out of the local optimum with a certain probability, and sets the termination condition as the clustering result no longer improving after several consecutive iterations.

8. The text clustering system for open source security intelligence according to claim 1, characterized in that: In the clustering evaluation module, the target text data is calculated To the cluster to which it belongs The average distance of other target text data , to other clusters Average distance between target text data , calculate the target text data The evaluation index value is: in, The target text data To the cluster to which it belongs The average distance of other target text data within The target text data To other clusters The average distance of the internal target text data is calculated, and the average value of the evaluation index values ​​of all target text data is obtained to obtain the evaluation index value of the clustering result. A threshold is set. If the average value of the evaluation index value is lower than the threshold, it is determined that the clustering effect is not ideal; Calculate the internal density of each cluster, assuming that the cluster The amount of target text data is , the eigenvector is , the cluster center feature vector is , the calculation formula is: in, For clustering The amount of target text data, For clustering Internal target text data The characteristic vector of is the cluster center feature vector, for The elements, for The elements; The clustering quality is judged by combining the evaluation index value and the internal compactness.

9. The text clustering system for open source security intelligence according to claim 1, characterized in that: In the result output module, the threshold of high-frequency words is determined based on the number of target text data and word distribution in the cluster. The frequency distribution of words in the cluster is set as The total number of words is , the calculation formula is: in, is the frequency distribution of words within the cluster, For cluster The frequency of words, is the total number of words, is the adjustment coefficient; By using the semantic relationship of words in the semantic analysis tool, the topic tags are selected based on the statistical high-frequency words for expansion and association, and the generated topic tags are modified.

10. The text clustering system for open source security intelligence according to claim 1, characterized in that: It also includes a data update module, which is used to determine the time interval based on the update frequency of open source security intelligence, regularly collect new data from the target text data source, use an incremental clustering algorithm to merge the new data with the clustered data and re-cluster them, use the clustered results to build a data index, and quickly locate the cluster range to which the new data belongs for classification and update; When the data is updated, the pre-screening conditions are determined based on the historical data characteristics and the current clustering requirements. The new data text is set to , the clustered data text set is The eigenvectors are and , calculate the similarity between the new data and the clustered data, the formula is: in, is the new data text, is the clustered data text set, for The characteristic vector of for The feature vector of the Chinese text, for The elements, for The elements; Pre-screening is performed based on the calculated similarity between new data and already clustered data.

Images