Method and device for clustering results of multi-source data

Abstract

The invention discloses an achievement clustering method and device for multi-source data. The method comprises the steps that initial crawler parameters are configured, characteristic data are generated for collected and stored multi-source data, and the characteristic data are clustered; performing risk assessment on the clustering result analysis, and optimizing crawler parameters according to an assessment result; and the optimized crawler parameters are fed back to the intelligent parameter linkage crawler module. According to the multi-source data achievement clustering method, crawler parameters and clustering analysis parameters are linked to form a dynamic feedback mechanism, and closed-loop optimization of data collection and clustering analysis is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a method and device for clustering results of multi-source data. Background Art

[0002] With the rapid development of information technology and the Internet, technological innovation results have emerged in large numbers in various forms such as various scientific and technological documents and industry reports. Enterprises and scientific research institutions are in urgent need of efficient methods to collect, organize and analyze these technical data to support technology transformation, competition analysis and strategy formulation. However, most existing methods rely on manual data collection and static clustering analysis, which have problems such as low efficiency, untimely data updates and inaccurate clustering results. In addition, the dynamic and diverse nature of technological innovation results makes real-time monitoring and analysis particularly important. In the process of data collection and analysis, the existing system lacks a dynamic feedback mechanism, making it difficult to optimize the data collection strategy based on the analysis results, resulting in limited data quality and analysis results. Therefore, there is an urgent need for a method for clustering results that links crawler technology with clustering analysis parameters to improve the efficiency and accuracy of data collection and analysis and support more accurate technology achievement transformation. Summary of the invention

[0003] In view of the defects in the above-mentioned prior art, the present invention provides a method for clustering results of multi-source data, the method comprising:

[0004] Configure initial crawler parameters, generate characterization data for the collected and stored multi-source data, and cluster the characterization data;

[0005] Conduct risk assessment on clustering results and optimize crawler parameters based on the assessment results;

[0006] And, the optimized crawler parameters are fed back to the intelligent parameter linkage crawler module.

[0007] The user enters the initial crawler parameters through the Web form, and the backend service module receives the initial crawler parameters and generates a configuration file for the crawler task;

[0008] The backend schedules the crawler scheduling module through the task queue;

[0009] The crawler scheduling module reads the configuration file and starts the crawler task;

[0010] The crawled multi-source data is sent to the multi-source data storage and management module and stored in the database.

[0011] The method of generating feature data for the collected and stored multi-source data includes: extracting features from the collected and stored multi-source data using TF-IDF based on clustering indicators to generate a text feature vector for each multi-source data.

[0012] Wherein, clustering is performed on the characterized data based on the DBSCAN algorithm.

[0013] The parameters defined in the DBSCAN algorithm include the neighborhood radius and the minimum number of samples, and the input of the user interface is used to set and adjust the above parameters.

[0014] The clustering result analysis is used to conduct risk assessment, including generating risk assessment results for each high-density cluster based on the information of the high-density clusters.

[0015] Among them, the number of samples in each cluster is calculated, and the number of samples corresponding to each cluster label is counted;

[0016] A high-density threshold is set to identify high-density clusters, and clusters whose sample numbers exceed the high-density threshold are screened out as high-density clusters.

[0017] The risk assessment module evaluates high-density clusters based on multi-dimensional indicators and generates risk scores, including:

[0018] Select risk indicators including cluster density D and cluster size S, adaptively adjust the risk indicator weights based on the entropy weight method, and use the risk indicator weights to combine the risk indicators into a comprehensive risk score and standardize it;

[0019] The entropy weight method determines the weight by calculating the information entropy of each risk indicator.

[0020] Among them, the preliminary comprehensive risk score R is calculated i , using the risk indicator weights to combine the risk indicators into a comprehensive risk score:

[0021] R i =w D ×D i +w S ×S i ',

[0022] in:

[0023] R i : The comprehensive risk score of the i-th high-density cluster.

[0024] w D : The weight of cluster density;

[0025] w S : weight of cluster size;

[0026] D i : The cluster density of the i-th high-density cluster;

[0027] S i ': Standardized cluster size of the i-th high-density cluster;

[0028] The comprehensive risk score R i Standardize the comprehensive risk score R i Normalize to the range 0 to 1:

[0029]

[0030] in:

[0031] R i ': Standardized risk score of the i-th high-density cluster;

[0032] R i : The original comprehensive risk score of the i-th high-density cluster;

[0033] R min : The minimum value of the original comprehensive risk score among all high-density clusters;

[0034] R max : The maximum value of the original composite risk scores among all high-density clusters.

[0035] Among them, the membership function is defined based on the threshold of the risk level;

[0036] According to the standardized risk score and membership function, each high-density cluster is divided into the corresponding risk level.

[0037] Among them, the membership function is as follows:

[0038]

[0039] In the formula, Risk Level is the risk level, that is, the membership function, LowRisk is low risk, Medium is medium risk, HighRisk is high risk, θ 1 ,θ 2 ,θ 3 These are all risk level thresholds.

[0040] Among them, the crawler parameters are optimized according to the evaluation results, including adjusting the following three crawler parameters based on the risk level:

[0041] Crawling rate CR: the number of crawling requests per unit time;

[0042] Crawling depth CD: the crawling level depth of the crawler;

[0043] Target frequency TF: The crawler's crawling frequency for a specific target.

[0044] Among them, the crawler parameters are adjusted based on the risk level, including:

[0045]

[0046] CD new =CD base +β·(1-L degree )

[0047]

[0048] In the formula, CR new is the adjusted crawl rate, CD new is the adjusted crawling depth, TF new is the adjusted target frequency, CR base is the basic crawl rate, CD base As the basic crawl depth, TF base is the basic target frequency, α, β, γ are adjustment coefficients, which determine the sensitivity of parameter adjustment, L degree It is a numerical representation of the risk level.

[0049] The present invention also provides a device for clustering results of multi-source data, comprising:

[0050] Parameter configuration module, which is used to configure the initial crawler parameters;

[0051] A feature extraction module, which is used to generate feature data from the collected and stored multi-source data;

[0052] A clustering analysis module, which is used to cluster the characterization data;

[0053] The risk assessment module is used to perform risk assessment on clustering result analysis and optimize crawler parameters based on the assessment results.

[0054] The achievement clustering device also includes a crawler scheduling module, which is used to read the configuration file and start the crawler task.

[0055] Among them, the achievement clustering device also includes an intelligent parameter linkage crawler module, which is used to realize multi-source data collection according to optimized crawler parameters.

[0056] Among them, the intelligent parameter linkage crawler module includes a backend service module and a crawler scheduling module.

[0057] The present invention provides a method and device for clustering results of multi-source data, which forms a dynamic feedback mechanism by linking crawler parameters with cluster analysis parameters to achieve closed-loop optimization of multi-source data collection and cluster analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0059] Figure 1 The figure is an overall flow chart of the method for clustering results of multi-source data of the present invention.

[0060] Figure 2 It is a schematic diagram of the device for clustering results of multi-source data of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0063] It should be understood that although the terms first, second, third, etc. may be used to describe ... in the embodiments of the present invention, these ... should not be limited to these terms. These terms are only used to distinguish .... For example, without departing from the scope of the embodiments of the present invention, the first ... may also be referred to as the second ..., and similarly, the second ... may also be referred to as the first ....

[0064] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0065] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0066] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0067] The present invention aims to provide a method for clustering results of multi-source data. By linking crawler parameters with clustering analysis parameters to form a dynamic feedback mechanism, the closed-loop optimization of multi-source data collection and clustering analysis can be achieved, thereby improving the efficiency of data processing and the accuracy of clustering results, and supporting the precise transformation of technological innovation results.

[0068] To achieve the above object, the present invention provides a method for clustering results of multi-source data, which is characterized by comprising the following steps:

[0069] Step 1: Data collection.

[0070] It is necessary to configure the initial crawler parameters, including crawling rate, crawling frequency, depth, keyword list, the region where the R&D institution is located, etc., and use crawler technology to collect data related to technological innovation achievements from public scientific and technological databases (such as CNKI, Web of Science, SpringerLink, etc.) based on the initial crawler parameters, and store the collected raw data in the database.

[0071] Step 2: Feature extraction.

[0072] Based on clustering indicators such as technical fields, application fields, and regions where R&D institutions are located, TF-IDF is used to extract features from the collected multi-source data and generate text feature vectors for each multi-source data.

[0073] Step 3: Cluster analysis.

[0074] Cluster the characterized data using a clustering algorithm.

[0075] Calculate clustering evaluation indicators and evaluate clustering effects.

[0076] According to the results of clustering evaluation indicators, the parameters of the clustering algorithm, such as the number of clusters K value, distance measurement method, etc., are dynamically adjusted to optimize the clustering effect.

[0077] Step 4: Parameter linkage and dynamic adjustment.

[0078] Perform risk assessment on clustering analysis results, including generating risk assessment results for each high-density cluster based on the information of high-density clusters. Optimize data collection strategies (crawler parameters) based on the assessment results, including adjusting crawler parameters based on the information of high-density clusters according to cluster analysis results, including adjusting crawling parameters such as crawling frequency of related content.

[0079] And, the optimized crawler parameters are fed back to the intelligent parameter linkage crawler module, which includes a backend service module and a crawler scheduling module.

[0080] Dynamic adjustment of crawler parameters is achieved through automated scripts, forming a closed-loop optimization process of data collection and clustering analysis.

[0081] In a certain embodiment, after the multi-source data collection, a data preprocessing operation may be included.

[0082] Clean the collected multi-source data, including deduplication and processing of missing values.

[0083] Perform pre-processing operations such as word segmentation, stop word removal, and word form restoration on text data to improve text quality.

[0084] In a certain embodiment, the present invention provides a user interaction interface, and the system supports users to query, filter and browse clustering results.

[0085] The present invention establishes a real-time feedback mechanism for crawler parameters and cluster analysis parameters, dynamically adjusts the crawler's crawling strategy according to the results of cluster analysis, and realizes intelligent optimization of multi-source data collection.

[0086] By improving the pertinence and efficiency of data collection and ensuring that cluster analysis is based on the latest and most relevant data, the system's adaptability and intelligence level can be enhanced.

[0087] The evaluation index of cluster analysis directly affects the configuration parameters of the crawler, forming a closed loop of continuous optimization between data collection and cluster analysis. This crawler parameter adjustment strategy significantly improves the system's adaptive ability and ensures that the data collection and analysis process is always in the best state.

[0088] In addition, the system can automatically identify and extract high-frequency keywords based on clustering results, and dynamically adjust the crawler's keyword list and other crawling parameters based on multi-dimensional indicators to more accurately crawl data in related fields.

[0089] And introduce a user interactive feedback mechanism, allowing users to verify and adjust the clustering results through the interface. The system automatically optimizes the clustering model parameters based on user feedback to improve the accuracy and practicality of the clustering results.

[0090] like Figure 2 As shown, the clustering of multi-source data results of the present invention includes the following main modules:

[0091] Intelligent parameter linkage crawler module: used to dynamically adjust the crawler's crawling strategy according to the cluster analysis results (optimized crawler parameters) to achieve accurate data collection.

[0092] Multi-source data storage and management module, used to efficiently store raw data and processed data from different data sources, supporting fast retrieval and analysis.

[0093] The feature extraction module is used to extract and enhance data features based on the selected clustering indicators to improve the clustering analysis effect.

[0094] The clustering analysis module is used to perform clustering analysis and dynamically adjust clustering parameters according to the clustering results to optimize the overall performance of the system.

[0095] The risk assessment module is used to conduct risk assessment on the clustering result analysis, including generating risk assessment results for each high-density cluster based on the information of the high-density clusters. The data collection strategy is optimized according to the assessment results, including adjusting the crawler parameters based on the information of the high-density clusters according to the cluster analysis results, including increasing the crawling frequency of related fields to obtain more relevant data.

[0096] And, the adjusted crawler parameters are fed back to the intelligent parameter linkage crawler module.

[0097] Optionally, adjusting the crawler parameters includes adjusting at least one combination of parameters such as technical field, application field, and region where the R&D institution is located.

[0098] The user interface is used to provide a user interaction interface to support the operation and viewing of various system functions.

[0099] In a certain embodiment, for the intelligent parameter linkage crawler module, initial crawler parameters are set, including crawling rate, crawling frequency, depth, keyword list, the region where the R&D institution is located, etc.

[0100] In one embodiment, for the multi-source data storage and management module, MongoDB is used to store semi-structured data, and Elasticsearch is used for full-text search and analysis.

[0101] In one embodiment, for the feature extraction module, improving the clustering analysis effect includes: generating a text feature vector using the TF-IDF method and performing one-hot encoding on categorical data.

[0102] In one embodiment, the clustering analysis module is used to perform clustering analysis and dynamically adjust clustering parameters according to clustering results to optimize the overall performance of the system.

[0103] The cluster analysis includes KMeans or hierarchical clustering algorithm, and the optimal number of clusters is determined through preliminary clustering.

[0104] The clustering effect is evaluated by using evaluation indicators such as silhouette coefficient, and the clustering algorithm parameters are adjusted dynamically.

[0105] The high-density cluster information in the clustering results is fed back to the risk assessment module to optimize the data collection strategy and achieve closed-loop optimization of data collection and clustering analysis.

[0106] In one embodiment, the intelligent parameter linkage crawler module is responsible for automatically collecting data related to technological innovation achievements from the science and technology database, and dynamically adjusting the crawling strategy based on the feedback from cluster analysis and risk assessment, including:

[0107] Initial configuration is achieved by inputting initial crawler parameters through the user interface, including crawling rate, crawling frequency, depth, keyword list, and the region where the R&D institution is located.

[0108] The Scrapy framework is used to crawl the configured data sources in parallel, such as CNKI, Web of Science, SpringerLink, etc. The collected data is sent to the multi-source data storage and management module through the API interface and stored in MongoDB.

[0109] After the cluster analysis is completed, the cluster analysis module feeds back the information of high-density clusters (such as high-frequency keywords, the location of R&D institutions, etc.) to the risk assessment module through a message queue (such as RabbitMQ).

[0110] The risk assessment module is used to conduct risk assessment on the clustering result analysis, including generating risk assessment results for each high-density cluster based on the information of the high-density clusters. The data collection strategy is optimized according to the assessment results, including adjusting crawler parameters based on the cluster analysis results and the information of the high-density clusters, including adjusting crawling parameters such as the crawling frequency of related content.

[0111] And, the adjusted crawler parameters are fed back to the intelligent parameter linkage crawler module.

[0112] After receiving the feedback, the intelligent parameter linkage crawler module automatically adjusts the crawler parameters, such as increasing the crawling frequency of related keywords, optimizing the keyword list, or adjusting the priority of the region where the R&D institution is located.

[0113] The adjusted parameters update the crawler configuration file through an automated script and automatically restart the crawler task to apply the new crawling strategy.

[0114] In one embodiment, the user inputs initial crawler parameters, including crawling rate, crawling frequency, depth, keyword list, and the region where the R&D institution is located, through the Web front end to achieve initial configuration.

[0115] Build the user form interface.

[0116] The form content includes:

[0117] Crawling frequency: Select the execution interval of the scheduled task (such as daily or weekly).

[0118] Crawling depth: defines the crawling depth of the crawler and determines the level of visited links.

[0119] Keyword list: Enter multiple keywords to search for related technical achievements.

[0120] R&D Institution Location: Select or enter a specific region (e.g., United States, China).

[0121] In one embodiment, the backend service module processes the initial crawler parameters submitted by the user, generates a crawler configuration file, and schedules the crawler tasks.

[0122] The backend web framework receives form data from the user interface, verifies the validity and integrity of the parameters, and generates configuration files or dynamic parameters required by the Scrapy spider.

[0123] Crawler scheduling uses Scrapy's Scrapy API to start and manage crawler tasks, and integrates task queues to support the execution of scheduled tasks.

[0124] In one embodiment, the crawler scheduling module starts the Scrapy crawler according to the configuration file or dynamic parameters to perform the data collection task.

[0125] The crawler scheduling module reads the parameters in the configuration file and uses Scrapy's asynchronous characteristics and multi-threading capabilities to crawl multiple data sources (such as CNKI, Web of Science, SpringerLink, etc.) in parallel.

[0126] In Python, you can use Scrapy's a parameter to pass configuration parameters to the spider to achieve dynamic parameter transfer, realize parameter linkage and dynamic adjustment mechanism, and dynamically adjust crawler parameters based on the results of cluster analysis and risk assessment. Adjust crawler parameters based on risk assessment results, and feed back the adjusted crawler parameters to the crawler scheduling module through RabbitMQ. According to the feedback results, for clusters with high potential benefits and low risks, increase the crawling frequency of related fields. For clusters with low potential benefits or high risks, reduce the crawling frequency of related fields.

[0127] Modify the crawler configuration file through an automated script and restart the crawler task to apply the new crawler parameter settings.

[0128] In one embodiment, the user inputs the initial crawler parameters through a Web form. The backend service module receives the initial crawler parameters and generates a jason configuration file for the crawler task.

[0129] The backend schedules the crawler scheduling module through the Celery task queue.

[0130] The crawler scheduling module reads the configuration file and starts the Scrapy crawler task.

[0131] Scrapy crawlers crawl multiple data sources in parallel (such as CNKI, Web of Science, SpringerLink, etc.).

[0132] The crawled multi-source data is sent to the multi-source data storage and management module through the API interface and stored in the database MongoDB.

[0133] The feature extraction module periodically (such as daily or weekly) reads new data from the MongoDB database, performs cleaning and feature extraction, and exports the processed feature data as a feature file.

[0134] The clustering analysis module loads feature data from the feature file, executes the clustering algorithm, evaluates the clustering effect, identifies high-density clusters, and sends relevant information to the risk assessment module through RabbitMQ.

[0135] The risk assessment module conducts risk assessment on high-density clusters based on multi-dimensional indicators, generates risk assessment results, and adjusts crawler parameters based on the risk assessment results.

[0136] The adjusted crawler parameters are fed back to the crawler scheduling module through RabbitMQ to achieve closed-loop optimization of data collection and analysis.

[0137] In one embodiment, the cluster analysis module needs to load data from a feature file, which is a CSV file, where each row represents a sample and each column represents a feature. After determining the feature file path and the location of the feature file, read and load the data file.

[0138] Afterwards, based on the DBSCAN algorithm, the clustering steps are performed, including:

[0139] Convert the loaded data into a numeric matrix for input to the clustering algorithm.

[0140] Define DBSCAN parameters:

[0141] ε (epsilon): Neighborhood radius, which determines the search range. Points within this radius are considered to be neighboring.

[0142] min_samples: The minimum number of samples, which determines the minimum number of points required for an area to be considered a high-density area.

[0143] Initialize DBSCAN: Configure the DBSCAN algorithm using the above parameters.

[0144] Perform clustering and input the feature matrix into DBSCAN to obtain the cluster label of each sample. Label 1 represents a noise point, and non-negative integers represent different clusters.

[0145] Append clustering results and add cluster labels to the original data for easy subsequent analysis. cluster_labels is the label after clustering, identifying the cluster to which each sample belongs.

[0146] Among them, the DBSCAN algorithm can identify clusters of arbitrary shapes and automatically identify noise points.

[0147] DBSCAN performs clustering based on the following concepts:

[0148] 1. Neighborhood (ε neighborhood): For a point in the data set, all points within its ε neighborhood constitute its neighborhood.

[0149] 2. Core point: If there are at least min_samples points (including the point itself) in the ε neighborhood of a point, then the point is a core point.

[0150] 3. Boundary points: points that have core points in their neighborhood, but the number of points in their own neighborhood is less than min_samples.

[0151] 4. Noise point: a point that is neither a core point nor a boundary point.

[0152] DBSCAN can identify dense areas and divide them into different clusters while identifying outliers.

[0153] The parameter definitions in the DBSCAN algorithm include:

[0154] epsilon (eps): Neighborhood radius, used to determine the neighborhood of a point.

[0155] min_samples: The minimum number of samples used to define high-density areas.

[0156] The parameters of the DBSCAN algorithm can be set and adjusted through user interface input.

[0157] In one embodiment, in order to evaluate the clustering effect, the contour coefficient is used. The value range of the contour coefficient is [-1, 1]. The evaluation steps are as follows:

[0158] Remove noise points: Since noise points are not assigned to any cluster, they should be excluded when calculating the silhouette coefficient.

[0159] Calculate the silhouette coefficient: Based on the samples of the core cluster, calculate the silhouette coefficient of each sample and take the average.

[0160] a(i): The average distance between a sample and other samples in the same cluster, reflecting the compactness within the cluster.

[0161] b(i): The average distance between sample G and the nearest neighbor cluster (the closest cluster), reflecting the separation between clusters.

[0162] Silhouette Coefficient:

[0163] in,

[0164] s(i) is close to 1: sample i is in a good cluster and is highly separated from other clusters.

[0165] s(i) is close to 0: sample i is between the two clusters and the boundary is unclear.

[0166] s(i) is close to 1: sample i may be incorrectly assigned to the wrong cluster.

[0167] In one embodiment, identifying high-density clusters finds clusters with a large number of samples. These clusters may represent the main patterns or trends in the data. Specifically:

[0168] Calculate the number of samples in each cluster and count the number of samples corresponding to each cluster label.

[0169] Determining a high density threshold includes setting a high density threshold, for example, the number of samples of a cluster is greater than the average number of samples of all clusters, or using other statistical methods to determine the high density threshold.

[0170] Identify high-density clusters and filter out clusters whose sample numbers exceed the high-density threshold. These clusters are high-density clusters.

[0171] Clusters with density lower than the high-density threshold will not be screened because their distribution is relatively scattered and they do not form clusters. These samples are unimportant parameters, and the default initial crawler parameters are used for multi-source data collection.

[0172] In one embodiment, the identified high-density cluster information is transmitted to the risk assessment module, which can be achieved through the message queue RabbitMQ. The specific steps are as follows:

[0173] The default host address of the RabbitMQ server is localhost. Define a queue to store information about high-density clusters, for example, named risk_evaluation_queue.

[0174] Establish a connection and use the RabbitMQ client library to establish a connection with the RabbitMQ server. Declare the queue and ensure that the target queue exists. If it does not exist, create it. Prepare the message content, list the numbers of all high-density clusters, count the number of samples of each high-density cluster in detail, and record the time when the message was sent for subsequent tracking and analysis. Convert the above information into a JSON format serialized message and send the serialized message to the specified RabbitMQ queue. After the message is sent, close the connection with the RabbitMQ server.

[0175] In one embodiment, the risk assessment module assesses the high-density clusters based on multi-dimensional indicators to generate a risk score, including:

[0176] Based on the comprehensive risk score, it is used to quantitatively assess the risk level of the cluster.

[0177] The following two key risk indicators are selected:

[0178] Cluster Density (D): reflects the density of points within the cluster. The higher the density, the more concentrated the data points within the cluster.

[0179] Cluster Size (S): The number of sample points in a cluster, reflecting the importance or influence of the cluster.

[0180] Based on the entropy weight method, the indicator weights are adaptively adjusted, and the risk indicators are combined into a comprehensive risk score using the risk indicator weights and standardized. The entropy weight method determines the weights by calculating the information entropy of each risk indicator, objectively reflecting the characteristics of the data itself, including:

[0181] Perform maximum and minimum standardization on the original indicator data:

[0182] in:

[0183] Z ij : The standardized value of the i-th high-density cluster on the j-th indicator.

[0184] X ij : The original value of the i-th high-density cluster on the j-th index.

[0185] and The minimum and maximum values ​​of the j-th indicator.

[0186] Next, calculate the proportion of each risk indicator in all samples: Here, n is the total number of clusters.

[0187] Afterwards, the entropy value of each risk indicator is calculated:

[0188] in, is a constant used to ensure that the entropy value is in the range of [0, 1].

[0189] Then determine the weight of the corresponding risk indicator according to the entropy value:

[0190] Where m is the total number of indicators.

[0191] For the case of cluster density and cluster size, the cluster size S is converted to the range of 0 to 1 using the maximum and minimum normalization method:

[0192] in:

[0193] S i ': The standardized cluster size of the i-th high-density cluster.

[0194] S i : The original cluster size of the i-th high-density cluster.

[0195] S min : The minimum value among all high-density clusters

[0196] S max : The maximum value among all high-density clusters

[0197] Afterwards, calculate the preliminary comprehensive risk score R i , using the risk indicator weights to combine the risk indicators into a comprehensive risk score:

[0198] R i =w D×D i +w S ×S i ',

[0199] in:

[0200] R i : The comprehensive risk score of the i-th high-density cluster.

[0201] w D : The weight of cluster density.

[0202] w S : The weight of cluster size.

[0203] D i : The cluster density of the i-th high-density cluster.

[0204] S i ': The standardized cluster size of the i-th high-density cluster.

[0205] Finally, the comprehensive risk score is normalized to a range of 0 to 1:

[0206] method:

[0207] in:

[0208] R i ': Standardized risk score of the ith high-density cluster.

[0209] R i : The original comprehensive risk score of the i-th high-density cluster.

[0210] R min : The minimum value among the original comprehensive risk scores among all high-density clusters.

[0211] R max : The maximum value of the original composite risk scores among all high-density clusters.

[0212] The entropy weight method adaptively adjusts weights to avoid the subjectivity caused by artificially setting weights, and objectively determines the weights of each indicator through the information distribution of the data itself. Combining the two key indicators of cluster density and cluster size, a comprehensive risk score is generated to facilitate subsequent risk management and decision-making.

[0213] In one embodiment, three risk level thresholds θ are set. 1 ,θ 2 ,θ 3 , the membership function can be defined as follows:

[0214]

[0215] In one embodiment, the threshold values ​​of the risk levels are 1 =0.2,θ 2 =0.4,θ 3 =0.6. The membership function can be defined as follows:

[0216]

[0217] The threshold values ​​of the above risk levels are 1 =0.2,θ 2 =0.4,θ 3 =0.6 is just an example. In actual use, any three other thresholds between 0.1 and 0.2, 0.4, and 0.6 can be used to replace the thresholds for the risk levels from small to large. 1 ,θ 2 ,θ 3 It should be reasonably set according to actual business needs, historical data and risk tolerance, so that the risk level warning effect can be optimized according to the actual application scenario.

[0218] For example, assume that there are three clusters whose cluster density and cluster size data are shown in Table 1:

[0219] Table 1 Corresponding data table of cluster density and cluster size of three clusters

[0220] Cluster number Cluster density D Cluster size S 1 0.8 150 2 0.5 200 3 0.9 180

[0221] The weight w is calculated by the above entropy weight method steps D and w S , assuming that W D =0.6,w S =0.4, then the risk score of each cluster is R 1 , R 2 , R 3 The calculation process includes:

[0222] Standardized cluster size S: Calculate the standardized size S of each cluster i 'Use minimax normalization.

[0223] Calculate the comprehensive risk score R i : The standardized cluster density D i and the normalized cluster size S i 'According to the weight w D and w S Combined, we get a preliminary risk score R i .

[0224] Standardized comprehensive risk score R i : Use the maximum and minimum standardization to convert R iConvert to R i ', ensuring that all scores are between 0 and 1.

[0225] For the above case:

[0226] Normalizing the cluster size S, we get:

[0227] Calculate the initial risk score R i , we get: R 1 =0.6×0.8+0.4×0=0.48, R 2 =0.6×0.5+0.4×1=0.70, R 3 =0.6×0.9+0.4×0.6=0.78.

[0228] Standardized comprehensive risk score R i ',get:

[0229] For example, the following risk score thresholds are set according to requirements: θ 1 =0.2,θ 2 =0.4,θ 3 =0.6.

[0230] According to the standardized risk score R', each cluster is divided into the corresponding risk level:

[0231] Cluster 1: R 1 '=0

[0232]

[0233] Risk level: Low Risk

[0234] Cluster 2: R 2 '≈0.733

[0235]

[0236] Risk level: High Risk

[0237] Cluster 3: R 3 '=1

[0238]

[0239] Risk level: High Risk

[0240] In one embodiment, a risk assessment result is generated for each high-density cluster, and a data collection strategy is optimized according to the assessment result, including adjusting crawler parameters based on the information of the high-density cluster according to the cluster analysis result, including:

[0241] Consider the following three crawler parameters:

[0242] Crawl Rate (CR): The number of crawl requests per unit time.

[0243] Crawl Depth (CD): The crawling depth of the crawler.

[0244] Target Frequency (TF): The crawler's crawling frequency for a specific target.

[0245] The ways to adjust the above crawler parameters are:

[0246]

[0247] CD new =CD base +β·(1-L degree )

[0248]

[0249] in,

[0250] CR base : Basic crawl rate.

[0251] CD base : Basic crawling depth.

[0252] TF base : Base target frequency.

[0253] α, β, γ: adjustment coefficients that determine the sensitivity of parameter adjustment.

[0254] L degree : Numerical representation of risk level (between 0 and 1).

[0255] Among them, L degree The numerical representation of the risk level may be, for example, Low Risk=0, Medium-LowRisk=0.33, Medium-High Risk=0.66, High Risk=1.

[0256] 1.2 Crawl Depth Adjustment (CD)

[0257] For example, assume that the following adjustment coefficients are set:

[0258] α=1,

[0259] β=2.0,

[0260] γ=0.5,

[0261] The basic parameters are set as follows:

[0262] CR base = 10 requests / second,

[0263] CD base =5 floors,

[0264] TF base = 1 time / hour,

[0265] Cluster 1: Low Risk (L degree =0)

[0266] CR new =10×e 1×0 =10×1=10 requests / second

[0267] CD new =5+2.0×0=5 layers

[0268]

[0269] Crawl rate: 10 requests / second (remains unchanged)

[0270] Crawling depth: 5 layers (remains unchanged)

[0271] Target frequency: 1 time / hour (remain unchanged)

[0272] Cluster 2 and Cluster 3: High Risk (L degree =1)

[0273] CR new =10×e 1×1 ≈10×2.7183≈27.183 requests / second

[0274] CD new =5+2.0×1=7 layers

[0275]

[0276] Crawl rate: 27.183 requests / second (significant increase)

[0277] Crawling depth: 7 layers (increasing)

[0278] Target frequency: 1.5 times / hour (increasing)

[0279] In one embodiment, the risk assessment module calculates new crawler parameters (CR_new, CD_new, TF_new) according to the risk scores of the high-density clusters, and sends them to the crawler scheduling module through RabbitMQ.

[0280] The crawler scheduling module receives messages sent by the risk assessment module through the RabbitMQ consumer.

[0281] Once new parameters are received, the automated script will dynamically update the crawler configuration based on the received CR_new, CD_new, and TF_new to achieve parameter adjustment.

[0282] After the adjustment is completed, the script will record the adjustment log and start the monitoring mechanism to ensure that the crawler runs according to the new parameters, while continuously monitoring the system performance and risk status.

[0283] After completing the risk assessment of high-density clusters, new crawler parameters (CR_new, CD_new, TF_new) are calculated. The new crawler parameters are usually packaged in JSON format and sent through RabbitMQ: The risk assessment module publishes the above message to the pre-defined RabbitMQ queue. The crawler scheduling module connects to the RabbitMQ server through the RabbitMQ client library and subscribes to the queue. The crawler scheduling module runs a continuously listening consumer to wait and receive messages from the risk assessment module.

[0284] As part of the crawler scheduling module, the automated script receives the message, parses the JSON content, and extracts new crawler parameters (CR_new, CD_new, TF_new). CR_new: new crawl rate. CD_new: new crawl depth. TF_new: new target frequency.

[0285] Crawlers usually have configuration files or manage their parameters through APIs. The automation script needs to update the corresponding parameters according to the specific implementation of the crawler.

[0286] Read the current crawler configuration file and modify it to the new values ​​CR_new, CD_new, and TF_new.

[0287] The present invention provides a method for clustering results of multi-source data, which forms a dynamic feedback mechanism by linking crawler parameters with cluster analysis parameters to achieve closed-loop optimization of multi-source data collection and cluster analysis.

[0288] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0289] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0290] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0291] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0292] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0293] The above introduces the preferred embodiments of the present invention, which is intended to make the spirit of the present invention clearer and easier to understand, but is not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection outlined by the claims attached to the present invention.

Claims

1. A method for clustering results of multi-source data, the method comprising: Configure initial crawler parameters, generate characterization data for the collected and stored multi-source data, and cluster the characterization data; Conduct risk assessment on clustering results and optimize crawler parameters based on the assessment results; And, the optimized crawler parameters are fed back to the intelligent parameter linkage crawler module.

2. A method for clustering results of multi-source data as claimed in claim 1, characterized in that: The user enters the initial crawler parameters through a web form; The backend service module receives the initial crawler parameters and generates the configuration file for the crawler task; The backend schedules the crawler scheduling module through the task queue; The crawler scheduling module reads the configuration file and starts the crawler task; The crawled multi-source data is sent to the multi-source data storage and management module and stored in the database.

3. The method for clustering results of multi-source data according to claim 1, characterized in that: Based on the DBSCAN algorithm, clustering is performed on the characterized data; The parameters defined in the DBSCAN algorithm include the neighborhood radius and the minimum number of samples, and the user interface inputs set and adjusts the above parameters.

4. A method for clustering results of multi-source data as described in any one of claims 1 to 3, characterized in that: The clustering result analysis is used to perform risk assessment, including generating risk assessment results for each high-density cluster based on the information of the high-density clusters.

5. A method for clustering results of multi-source data as claimed in claim 4, characterized in that: Calculate the number of samples in each cluster and count the number of samples corresponding to each cluster label; A high-density threshold is set to identify high-density clusters, and clusters whose sample numbers exceed the high-density threshold are screened out as high-density clusters.

6. A method for clustering results of multi-source data as claimed in claim 4, characterized in that: Evaluate high-density clusters based on multi-dimensional indicators to generate risk scores, including: Select risk indicators including cluster density D and cluster size S, adaptively adjust the risk indicator weights based on the entropy weight method, and use the risk indicator weights to combine the risk indicators into a comprehensive risk score and standardize it; The weight is determined by calculating the information entropy of the risk indicator.

7. A method for clustering results of multi-source data as claimed in claim 6, characterized in that: Risk indicators are combined into a composite risk score using risk indicator weights: R i =w D ×D i +w S ×S i ', in: R i : The comprehensive risk score of the i-th high-density cluster; w D : weight of cluster density; w S : weight of cluster size; D i : The cluster density of the i-th high-density cluster; S i ': Standardized cluster size of the i-th high-density cluster; The comprehensive risk score R i Standardize the comprehensive risk score R i Normalize to the range 0 to 1: in: R i ': Standardized risk score of the i-th high-density cluster; R i : The original comprehensive risk score of the i-th high-density cluster; R min : The minimum value of the original comprehensive risk score among all high-density clusters; R max : The maximum value of the original comprehensive risk scores among all high-density clusters.

8. A method for clustering results of multi-source data as claimed in claim 7, characterized in that: Define membership functions based on risk level thresholds; According to the standardized risk score R i ' and membership function, each high-density cluster is divided into the corresponding risk level.

9. A method for clustering results of multi-source data as claimed in claim 8, characterized in that: Crawler parameters include crawling rate, crawling depth, and target frequency; Adjust crawler parameters based on risk level, including: CD neW =CD base +β (1-L degree ) In the formula, CR new is the adjusted crawl rate, CD new is the adjusted crawling depth, TF new is the adjusted target frequency, CR base is the basic crawl rate, CD base As the basic crawl depth, TF base is the basic target frequency, α, β, γ are adjustment coefficients, which determine the sensitivity of parameter adjustment, L degree It is a numerical representation of the risk level.

10. A result clustering device for multi-source data, whose characteristic value is, According to any one of claims 1 to 9, the method for clustering results of multi-source data is adopted, and the result clustering device comprises: Parameter configuration module, which is used to configure the initial crawler parameters; A feature extraction module, which is used to generate feature data from the collected and stored multi-source data; A clustering analysis module, which is used to cluster the characterization data; The risk assessment module is used to perform risk assessment on clustering result analysis and optimize crawler parameters based on the assessment results.

Images