Deep network news data acquisition method and device, equipment, medium and product

Abstract

The invention relates to the technical field of data collection, in particular to a deep network news data collection method and device, equipment, a medium and a product, and the method comprises the steps: collecting surface news data under each news column of different news platforms; performing first-stage clustering processing on the surface news data in a mode of calculating data density to obtain a first clustering result, and performing second-stage clustering processing on the first clustering result to obtain a second clustering result; classifying the surface news data into each category of the second clustering result, respectively extracting keywords in each category, and generating a keyword list of each category; and based on the keywords in the keyword list, collecting deep network news data under each category. Through the first-stage clustering processing and the second-stage clustering processing, cross-type event clustering and same-type event clustering are carried out respectively, overlapping and fuzziness between categories are reduced, and the collection integrity and diversity of deep network news data are effectively improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data collection, and in particular to a deep web news data collection method, device, equipment, medium and product. BACKGROUND

[0002] With the development of new media field and digital technology, news data of news platforms has been widely used in hot topic mining, public opinion monitoring and guiding and other fields, and at the same time, higher requirements have been put forward for the completeness and efficiency of news data collection.

[0003] Although the news platform provides a classification mechanism for news columns, the number of news displayed in the news column is limited, and many news data are hidden in the deep web of the news platform, so deep web data collection is needed. For this, the commonly used method is to perform coarse-grained clustering according to the surface information such as news title and content, but collecting deep web news data according to this clustering method may miss some deep web news data under the news theme, which seriously affects the completeness of deep web news data collection. SUMMARY

[0004] The present disclosure is proposed in view of the above problems, and provides a deep web news data collection method, device, equipment, medium and product.

[0005] According to one aspect of the present disclosure, a deep web news data collection method is provided, comprising:

[0006] Collecting surface layer news data under each news column of different news platforms, the surface layer news data being used to represent the theme content of news;

[0007] Performing first-stage clustering processing on the surface layer news data by calculating data density to obtain a first clustering result, and performing second-stage clustering processing on the first clustering result to obtain a second clustering result;

[0008] Classifying the surface layer news data into each category of the second clustering result, respectively extracting keywords in each category to generate a keyword list of each category;

[0009] Collecting deep web news data under each category based on the keywords in the keyword list.

[0010] Through first-stage and second-stage clustering processing, cross-type event clustering and same-type event clustering are respectively performed, reducing the overlap and ambiguity between categories, achieving fine-grained category division, ensuring that the keyword list always reflects the latest news hotspots and changes in news themes, and being able to dynamically capture changes in news content, effectively improving the completeness and diversity of deep web news data collection.

[0011] Further, according to the deep web news data collection method of one aspect of the present disclosure, the surface news data is subjected to first-stage clustering processing by calculating data density to obtain a first clustering result, and the first clustering result is subjected to second-stage clustering processing to obtain a second clustering result, comprising:

[0012] Based on the surface news data, a feature vector is obtained;

[0013] The data density of each feature vector is obtained through the Euclidean distance between each feature vector;

[0014] Based on the data density, first-stage clustering processing is performed to obtain a coarse-grained category;

[0015] The surface news data is classified into each coarse-grained category to obtain a first clustering result;

[0016] The surface news data included in each category in the first clustering result is subjected to second-stage hierarchical clustering processing to obtain a fine-grained category, and the fine-grained category is taken as a second clustering result.

[0017] The first-stage clustering processing is density clustering, which is used for cross-type event clustering, and the second-stage clustering processing is hierarchical clustering, which is used for same-type event clustering. Through this two-stage clustering, a fine-grained category is obtained, which avoids the overlap and ambiguity between coarse-grained categories, and is beneficial to improve the completeness and diversity of deep web news data collection.

[0018] Further, according to the deep web news data collection method of one aspect of the present disclosure, based on the surface news data, a feature vector is obtained, comprising:

[0019] The surface news data under each news column is preprocessed;

[0020] The preprocessed surface news data is subjected to feature extraction by a pre-trained embedding model to obtain an initial feature vector, and the initial feature vector is used to represent the global semantic information of the surface news data;

[0021] The initial feature vector is subjected to dimension reduction processing to obtain a feature vector.

[0022] Further, according to the deep web news data collection method of one aspect of the present disclosure, the surface news data is classified into each category of the second clustering result, and keywords in each category are extracted to generate a keyword list for each category, comprising:

[0023] The surface news data is classified into each category of the second clustering result to obtain text data of each category;

[0024] The text data of each category is converted into sparse vectorized features by using a bag-of-words model, and keywords under each category are extracted;

[0025] Based on the frequency of occurrence of each keyword in the category to which it belongs, the weight of each keyword is calculated;

[0026] The keywords under each category are arranged according to the weights to obtain a keyword list for each category.

[0027] In addition, according to the deep web news data collection method of one aspect of the present disclosure, based on the keywords in the keyword list, deep web news data is collected, including:

[0028] Based on the keyword list, a current selection order is determined, and a target keyword is selected from the keyword list;

[0029] The target keyword is submitted to a search form of a news platform to obtain a collection result of deep web news data corresponding to the target keyword;

[0030] The degree of repetition of the collection result and already collected data is calculated, and the already collected data is historical collection results stored in a database;

[0031] When the degree of repetition is less than or equal to a threshold value, the current selection order is used as a next selection order, and a next target keyword is selected from the keyword list;

[0032] When the degree of repetition is greater than the threshold value, the reverse order of the current selection order is used as the next selection order, a next target keyword is selected from the keyword list, and the threshold value is updated.

[0033] After submitting the target keyword to the search form each time, the relationship between the degree of repetition and the threshold value is compared. If the degree of repetition is greater than the threshold value, it indicates that the addition rate of news data in the collection result corresponding to the target keyword is relatively low. If the next target keyword is still selected according to the current selection order, the addition rate may be further reduced, which can easily cause repeated grabbing. Therefore, when the degree of repetition is greater than the threshold value, the selection order of the next target keyword and the threshold value are dynamically adjusted. For example, the current selection order is the order of the keyword list (from top to bottom), and the next selection order is the reverse order of the keyword list (from bottom to top). This cross-retrieval strategy can solve the problem of repeated grabbing caused by semantic overlap of keywords and effectively improve the collection efficiency of deep web news data within a unit of time.

[0034] In addition, according to the deep web news data collection method of one aspect of the present disclosure, after collecting deep web news data under each category based on the keywords in the keyword list, the method further includes:

[0035] Based on the collected deep web news data of each category, a first similarity of each deep web news data and the collected data is calculated, and target deep web news data with a first similarity less than or equal to a preset threshold is obtained;

[0036] A second similarity of the target deep web news data and each category is calculated, and target deep web news data with a second similarity less than a threshold for each category is taken as new data;

[0037] Based on the new data, new keywords are extracted;

[0038] The new keywords are classified into the keyword list of each category, and the keyword list of each category is updated based on the weight of the new keywords.

[0039] After the deep web news data collection of a certain category is completed, the existing keyword list is updated and optimized through newly collected deep web news data, new keywords are extracted to update the keyword list to capture the new dynamics of the news field, which is conducive to further improving the collection integrity of the deep web news data.

[0040] According to another aspect of the present disclosure, a deep web news data collection device is provided, comprising:

[0041] A collection module is configured to collect surface news data under each news column of different news platforms, and the surface news data is used to represent the theme content of news;

[0042] A clustering module is configured to perform first-stage clustering processing on the surface news data by calculating data density to obtain a first clustering result, and perform second-stage clustering processing on the first clustering result to obtain a second clustering result;

[0043] A keyword generation module is configured to classify the surface news data into each category of the second clustering result, extract keywords in each category respectively, and generate a keyword list of each category;

[0044] A deep web collection module is configured to collect deep web news data under each category based on the keywords in the keyword list.

[0045] According to still another aspect of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method of the above aspect.

[0046] According to still another aspect of the present disclosure, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the method of the above aspect.

[0047] According to still another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method of the above aspect.

[0048] It is to be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further explanation of the subject technology. BRIEF DESCRIPTION OF DRAWINGS

[0049] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0050] Figure 1 is a system architecture diagram illustrating application of a deep web news data collection method according to embodiments of the present disclosure.

[0051] Figure 2 is a flowchart illustrating application of a deep web news data collection method according to embodiments of the present disclosure.

[0052] Figure 3 is still another flowchart illustrating application of a deep web news data collection method according to embodiments of the present disclosure.

[0053] Figure 4 is a structural schematic diagram of a deep web news data collection apparatus according to embodiments of the present disclosure.

[0054] Figure 5 is a structural schematic diagram of a computer device according to embodiments of the present disclosure.

[0055] Figure 6 is a schematic diagram of a computer program product according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present disclosure more apparent, the following will describe example embodiments according to the present disclosure in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited by the example embodiments described herein.

[0057] With the development of new media field and digital technology, news data of news platforms has been widely applied in hot topic mining, public opinion monitoring and guiding, and many other fields, and meanwhile, higher requirements have been put forward for the completeness and efficiency of news data collection.

[0058] The news category classification mechanisms provided by news platforms are usually coarse-grained, lacking fine-grained hierarchical divisions. For example, the "Digital" category typically includes more detailed subcategories such as "Mobile Phone Themes," "Computers," "Home Appliances," and "Cameras." Existing data collection methods only perform coarse-grained clustering, easily overlooking the more subtle thematic differences between different news items within the same category. This may lead to omissions in the collection of deep web news data for certain news topics, seriously affecting the completeness of deep web news data collection.

[0059] The above description, with reference to the accompanying drawings, illustrates a method, apparatus, device, medium, and product for collecting deep web news data according to embodiments of the present disclosure. Through first-stage and second-stage clustering processes, cross-type event clustering and same-type event clustering are performed respectively, reducing overlap and ambiguity between categories, achieving fine-grained category division, ensuring that the keyword list always reflects the latest news hotspots and changes in news themes, dynamically capturing changes in news content, and effectively improving the completeness and diversity of deep web news data collection.

[0060] To facilitate understanding of this embodiment, a method for collecting deep web news data disclosed in this disclosure will first be described in detail. The execution entity of this method for collecting deep web news data is generally a computer device with a certain computing power. In some possible implementations, this method for collecting deep web news data can be implemented by a processor calling computer-readable instructions stored in memory.

[0061] like Figure 1 The diagram shown is a system architecture diagram of the deep web news data collection method provided in this embodiment of the disclosure, including a surface data collection module, a multi-level clustering module, and a deep web data collection module.

[0062] Surface data acquisition module: This module collects surface news data from various news sections across different news platforms using the Scrapy framework. The collected results are formatted and categorized according to the news section, such as finance, military, or technology. The stored surface news data includes the news title, content, publication time, and publisher. Optionally, for dynamic interactive webpage structures, this embodiment utilizes Selenium to simulate user interaction to achieve surface data acquisition.

[0063] The multi-level clustering module, targeting surface news data collected from various news sections, generates a keyword list through three sub-modules: data preprocessing, feature extraction, and text clustering. The data preprocessing sub-module performs data cleaning, word segmentation, and stop word removal on the surface news data. The feature extraction sub-module uses a pre-trained Sentence Bidirectional Encoder Representations from Transformers (Sentence-BERT) model for text embedding. Considering the characteristics of non-linear and high-dimensional data, this embodiment further utilizes the Uniform Manifold Approximation and Projection (UMAP) algorithm to reduce the dimensionality of the feature vectors. The text clustering sub-module uses the Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) algorithm to extract cross-type event topics and further utilizes hierarchical clustering algorithms to achieve fine-grained news topic clustering for similar events.

[0064] The Deep Web Data Acquisition Module: This module generates keyword lists for different categories. The keyword search and data extraction submodules simulate user search requests to collect deep web news data, extracting information such as news titles, content, publication time, and publisher. In the initial stage, searches are performed sequentially from highest to lowest ranking in the keyword list. After each search and data collection, the duplicate data evaluation and dynamic feedback submodule calculates the duplication rate of the collected deep web news data in real time and dynamically switches search strategies based on thresholds. Finally, after completing data collection for a specific category, the new data extraction and keyword update submodule dynamically feeds all newly added deep web news data to the multi-level clustering module. For new data not included in existing categories, keyword extraction and re-sorting are performed to generate a new keyword list.

[0065] like Figure 2 The flowchart shown is a method for collecting deep web news data provided in this embodiment of the disclosure, including:

[0066] S201: Surface News Data Collection: Collect surface news data from each news section on different news platforms, and format and store the surface news data according to the news section.

[0067] Specifically, a collection script can be written using a Scrapy crawler framework and a Selenium test tool to collect surface news data. Among them, news columns such as finance, military, and entertainment columns, surface news data are used to represent the theme content of news, such as news title, news content, author, and other information.

[0068] S202: Multi-level clustering: After data preprocessing, feature extraction and dimensionality reduction processing of the collected surface news data, two-stage clustering is performed on the feature vectors obtained after dimensionality reduction processing, and finally the keyword list of each fine-grained category is obtained.

[0069] Data preprocessing: The surface news data is cleaned, including removing HTML tags, unnecessary line breaks, and irrelevant characters and other noise information. Secondly, the text data in the surface news data is tagged with word features and filtered with stop words to remove high-frequency meaningless words while retaining important words that help semantic analysis.

[0070] Feature extraction: The embedded model Sentence-BERT is used to capture the context semantic relationship in the text data, and the text data is converted into an embedded vector. Specifically, the text data after data preprocessing is input into the embedded model, and the embedded model generates an embedded vector (i.e., an initial feature vector) by encoding the text. The embedded vector not only contains the local semantic relationship between words, but also integrates global semantic information, laying a foundation for subsequent category division. In addition, considering that news content is usually long text, this embodiment adopts a segmented processing method, extracts the embedded vector of each segment and aggregates it to obtain the final embedded vector (i.e., the initial feature vector), thereby generating a semantic representation of the entire news content.

[0071] Dimensionality reduction processing: For the initial feature vector output by S22, further dimensionality reduction processing is performed using the UMAP algorithm. The UMAP algorithm is used to map high-dimensional text data to a low-dimensional space based on its advantages in processing linear data and maintaining local structure, and the feature vector is obtained after dimensionality reduction.

[0072] Two-stage clustering includes HDBSCAN density clustering and hierarchical clustering:

[0073] HDBSCAN density clustering: The HDBSCAN algorithm is used to perform first-stage clustering processing on the feature vector to extract cross-type events and obtain coarse-grained category division.

[0074] First, based on the data points representing the feature vectors, the mutual reachable distance d mutual between each data point is calculated, and the calculation formula is as follows:

[0075] d mutual(p, q) = max(core_dist(p), core_dist(q), d(p, q))

[0076] Where core_dist(p) is the Euclidean distance from data point p to its kth nearest neighbor, the smaller the distance is, the higher the density around the data point is, and d(p, q) is the Euclidean distance between data point p and data point q.

[0077] Then, these Euclidean distances are regarded as the edge weights of a complete graph, and a minimum spanning tree T of the complete graph is constructed by using Kruskal algorithm. Secondly, a distance threshold β = max(d mutual (x, y)), x, y ∈ V(T) is set, and when the edge weight of an edge in the minimum spanning tree T exceeds β, the edge is disconnected, and different connected branches of the tree are regarded as a new cluster. By continuously reducing the value of β to min(d mutual (x, y)), x, y ∈ V(T), and a minimum cluster value m is set in the process, and when the number of points contained in a newly generated cluster is less than m, the cluster is regarded as noise, and the value of m is for example 10, which can be adjusted according to actual requirements.

[0078] Finally, the stability λ stability of the cluster under different β is calculated to obtain the coarse-grained categories of the final clustering, and the surface news data is classified into the coarse-grained categories to obtain a first clustering result.

[0079]

[0080] Where λ p represents the reciprocal of the distance threshold β when data point p exits the cluster, and λ birth represents the reciprocal of the distance threshold β when the cluster is first split out. If the stability of the final cluster is greater than the sum of the stabilities of the sub-clusters, the cluster is retained.

[0081] Hierarchical clustering: In order to realize more fine-grained category division, the embodiment further adopts a bottom-up agglomerative hierarchical clustering algorithm to perform second-stage clustering processing on the same-type event sets to obtain a second clustering result.

[0082] Specifically, the surface news data belonging to the same coarse-grained category is classified into a same-type event set, an appropriate number of clusters K is selected for each same-type event set, the same-type event set is subjected to clustering analysis, and a second clustering result, i.e., a fine-grained category, is obtained. The number of clusters K is used to determine the division of the fine-grained category, and in order to find the optimal number of clusters K, the silhouette coefficient is used as an evaluation index in the embodiment, the K value that makes the silhouette coefficient maximum is selected, and a more accurate division result of the fine-grained category is obtained. The silhouette coefficient s(i) is calculated according to the following formula:

[0083]

[0084] where a(i) represents the cohesion within the cluster, i.e. the average Euclidean distance of data point i to all other points in the same cluster; b(i) represents the separation between clusters, i.e. the average distance of data point i to all points in the nearest other cluster.

[0085] For the silhouette coefficient of the entire data set, it can be represented by the average value S of the silhouette coefficients of all data points:

[0086]

[0087] where N is the total number of data points, and by calculating the silhouette coefficient for different cluster numbers, the cluster number with the maximum silhouette coefficient can be found.

[0088] The density clustering algorithm is used for cross-type event clustering, and the hierarchical clustering algorithm is used for same-type event clustering, so as to obtain fine-grained categories through the two-stage clustering, avoid the overlap and ambiguity between coarse-grained categories, and ensure that the keyword list always reflects the latest news hotspots and news theme changes, so as to dynamically capture the changes of news content and effectively improve the completeness and diversity of deep web news data collection.

[0089] Generating a keyword list includes the following steps:

[0090] (1) Classifying the surface news data into each category of the second clustering result to obtain text data of each category;

[0091] (2) Converting the text data of each category into sparse vectorized features using the bag-of-words model, and extracting keywords under each category using an improved Term Frequency-Inverse Document Frequency (c-TF-IDF) method, the calculation formula of c-TF-IDF is as follows:

[0092]

[0093] where W t,c represents the weight of word t in cluster c, tf t,c represents the frequency of word t appearing in cluster c, A represents the average number of words per cluster, tf t represents the number of times word t appears in all clusters.

[0094] (3) Calculating the weight of each keyword based on the frequency of each keyword appearing in the category to which it belongs;

[0095] (4) Arranging the keywords under each category according to the weights to obtain a keyword list of each category.

[0096] S203: Deep web news data collection: based on the keyword list under each category, the keywords are selected one by one according to the selected order, and submitted to the search form of the news platform for deep web news data collection.

[0097] In order to reduce the influence of excessive repeated content on the collection efficiency, the embodiment provides a cross-search strategy based on repeated data evaluation, which specifically includes:

[0098] (1) Based on the keyword list, determine the current selection order, and select the target keyword from the keyword list, wherein if it is the first selection, the current selection order can be set as the order of the keyword list.

[0099] (2) Submit the target keyword to the search form of the news platform to obtain the collection result of the deep web news data corresponding to the target keyword.

[0100] (3) Calculate the repetition degree of the collection result and the collected data.

[0101] When the repetition degree is less than or equal to the threshold value, the current selection order is used as the next selection order, and the next target keyword is selected from the keyword list.

[0102] When the repetition degree is greater than the threshold value, the reverse order of the current selection order is used as the next selection order, and the next target keyword is selected from the keyword list, and the threshold value is updated.

[0103] Among them, the collected data is the historical collection result stored in the database, after each keyword is submitted, the corresponding collection result is compared with the collected data and the repetition degree is calculated, when the repetition degree is greater than the threshold value, the selection order is changed to reverse order, and a new threshold value is reset. During the collection process, the selection order and the threshold value are constantly adjusted until the entire keyword list is completely traversed. The updating formula of the threshold value is:

[0104] θ n = θ0·(1+α·n) k

[0105] Among them, θ n is the threshold value of the nth iteration, θ0 is the initial threshold value, α and k are constants, α controls the rate of increase of the threshold value, and k is used to adjust the increase speed of the slope. By adjusting α and k, the increase speed of the threshold value can be flexibly controlled, which can adapt to the needs of different data collection. For example, θ0 can be set to 20%, α is 0.05, and k is 1. Users can adjust θ0, α and k according to the actual collection efficiency requirements.

[0106] After each target keyword is submitted to the search form, the relationship between the repetition degree and the threshold value is compared. If the repetition degree is greater than the threshold value, it indicates that the increase rate of news data in the collection result corresponding to the target keyword is low. If the next target keyword is still selected according to the current selection order, the increase rate may be further reduced, which is easy to cause repeated crawling. Therefore, when the repetition degree is greater than the threshold value, the selection order of the next target keyword and the threshold value are dynamically adjusted. For example, the current selection order is the order of the keyword list (from top to bottom), and then the next selection order is the reverse order of the keyword list (from bottom to top). This cross-retrieval strategy can solve the problem of repeated crawling caused by semantic overlap of keywords and effectively improve the collection efficiency of deep web news data in unit time.

[0107] S204: Extracting new data and updating the keyword list: when the deep web news data collection of a certain category is completed, the new data is further extracted and the keyword list is updated.

[0108] Specifically, the following steps are included:

[0109] News deduplication: based on the collected deep web news data of each category, the first similarity between each deep web news data and the collected data is calculated, and the target deep web news data with a first similarity less than or equal to a preset threshold value is obtained.

[0110] Specifically, in order to identify the new deep web news data, the title and publication time information of each newly collected deep web news data are extracted and compared with the collected data. The cosine similarity algorithm is used to compare the titles. If the similarity between the title of the newly collected deep web news data and the title of the existing news is greater than the set threshold value, and the publication time is close, it is considered that the two news are repeated content. The repeated news is deduplicated, and only the target deep web news data is retained.

[0111] Extracting new data: calculating the second similarity of the target deep web news data with each category, and taking the target deep web news data with a second similarity less than a threshold value as new data.

[0112] Specifically, the target deep web news data is classified, and compared with each category according to its characteristics. It is classified into the closest category. If the similarity of the target deep web news data with each category is low, it is considered as new data.

[0113] Updating the keyword list: based on the new data, extracting new keywords; classifying the new keywords into the keyword list of each category, and updating the keyword list of each category based on the weight of the new keywords.

[0114] Specifically, to improve the completeness of deep web news data collection, the keyword list is updated according to the new data, the weights of new keywords and old keywords are re-adjusted, and the keyword list is reordered. For the new keywords, the initial weight is assigned by the following formula:

[0115]

[0116] wherein f new represents the frequency of the new keyword appearing in the new data, F total represents the total frequency of all keywords in the new data, R new represents the relevance score of the new keyword to the category, and β represents a factor adjusting the influence of relevance on weight.

[0117] The updating formula of the old keyword weight is as follows:

[0118] W updated = W old ·(1+γ·Δf)·(1-λ·ΔT)

[0119] wherein W old represents the original weight of the old keyword, Δf represents the frequency change rate of the old keyword in the new data, γ represents a factor adjusting the influence of frequency change on weight, ΔT represents the time decay factor of the old keyword, represents the time span since the last use of the keyword (0-1), and λ represents a factor adjusting the influence of time decay on weight.

[0120] Finally, the keyword list is reordered according to the new keywords and the updated keyword weights, and is applied as feedback to the next deep web data collection.

[0121] By introducing the dynamic updating mechanism of the keyword list, after completing the deep web news data collection task of a single category, the keyword list is dynamically adjusted based on the new news to improve the completeness of deep web news data collection.

[0122] Based on the above embodiment, in another embodiment provided by the present disclosure, a deep web news data collection method is also provided. As shown in Figure 3 Fig. 4 is another flowchart of the deep web news data collection method provided by the embodiment of the present disclosure, which comprises S301-S304:

[0123] S301: Collecting surface news data under each news column of different news platforms.

[0124] The surface news data is used to represent the theme content of the news, for example, including news title, news content, publication time and publisher information, etc.

[0125] S302: Perform first-stage clustering processing on the surface news data by calculating the data density to obtain a first clustering result, and perform second-stage clustering processing on the first clustering result to obtain a second clustering result.

[0126] S303: Classify the surface news data into each category of the second clustering result, extract keywords in each category respectively, and generate a keyword list for each category.

[0127] S304: Collect deep web news data under each category based on the keywords in the keyword list.

[0128] In one or more embodiments, S302 includes the following steps:

[0129] Based on the surface news data, obtain a feature vector;

[0130] Obtain the data density of each feature vector through the Euclidean distance between each feature vector;

[0131] Perform first-stage clustering processing based on the data density to obtain a coarse-grained category;

[0132] Classify the surface news data into each coarse-grained category to obtain a first clustering result;

[0133] Perform second-stage hierarchical clustering processing on the surface news data included in each category of the first clustering result to obtain a fine-grained category, and take the fine-grained category as a second clustering result.

[0134] In one or more embodiments, based on the surface news data, a feature vector is obtained, including the following steps:

[0135] Preprocess the surface news data under each news column;

[0136] Perform feature extraction on the preprocessed surface news data through a pre-trained embedding model to obtain an initial feature vector, which is used to represent the global semantic information of the surface news data;

[0137] Perform dimension reduction processing on the initial feature vector to obtain a feature vector.

[0138] In one or more embodiments, S303 includes the following steps:

[0139] Classify the surface news data into each category of the second clustering result to obtain text data for each category;

[0140] Convert the text data of each category into sparse vectorized features using a bag-of-words model, and extract keywords under each category;

[0141] a weight of each keyword is calculated based on a frequency of occurrence of the keyword in the category to which the keyword belongs;

[0142] the keywords in each category are arranged according to the weights to obtain a keyword list of each category.

[0143] In one or more embodiments, S304 comprises the following steps:

[0144] a target keyword is selected from the keyword list based on the current selection order;

[0145] the target keyword is submitted to a search form of a news platform to obtain a collection result of deep web news data corresponding to the target keyword;

[0146] a degree of repetition of the collection result and already collected data is calculated, the already collected data being historical collection results stored in a database;

[0147] when the degree of repetition is less than or equal to a threshold value, the current selection order is taken as a next selection order, and a next target keyword is selected from the keyword list;

[0148] when the degree of repetition is greater than the threshold value, a reverse order of the current selection order is taken as the next selection order, a next target keyword is selected from the keyword list, and the threshold value is updated.

[0149] In one or more embodiments, the method further comprises the following steps:

[0150] based on the collected deep web news data in each category, a first similarity of each of the deep web news data and already collected data is calculated, and target deep web news data with a first similarity less than or equal to a preset threshold value is obtained;

[0151] a second similarity of the target deep web news data and each category is calculated, and target deep web news data with a second similarity less than a threshold value for each category is taken as new data;

[0152] based on the new data, new keywords are extracted;

[0153] the new keywords are classified into the keyword list of each category, and the keyword list of each category is updated based on a weight of the new keywords.

[0154] According to another aspect of the embodiments of the present disclosure, a deep web news data collection device is provided, as shown in Figure 4 The device comprises:

[0155] The collection module 401 is configured to collect surface news data under each news column of different news platforms, and the surface news data is used to represent the theme content of news.

[0156] The clustering module 402 is configured to perform first-stage clustering processing on the surface news data by calculating data density, to obtain a first clustering result, and perform second-stage clustering processing on the first clustering result, to obtain a second clustering result.

[0157] The keyword generation module 403 is configured to classify the surface news data into each category of the second clustering result, extract keywords in each category respectively, and generate a keyword list of each category.

[0158] The deep web collection module 404 is configured to collect deep web news data under each category based on the keywords in the keyword list.

[0159] The deep web news data collection device provided in the embodiments of the present disclosure and the deep web news data collection method provided in the embodiments of the present disclosure have the same beneficial effects as the method adopted, run or implemented by them.

[0160] The embodiments of the present disclosure further provide a computer device for executing the deep web news data collection method described above. Please refer to Figure 5 which shows a schematic diagram of a computer device provided by some embodiments of the present disclosure. As shown in Figure 5 The computer device 50 includes a processor 500, a memory 501, a bus 502 and a communication interface 503, wherein the processor 500, the communication interface 503 and the memory 501 are connected through the bus 502; the memory 501 stores a computer program which can run on the processor 500, and the processor 500 runs the computer program to execute the deep web news data collection method provided by any of the preceding embodiments of the present disclosure.

[0161] The memory 501 can include a high-speed random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. The communication between the device network element and at least one other network element is realized through at least one communication interface 503 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0162] The bus 502 can be an ISA bus, a PCI bus, an EISA bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. The memory 501 is configured to store programs, and the processor 500 executes the programs after receiving execution instructions. The method for collecting deep web news data according to any of the embodiments of the present disclosure can be applied to the processor 500 or implemented by the processor 500.

[0163] The processor 500 can be an integrated circuit chip with processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 500 or an instruction in the form of software. The processor 500 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), and the like; can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready programmable gate array (FPTA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step, and logic block disclosed in the embodiments of the present disclosure can be implemented or executed. The general processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, or other mature storage media in the art. The storage medium is located in the memory 501, and the processor 500 reads the information in the memory 501 and combines the hardware to complete the steps of the above method.

[0164] The computer device provided by the embodiments of the present disclosure and the method for collecting deep web news data provided by the embodiments of the present disclosure have the same beneficial effects as the method they adopt, run, or implement.

[0165] The embodiments of the present disclosure also provide a computer readable storage medium corresponding to the method for collecting deep web news data provided by the preceding embodiments. The computer readable storage medium is an optical disc, and a computer program (i.e., a computer program product) is stored on the optical disc. When the computer program is run by a processor, the method for collecting deep web news data provided by any of the preceding embodiments is executed.

[0166] It should be noted that examples of the computer-readable storage medium can also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical, magnetic storage media, and the like, which will not be listed one by one here.

[0167] The computer-readable storage medium provided by the above embodiments of the present disclosure has the same beneficial effects as the method adopted, run or implemented by the application program stored therein, based on the same inventive concept as the method of collecting deep web news data provided by the embodiments of the present disclosure.

[0168] The embodiments of the present disclosure also provide a computer program product, please refer to Figure 6 The computer program product 600 carries a program code, that is, a computer program 601, and the instructions included in the computer program 601 can be used to execute the steps of the method of collecting deep web news data described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0169] The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0170] The above describes the basic principles of the present disclosure in combination with specific embodiments, but it should be noted that the advantages, advantages, effects and the like mentioned in the present disclosure are only examples and not limitations, and these advantages, advantages, effects and the like cannot be considered as the must-have of each embodiment of the present disclosure. In addition, the above specific details are only for the purpose of example and understanding, and are not limited to the specific details disclosed above for the implementation of the present disclosure.

[0171] The block diagrams of devices, apparatuses, equipment, systems referred to in the present disclosure are merely illustrative examples and are not intended to require or imply that the connection, arrangement, configuration must be as shown in the block diagrams. These devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner as will be appreciated by those skilled in the art. Words such as "include," "contain," "have," etc. are open-ended words that are to be interpreted to mean "including but not limited to," and are to be interpreted not to exclude other items. The words "or" and "and" as used herein are to be interpreted as the word "and / or," and are to be interpreted not to exclude other items. The word "such as" as used herein is to be interpreted as the phrase "such as but not limited to," and is to be interpreted not to exclude other items.

[0172] In addition, as used herein, the "or" as used in the context "at least one of A, B, or C" : means A or B or C or any combination thereof. Further, the phrase "example of" is not meant to be limiting in terms of the examples described. For example, the phrase "example of A, B, or C" means A or B or C, or any combination thereof.

[0173] It is also important to note that the systems and methods of the present disclosure can be embodied in a variety of forms including, but not limited to, a data processor, a computer program product, a computer, one or more tangible computer readable storage devices, one or more computer-implemented methods, information, or a bit of information. Additionally the systems and methods of the present disclosure can be embodied as one or more computers or computer implementations that include one or more processors or one or more memory modules.

[0174] Various changes, modifications and alterations in the teachings and techniques described herein can be made without departing from the teachings that are defined by the appended claims. Further, the scope of the claims of the present disclosure is not limited to the specific aspects described herein. Processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

[0175] The above description of the disclosed aspects is meant to be illustrative of the application and not limiting. Various modifications of the aspects will be apparent to those with ordinary skill in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0176] The foregoing description has been presented for the purposes of illustration and description. Furthermore, the description is not intended to limit the embodiments of the disclosure to the forms disclosed herein. Although the various example aspects and embodiments have been described herein with regard to particular aspects and embodiments, those skilled in the art will recognize that certain modifications, changes, substitutions, additions and sub-combinations can be made without departing from the spirit of the disclosure.

Claims

1. A method for collecting deep web news data, characterized in that, include: Collect surface news data from various news sections on different news platforms; the surface news data is used to characterize the main content of the news. The surface news data is subjected to a first-stage clustering process by calculating data density to obtain a first clustering result, and the first clustering result is subjected to a second-stage clustering process to obtain a second clustering result. The surface news data is categorized into the categories of the second clustering results, and keywords are extracted from each category to generate a keyword list for each category. Based on the keywords in the keyword list, collect deep web news data under each category.

2. The method for collecting deep web news data as described in claim 1, characterized in that, The surface news data is subjected to a first-stage clustering process by calculating data density to obtain a first clustering result. The first clustering result is then subjected to a second-stage clustering process to obtain a second clustering result, including: Based on the aforementioned surface news data, a feature vector is obtained; The data density of each feature vector is obtained by using the Euclidean distance between each feature vector; Based on the data density, a first-stage clustering process is performed to obtain coarse-grained categories; The surface news data is categorized into the coarse-grained categories to obtain the first clustering result; A second stage of hierarchical clustering is performed on the surface news data included in each category of the first clustering result to obtain fine-grained categories, and the fine-grained categories are used as the second clustering result.

3. The method for collecting deep web news data as described in claim 2, characterized in that, Based on the aforementioned surface news data, a feature vector is obtained, including: Preprocess the surface news data under each of the aforementioned news categories; The pre-trained embedding model is used to extract features from the pre-processed surface news data to obtain an initial feature vector, which is used to characterize the global semantic information of the surface news data. The initial feature vector is subjected to dimensionality reduction processing to obtain the feature vector.

4. The method for collecting deep web news data as described in claim 1, characterized in that, The surface news data is categorized into the categories of the second clustering results, and keywords are extracted from each category to generate a keyword list for each category, including: The surface news data is categorized into the categories of the second clustering result to obtain the text data of each category; The bag-of-words model is used to transform the text data of each category into sparse vectorized features, and keywords under each category are extracted; The weight of each keyword is calculated based on its frequency of occurrence in its respective category; The keywords under each category are arranged according to the weights to obtain a list of keywords for each category.

5. The method for collecting deep web news data as described in claim 1, characterized in that, Based on the keywords in the keyword list, deep web news data is collected, including: Determine the current selection order based on the keyword list, and select the target keyword from the keyword list; Submit the target keywords to the search form of the news platform to obtain the collection results of deep web news data corresponding to the target keywords; Calculate the overlap between the collected results and the collected data, wherein the collected data is the historical collected results stored in the database; When the repetition rate is less than or equal to the threshold, the current selection order is used as the next selection order, and the next target keyword is selected from the keyword list; When the repetition rate is greater than the threshold, the reverse of the current selection order is used as the next selection order, the next target keyword is selected from the keyword list, and the threshold is updated.

6. The method for collecting deep web news data as described in claim 1, characterized in that, After collecting deep web news data under each category based on the keywords in the keyword list, the process also includes: Based on the collected deep web news data under each of the aforementioned categories, calculate the first similarity between each of the deep web news data and the collected data, and obtain target deep web news data whose first similarity is less than or equal to a preset threshold. Calculate the second similarity between the target deep web news data and each of the categories, and take the target deep web news data whose second similarity with each of the categories is less than the threshold as new data; Based on the newly added data, extract new keywords; The newly added keywords are assigned to the keyword lists of each category, and the keyword lists of each category are updated based on the weight of the newly added keywords.

7. A device for collecting deep web news data, characterized in that, include: The data collection module is used to collect surface news data from various news sections on different news platforms. The surface news data is used to represent the main content of the news. The clustering module is used to perform a first-stage clustering process on the surface news data by calculating data density to obtain a first clustering result, and then perform a second-stage clustering process on the first clustering result to obtain a second clustering result. The keyword generation module is used to classify the surface news data into the categories of the second clustering result, extract keywords from each category, and generate a keyword list for each category. The deep web data collection module is used to collect deep web news data under each category based on the keywords in the keyword list.

8. A computer embedded device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 6.

Images