Data acquisition method based on SpiderFlow

Abstract

The invention discloses a SpiderFlow-based data acquisition method, which is characterized by comprising the following steps of: acquiring a data acquisition request, and determining information such as a target URL (Uniform Resource Locator), an access path, a target element position and the like which need to be acquired; the method comprises the following steps: creating a data acquisition task on a SpiderFlow platform, and configuring corresponding task information; performing collection rule development of specific elements on the target website according to requirements; setting the thread number and the scheduling frequency of the collection task; starting a collection thread at regular time, accessing a website URL (Uniform Resource Locator) specified by a task, collecting specified element contents, and cleaning the collected page data to form normalized data; after the acquisition is completed, updating a URL state, and reporting detailed data through a Kafka message queue; a Kafka message is consumed at a ClickHouse end and is stored in a persistent manner; the collected data can be subjected to targeted logic processing again in ClickHouse, and finally data analysis of various dimensions is carried out according to needs.

Description

Technical Field

[0001] The present invention relates to the field of data collection and analysis of big data, and in particular to a data collection method based on SpiderFlow. Background Art

[0002] In the era of information explosion, a large amount of data of all kinds is surging on the Internet. How to quickly and efficiently collect specific public data that is valuable to enterprises and analyze and mine it is the key and difficulty in data collection and analysis. Traditional data collection methods store crawled data in various relational databases, which have bottlenecks such as slow timeliness and limited storage capacity, and are difficult to support logical processing and real-time data analysis under large amounts of data.

[0003] In the existing technology, there are many tools for data collection and analysis. In the data collection direction, tools similar to SpiderFlow include Octoparse, ParseHub, and WebScraper, while in the real-time data analysis and processing direction, tools similar to ClickHouse include Apache Druid, Presto, and Kylin.

[0004] Octoparse and ParseHub are both powerful visual crawler tools that can be used to design crawlers to crawl network data with simple clicks and drags. WebScraper is a Chrome browser plug-in that allows users to design and run crawlers directly in the browser. Although Octoparse and ParseHub are easy to use, they may require writing code when dealing with complex crawler tasks, which is a challenge for non-professional users. Although WebScraper can be run directly in the browser, its performance is limited by the browser and cannot handle large-scale data crawling tasks.

[0005] Apache Druid, Presto, and Apache Kylin are all powerful real-time data analysis tools that perform well in specific application scenarios. For example, Apache Druid excels in real-time data ingestion and ad hoc queries, Presto is suitable for interactive queries of different data sources, and Apache Kylin excels in multi-dimensional analysis of big data. However, these tools still cannot compete with ClickHouse in some aspects. For example, although Apache Druid's data ingestion is real-time, its data storage and query optimization may not be as efficient as ClickHouse. Although Presto supports multiple data sources, its performance may not be as good as ClickHouse, which is optimized for columnar storage, when facing a single large-scale columnar data source. Although Apache Kylin has advantages in multi-dimensional analysis, its pre-calculation model may lead to insufficient flexibility in some real-time analysis scenarios.

[0006] Therefore, the existing technologies / combinations cannot meet the needs of this case, that is, in the direction of data collection, a tool that can handle complex crawler tasks and is easy to use is needed; in the direction of real-time data analysis and processing, a tool that can handle big data and has powerful query functions is needed. Summary of the invention

[0007] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art, solve the problem that the existing data collection and analysis methods cannot meet the storage requirements of large data sets and the timeliness requirements of data analysis, and provide a data collection method based on SpiderFlow.

[0008] In order to solve the above technical problems, the present invention provides a data collection method based on SpiderFlow. After the collection is completed, the data is connected to the Kafka engine table of ClickHouse, and the data is incrementally persisted and stored in a streaming manner and analyzed and calculated in real time. Combining the above characteristics, the two are combined to realize data collection and real-time analysis under large data volumes. Specifically, the following steps are included:

[0009] Step 1: Obtain a data collection request, which includes a URL, an access path, and an element location;

[0010] Step 2: Create a data collection task corresponding to the collection request on the SpiderFlow platform, and configure the task information corresponding to the data collection task. The task information includes the following mandatory items: task name, seed URL, task type, scheduling frequency, and start / stop status; the task information also includes the following optional items: scheduling priority, site category, and remarks;

[0011] Step 3: Develop rules for specific data collection tasks on the target website based on the data collection request, describe the path through Xpath and Selectors expressions, and locate the target;

[0012] Step 4: Configure the number of threads and task scheduling frequency of the data collection task to automatically trigger the execution of the data collection task in multiple threads, and achieve different data crawling capacities by setting different numbers of threads and scheduling frequencies;

[0013] Step 5: The data collection task is started regularly, the designated website URL is accessed, the corresponding page element is located according to the path rule configured in step 3, its data content is collected, and data cleaning is performed. The data cleaning includes setting the program execution order and adjusting the data format to form relatively standardized page detail data;

[0014] Step 6: Send the page details data to the specified Topic through the Kafka message queue, and update the collection status of this URL;

[0015] Step 7: Access the Kafka message queue in step 6 through the Kafka table engine of ClickHouse, and use materialized views to persist the Kafka data source data into the ClickHouse table data of the MergeTree engine;

[0016] Step 8: Clean and aggregate the ClickHouse table data in step 7 through the materialized view of ClickHouse. Taking forum data as an example, you can analyze the number of posts, update frequency, article length, comment area popularity, keyword appearance ratio, etc. in different forums, and perform data analysis on the collected data.

[0017] 2. A data collection method based on SpiderFlow as claimed in claim 1, characterized in that: said step 4 comprises:

[0018] Step 4-1: Setting the number of threads of the acquisition task, wherein the acquisition task executes the same code logic;

[0019] Step 4-2: The task scheduling frequency is achieved by configuring the CRON expression;

[0020] Step 4-3: For the collection of large amounts of data, use proxy IP and Cookie to manage it at the same time.

[0021] The step 5 includes:

[0022] Step 5-1: Parse the webpage source code according to the path rule configured in step 3, locate the content in the target HTML tag or Script element, and obtain the native data;

[0023] Step 5-2: Further clean the acquired native data, add loop or branch conditions according to data characteristics in setting the program execution order, and parse and reorganize the data in array or Json format to form standardized page detail data.

[0024] The beneficial effects achieved by the present invention are: through the SpiderFlow platform (supporting HTTP interface, proxy IP, automatic Cookie management, multi-data source docking, task log monitoring and other features), crawler task configuration can be customized as needed, realizing the data collection capability of large data volumes in most scenarios; at the same time, by docking with the Kafka engine table of ClickHouse, the data is incrementally persistently stored and analyzed and calculated in real time in a streaming manner, meeting the storage requirements of large data sets and the timeliness requirements of data analysis; providing convenience for subsequent docking with enterprise data warehouses for further data analysis and mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of a method flow of an exemplary embodiment of the present invention;

[0026] Figure 2 A data processing flow chart in an exemplary embodiment of the present invention;

[0027] Figure 3 A data collection task scheduling diagram in an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0028] SpiderFlow is an open source, highly flexible and configurable data collection platform. It defines crawler logic through flowcharts, and crawler tasks can be configured with low-code development. The platform complies with the Spider Protocol, supports Xpath / JsonPath / CSS selectors / regular extraction / mashup extraction, supports crawling JS dynamically rendered pages, supports HTTP interfaces, supports proxy IPs, supports automatic Cookie management, supports multi-data source docking, supports task monitoring and task logs, etc., and can customize the collection of large amounts of data.

[0029] ClickHouse is an open source distributed columnar database management system (DBMS) designed for online analytical processing (OLAP). It performs well in processing large-scale data sets and has high performance and high scalability. Its key features are: columnar storage, high performance, distributed architecture, real-time data access, and strong scalability; ClickHouse is suitable for scenarios that require large-scale data analysis and query, such as data warehouses, log analysis, time series data analysis, etc.

[0030] SpiderFlow, with its visual operating interface and powerful scalability, can easily handle various complex crawler tasks; ClickHouse, with its columnar storage design and efficient query engine, can easily handle TB-level data analysis tasks, solving the problem that existing data collection and analysis methods cannot meet the storage requirements of large data sets and the timeliness requirements of data analysis.

[0031] The following is combined with Figure 1 The data processing flow chart further illustrates the present invention:

[0032] like Figure 1 The data collection method based on SpiderFlow is shown, which is characterized by being able to meet the needs of large data volume collection in most scenarios, and being able to perform incremental persistent storage and real-time analysis and calculation on the data, and specifically includes the following steps:

[0033] Step 1: According to the collection requirements, clarify the entire collection page access path, which URLs need to be collected, which specific elements to collect on each URL page, what kind of cleaning needs to be done, and other information.

[0034] Step 2: Create the corresponding data collection task on the SpiderFlow platform and configure the corresponding task information according to the content described in step 1.

[0035] Step 3: Develop rules for collecting specific elements on the target website, and use Xpath and Selectors expressions to locate the specified elements through element path characteristics.

[0036] Step 4: Configure the number of threads and task scheduling frequency of the collection task to automatically trigger multi-threaded execution, and achieve different data crawling capacities by controlling the number of threads and scheduling frequency.

[0037] Step 4-1: The number of threads for the collection task can be set, and the same code logic is executed;

[0038] Step 4-2: Task scheduling frequency can be achieved by configuring CRON expressions;

[0039] Step 4-3: For the collection of large amounts of data, use proxy IP and Cookie management to improve the collection success rate.

[0040] Step 5: The collection thread starts regularly and accesses the website URL specified by the task to collect and clean it.

[0041] Step 5-1: Locate the corresponding page elements according to the rules configured in step 3 and collect their native content;

[0042] Step 5-2: After obtaining the original data on the page, clean it up again, such as removing leading and trailing spaces, judging branch conditions, calling interface services, and assembling it into Json format, etc., to form standardized page detail data.

[0043] Step 6: The detailed data collected in step 5 will be reported to the Kafka message queue, and the URL status bit in the corresponding task will be updated.

[0044] Step 7: Access the Kafka message in step 6 through the Kafka table engine of ClickHouse, and store the Kafka data source data persistently as ClickHouse table data of the MergeTree engine through materialized views

[0045] Step 8: Use the materialized view of ClickHouse to clean and aggregate the data in step 7 again, and finally perform various data analyses on the collected data.

[0046] The following is combined with Figure 2 The data processing flow diagram further illustrates the data processing flow of the present invention:

[0047] like Figure 2 The data processing flow diagram is shown as follows: the business system pushes the collection request to Mysql. After the SpiderFlow collection platform listens to the request, it will collect data. The detailed data is stored in ClickHouse and then extracted to the data warehouse for various business analyses. Finally, it can be imported back to the business system, produce related reports, or serve other applications.

[0048] The following is an example of collecting forum data. Figure 3 The data acquisition task scheduling diagram further illustrates the present invention:

[0049] Suppose we need to collect the hot search titles and popularity values ​​of 4 forums and require them to be updated once a minute. Considering the probability of failure in a single access, we set the frequency to execute once every 20 seconds, that is, it can be executed at most 3 times in 1 minute, such as Figure 3As shown in the data collection task scheduling diagram: the number of configured task threads is 4, and the single thread scheduling frequency is once every 20 seconds through the CRON expression. The first two tasks are started at the 0th second of every 20 seconds, and the last two tasks are started at the 10th second of every 20 seconds. Of course, the scheduling frequency can be dynamically adjusted according to business needs.

[0050] The beneficial effects achieved by the present invention are: through the SpiderFlow platform (supporting HTTP interface, proxy IP, automatic Cookie management, multi-data source docking, task log monitoring and other features), crawler task configuration can be customized as needed, realizing the data collection capability of large data volumes in most scenarios; at the same time, by docking with the Kafka engine table of ClickHouse, the data is incrementally persistently stored and analyzed and calculated in real time in a streaming manner, meeting the storage requirements of large data sets and the timeliness requirements of data analysis; and providing convenience for subsequent docking with enterprise data warehouses for further data analysis and mining.

[0051] The present invention may also have many other implementation modes. The above embodiments do not limit the present invention in any way. Without departing from the spirit and essence of the present invention, technicians familiar with the field may make various corresponding changes and modifications based on the present invention. All other improvements and applications made to the above embodiments in an equivalent transformation manner should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A data collection method based on SpiderFlow, characterized in that: The steps include: Step 1: Obtain a data collection request, which includes a URL, an access path, and an element location; Step 2: Create a data collection task corresponding to the collection request on the SpiderFlow platform, and configure the task information corresponding to the data collection task. The task information includes the following mandatory items: task name, seed URL, task type, scheduling frequency, and start / stop status; the task information also includes the following optional items: scheduling priority, site category, and remarks; Step 3: Develop rules for specific data collection tasks on the target website based on the data collection request, describe the path through Xpath and Selectors expressions, and locate the target; Step 4: Configure the number of threads and task scheduling frequency of the data collection task to automatically trigger the execution of the data collection task in multiple threads, and achieve different data crawling capacities by setting different numbers of threads and scheduling frequencies; Step 5: The data collection task is started regularly, the designated website URL is accessed, the corresponding page element is located according to the path rule configured in step 3, its data content is collected, and data cleaning is performed. The data cleaning includes setting the program execution order and adjusting the data format to form relatively standardized page detail data; Step 6: Send the page details data to the specified Topic through the Kafka message queue, and update the collection status of this URL; Step 7: Access the Kafka message queue in step 6 through the Kafka table engine of ClickHouse, and use materialized views to persist the Kafka data source data into the ClickHouse table data of the MergeTree engine; Step 8: Clean and aggregate the ClickHouse table data in step 7 through the materialized view of ClickHouse. Taking forum data as an example, you can analyze the number of posts, update frequency, article length, comment area popularity, keyword appearance ratio, etc. in different forums, and perform data analysis on the collected data.

2. A data collection method based on SpiderFlow as claimed in claim 1, characterized in that: The step 4 includes: Step 4-1: Setting the number of threads of the acquisition task, wherein the acquisition task executes the same code logic; Step 4-2: The task scheduling frequency is achieved by configuring the CRON expression; Step 4-3: For the collection of large amounts of data, use proxy IP and Cookie to manage it at the same time.

3. A data collection method based on SpiderFlow as claimed in claim 2, characterized in that: The step 5 includes: Step 5-1: Parse the webpage source code according to the path rule configured in step 3, locate the content in the target HTML tag or Script element, and obtain the native data; Step 5-2: Further clean the acquired native data, add loop or branch conditions according to data characteristics in setting the program execution order, and parse and reorganize the data in array or Json format to form standardized page detail data.

Images