One-stop acquisition system and method for multi-source heterogeneous data

Abstract

The invention discloses a one-stop acquisition system and method for multi-source heterogeneous data, and belongs to the technical field of big data processing. The system constructs a unified acquisition platform by integrating web crawlers, Sqoop, Flume and Kafka components, adopts a task management module to realize dynamic scheduling of the components, and supports configuration of structured data extraction rules, unstructured data analysis templates and real-time data filtering conditions through a visual interface. According to the technical scheme, data features are automatically recognized through a data source adaptation layer, corresponding collection assemblies are called to complete data extraction, and the data are stored to a target system after format standardization processing. According to the invention, the problem of low efficiency caused by multi-tool switching in the prior art is solved, and the operation and maintenance complexity of a data acquisition scene is reduced.

Description

Technical Field

[0001] This invention relates to the field of big data acquisition and processing, and specifically proposes a one-stop acquisition system and method for multi-source heterogeneous computer data. Background Technology

[0002] In recent years, with the popularization and development of technologies such as the Internet, mobile Internet, and 5G, data is growing at an exponential rate, and the sources of data are becoming more and more diverse, including network data, database data, and log data. At the same time, the types and formats of data have gradually expanded from the initial structured data to semi-structured / unstructured data such as text, video, images, and audio. Therefore, one-stop, undifferentiated multi-source heterogeneous data acquisition systems and methods have once again attracted great attention.

[0003] Web crawlers are an important method for acquiring web data. Their basic workflow is as follows: 1. First, select a carefully chosen set of seed URLs; 2. Add these URLs to a queue of URLs to be crawled; 3. Retrieve URLs from the queue, resolve DNS, obtain the host's IP address, download the corresponding web pages, and store them in a downloaded webpage database. Additionally, add these URLs to a queue of crawled URLs; 4. Analyze the URLs in the queue, analyze other URLs within them, and add these URLs to the queue of URLs to be crawled, thus starting the next iteration.

[0004] Sqoop (pronounced skup) is an open-source tool primarily used for transferring data between Hadoop (Hive) and traditional relational databases (such as MySQL, Oracle, Postgres, etc.). It can import data from a relational database into Hadoop Distributed File System (HDFS) and vice versa. It also provides connectors for some NoSQL databases. Similar to other ETL (Extract-Transform-Load) tools, Sqoop is designed for large-scale data transfer, capable of partitioning datasets and creating parallel Hadoop tasks to process each partition.

[0005] Flume is a distributed log collection, aggregation, and transmission system that supports customizing various data senders within the log system for data collection. Flume also provides the ability to perform simple data processing and write data to various customizable data receivers.

[0006] Kafka is a high-throughput distributed publish-subscribe messaging system with built-in storage, capable of handling all action streams from consumers on a website. These actions (web browsing, searching, and other user actions) are a key element of many social functions on the modern web. This data is typically handled through log processing and log aggregation due to throughput requirements. Therefore, in most log collection scenarios, Kafka and Flume are used together, solving both the flexibility of data collection and meeting the requirements of high throughput and high reliability.

[0007] The shortcomings of existing technologies: Currently, specific technologies or components are used to adapt and collect data for different data types. For web data, web crawling is commonly used, with dozens to hundreds of existing web crawling software programs, each with its own characteristics: some excel at crawling HTML pages, some at crawling JSON pages, some at horizontal breadth crawling, some at vertical depth crawling, and so on. For structured data, data integration components such as Sqoop are used to transfer and exchange data between traditional databases and the HDFS file system. For log-type real-time data, Flume + Kafka is used for data extraction, processing, and forwarding. Developers develop and configure different tools and components each time based on different data scenarios and business needs. When requirements change and the page structure changes, these tools must be replaced, resulting in heavy and repetitive work. Currently, there is no system that integrates web crawlers, Sqoop, Flume, and Kafka as components into a single system for coordinated management to adapt to the data collection needs of various data types and achieve a one-stop collection solution.

[0008] The technical problem this invention aims to solve is to provide users with a real-time and offline data acquisition system that integrates various web crawler components, Sqoop components, Flume, and Kafka components. This system features an open architecture, allowing easy integration with existing web crawler tools for web data scraping, and also utilizes tools such as Sqoop, Flume, and Kafka to collect and process offline database data and real-time log data. The system manages these components uniformly through task configuration, including starting, pausing, continuing, and terminating functions. It dynamically selects specific acquisition components based on the data type, achieving a "one-stop" data acquisition and processing function. Developers only need to spend a small initial investment of time integrating the necessary tools into the system to easily acquire multi-source heterogeneous data.

[0009] 2. Detailed description of the technical solution of the present invention

[0010] 2.1 Technical Field of the Invention

[0011] This invention relates to the field of big data data acquisition. In order to solve the technical problems of existing technologies, such as inflexibility in adapting to changes in data scenarios and business needs, inconvenient management, low efficiency, high application threshold and high maintenance difficulty, this invention proposes a one-stop acquisition system and method for multi-source heterogeneous data.

[0012] 2.2 Complete Technical Solution Provided by the Invention Summary of the Invention

[0013] The purpose of this invention is to solve the problem of collecting multi-source heterogeneous data. It integrates various web crawler components for collecting network data, Sqoop for collecting structured data, and Flume+Kafka for collecting and processing real-time log data. Through unified system coordination and scheduling, it achieves a "one-stop" collection and processing function for multi-source heterogeneous data. Specific functions are as follows:

[0014] (1) Management of data collection components, including dynamic addition, configuration, activation, deactivation and deletion of different crawler components, to meet the data collection function of various websites.

[0015] (2) Task management, including operations such as adding, querying, modifying and deleting data collection tasks.

[0016] (3) Task synchronization: The system database is responsible for storing all user-created tasks, including those that have not been executed, those that have been executed, and those that are currently being executed. The task queue stores those that have not been executed and those that are currently being executed. Every time the system starts, the corresponding tasks need to be synchronized from the database to the queue.

[0017] (4) Task start, stop, pause, and continue functions.

[0018] (5) View task execution history and task instances.

[0019] (6) Network data collection function, including but not limited to the collection and download of weather data, online shopping data and images.

[0020] (7) Batch acquisition of structured data, supporting data acquisition from various relational databases (such as MySQL and Oracle).

[0021] HDFS migration.

[0022] (8) Local directory monitoring: Monitor files in a specified local directory. When files (such as documents, videos, audio, images, etc.) exist in the directory, they are automatically uploaded to HDFS.

[0023] (9) Real-time data collection, supporting real-time collection of log data.

[0024] The system has the following characteristics:

[0025] (1) The front-end is developed using Vue language, and the back-end is developed using Java language.

[0026] (2) The RESTful interface is used to achieve front-end and back-end separation and system decoupling.

[0027] (3) The framework is open and scalable, making it convenient to integrate and manage different types of acquisition components.

[0028] (4) It also supports “one-stop” collection of multi-source heterogeneous data such as network data, database table data and log data.

[0029] (5) It supports the collection of both real-time and offline data.

[0030] (6) The system has certain cleaning and conversion processing functions.

[0031] (7) The system adopts a distributed architecture and has horizontal scaling capability.

[0032] (8) The system has high throughput and high availability.

[0033] (9) The system has load balancing capabilities, which solves the problem of data loss caused by network bottlenecks.

[0034] (10) The system has the function of transmitting data to different targets, such as HDFS, HBase, Hive, etc. Detailed Implementation

[0035] The detailed embodiments of the present invention are described below with reference to the accompanying drawings.

[0036] Figure 1This is the system architecture diagram of the present invention. The present invention mainly targets network data, real-time log data, and offline database table data, proposing a "one-stop" collection and processing system and method. As shown in the diagram, the system integrates various crawler components (crawler component 1, crawler component 2... crawler component n) to solve network data collection, integrates Flume+Kafka components to solve real-time log data collection, and integrates Sqoop components to solve the need for uploading structured data to HDFS. Each component is uniformly managed and scheduled by the system's task management module, providing data input to the upper layer through a unified interface. According to business needs, data can be cleaned, filtered, and format converted, and finally stored in HDFS, Hive, or HBase big data storage systems, further providing necessary data support for upper-layer businesses. In addition, the task management module, which performs unified scheduling and management, communicates with the front-end system through a RESTful interface, receives collection tasks from the front-end, pushes the tasks to be executed into a memory queue, and begins the actual collection work when the user issues a command to start collection. The system first parses the task parameters and distributes them to different acquisition services according to the specific task type. It records and updates the current task status based on the reports from the specific acquisition services, continuously monitoring until the user terminates all acquisition operations. The system also has the function of monitoring specified directories, allowing the transfer of acquired resource data (videos, documents, images, etc.) to a designated target system at any time.

[0037] Figure 2 This is a schematic diagram of the data flow of the present invention. From a data perspective, the system consists of a data input module, a data acquisition module, an interface adaptation module, a data processing module, a data storage module, and a data output module.

[0038] Figure 3 This is a data processing flowchart of the present invention. Log data (LogData) is proxied to different Flume servers via the Nginx reverse proxy module. After Flume collects the data, it is transferred to a Kafka cluster to achieve data smoothing. The data output from Kafka is then consumed again by Flume and finally stored in a backend storage system, such as HDFS or Hive. Web data (WebData) is crawled and processed by different crawler components according to the web page type. The crawled data is still input into the Kafka cluster and Flume for consumption and processing. Database data (DataBase) is directly transferred to the subsequent storage system offline via the Sqoop component.

[0039] Figure 4 This is the unified task scheduling diagram of the present invention. Specific execution steps:

[0040] S1: Start the service.

[0041] S2: Load unexecuted tasks from the task database into the task queue.

[0042] S3: Determine if there are any tasks to be executed in the task queue, i.e., whether it is empty.

[0043] S4: Repeat S2 to S3 when there are no tasks in the queue.

[0044] S5: When there are tasks in the queue, retrieve one task.

[0045] S6: Read task parameters and parse task type.

[0046] S7: Determine the task type.

[0047] S8: When the task type is a log data collection task, the task is sent to the Flume server for Flume to collect log data, and the task status is updated to "running".

[0048] S9: When the task type is a web page data collection task, the task is sent to the crawler server for data crawling and processing, and the task status is updated to "running".

[0049] S10: When the task type is a database table data task, the task is sent to the Sqoop server to migrate the database table data, and the task status is updated to "running".

[0050] S11: Repeat S2 to S10.

[0051] The data collection tasks are executed separately on the Flume server, crawler server, and Sqoop server. The execution flow on the Flume server is as follows:

[0052] S8-1: Perform task parameter parsing.

[0053] S8-2: Start the Flume service.

[0054] S8-3: Update Flume configuration file.

[0055] S8-4: Perform data acquisition and processing.

[0056] S8-5: Store according to configuration.

[0057] S8-6: The task is completed, and the task status is updated to "executed".

[0058] The execution flow on the crawler server is as follows:

[0059] S9-1: Perform task parameter parsing.

[0060] S9-2: Selection of web crawler components.

[0061] S9-3: Start the corresponding crawler component.

[0062] S9-4: Perform data crawling and processing.

[0063] S9-5: Store according to configuration.

[0064] S9-6: The task is completed, and the task status is updated to "executed".

[0065] The execution flow on the Sqoop server is as follows:

[0066] S10-1: Perform task parameter parsing.

[0067] S10-2: Start the Sqoop service.

[0068] S10-3: Perform data migration processing.

[0069] S10-4: Store according to configuration.

[0070] S10-5: The task is completed, and the task status is updated to "executed". Attached Figure Description

[0071] Figure 1 Software system architecture diagram of the present invention

[0072] Figure 2 : A schematic diagram of the data flow of this invention

[0073] Figure 3 Flowchart of the data processing of this invention

[0074] Figure 4 The unified task scheduling diagram of the present invention

[0075] 2.4 Beneficial effects that will result from the implementation of the technical solution of this invention

[0076] This invention proposes a one-stop data acquisition system and method for multi-source heterogeneous data. It achieves flexible integration, automatic selection, unified scheduling, and management of various components for different data acquisition tasks (log data, web page data, and database table data), enabling users to complete the acquisition of required business data in a "one-stop" manner. This effectively avoids the heavy repetitive work of developers configuring and developing different components and other acquisition systems, improving development and usage efficiency.

[0077] 3. Regarding the technical solution of this invention, are there any other alternative solutions that can also achieve the purpose of the invention?

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0079] 4. Key technical points and points to be protected in this invention

[0080] The key technology of this invention adopts a unified task approach to transparently and uniformly manage Sqoop, web crawler, and Flume+Kafka components. It combines a dual storage method of database and memory queue, enabling efficient real-time monitoring of the task queue while also providing a system "memory" function for tasks. The points to be protected by this invention are as follows:

[0081] (1) Effective integration and unified management of crawler components and Sqoop tools.

[0082] (2) Effective integration and unified management of crawler components and Flume+Kafka tools.

[0083] (3) The RESTful API call method enables task transfer with the front end.

[0084] (4) Management of real-time and offline tasks, including adding, deleting, modifying and querying.

[0085] (5) The system monitors the specified local directory and asynchronously uploads newly added files in the directory.

[0086] (6) Dual storage and synchronization technology of database and memory queue.

[0087] (7) Kafka's data smoothing and caching techniques.

Claims

1. A one-stop acquisition system for multi-source heterogeneous data, characterized in that, include: (1) Data source adaptation layer, used to connect to network data sources, structured databases and real-time log data sources; (2) Component integration layer, which integrates various web crawler components, Sqoop components, Flume components and Kafka components; (3) Task management module, which provides a unified task configuration interface and supports the start, pause, continue and terminate operations of data collection tasks; (4) Scheduling engine, which dynamically selects and adapts components and coordinates resource allocation based on the data source type; (5) Data preprocessing module, which performs format conversion and metadata annotation on the collected heterogeneous data; (6) Storage interface layer, which supports outputting processed data to HDFS, relational databases and message queues.

2. The system according to claim 1, characterized in that, The web crawler component supports plug-in extensions, including HTML parsing plugins, JSON parsing plugins, and dynamic rendering plugins.

3. The system according to claim 1, characterized in that, The task management module provides a visual configuration interface, supporting parameterized configuration of data sources, setting of collection frequency, and definition of abnormal alarm rules.

4. The system according to claim 1, characterized in that, The scheduling engine uses a priority queue mechanism to allocate higher resource weights to real-time acquisition tasks than to offline tasks.

5. A data acquisition method for the system as described in any one of claims 1-4, characterized in that, Includes the following steps: (1) S1: Receive the data source type, collection parameters and target storage path through the task configuration interface; (2) S2: Automatically activate the corresponding acquisition component based on the data source type; (3) S3: Perform incremental extraction strategy on structured data and content feature extraction on unstructured data; (4) S4: Distribute the standardized data stream to the specified storage system through the storage interface layer.

Images