Implementation method and device of lightweight data acquisition framework

Abstract

The invention discloses an implementation method and device for a lightweight data acquisition framework, and the method comprises the steps: a data reading module configures a plurality of data sources of different types, starts parallel slices to read the data sources of different types, converts the data read by each data source and each slice into a data stream ArrowStreamReader object of Arrow, and registers the data stream ArrowStreamReader object as a DuckDB table; the data cleaning and conversion module executes association of multiple tables and an SQL function conversion function by utilizing an SQL statement of the DuckDB, and registers an SQL association result as a new DuckDB table; and the data writing module writes the data converted by the data cleaning conversion module into any target storage. According to the method and the device, a lightweight data acquisition framework is realized by using DuckDB, efficient data acquisition from various heterogeneous data sources is supported through a configuration mode, unified processing, conversion and cleaning of the data are carried out by using the DuckDB, the processed data can be output to various target storage systems, concurrency control and resource management are optimized at the same time, and the data acquisition efficiency is improved. And high performance and stability of data processing are ensured.

Description

Technical Field

[0001] The present invention relates to the field of data collection and cleaning, and in particular to a method and device for realizing a lightweight data collection framework. Background Art

[0002] 1. Addressing the challenges of data collection and cleaning

[0003] Data collection and cleaning usually involve processing various heterogeneous data sources, processing large amounts of data, executing complex transformation and cleaning logic, etc., which poses a huge challenge to traditional ETL tools. Taking DataX or Seatunnel as an example, both can provide high-performance data collection and conversion processing, and have good single-machine performance, but they do not have the ability to associate data between heterogeneous data sources.

[0004] 2. Pursue lightweight and high performance

[0005] In the data collection and cleaning process, high-performance, low-latency data processing is usually pursued. Traditional large-scale ETL tools are usually too large and inefficient. For example, although Spark or Flink computing engines have good scalability and can link heterogeneous data sources, Spark and Flink have high requirements for host memory and CPU resources, and have a long startup time, which is not suitable for low-latency data processing scenarios.

[0006] DuckDB's high-performance embedded analytical database features enable it to perform well in ETL and data cleaning transformation. Its columnar storage and vectorized execution engine optimize the processing speed of large-scale data sets. Memory optimization and low-latency design simplify the real-time analysis process. DuckDB supports standard SQL, which simplifies the writing of complex queries. Multi-threading and parallel processing further improve efficiency, while built-in query optimization technology intelligently reduces data scanning and speeds up queries. The richness of the plug-in system and API interface allows DuckDB to be easily integrated into various applications for efficient data management. Nevertheless, when using DuckDB, you still need to pay attention to its challenges in parallel processing of large amounts of data and user-friendliness. Summary of the invention

[0007] In view of the above situation, the present invention provides a method and device for implementing a lightweight data acquisition framework, which uses DuckDB to implement a lightweight data acquisition framework. It supports efficient data acquisition from multiple heterogeneous data sources through configuration, uses DuckDB to perform unified data processing, conversion and cleaning, and can output the processed data to multiple target storage systems. At the same time, it optimizes concurrency control and resource management to ensure high performance and stability of data processing.

[0008] To achieve the above purpose, the present invention adopts the following technical solutions:

[0009] In one embodiment of the present invention, a method for implementing a lightweight data acquisition framework is proposed, the method comprising:

[0010] The data reading module configures multiple different types of data sources, enables parallel slicing to read different types of data sources, converts the data read by each slice of each data source into an Arrow data stream ArrowStreamReader object, and registers it as a DuckDB table;

[0011] The data cleaning and conversion module uses DuckDB's SQL statements to perform multi-table association and SQL function conversion functions, and registers the results of SQL association as a new DuckDB table for subsequent SQL association and conversion;

[0012] The data writing module writes the data converted by the data cleaning conversion module into different types of target storage.

[0013] Furthermore, the following process is used to slice data for different types of data sources:

[0014] (1) Relational Database

[0015] Determine the slice expression according to the database type, which is used to convert the value of the slice field into a number;

[0016] Use the slice expression to modulo the number of slices. The modulo value is a value ranging from 0 to the number of slices. Generate a corresponding SQL filter condition list based on different database types.

[0017] Generate a list of SQL statements for the database based on the SQL filtering condition list;

[0018] (2) File type data source

[0019] Read the files that meet the conditions in the directory, and specify the slicing strategy to cut the data by file or by file size;

[0020] When the slice strategy specifies files as the unit, each file is used as a data slice;

[0021] When the slicing strategy specifies to slice data according to file size, a file is sliced ​​into multiple data slices according to the file offset.

[0022] Furthermore, the following steps are used to control the number of concurrent data reads:

[0023] Set a maximum number of concurrent tasks, i.e. the concurrency threshold;

[0024] When the number of data slices to be processed exceeds the preset concurrency threshold, the number of tasks running simultaneously is automatically limited to not exceed the concurrency threshold;

[0025] Tasks that exceed the concurrency threshold will be added to the waiting queue and wait for resources to be released;

[0026] Once a task is completed and resources are released, the tasks in the waiting queue will obtain resources in order and start execution.

[0027] In one embodiment of the present invention, a device for implementing a lightweight data acquisition framework is also proposed, and the device includes:

[0028] The data reading module is used to configure multiple different types of data sources, enable parallel slicing to read different types of data sources, convert the data read by each slice of each data source into an Arrow data stream ArrowStreamReader object, and register it as a DuckDB table;

[0029] The data cleaning and conversion module is used to use DuckDB's SQL statements to perform multi-table association and SQL function conversion functions, and register the results of SQL association as a new DuckDB table for subsequent SQL association and conversion;

[0030] The data writing module is used to write the data converted by the data cleaning conversion module into different types of target storage.

[0031] Furthermore, the following process is used to slice data for different types of data sources:

[0032] (1) Relational Database

[0033] Determine the slice expression according to the database type, which is used to convert the value of the slice field into a number;

[0034] Use the slice expression to modulo the number of slices. The modulo value is a value ranging from 0 to the number of slices. Generate a corresponding SQL filter condition list based on different database types.

[0035] Generate a list of SQL statements for the database based on the SQL filtering condition list;

[0036] (2) File type data source

[0037] Read the files that meet the conditions in the directory, and specify the slicing strategy to cut the data by file or by file size;

[0038] When the slicing strategy specifies that the file is used as the unit, each file is used as a data slice; when the slicing strategy specifies that the data is sliced ​​according to the file size, a file is cut into multiple data slices according to the file offset.

[0039] Furthermore, the following steps are used to control the number of concurrent data reads:

[0040] Set a maximum number of concurrent tasks, i.e. the concurrency threshold;

[0041] When the number of data slices to be processed exceeds the preset concurrency threshold, the number of tasks running simultaneously is automatically limited to not exceed the concurrency threshold;

[0042] Tasks that exceed the concurrency threshold will be added to the waiting queue and wait for resources to be released;

[0043] Once a task is completed and resources are released, the tasks in the waiting queue will obtain resources in order and start execution.

[0044] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, an implementation method of the aforementioned lightweight data acquisition framework is designed.

[0045] In one embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program for executing the implementation method of the lightweight data acquisition framework.

[0046] Beneficial effects:

[0047] 1. The framework of the present invention can process data from different heterogeneous data sources, such as file systems and relational databases, thereby enhancing the flexibility and extensiveness of data collection.

[0048] 2. The present invention realizes efficient data stream processing by uniformly serializing data into ArrowStreamReader objects of Arrow, laying a foundation for subsequent ETL operations.

[0049] 3. The present invention adopts an intuitive configuration method. Users can implement the ETL process of data through configuration rather than writing code, which lowers the technical threshold and improves development efficiency.

[0050] 4. The present invention utilizes the SQL function and multi-table association capability of DuckDB to clean and convert data, thus simplifying the implementation of data processing logic.

[0051] 5. The framework of the present invention has dynamic task scheduling and concurrent number control mechanisms, which effectively balances CPU load and memory usage, avoids resource overload, and ensures the stability of data processing.

[0052] 6. According to the data source type, the framework of the present invention supports flexible data slicing strategies, including file-based or record-based slicing, to adapt to different data processing requirements.

[0053] 7. The present invention configures multiple SQL conversion steps, and the framework supports multi-stage data cleaning and conversion processes, so that complex data processing logic can be implemented.

[0054] 8. The data processed by the present invention can be written into a variety of target storages, such as databases, file systems, message queues, etc., providing flexibility in data output. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flowchart of a method for implementing the lightweight data acquisition framework of the present invention;

[0056] Figure 2 It is a schematic diagram of the structure of the implementation device of the lightweight data acquisition framework of the present invention;

[0057] Figure 3 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0058] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and design the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] Those skilled in the art will appreciate that the embodiments of the present invention may be designed as a device, apparatus, equipment, method or computer program product. Therefore, the present disclosure may be specifically designed in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0060] According to an implementation of the present invention, a method for implementing a lightweight data acquisition framework is proposed. A lightweight data acquisition framework is implemented using DuckDB. Through configuration, it supports efficient data acquisition from multiple heterogeneous data sources, uses DuckDB for unified data processing, conversion and cleaning, and can output the processed data to multiple target storage systems. At the same time, it optimizes concurrency control and resource management to ensure high performance and stability of data processing.

[0061] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0062] Figure 1FIG. 1 is a flow chart of the implementation method of the lightweight data acquisition framework of the present invention. Figure 1 As shown, the method includes:

[0063] 1. The data reading module configures multiple different types of data sources, enables parallel slicing to read different types of data sources, converts the data read by each slice of each data source into an Arrow data stream ArrowStreamReader object, and registers it as a DuckDB table;

[0064] Apache Arrow is an open source, cross-language in-memory data layer for efficiently processing large data sets. Arrow defines an in-memory format that enables data to be efficiently transferred and processed between different systems and languages ​​without the need for format conversion. It supports multiple programming languages, including Java, Python, C++, etc.

[0065] In Java, ArrowStreamReader is a class for reading Arrow format data. It allows developers to read Arrow data from a stream and serialize the data into an in-memory structure for further processing.

[0066] 2. The data cleaning and conversion module uses DuckDB's SQL statements to perform multi-table associations and SQL function conversion functions, and registers the results of SQL associations as new DuckDB tables for subsequent SQL associations and conversions.

[0067] 3. The data writing module writes the data converted by the data cleaning conversion module into different types of target storage.

[0068] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0069] In order to explain the implementation method of the above-mentioned lightweight data acquisition framework more clearly, a specific embodiment is described below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0070] Example:

[0071] 1. Configuration of the acquisition framework

[0072] The collection framework adopts a classic three-module structure: data reading (Source), data cleaning and transformation (Transformer), and data writing (Sink). This design allows users to easily implement the data ETL process through intuitive configuration rather than writing code.

[0073] (1) Data reading Source module

[0074] The data reading Source module can configure multiple data sources without limiting the data source type. The result of each data source reading is converted into an Arrow data stream ArrowStreamReader object. For example, you can configure the file data source and the MySQL database data source at the same time.

[0075] The configuration items for each data source include:

[0076] General configuration: General configuration applies to all data sources, regardless of the data source type.

[0077] Data source exclusive configuration: Data source exclusive configuration is applicable only to specific data source configurations, depending on the data source type. For example, the jdbc url configuration is applicable to relational databases, while the file path is only applicable to file-type data sources.

[0078] Common configuration items include:

[0079] Field mapping: Map the read data to the fields and field data types of the DuckDB table.

[0080] DuckDB mapping table name: used to register the read data as a DuckDB table.

[0081] Parallelism size: Specify the parallelism size to control the maximum number of slices read each time.

[0082] Data source exclusive configuration, taking MySQL as an example, includes:

[0083] URL: The JDBC URL of the database.

[0084] Username: The database user name.

[0085] Password: The password for the database.

[0086] Slice field: Divide the data into multiple data segments according to the value of the slice field and read them separately.

[0087] Number of slices: Specify the number of data slices.

[0088] MySQL query statement or database table.

[0089] Data source specific configuration, taking files as an example, includes:

[0090] File path: The directory where the file is located.

[0091] File name wildcard: Read files whose file names meet the wildcard.

[0092] File slicing strategy: slicing by file or by file size, etc.

[0093] File format: JSON, CSV, Parquet, etc. Parse file content according to the file format.

[0094] Other file configurations: for example, the field separator of CSV, whether to include the header, time data parsing of JSON, etc.

[0095] Configuration example: The following configuration defines two data sources: a file data source and a MySQL data source.

[0096]

[0097]

[0098]

[0099] (2) Configuration of data cleaning transformation

[0100] Use duckdb's SQL and functions to clean and transform data or perform SQL associations, etc. The result of SQL association can be registered as a new DuckDB mapping table name for subsequent SQL conversion and association.

[0101] Data cleaning and transformation Transformer module configuration, as shown below:

[0102]

[0103]

[0104] (3) Data is written into the Sink module

[0105] Used to write data converted by the Transformer module into any target storage such as database, file system, message queue, etc.

[0106] The Sink module uses the corresponding configuration items according to the type of target storage being written.

[0107] Take writing data into MySQL as an example:

[0108] Sink{

[0109] mysql{

[0110] / / Database URL

[0111] url="jdbc:mysql: / / 192.168.118.200:3306 / test"

[0112] / / Database username

[0113] user="test"

[0114] / / Database password

[0115] password="123456"

[0116] / / MySQL library table

[0117] db_table="dame.device"

[0118] / / Write mode: append, overwrite, update

[0119] save_mode="append"

[0120] }

[0121] }

[0122] 2. Collection framework workflow

[0123] The workflow of the acquisition framework, such as Figure 1 As shown, including:

[0124] Data source reading: read file data source or table data of relational database.

[0125] Convert to ArrowStreamReader object: Convert the data read from the data source to an ArrowStreamReader object.

[0126] Register DuckDB table: Register the ArrowStreamReader object as a DuckDB table.

[0127] SQL association conversion: Use DuckDB's SQL to perform multi-table association and data cleaning conversion.

[0128] Write to target: Write the processed data to the target storage.

[0129] (1) Data source reading

[0130] Pain points of using DuckDB:

[0131] First, if the data source has a huge amount of data, the host's memory may not be able to handle it if it is read and loaded into a data stream ArrowStreamReader object at one time, resulting in memory overflow. Therefore, it is necessary to read in batches with multiple concurrent reads to solve the performance and stability issues of reading massive data.

[0132] Second, too much concurrency will lead to excessive CPU load. Therefore, it is necessary to have the ability to control the number of concurrent reads.

[0133] Different data source types have different ways of slicing data. This article uses file data sources and relational databases to describe how to solve the data slicing problem. Other types of data sources can refer to these two data sources for implementation.

[0134] Data slicing process of relational database:

[0135] First, determine the slice expression based on the database type to convert the value of the slice field to a number.

[0136] For example: read the device table of the database, the slice field is device, and the number of slices is 3.

[0137] If it is a MySQL database, the slice expression can be specified as: abs(crc32(deviceid)). The crc32 function converts the deviceid field value to a number, but this number may be negative. To ensure that the value is positive, use abs to convert the number to an absolute value.

[0138] If it is an Oracle database, the slice expression can be specified as: ora_hash(deviceid) to convert the deviceid field value into a number.

[0139] For other types of databases, the idea is similar to determine the slice expression based on the corresponding functions of the database.

[0140] Secondly, generate a SQL filter condition list based on the slice expression and the number of slices.

[0141] Use the slice expression to modulo the number of slices. The modulo value is a value ranging from 0 to the number of slices. According to different database types, a corresponding SQL filter condition list is generated.

[0142] Here, it is assumed that the number of slices is specified as 3. Taking MySQL as an example, the SQL filtering conditions are:

[0143] abs(crc32(deviceid))%3=0;

[0144] abs(crc32(deviceid))%3=1;

[0145] abs(crc32(deviceid))%3=2;

[0146] The above three SQL filter expressions cover all data.

[0147] If it is an Oracle database, the SQL filtering condition is:

[0148] mod(ora_hash(deviceid),3)=0;

[0149] mod(ora_hash(deviceid),3)=1;

[0150] mod(ora_hash(deviceid),3)=2;

[0151] Finally, based on the SQL filtering condition list, a list of SQL statements for the database is generated.

[0152] Generates the SQL query statement of the database based on the read table.

[0153] For example, to generate a MySQL SQL statement:

[0154] select*from device where abs(crc32(deviceid))%3=0;

[0155] select*from device where abs(crc32(deviceid))%3=1;

[0156] select*from device where abs(crc32(deviceid))%3=2;

[0157] Data slicing process for file type data sources:

[0158] Read the files that meet the conditions in the directory, and specify the slicing strategy: cut the data by file or by file size.

[0159] When the slice strategy is specified in files, each file is used as a data slice.

[0160] When the slicing strategy specifies to slice data according to file size, a file is sliced ​​into multiple data slices according to the file offset.

[0161] Data reading concurrency control:

[0162] Define concurrency threshold: Set a maximum number of concurrent tasks, namely the concurrency threshold.

[0163] Dynamic task scheduling: When the number of data slices to be processed exceeds the preset concurrency threshold, the system will automatically limit the number of tasks running simultaneously to no more than the concurrency threshold.

[0164] Task queue management: Tasks that exceed the concurrency threshold will be added to the waiting queue and wait for resources to be released.

[0165] Resource release and task scheduling: Once a task is completed and resources are released, the tasks in the waiting queue will obtain resources in sequence and start execution.

[0166] Concurrency control: In this way, the concurrency of data reading operations is ensured to be maintained within a reasonable and controllable range to avoid resource overload.

[0167] (2) ArrowStreamReader object conversion

[0168] All read data sources are uniformly converted into ArrowStreamReader objects. If the data source enables parallel slice reading of data, the data of each slice is registered as a temporary table of DuckDB, that is, one data source corresponds to multiple DuckDB temporary tables. If parallel slice reading of data is not enabled, there is only one DuckDB temporary table.

[0169] Based on multiple SQL statements generated by the data reading module, data is read in parallel, converted into Arrow data stream ArrowStreamReader objects, and registered as temporary tables of DuckDB respectively.

[0170] When specifying the JDBC URL, you can specify the file path of DUCKDB. When DuckDB reads data, it stores the data in memory by default. If the file path is specified, if the memory is insufficient, the local disk will be used for storage, thus ensuring the stability of data reading.

[0171] If the url is "jdbc:duckdb", it means that the read data is stored in the disk. If it is specified as "jdbc:duckdb: / tmp / my_database", the memory space is insufficient and the data is stored in the / tmp / my_database directory.

[0172] / / Create a root allocator instance for memory management

[0173] var allocator=new RootAllocator();

[0174] / / Create an ArrowStreamReader instance, which should be initialized instead of assigned to null.

[0175] / / Read data from a slice of a data source such as a file or relational database and convert it into an ArrowStreamReader object

[0176] ArrowStreamReader reader=null;

[0177]

[0178] (3) Register Duckdb table

[0179] In the ArrowStreamReader object conversion step, the data of each slice has been registered as a temporary table of duckdb, but the data may be scattered in multiple tables, and centralized data cleaning and conversion cannot be performed. Therefore, here we merge multiple duckdb temporary tables into a duckdb view, and use the view as a duckdb table for subsequent SQL processing. The view name is the DuckDB mapping table name in the general configuration item.

[0180] Create a view example, merge the data of multiple DuckDB temporary tables, and use java jdbc to execute SQL statements.

[0181] create or replace view duckdb_table

[0182] as

[0183] select * from duckdb_table_tmp1

[0184] union all

[0185] select * from duckdb_table_tmp2

[0186] union all

[0187] select * from duckdb_table_tmp3

[0188] (4) SQL association conversion

[0189] SQL Execution: Executes queries using the configured SQL statement using DuckDB's JDBC driver to obtain result data sets.

[0190] Result acquisition: After executing the SQL statement, the system will return the query results, which can be used for further data processing and analysis.

[0191] Result table registration: If the output table configuration item is set, the SQL query results can be registered as a new table in DuckDB.

[0192] Subsequent processing: The SQL results registered as tables can be used for subsequent data processing tasks, including multi-table association, data cleaning and conversion, and other operations.

[0193] (5) Target storage

[0194] After the Transformer module processes and transforms the data, it outputs the resulting data to a variety of target storage systems, including databases, file systems, message queues, etc.

[0195] Data acquisition: The converted data can be accessed through DuckDB's JDBC interface to ensure the queryability and operability of the data.

[0196] Target storage adaptation: Develop customized write logic using Java language for different target storage systems to ensure that data can be written correctly and efficiently.

[0197] Flexible integration: The write logic implemented through Java can be flexibly integrated with various storage systems to meet different data storage requirements.

[0198] Based on the same inventive concept, the present invention also proposes a device for implementing a lightweight data acquisition framework. The implementation of the device can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" used below can be a combination of software and / or hardware designed for a predetermined function. Although the device described in the following embodiments is preferably designed with software, the design of hardware, or a combination of software and hardware is also possible and conceivable.

[0199] Figure 2 Schematic diagram of the structure of the implementation device of the lightweight data acquisition framework of the present invention. Figure 2 As shown, the device comprises:

[0200] The data reading module 101 is used to configure multiple different types of data sources, enable parallel slicing to read different types of data sources, convert the data read by each slice of each data source into an Arrow data stream ArrowStreamReader object, and register it as a DuckDB table.

[0201] Use the following process to slice data for different types of data sources:

[0202] (1) Relational Database

[0203] Determine the slice expression according to the database type, which is used to convert the value of the slice field into a number;

[0204] Use the slice expression to modulo the number of slices. The modulo value is a value ranging from 0 to the number of slices. Generate a corresponding SQL filter condition list based on different database types.

[0205] Generate a list of SQL statements for the database based on the SQL filtering condition list;

[0206] (2) File type data source

[0207] Read the files that meet the conditions in the directory, and specify the slicing strategy to cut the data by file or by file size;

[0208] When the slice strategy specifies files as the unit, each file is used as a data slice;

[0209] When the slicing strategy specifies to slice data according to file size, a file is sliced ​​into multiple data slices according to the file offset.

[0210] Use the following steps to control the number of concurrent data reads:

[0211] Set a maximum number of concurrent tasks, i.e. the concurrency threshold;

[0212] When the number of data slices to be processed exceeds the preset concurrency threshold, the number of tasks running simultaneously is automatically limited to not exceed the concurrency threshold;

[0213] Tasks that exceed the concurrency threshold will be added to the waiting queue and wait for resources to be released;

[0214] Once a task is completed and resources are released, the tasks in the waiting queue will obtain resources in order and start execution.

[0215] The data cleaning and conversion module 102 is used to use the SQL statement of DuckDB to perform multi-table association and SQL function conversion functions, and register the result of SQL association as a new DuckDB table for subsequent SQL association and conversion;

[0216] The data writing module 103 is used to write the data converted by the data cleaning conversion module into different types of target storage.

[0217] It should be noted that although several modules of the implementation device of the lightweight data acquisition framework are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be concretized.

[0218] Based on the above invention concept, Figure 3 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220, and when the processor 220 executes the computer program 230, an implementation method of the aforementioned lightweight data acquisition framework is designed.

[0219] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the implementation method of the aforementioned lightweight data acquisition framework.

[0220] The implementation method and device of the lightweight data acquisition framework proposed in the present invention have the following highlights:

[0221] 1. Compatibility with multiple data sources: The framework can process data from different heterogeneous data sources, such as file systems and relational databases, which enhances the flexibility and breadth of data collection.

[0222] 2. Efficient data stream processing: By uniformly serializing data into ArrowStreamReader objects of Arrow, efficient data stream processing is achieved, laying the foundation for subsequent ETL operations.

[0223] 3. ETL configuration without coding: With an intuitive configuration method, users can implement the ETL process of data through configuration rather than writing code, which lowers the technical threshold and improves development efficiency.

[0224] 4. DuckDB integration: Use DuckDB's SQL functions and multi-table association capabilities to clean and convert data, simplifying the implementation of data processing logic.

[0225] 5. Dynamic concurrency control: The framework has dynamic task scheduling and concurrency control mechanisms, which effectively balances CPU load and memory usage, avoids resource overload, and ensures the stability of data processing.

[0226] 6. Flexible data slicing strategies: Depending on the data source type, the framework supports flexible data slicing strategies, including file-based or record-based slicing, to adapt to different data processing requirements.

[0227] 7. Multi-stage data processing: By configuring multiple SQL transformation steps, the framework supports multi-stage data cleaning and transformation processes, enabling complex data processing logic to be implemented.

[0228] 8. Diversity of target storage: The processed data can be written to a variety of target storages, such as databases, file systems, message queues, etc., providing flexibility in data output.

[0229] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0230] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A method for implementing a lightweight data acquisition framework, characterized in that: The method includes: The data reading module configures multiple different types of data sources, enables parallel slicing to read different types of data sources, converts the data read by each slice of each data source into an Arrow data stream ArrowStreamReader object, and registers it as a DuckDB table; The data cleaning and conversion module uses DuckDB's SQL statements to perform multi-table association and SQL function conversion functions, and registers the results of SQL association as a new DuckDB table for subsequent SQL association and conversion; The data writing module writes the data converted by the data cleaning conversion module into different types of target storage.

2. The method for implementing the lightweight data acquisition framework according to claim 1, characterized in that: Use the following process to slice data for different types of data sources: (1) Relational Database Determine the slice expression according to the database type, which is used to convert the value of the slice field into a number; Use the slice expression to modulo the number of slices. The modulo value is a value ranging from 0 to the number of slices. Generate a corresponding SQL filter condition list based on different database types. Generate a list of SQL statements for the database based on the SQL filtering condition list; (2) File type data source Read the files that meet the conditions in the directory, and specify the slicing strategy to cut the data by file or by file size; When the slice strategy specifies files as the unit, each file is used as a data slice; When the slicing strategy specifies to slice data according to file size, a file is sliced ​​into multiple data slices according to the file offset.

3. The method for implementing the lightweight data acquisition framework according to claim 1, characterized in that: Use the following steps to control the number of concurrent data reads: Set a maximum number of concurrent tasks, i.e. the concurrency threshold; When the number of data slices to be processed exceeds the preset concurrency threshold, the number of tasks running simultaneously is automatically limited to not exceed the concurrency threshold; Tasks that exceed the concurrency threshold will be added to the waiting queue and wait for resources to be released; Once a task is completed and resources are released, the tasks in the waiting queue will obtain resources in order and start execution.

4. A device for implementing a lightweight data acquisition framework, characterized in that: The device includes: The data reading module is used to configure multiple different types of data sources, enable parallel slicing to read different types of data sources, convert the data read by each slice of each data source into an Arrow data stream ArrowStreamReader object, and register it as a DuckDB table; The data cleaning and conversion module is used to use DuckDB's SQL statements to perform multi-table association and SQL function conversion functions, and register the results of SQL association as a new DuckDB table for subsequent SQL association and conversion; The data writing module is used to write the data converted by the data cleaning conversion module into different types of target storage.

5. The device for implementing the lightweight data acquisition framework according to claim 4, characterized in that: Use the following process to slice data for different types of data sources: (1) Relational Database Determine the slice expression according to the database type, which is used to convert the value of the slice field into a number; Use the slice expression to modulo the number of slices. The modulo value is a value ranging from 0 to the number of slices. Generate a corresponding SQL filter condition list based on different database types. Generate a list of SQL statements for the database based on the SQL filtering condition list; (2) File type data source Read the files that meet the conditions in the directory, and specify the slicing strategy to cut the data by file or by file size; When the slice strategy specifies files as the unit, each file is used as a data slice; When the slicing strategy specifies to slice data according to file size, a file is sliced ​​into multiple data slices according to the file offset.

6. The device for implementing the lightweight data acquisition framework according to claim 4, characterized in that: Use the following steps to control the number of concurrent data reads: Set a maximum number of concurrent tasks, i.e. the concurrency threshold; When the number of data slices to be processed exceeds the preset concurrency threshold, the number of tasks running simultaneously is automatically limited to not exceed the concurrency threshold; Tasks that exceed the concurrency threshold will be added to the waiting queue and wait for resources to be released; once a task is completed and resources are released, the tasks in the waiting queue will obtain resources in sequence and start execution.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 3.

Images