Data acquisition method, device and system and medium

Abstract

The invention discloses a data acquisition method, device and system and a medium, and the method comprises the steps: carrying out the architecture analysis of a source end database, and obtaining the architecture information of the source end database; creating a corresponding number of data acquisition engines according to the architecture information of the source end database; performing multi-thread parallel data acquisition on the source end database through the data acquisition engine to obtain corresponding acquisition data; and sending the collected data to a specified target data bin. A corresponding number of data acquisition engines are created by analyzing architecture information of a source end database, so that data acquisition requirements of databases of different architecture types are met, the compatibility of data acquisition is improved, multi-thread parallel data acquisition is performed through the created data acquisition engines, and the data acquisition efficiency is improved. The multi-thread cooperation improves the data acquisition performance and better avoids the resource waste at the same time.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data acquisition method, device, system and medium. Background Art

[0002] With the development of database technology, various databases have emerged, so database replacement is becoming more and more common. Database replacement requires data migration, which is to migrate data from one database to another in full or incremental manner. Data migration needs to meet high availability, security, and consistency.

[0003] Data collection is the first step in data migration, and its performance plays an important role in the data migration process. Currently, there are many data migration methods, but they are usually only targeted at a specific data source and lack performance improvement in the data collection stage, resulting in poor compatibility and performance of data collection, which in turn affects data migration performance. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, an object of the present invention is to provide a data collection method, device, system and medium that can be applied to financial technology or other related fields, aiming to improve the data source compatibility and collection performance of data collection.

[0005] The technical solution of the present invention is as follows:

[0006] A data collection method, comprising:

[0007] Performing architecture analysis on the source database to obtain architecture information of the source database;

[0008] Creating a corresponding number of data collection engines according to the architecture information of the source database;

[0009] Performing multi-threaded parallel data collection on the source database through the data collection engine to obtain corresponding collected data;

[0010] The collected data is sent to a designated target data warehouse.

[0011] In one embodiment, performing architecture analysis on the source database to obtain architecture information of the source database specifically includes:

[0012] Perform an architecture analysis on the source database to confirm whether the source database is a centralized database or a distributed database, and the number of data nodes included in the distributed database.

[0013] In one embodiment, creating a corresponding number of data acquisition engines according to the architecture information of the source database specifically includes:

[0014] When the source database is a centralized database, creating a data collection engine;

[0015] When the source database is a distributed database, a data acquisition engine is created for each data node according to the number of the data nodes.

[0016] In one embodiment, the data acquisition engine performs multi-threaded parallel data acquisition on the source database to obtain corresponding acquired data, including:

[0017] Acquire table information to be collected from the source database, and generate a queue of tables to be collected according to the table information to be collected;

[0018] Creating a plurality of table splitting threads, extracting tables from the table column pairs to be collected and performing segment processing on the tables through the plurality of table splitting threads, generating a plurality of table segment queues, wherein each segment in the table segment queue is encapsulated as a data collection task;

[0019] A plurality of data collection threads are created, and the data collection tasks in the table segment queue are executed in parallel by the plurality of data collection threads according to a preset thread cooperation strategy to obtain corresponding collection data.

[0020] In one embodiment, the acquiring the table information to be collected from the source database and generating the table queue to be collected according to the table information to be collected specifically includes:

[0021] The table information to be collected is obtained from the source database, and the table information to be collected is filtered through a preset black and white list to obtain the table to be collected and generate a corresponding queue of tables to be collected.

[0022] In one embodiment, the creating of multiple data collection threads, and executing the data collection tasks in the table segment queue in parallel by the multiple data collection threads according to a preset thread cooperation strategy to obtain corresponding collection data includes:

[0023] Create multiple data collection threads and bind a table segment queue to each data collection thread;

[0024] Each data collection thread first checks whether there is a data collection task in the currently bound table segment queue according to the binding relationship;

[0025] If there is a data collection task in the currently bound table segment queue, the task is obtained from the currently bound table segment queue and executed;

[0026] If there is no data collection task in the currently bound table segment queue, a queue to be coordinated is determined in all table segment queues, and tasks are obtained and executed from the queue to be coordinated to obtain corresponding collected data.

[0027] In one embodiment, the number of table splitting threads, the number of table segmentation queues, and the number of data collection threads are all the same.

[0028] A data acquisition device, comprising:

[0029] An analysis module is used to perform architecture analysis on a source database to obtain architecture information of the source database;

[0030] A creation module, used to create a corresponding number of data acquisition engines according to the architecture information of the source database;

[0031] A data collection module, used to perform multi-threaded parallel data collection on the source database through the data collection engine to obtain corresponding collected data;

[0032] A data sending module is used to send the collected data to a specified target data warehouse. A data collection system includes at least one processor; and

[0033] a memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned data collection method.

[0035] A non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned data acquisition method.

[0036] Beneficial effects: The present invention discloses a data collection method, device, system and medium. Compared with the prior art, the embodiments of the present invention create a corresponding number of data collection engines by analyzing the architecture information of the source database to meet the data collection requirements of databases of different architecture types, improve the compatibility of data collection, and perform multi-threaded parallel data collection through the created data collection engine. The multi-threaded collaboration improves the data collection performance while also better avoiding resource waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0038] Figure 1A flow chart of a data collection method provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the functional modules of a data acquisition device provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the hardware structure of a data acquisition system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0042] like Figure 1 As shown, the data collection method provided in this embodiment includes the following steps:

[0043] S101: Perform architecture analysis on a source database to obtain architecture information of the source database.

[0044] In this embodiment, when data migration is required for the source database, it is desired to perform an architecture analysis on the source database before data collection, and obtain the architecture information of the source database to determine its architecture characteristics, such as analyzing and obtaining the database type (centralized or distributed), version, supported parallel operations, table structure, index, partition and other information, so as to ensure that a fully compatible data collection strategy is provided for the current source database to improve the compatibility of data collection.

[0045] S102: Create a corresponding number of data acquisition engines according to the architecture information of the source database.

[0046] In this embodiment, based on the architecture information of the source database obtained, a corresponding number of data acquisition engines are created, and the data acquisition engines can independently and parallelly extract data from the database to complete data acquisition. For example, for a centralized or distributed source database, a distributed database can be used as multiple data sources, while a centralized one can be used as a single data source. Therefore, a single or multiple data acquisition engines can be created based on the architecture information, so that data acquisition from a single or multiple data sources can be performed according to the scale and architecture of the source database, and resources can be adjusted flexibly and dynamically, so that the data acquisition system can adapt to different database architectures, and enhance the versatility and flexibility of data acquisition.

[0047] S103: Perform multi-threaded parallel data collection on the source database through the data collection engine to obtain corresponding collected data.

[0048] In this embodiment, after creating a corresponding data collection engine according to the actual architecture of the source database, data is collected from tables in the source database through a single or multiple data collection engines. During the data collection process, the data collection engine creates multiple threads to perform parallel collection between and within tables in the source database, that is, multiple threads collect data from a table at the same time, improving the performance of single-table collection, and multiple tables are collected in parallel to avoid the situation where one table abnormality causes multi-threaded work abnormality when all threads collect the same table. Performing multi-threaded parallel collection through the data collection engine can greatly improve the throughput of data collection and make full use of resources, thereby improving the performance and efficiency of data collection.

[0049] S104: Send the collected data to a designated target data warehouse.

[0050] In this embodiment, after data collection is completed, the collected data is sent to the target data warehouse, which can be a file, middleware or target database, etc. This embodiment does not limit this. By analyzing the architecture information of the source database to create a corresponding number of data collection engines, it can be applied to database collection needs of various sizes and types, improve the compatibility of data collection, and perform multi-threaded parallel data collection through the created data collection engine. Multi-threaded collaboration improves data collection performance while also better avoiding resource waste.

[0051] In one embodiment, step S101 specifically includes:

[0052] Perform an architecture analysis on the source database to confirm whether the source database is a centralized database or a distributed database, and the number of data nodes included in the distributed database.

[0053] In this embodiment, when performing architecture analysis on the source database, the type of database is specifically determined, that is, the source database is a centralized database such as MySQL, ORACLE, DB2, etc., which usually has a central node to store all data, or a distributed database such as GoldenDB, etc., whose data is stored on multiple nodes, improving the availability and scalability of the data. And when the source database is a distributed database, it is also necessary to obtain and analyze the number of data nodes contained in the distributed database to provide a flexible basis for subsequent data acquisition resource configuration.

[0054] In one embodiment, step S102 specifically includes:

[0055] When the source database is a centralized database, creating a data collection engine;

[0056] When the source database is a distributed database, a data acquisition engine is created for each data node according to the number of the data nodes.

[0057] In this embodiment, when creating a data acquisition engine, if the source database is a centralized database, single-source data acquisition is performed centrally, and there is only one data node. Therefore, only one data acquisition engine needs to be created for data acquisition, and all acquisition resources are concentrated on one engine, which can optimize resource utilization and improve acquisition efficiency.

[0058] If the source database is a distributed database, multi-source data collection is performed in a distributed manner, which includes multiple data nodes. A corresponding data collection engine is created for each data node to perform data collection, so that data collection resources can be reasonably allocated according to the number of data nodes, ensuring that the data collection load is evenly distributed among all nodes to avoid single-point overload, and each data collection engine can work in parallel, significantly improving the overall data collection speed and efficiency. Preferably, when the source database is a distributed database, as the data nodes increase or decrease, the number of data collection engines can be flexibly increased or decreased, that is, the number of data nodes is monitored and the number of data collection engines is synchronously adjusted to adapt to changing data collection needs.

[0059] In one embodiment, step S103 includes:

[0060] Acquire table information to be collected from the source database, and generate a queue of tables to be collected according to the table information to be collected;

[0061] Creating a plurality of table splitting threads, extracting tables from the table column pairs to be collected and performing segment processing on the tables through the plurality of table splitting threads, generating a plurality of table segment queues, wherein each segment in the table segment queue is encapsulated as a data collection task;

[0062] A plurality of data collection threads are created, and the data collection tasks in the table segment queue are executed in parallel by the plurality of data collection threads according to a preset thread cooperation strategy to obtain corresponding collection data.

[0063] In this embodiment, the created data acquisition engine performs a multi-threaded parallel data acquisition process for the corresponding data nodes in the source database. The data acquisition engine adds two intermediate cache queues (queue for tables to be collected and table segmentation queues) and two threads (table splitting thread and data acquisition thread) during the acquisition process, splitting a table data acquisition process into three actions, namely table information acquisition, table splitting, and table data acquisition, thereby realizing multi-threaded parallel acquisition and improving the acquisition performance as much as possible. Among them, the number of table splitting threads, the number of table segmentation queues, and the number of data acquisition threads are the same, and the three correspond one to one, that is, one table splitting thread corresponds to one table segmentation queue and one data acquisition thread. The specific number can be flexibly set according to the data acquisition requirements, and this embodiment does not limit this.

[0064] Specifically, first query the source database to obtain the table information to be collected, which may include, for example, table name, table structure, index, etc., filter the table information to be collected through the preset black and white lists, so as to obtain the table to be collected, and put the table to be collected into the table queue of the intermediate cache to form a queue of tables to be collected. As a queue waiting to be processed, the collection table queue contains references or identifiers of all tables to be collected, so as to clarify the target and scope of data collection, making the data collection process more efficient and orderly.

[0065] By creating multiple table splitting threads, tables are extracted from the queue of tables to be collected and the extracted tables are segmented. Specifically, segmentation can be performed according to identifiers such as primary keys and indexes, so that a table is divided into multiple segments, and each segment is encapsulated as a data collection task and placed in the table segment queue. After a table is segmented, tables continue to be taken from the table queue until all tables are taken. Multiple table segment queues are generated, and each table segment queue contains data collection tasks corresponding to the segments of each table. Splitting a large table into multiple small segments can control the data collection process in a finer granularity and improve the flexibility of data processing.

[0066] And based on the created multiple data collection threads, the data collection tasks in the table segment queue are executed in parallel according to the preset thread cooperation strategy. The number of data collection threads created is the same as the number of table segment queues. Therefore, the tasks in the corresponding table segment queue can be bound and listened one-to-one. If there is a task, the task is taken to collect the data of the segment, and the corresponding segment data is collected. Alternatively, on this basis, the idle threads can cooperate according to the different numbers of tasks in different table segment queues. For example, the data collection thread that has been executed continues to poll the tasks in the remaining table segment queues. The data collection tasks are executed in parallel through multiple data collection threads, and the thread cooperation strategy is used to ensure the full utilization of threads and system resources, thereby improving the data collection speed while avoiding resource waste.

[0067] In one embodiment, the creating of multiple data collection threads, and executing the data collection tasks in the table segment queue in parallel by the multiple data collection threads according to a preset thread cooperation strategy to obtain corresponding collection data includes:

[0068] Create multiple data collection threads and bind a table segment queue to each data collection thread;

[0069] Each data collection thread first checks whether there is a data collection task in the currently bound table segment queue according to the binding relationship;

[0070] If there is a data collection task in the currently bound table segment queue, the task is obtained from the currently bound table segment queue and executed;

[0071] If there is no data collection task in the currently bound table segment queue, a queue to be coordinated is determined in all table segment queues, and tasks are obtained and executed from the queue to be coordinated to obtain corresponding collected data.

[0072] In this embodiment, multiple data collection threads are created. The number of data collection threads is the same as the number of table segment queues, and the data collection threads correspond to the table segment queues one by one. Therefore, it is possible that the tasks of some table segment queues are completed first, and the corresponding data collection threads may be idle. In order to ensure full utilization of resources, a thread cooperation mechanism is added to ensure resource utilization. Specifically, multiple data collection threads are initialized first to realize parallel data processing, and a table segment queue is assigned (or bound) to each data collection thread. The binding relationship means that each data collection thread is responsible for obtaining data collection tasks from the queue to which it is bound. Each data collection thread is responsible for checking the table segment queue to which it is bound to confirm whether there are data collection tasks to be processed. For example, data collection thread 1 will check table segment queue 1, data collection thread 2 will check table segment queue 2, and so on. It is ensured that the data collection thread can immediately process the tasks in its bound queue, reducing the delay in task allocation.

[0073] If there are data collection tasks in the currently bound table segment queue, the task is obtained from the currently bound table segment queue and executed. For example, data collection thread 1 verifies that there are tasks in table segment queue 1, then the task will be taken out and the data collection process will be started to ensure the timeliness of data collection; if there are no data collection tasks in the currently bound table segment queue, then the queue to be coordinated is determined in all table segment queues. The queue to be coordinated can be, for example, the table segment queue with the largest number of remaining tasks, or the table segment queue with the largest data flow to be collected, etc. For example, data collection thread 1 verifies that there are no tasks in table segment queue 1, and confirms from other table segment queues that the queue with the largest number of tasks is table segment queue 5, which is used as the current queue to be coordinated. Then data collection thread 1 takes tasks from table segment queue 5 and starts the data collection process, trying to avoid idleness of each data collection thread, improving overall resource utilization, and effectively sharing tasks of other queues, reducing the risk of system overload, and improving the risk resistance of data collection.

[0074] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0075] Another embodiment of the present invention provides a data acquisition device, such as Figure 2 As shown, the device 1 comprises:

[0076] A data acquisition device, comprising:

[0077] The analysis module 11 is used to perform architecture analysis on the source database to obtain architecture information of the source database;

[0078] A creation module 12, used to create a corresponding number of data acquisition engines according to the architecture information of the source database;

[0079] The data collection module 13 is used to perform multi-threaded parallel data collection on the source database through the data collection engine to obtain corresponding collected data;

[0080] The data sending module 14 is used to send the collected data to the specified target data warehouse. The module referred to in the present invention refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of data collection than a program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0081] Another embodiment of the present invention provides a data acquisition system, such as Figure 3 As shown, the system 10 includes:

[0082] One or more processors 110 and memory 120, Figure 3 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0083] The processor 110 is used to complete various control logics of the system 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0084] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the data acquisition method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the data acquisition method in the above method embodiment.

[0085] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0086] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:

[0087] Performing architecture analysis on the source database to obtain architecture information of the source database;

[0088] Creating a corresponding number of data collection engines according to the architecture information of the source database;

[0089] Performing multi-threaded parallel data collection on the source database through the data collection engine to obtain corresponding collected data;

[0090] The collected data is sent to a designated target data warehouse.

[0091] In one embodiment, performing architecture analysis on the source database to obtain architecture information of the source database specifically includes:

[0092] Perform an architecture analysis on the source database to confirm whether the source database is a centralized database or a distributed database, and the number of data nodes included in the distributed database.

[0093] In one embodiment, creating a corresponding number of data acquisition engines according to the architecture information of the source database specifically includes:

[0094] When the source database is a centralized database, creating a data collection engine;

[0095] When the source database is a distributed database, a data acquisition engine is created for each data node according to the number of the data nodes.

[0096] In one embodiment, the data acquisition engine performs multi-threaded parallel data acquisition on the source database to obtain corresponding acquired data, including:

[0097] Acquire table information to be collected from the source database, and generate a queue of tables to be collected according to the table information to be collected;

[0098] Creating a plurality of table splitting threads, extracting tables from the table column pairs to be collected and performing segment processing on the tables through the plurality of table splitting threads, generating a plurality of table segment queues, wherein each segment in the table segment queue is encapsulated as a data collection task;

[0099] A plurality of data collection threads are created, and the data collection tasks in the table segment queue are executed in parallel by the plurality of data collection threads according to a preset thread cooperation strategy to obtain corresponding collection data.

[0100] In one embodiment, the acquiring the table information to be collected from the source database and generating the table queue to be collected according to the table information to be collected specifically includes:

[0101] The table information to be collected is obtained from the source database, and the table information to be collected is filtered through a preset black and white list to obtain the table to be collected and generate a corresponding queue of tables to be collected.

[0102] In one embodiment, the creating of multiple data collection threads, and executing the data collection tasks in the table segment queue in parallel by the multiple data collection threads according to a preset thread cooperation strategy to obtain corresponding collection data includes:

[0103] Create multiple data collection threads and bind a table segment queue to each data collection thread;

[0104] Each data collection thread first checks whether there is a data collection task in the currently bound table segment queue according to the binding relationship;

[0105] If there is a data collection task in the currently bound table segment queue, the task is obtained from the currently bound table segment queue and executed;

[0106] If there is no data collection task in the currently bound table segment queue, a queue to be coordinated is determined in all table segment queues, and tasks are obtained and executed from the queue to be coordinated to obtain corresponding collected data.

[0107] In one embodiment, the number of table splitting threads, the number of table segmentation queues, and the number of data collection threads are all the same.

[0108] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the above-described Figure 1 The method comprises steps S101 to S104.

[0109] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0110] In summary, the present invention discloses a data collection method, device, system and medium. The method performs architecture analysis on the source database to obtain the architecture information of the source database; creates a corresponding number of data collection engines according to the architecture information of the source database; performs multi-threaded parallel data collection on the source database through the data collection engine to obtain corresponding collected data; and sends the collected data to a designated target data warehouse. By analyzing the architecture information of the source database to create a corresponding number of data collection engines, the data collection requirements of databases of different architecture types are met, the compatibility of data collection is improved, and multi-threaded parallel data collection is performed through the created data collection engine. Multi-threaded collaboration improves data collection performance while also better avoiding resource waste.

[0111] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0112] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A data collection method, characterized in that: include: Performing architecture analysis on the source database to obtain architecture information of the source database; Creating a corresponding number of data collection engines according to the architecture information of the source database; Performing multi-threaded parallel data collection on the source database through the data collection engine to obtain corresponding collected data; The collected data is sent to a designated target data warehouse.

2. The data collection method according to claim 1, characterized in that: The performing of architecture analysis on the source database to obtain architecture information of the source database specifically includes: Perform an architecture analysis on the source database to confirm whether the source database is a centralized database or a distributed database, and the number of data nodes included in the distributed database.

3. The data collection method according to claim 2, characterized in that: The step of creating a corresponding number of data acquisition engines according to the architecture information of the source database specifically includes: When the source database is a centralized database, creating a data collection engine; When the source database is a distributed database, a data acquisition engine is created for each data node according to the number of the data nodes.

4. The data collection method according to claim 1, characterized in that: The data acquisition engine performs multi-threaded parallel data acquisition on the source database to obtain corresponding acquired data, including: Acquire table information to be collected from the source database, and generate a queue of tables to be collected according to the table information to be collected; Creating a plurality of table splitting threads, extracting tables from the table column pairs to be collected and performing segment processing on the tables through the plurality of table splitting threads, generating a plurality of table segment queues, wherein each segment in the table segment queue is encapsulated as a data collection task; A plurality of data collection threads are created, and the data collection tasks in the table segment queue are executed in parallel by the plurality of data collection threads according to a preset thread cooperation strategy to obtain corresponding collection data.

5. The data collection method according to claim 4, characterized in that: The step of acquiring the table information to be collected from the source database and generating a table queue to be collected according to the table information to be collected specifically includes: The table information to be collected is obtained from the source database, and the table information to be collected is filtered through a preset black and white list to obtain the table to be collected and generate a corresponding queue of tables to be collected.

6. The data collection method according to claim 4, characterized in that: The step of creating a plurality of data collection threads and executing the data collection tasks in the table segment queue in parallel according to a preset thread cooperation strategy through the plurality of data collection threads to obtain corresponding collected data includes: Create multiple data collection threads and bind a table segment queue to each data collection thread; Each data collection thread first checks whether there is a data collection task in the currently bound table segment queue according to the binding relationship; If there is a data collection task in the currently bound table segment queue, the task is obtained from the currently bound table segment queue and executed; If there is no data collection task in the currently bound table segment queue, a queue to be coordinated is determined in all table segment queues, and tasks are obtained and executed from the queue to be coordinated to obtain corresponding collected data.

7. The data collection method according to any one of claims 4 to 6, characterized in that: The number of table splitting threads, the number of table segmentation queues, and the number of data collection threads are all the same.

8. A data acquisition device, characterized in that: include: An analysis module is used to perform architecture analysis on a source database to obtain architecture information of the source database; A creation module, used to create a corresponding number of data acquisition engines according to the architecture information of the source database; A data collection module, used to perform multi-threaded parallel data collection on the source database through the data collection engine to obtain corresponding collected data; The data sending module is used to send the collected data to a specified target data warehouse.

9. A data acquisition system, characterized in that: The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data collection method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the data collection method according to any one of claims 1 to 7.

Images