Multi-thread data cache read-write parallel high-performance data extraction method and system

Abstract

The application belongs to the technical field of data extraction, and discloses a multi-thread data cache read-write parallel high-performance data extraction method and system. The method comprises the following steps: establishing an extraction plan, and establishing one or more extraction tasks for each extraction plan; setting each extraction task in the extraction plan as pre-extraction execution or post-extraction execution; triggering execution of an established extraction plan; calculating the maximum number of parallel tasks according to the number of CPU cores, starting the extraction tasks in the triggered extraction plan based on the maximum number of parallel tasks; monitoring the started extraction tasks, if the number of continuous waiting times of the read threads in the extraction tasks for writing into the cache is greater than or equal to the waiting threshold, and the number of idle CPU cores is not zero, starting a new write thread of the extraction task, clearing the number of continuous waiting times of the read threads for writing into the cache, and continuously monitoring until the extraction task is completed. The application realizes high-performance, low-cost, centralized and easy-to-deploy data extraction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data extraction, and particularly relates to a multi-thread data cache read-write parallel high-performance data extraction method and system. BACKGROUND

[0002] In the face of massive data resources, how to efficiently and accurately extract valuable information has become a problem to be solved in the process of current informatization moving towards dataization; enterprises and institutions often have multiple systems and different database products; the current mainstream data extraction tools often have the disadvantages of low efficiency, huge resource investment, etc., and cannot meet the timeliness and economy requirements of modern enterprises and governments on data processing.

[0003] System running cost, read-write synchronization speed, ease of use, support of different databases, task intervention, and monitoring are important factors for selecting data synchronization software. The current software technology data synchronization mode mainly includes the following five modes:

[0004] Database characteristic mode: through database characteristic connection, data is read through Select and data is written through Insert, such as data federation (Db2), DBlink (Oracle, Dream, PostgreSql, Mysql), etc. This mode has fast speed and low cost, but it does not support the convergence of complex and various heterogeneous databases, internal read-write serialization, and intervention to improve large synchronization tasks.

[0005] Multi-task serial mode: multi-thread read-after-write form, read while waiting to write, write while waiting to read. This mode has simple system logic and low cost. It is mainly used in government department data centers for data development and convergence. However, the multi-task serial mode has low server resource utilization, slow speed, and no means to intervene to improve large synchronization tasks.

[0006] Multi-task parallel mode: multi-thread read-write parallel is adopted, read threads write data into memory, and read services continuously read. Write threads take data from memory, and write threads continuously write, such as IBM DataStage, open source Kettle, and other products. This mode has high resource utilization, fast speed of commercial products, but high cost, and general speed of open source products. The number of parallel tasks cannot be controlled, the memory occupancy rate cannot be controlled, too many threads cause CPU time slice frequent switching, which reduces synchronization speed, and there is no means to intervene to improve large synchronization tasks.

[0007] Distributed consumption mode: service scheduling through applications such as ZooKeeper, multi-machine multi-threaded data reading, one task assigned to one machine, and one read task can set one or more write tasks. Get the read task machine to start the read thread, and the read data is packaged according to a certain number of shards, and the packaged data is sent to the MQ message middleware; get the write task machine to start the write thread, and get a data shard from the MQ message middleware, and write the data to the target library. This mode supports multiple different databases, fast synchronization speed, and limited horizontal expansion of read and write capabilities, but requires multiple server hardware stacking, high investment cost, complex structure, difficult deployment, poor single machine processing capability, scheduling service brain split risk, and no means to intervene to improve large synchronization tasks.

[0008] Streaming synchronization mode: using database log analysis, often through log operation time point dotting, first serially synchronizing inventory data, then using log analysis from the recorded time point, and performing streaming incremental processing, generally CDC log data analysis, using frameworks such as Spark and Flink to analyze logs for streaming processing, such as the Apache SeaTunne system. This mode is mostly used in e-commerce user behavior analysis recommendation systems, aiming to process incremental data, and has slow full data processing, relies on database archive logs, complex structure, difficult deployment, and no means to intervene to improve large synchronization tasks. SUMMARY

[0009] One of the purposes of the present application is to provide a multi-threaded data cache read-write parallel high-performance data extraction method, which realizes high-performance, low-cost, centralized, and easy-to-deploy implementation of data extraction.

[0010] To achieve the above purpose, the technical solution adopted by the present application is:

[0011] A multi-threaded data cache read-write parallel high-performance data extraction method, the multi-threaded data cache read-write parallel high-performance data extraction method, comprising:

[0012] Establish an extraction plan, and establish one or more extraction tasks for each extraction plan;

[0013] Each extraction task in the extraction plan is set to be executed before extraction or executed after extraction;

[0014] Trigger an established extraction plan to execute;

[0015] Calculate the maximum number of parallel tasks according to the number of CPU cores, start the extraction tasks in the triggered extraction plan based on the maximum number of parallel tasks, and each extraction task initially starts 1 read thread and N write threads;

[0016] The monitoring of the started extraction task is performed, if the number of continuous waiting of the read thread in the extraction task for the write cache is greater than or equal to a waiting threshold value, and the number of idle CPU cores is not zero, a new write thread of the extraction task is started, the number of continuous waiting of the read thread for the write cache is cleared, and the monitoring is continuously performed until the extraction task is completed.

[0017] The following also provides several optional modes, but not as an additional limitation of the above general scheme, just a further supplement or preferred, without technical or logical contradiction, each optional mode can be combined with the above general scheme alone, but also can be combined between multiple optional modes.

[0018] As preferred, the calculation method of the parallel maximum task number is: parallel maximum task number = (CPU core number - system operation interaction core number) / (1 + N).

[0019] As preferred, the read thread shards the data to be read according to the number of data, judges whether it meets the writing of the data shard into the memory cache, if it meets, caches the data shard, at this time, the total cache shard count and the target table cache shard count are each added by 1, otherwise the read thread waits, and the number of continuous waiting of the read thread for the write cache is added by 1.

[0020] The write thread consumes the data shard in the memory cache, at this time, the total cache shard count and the target table cache shard count are each reduced by 1.

[0021] As preferred, the judging whether it meets the writing of the data shard into the memory cache includes:

[0022] When the total cache shard count does not reach the set maximum total cache shard number DM1, and the target table cache shard count does not reach the set single table maximum cache shard number DM2, it is judged as meeting, otherwise it is judged as not meeting.

[0023] As preferred, the dynamic adjustment mode of the maximum total cache shard number DM1 and the single table maximum cache shard number DM2 is as follows:

[0024] Every interval of a preset time, it is judged whether the proportion of the total cache shard occupying the server memory exceeds a proportion threshold value, if it does not exceed, the maximum total cache shard number DM1 and the single table maximum cache shard number DM2 are not adjusted, otherwise the maximum total cache shard number DM1 and the single table maximum cache shard number DM2 are adjusted as follows:

[0025]

[0026] Single table maximum cache shard number DM2 = single table maximum cache shard number DM2 - NJ

[0027] In the formula, NJ is a preset decrement step.

[0028] As preferred, the multiple data sources further include a data source name, a database type, a database corresponding driver, an address, a port, a username and a password.

[0029] As preferred, the establishing the extraction plan includes establishing an extraction plan name, selecting a source data source and selecting a target data source.

[0030] The establishing the extraction task includes setting a task name, an extraction source data acquisition mode and a target data writing mode.

[0031] As preferred, the calculation method of the idle CPU core number is: idle CPU core number = CPU core number - total number of read threads - total number of write threads - system operation interactive core number.

[0032] The second object of the present application is to provide a multi-thread data cache read-write parallel high-performance data extraction system, which provides a system that is independent of database archiving, fast, low deployment cost, low structural complexity, controllable memory data volume, automatic monitoring and deployment of CPU resources to improve extraction speed.

[0033] To achieve the above object, the technical solution adopted by the present application is:

[0034] A multi-thread data cache read-write parallel high-performance data extraction system, the multi-thread data cache read-write parallel high-performance data extraction system comprising:

[0035] An extraction plan module for establishing an extraction plan and establishing one or more extraction tasks for each extraction plan;

[0036] An extraction processing setting module for setting each extraction task in the extraction plan as pre-extraction execution or post-extraction execution;

[0037] A plan execution module for triggering execution of an established extraction plan;

[0038] A task execution module for calculating the maximum number of parallel tasks according to the CPU core number, starting the extraction tasks in the extraction plan that has been triggered based on the maximum number of parallel tasks, and initially starting one read thread and N write threads for each extraction task;

[0039] A task monitoring module for monitoring the started extraction tasks, starting a new write thread for the extraction task if the number of consecutive read thread waiting for writing cache in the extraction task is greater than or equal to the waiting threshold value and the idle CPU core number is not zero, clearing the number of consecutive read thread waiting for writing cache, and continuously monitoring until the extraction task is completed.

[0040] The present invention provides a multi-threaded data cache read and write parallel high-performance data extraction method and system. In order to improve the extraction speed, the parallel reading and writing threads are used to eliminate the serial waiting time of reading and writing; the reading thread writes data to the memory cache, and the writing thread obtains data from the memory cache, reducing the network consumption of writing and obtaining data; multi-tasking and multi-threading improve CPU resource utilization and reduce the total synchronization time of multiple tasks; the maximum number of parallel tasks is controlled according to the number of CPU cores of the server to ensure that the threads have the corresponding number of CPU cores, eliminating the time consumption and memory loss of CPU time slice switching caused by frequent thread switching. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of a multi-threaded data cache read-write parallel high-performance data extraction method of the present invention;

[0042] Figure 2 Schematic diagram of the task execution process of the present invention;

[0043] Figure 3 This is a schematic diagram of the task monitoring process of the present invention;

[0044] Figure 4 This is a first structural diagram of a multi-threaded data cache read-write parallel high-performance data extraction system of the present invention;

[0045] Figure 5 This is a second structural diagram of a multi-threaded data cache read-write parallel high-performance data extraction system of the present invention;

[0046] Figure 6 Schematic diagram of the extraction speed of the system of the present invention under 5-task parallelism;

[0047] Figure 7 Schematic diagram of the extraction speed of the system of the present invention under 1 task parallelism;

[0048] Figure 8 This is a diagram of the extraction speed of the database migration tool of the domestic database manufacturer Dameng;

[0049] Figure 9 This is a diagram of the extraction speed of Kettler, Hitachi's open source database migration product. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application.

[0052] Embodiment 1

[0053] In order to overcome the low efficiency, large resource investment and other drawbacks of existing data extraction tools, the embodiment proposes a high-performance, low-cost, centralized and easy-to-deploy data extraction method, which is of great significance for improving data processing efficiency, reducing cost and promoting data-driven decision-making.

[0054] As shown in Figure 1 The multi-threaded data cache read-write parallel high-performance data extraction method of the embodiment includes the following steps:

[0055] (1) Establish an extraction plan and one or more extraction tasks for each extraction plan.

[0056] Establishing an extraction plan includes the extraction plan name, selecting the source data source and selecting the target data source. After establishing the extraction plan, one or more extraction tasks are established, which includes setting the extraction source data acquisition method in the source data source and the target data writing method in the target data source and the task name. Specifically, the above-mentioned acquisition method can be understood as the data item required by SQL query, and the writing method can be understood as the target table and field of SQL writing.

[0057] In order to perform the task normally, when the source data volume is greater than the data volume threshold, for example, the source data volume is greater than 100 million, Task Sharding can be set to split a large extraction task into multiple parallel extraction tasks. It should be noted that the extraction tasks mentioned in the subsequent steps include the originally established extraction tasks and the sliced extraction tasks.

[0058] Task Sharding is a process of splitting large tasks or data sets into multiple smaller, independent sub-tasks or data segments. These sub-tasks or data segments can be run independently on different machines to speed up the overall processing speed, improve system throughput and scalability.

[0059] (2) Set each extraction task in the extraction plan to be executed before extraction or executed after extraction, i.e. set the pre-extraction and post-extraction tasks such as emptying data, deleting index, creating index, etc., wherein the post-extraction execution is an operation on the target table written.

[0060] (3) Trigger an established extraction plan execution. Get all extraction tasks in the extraction plan to start synchronous task execution, and further record the state of the extraction plan after triggering, and view the number of extraction tasks, extraction task execution records, etc.

[0061] (4) Calculate the maximum number of parallel tasks according to the number of CPU cores, start the extraction tasks in the extraction plan that have been triggered based on the maximum number of parallel tasks, and each extraction task initially starts 1 read thread and N write threads.

[0062] The number of CPU cores refers to the number of processing units integrated in the central processing unit, and each processing unit is an independent computing engine that can execute instructions and process data. In order to set an effective maximum number of parallel tasks, the maximum number of parallel tasks is associated with the number of CPU cores in this embodiment, and the maximum number of parallel tasks is calculated according to the number of CPU cores to confirm the maximum number of parallel tasks, and the extraction tasks are executed in a queue to synchronize data extraction. The maximum number of parallel tasks started is related to the number of CPU cores. In order to improve the efficiency of task execution, each thread in the extraction task occupies an independent CPU, avoiding waiting caused by frequent switching of CPU time slices between threads, and leaving one CPU for system operation interaction to ensure normal interaction execution. The calculation method of the maximum number of parallel tasks is: maximum number of parallel tasks = (number of CPU cores - number of system operation interaction cores) / (1 + N). Each extraction task requires 1 read thread and N write threads. The number of system operation interaction cores is 1 by default; N is the initial setting value, which is 2 by default.

[0063] CPU time slice is the time period allocated by the operating system to each process or thread for execution when scheduling multiple processes or threads. Multiple processes or threads are usually running in a computer system, and in order to fairly and effectively manage the execution of these processes or threads, the operating system adopts a strategy called "time slice scheduling". When switching from one process or thread to another, the operating system needs to perform context switching. Context switching consumes a certain amount of CPU time, saves the state and data of the switched process or thread, and the CPU time slice of the process / thread is frequently switched, which affects system performance. Therefore, this embodiment sets that each thread occupies an independent CPU, which can guarantee system accessibility, reduce frequent switching of CPU time slices between threads, and improve thread efficiency.

[0064] As Figure 2As shown, within the range of the maximum number of parallel tasks, the parallel task running count is increased by 1 for each extraction task started. Each parallel task starts with 1 read thread and N write threads by default; the data is cached through the singleton singleton mode, wherein the read thread shards the source data to be processed according to the number of data, reads the data shards, and writes the memory cache through the synchronized addList method of the singleton singleton mode controller after the data shards are read, and the total cache shard count and the target table cache shard count are each increased by 1; the write thread consumes the memory cache data through the synchronized getList method of the singleton singleton mode controller and writes it to the target database, at which time the total cache shard count and the target table cache shard count are each decreased by 1. The singleton singleton mode is a common design pattern that ensures that there is only one instance of a class and provides a global access point to obtain the instance.

[0065] When multithreading is enabled, the present embodiment introduces synchronized lock synchronization control of threads, which is an effective method for realizing thread mutual exclusion and synchronization; ensures that thread contention will not produce deadlocks and errors, and is an effective control algorithm for avoiding thread deadlocks. When the system has A, B, and C threads, all of which use method f1, regardless of which thread (for example, thread A), when running to method f1, it must check whether there is another thread B (or C, etc.) using method f1, if there is, A must wait for the thread using f1, such as B (or C), to run the method f1, and then thread A can use method f1; if not, A locks the method f1 and directly runs the method f1.

[0066] The read thread of the extraction task entering parallel processing starts a read thread, and the read thread count is incremented by 1. The read thread judges whether it is allowed to write data shards into the memory cache through the (synchronized) canAdd method of the singletone singleton mode controller. If it is allowed, the data shards are cached through the (synchronized) addList method of the singletone singleton mode controller, and the cached data is increased. Otherwise, the read thread waits for M seconds and then judges whether it is allowed to cache data shards. The method for judging whether it is allowed to write data shards into the memory cache is as follows: when the total cache shard count does not reach the set maximum total cache shard count DM1, and the target table cache shard count does not reach the set single table maximum cache shard count DM2, it is judged that it is allowed. Otherwise, it is judged that it is not allowed. If the read thread finishes reading data, the state of the Task extraction task in which the current thread is located is set to “read complete”, the read thread count is decremented by 1, and the read thread is ended. The read thread records the total amount of read data and the start time of the thread while reading data, and updates the recorded total amount of read data to the Task extraction task in which the current thread is located after submitting the data cache, which is used for monitoring access.

[0067] The write thread of the extraction task entering parallel processing is automatically started. Every time a write thread is started, the write thread count is incremented by 1. The write thread obtains the data shards of the current task through the (synchronized) getList method of the singletone singleton mode controller. If the data shards are obtained, the data shards are written into the target database through the writing mode of the target data set when the task is established. If the data shards are not obtained, it is further judged whether the state of the Task extraction task in which the current thread is located is “read complete” and whether the target table cache shard count is 0. If the state of the Task extraction task in which the current thread is located is “read complete” and the target table cache shard count is 0, the state of the Task extraction task in which the current thread is located is set to “write complete”, the write thread count is decremented by 1, the number of parallel tasks is decremented by 1, and the current write thread is ended. Otherwise, the write thread waits for X seconds (X < M), and then judges whether the data shards of the current task are obtained. The write thread records the total amount of written data and the start time of the thread while writing data, and updates the total amount of written data to the Task extraction task in which the current thread is located after writing the shard data, which is used for monitoring access. In the embodiment, M is 1 and X is 0.2 by default.

[0068] The task queue judges that the number of current parallel tasks is less than the maximum number of parallel tasks, and there is a new extraction task to be executed. A new extraction task is started for data synchronization, and the number of parallel task runs is incremented by 1.

[0069] (5) Monitoring the started extraction task, if the number of consecutive waiting for write cache of read thread in the extraction task is greater than the waiting threshold, and the number of idle CPU cores is not zero, then a new write thread of the extraction task is started, the number of consecutive waiting for write cache of read thread is cleared, and the monitoring is continued until the extraction task is completed.

[0070] As shown in Figure 3 The data monitoring of the running extraction task in the started extraction plan is performed, including the start time, running time, read number, write number, number of data to be written, extraction speed, etc.

[0071] In this embodiment, the writing thread is increased to improve the cache data consumption speed and increase the writing speed by monitoring and evaluating whether it is necessary to increase the writing thread. Specifically, if the read thread waits for M seconds, the number of consecutive waiting for write cache of read thread is increased by 1. If the number of consecutive waiting for write cache of read thread is greater than P times, the processing capacity of the current task writing thread is not enough, the "need_new_wr" attribute in the current task attribute is set to 1, and the number of consecutive waiting for write cache of read thread is set to 0. Usually, P is a multiple of 10, and the default value in this embodiment is 20.

[0072] In the extraction task monitoring stage, the running parallel extraction task is judged in a cycle, for example, the default setting is 1 second. If the task attribute "need_new_wr" is 1, it is judged that the extraction task needs to increase the writing thread. If the number of idle CPU cores is not 0, a new writing thread of the task is automatically started, and the task "need_new_wr" attribute is set to 0. If the number of idle CPU cores is 0, a new writing thread is not started, and the next monitoring is waited for to judge again. At the same time, a corresponding prompt can also be given. In this embodiment, the writing thread is automatically increased according to the resources to eliminate data blocking and increase the synchronization speed. The calculation method of the number of idle CPU cores is: the number of idle CPU cores = the number of CPU cores - the total number of read threads - the total number of writing threads - the number of system operation interactive cores.

[0073] To ensure the stability and efficiency of task running, to realize scientific and reasonable use of cache, and to ensure that the cache occupies the system in a reasonable range in the case of multi-task parallelism, the operation system uses the disk for cache is avoided. In this embodiment, the total number of maximum cache shards DM1 and the maximum cache shard number DM2 of a single table are precisely controlled. The dynamic adjustment method is as follows:

[0074] Every interval of preset time, for example, every 1 minute, it is judged whether the proportion of total cache shard occupying server memory exceeds the proportion threshold. The proportion threshold is set to 70% by default. If it does not exceed, the maximum cache shard total number DM1 and the maximum cache shard number DM2 of a single table are not adjusted. Otherwise, the maximum cache shard total number DM1 and the maximum cache shard number DM2 of a single table are adjusted as follows:

[0075]

[0076] Single table maximum cache shard number DM2 = single table maximum cache shard number DM2-NJ

[0077] In the formula, NJ is a preset decrement step, and the default is 10. When the single table maximum cache shard number DM2 is less than 20, the decrement of the single table maximum cache shard number DM2 is not performed. It should be noted that after the division operation is performed in the embodiment, if the division is not exact, a down rounding operation is performed.

[0078] The embodiment automatically monitors and allocates CPU resources to improve the extraction speed, monitors the speed of each parallel processing task thread, automatically identifies insufficient writing processing tasks, and automatically increases the writing thread according to the number of idle CPU cores to improve the extraction speed. Further, the embodiment further includes configuring a plurality of data sources, the data sources including a data source name, a database type, a database corresponding driver, an address, a port, a username, and a password, and the configured data sources need to be connected for testing. The embodiment supports various relational databases with jdbc drivers, the system manages the data sources through the jdbc drivers, and obtains javaConnection connection through the data sources to perform data query and writing.

[0079] Embodiment 2

[0080] In order to overcome the drawbacks of low efficiency and large resource investment of the existing data extraction tool, the embodiment provides a high-performance data extraction system based on adjustable CPU resources, supports multiple databases, has a simple structure and is easy to deploy, has a low hardware cost, has high-speed synchronization of data memory cache, has multi-task monitoring, and significantly reduces synchronization time.

[0081] As shown in FIG. 1, the multi-threaded data cache read-write parallel high-performance data extraction system of the embodiment includes the following modules: Figure 4

[0082] A, an extraction plan module, for establishing an extraction plan, and establishing one or more extraction tasks for each extraction plan.

[0083] The extraction plan includes an extraction plan name, a selected source data source, and a selected target data source. After the extraction plan is established, one or more extraction tasks are established, and the extraction task includes setting the acquisition mode of the source data in the extraction source data source and the writing mode of the target data in the target data source, and the task name. Specifically, the above acquisition mode can be understood as the data item required by SQL query, and the writing mode can be understood as the target table and field of SQL writing.

[0084] ​In order to perform the task normally, when the source data volume is greater than the data volume threshold, for example, the source data volume is greater than 100 million, the Task Sharding task can be set to split a large extraction task into multiple parallel extraction tasks. It should be noted that the extraction task mentioned in the subsequent steps includes the originally established extraction task and the sliced extraction task.

[0085] Task Sharding is a process of splitting large tasks or data sets into multiple smaller, independent sub-tasks or data segments. These sub-tasks or data segments can be independently run on different machines to speed up the overall processing speed, improve system throughput and scalability.

[0086] B, an extraction processing setting module, configured to set each extraction task in the extraction plan as pre-extraction execution or post-extraction execution, i.e., set the pre-extraction task and the post-extraction task, such as emptying data, deleting index, creating index, etc., wherein the post-extraction execution is an operation on the target table written.

[0087] C, a plan execution module, configured to trigger an established extraction plan to execute. All extraction tasks in the extraction plan are sent to the task execution module to start synchronous task execution, and further can record the state of the extraction plan after the extraction plan is triggered, and view the number of extraction tasks, extraction task execution records, etc.

[0088] D, a task execution module, configured to calculate the maximum number of parallel tasks according to the number of CPU cores, start the extraction tasks in the extraction plan that have been triggered based on the maximum number of parallel tasks, and initially start one read thread and N write threads for each extraction task.

[0089] The number of CPU cores refers to the number of processing units integrated in the central processing unit. Each processing unit is an independent computing engine that can execute instructions and process data. In order to set an effective maximum number of parallel tasks, the maximum number of parallel tasks is associated with the number of CPU cores in this embodiment. The maximum number of parallel tasks is calculated based on the number of CPU cores, and the extraction tasks are executed in a queue to synchronize data extraction. The maximum number of parallel tasks started is related to the number of CPU cores. In order to improve the efficiency of task execution, each thread in the extraction task occupies an independent CPU, avoiding the waiting caused by frequent switching of CPU time slices between threads, and leaving one CPU for system operation interaction to ensure normal interaction execution. The calculation method of the maximum number of parallel tasks is: maximum number of parallel tasks = (number of CPU cores - number of system operation interaction cores) / (1 + N). Each extraction task requires one read thread and N write threads. The initial value of the number of system operation interaction cores is 1 by default; N is the initial setting value, which is 2 by default.

[0090] CPU time slice is the time period allocated by the operating system to each process or thread for execution when scheduling multiple processes or threads. In a computer system, usually multiple processes or threads are running, in order to fairly and efficiently manage the execution of these processes or threads, the operating system will adopt a strategy called "time slice scheduling". When switching from one process or thread to another, the operating system needs to perform context switching. Context switching will consume some CPU time, save the state and data of the switched process or thread, and the CPU time slice of the process / thread will be frequently switched, which will affect the system performance. Therefore, the embodiment sets up an independent CPU for each thread, which can guarantee system accessibility, reduce the frequent switching of CPU time slice between threads, and improve thread efficiency.

[0091] As shown in Figure 2 Within the range of the maximum number of parallel tasks, each started extraction task increases the parallel task running count by 1. Each parallel task starts with 1 read thread and N write threads by default; the data is cached through the singletone singleton mode, wherein the read thread shards the source data to be processed according to the number of data, reads the data shards, and writes the memory cache through the synchronized addList method of the singletone singleton mode controller, and the total cache shard count and the target table cache shard count are each increased by 1; the write thread consumes the memory cache data through the synchronized getList method of the singletone singleton mode controller, and writes to the target database, at which time the total cache shard count and the target table cache shard count are each decreased by 1. The singletone singleton mode is a common design pattern that ensures that there is only one instance of a class and provides a global access point to obtain the instance.

[0092] When multi-threading is enabled, the embodiment introduces synchronized lock synchronization control thread execution, which is an effective method to realize thread mutual exclusion and synchronization; ensures that thread contention will not produce deadlock and error, and is an effective control algorithm to avoid thread deadlock. When the system has A, B, and C threads that need to use method f1, no matter which thread (for example, thread A) runs to method f1, it needs to check whether there is another thread B (or C, etc.) using method f1, if there is, A needs to wait for the thread using f1, for example, B (or C) to run the method f1, and then thread A can use method f1; if not, A locks the method f1 and directly runs the method f1.

[0093] The read thread of the extraction task entering parallel processing starts a read thread, and the read thread count is incremented by 1. The read thread judges whether it is allowed to write data shards into the memory cache through the (synchronized) canAdd method of the singletone singleton mode controller. If it is allowed, the data shards are cached through the (synchronized) addList method of the singletone singleton mode controller, and the cached data is increased. Otherwise, the read thread waits for M seconds and then judges whether it is allowed to cache data shards. The method for judging whether it is allowed to write data shards into the memory cache is as follows: when the total cache shard count does not reach the set maximum total cache shard count DM1, and the target table cache shard count does not reach the set single table maximum cache shard count DM2, it is judged that it is allowed. Otherwise, it is judged that it is not allowed. If the read thread finishes reading data, the state of the Task extraction task in which the current thread is located is set to “read complete”, the read thread count is decremented by 1, and the read thread is ended. The read thread records the total amount of read data and the start time of the thread while reading data, and updates the recorded total amount of read data to the Task extraction task in which the current thread is located after submitting the data cache, which is used for monitoring access.

[0094] The write thread of the extraction task entering parallel processing is automatically started. Every time a write thread is started, the write thread count is incremented by 1. The write thread obtains the data shards of the current task through the (synchronized) getList method of the singletone singleton mode controller. If the data shards are obtained, the data shards are written into the target database through the writing mode of the target data set when the task is established. If the data shards are not obtained, it is further judged whether the state of the Task extraction task in which the current thread is located is “read complete” and whether the target table cache shard count is 0. If the state of the Task extraction task in which the current thread is located is “read complete” and the target table cache shard count is 0, the state of the Task extraction task in which the current thread is located is set to “write complete”, the write thread count is decremented by 1, the number of parallel tasks is decremented by 1, and the current write thread is ended. Otherwise, the write thread waits for X seconds (X < M), and then judges whether the data shards of the current task are obtained. The write thread records the total amount of written data and the start time of the thread while writing data, and updates the total amount of written data to the Task extraction task in which the current thread is located after writing the shard data, which is used for monitoring access. In the embodiment, M is 1 and X is 0.2 by default.

[0095] The task queue judges that the number of current parallel tasks is less than the maximum number of parallel tasks, and there is a new extraction task that has not been executed. Then, a new extraction task is started to synchronize data, and the number of parallel task runs is incremented by 1.

[0096] E. Task monitoring module, used to monitor the started extraction tasks. If the number of times the read thread in the extraction task waits continuously for writing to the cache is greater than the waiting threshold, and the number of idle CPU cores is not zero, a new write thread for the extraction task is started, the number of times the read thread waits continuously for writing to the cache is cleared, and continuous monitoring is continued until the extraction task is completed.

[0097] like Figure 3 As shown, the task monitoring module monitors the data of the running extraction tasks in the extraction plan, including the start time, running time, number of reads, number of writes, number to be written, extraction speed, etc.

[0098] This embodiment monitors and evaluates whether additional write threads are needed to increase cache data consumption and improve write speed. Specifically, if a read thread waits for M seconds, the number of consecutive write-cache waits is incremented by 1. If the number of consecutive write-cache waits exceeds P, the current task's write thread processing capacity is insufficient. The "need_new_wr" attribute in the current task's properties is set to 1, and the number of consecutive write-cache waits is set to 0. Typically, P is a multiple of 10; in this embodiment, it defaults to 20.

[0099] The task monitoring module is timed, for example, the default setting is 1 second loop to judge the extraction tasks that are running in parallel. If the task attribute "need to add write thread need_new_wr" is 1, it is judged that the extraction task needs to add a write thread, and further judges whether the current number of idle CPU cores is 0. If it is not 0, a new write thread for the task is automatically opened, and the task "need to add write thread need_new_wr" attribute is set to 0; if it is 0, a new write thread is not opened, and the judgment is made after the next monitoring, and corresponding prompts can also be given. This embodiment automatically adds write threads according to resources to eliminate data blockage and increase synchronization speed. The calculation method of the number of idle CPU cores is: number of idle CPU cores = number of CPU cores - total number of read threads - total number of write threads - number of system operation interaction cores.

[0100] To ensure stable and efficient task execution, it is necessary to scientifically and rationally utilize cache memory. When multiple tasks are running in parallel, the cache usage of system memory must be kept within a reasonable range to prevent the operating system from using disk cache. This embodiment precisely controls the maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2. The dynamic adjustment method is as follows:

[0101] At preset intervals, for example, every minute, it is determined whether the percentage of server memory occupied by the total cache shards exceeds the percentage threshold. The percentage threshold is set to 70% by default. If it does not exceed the percentage threshold, the maximum number of cache shards DM1 and the maximum number of cache shards per table DM2 will not be adjusted. Otherwise, the maximum number of cache shards DM1 and the maximum number of cache shards per table DM2 will be adjusted as follows:

[0102]

[0103] Single table maximum cache shard number DM2 = single table maximum cache shard number DM2-NJ

[0104] In the formula, NJ is a preset decrement step, and the default is 10. When the single table maximum cache shard number DM2 is less than 20, the decrement of the single table maximum cache shard number DM2 is not performed. It should be noted that after the division operation is performed in the embodiment, if the division is not exact, a down rounding operation is performed.

[0105] The system logical structure of the embodiment is simple, a web system is developed by using pure java, there is no service dependence such as a middleware service like Zookeeper, MQ, Elasticsearch and the like, and deployment is simple. CPU resources are automatically monitored and adjusted to improve the extraction speed, the speed of each parallel processing task thread is monitored, insufficient writing processing tasks are automatically identified, and writing threads are automatically increased according to the number of idle CPU cores to improve the extraction speed.

[0106] Further, as shown in Figure 5 The embodiment further includes a data source management module for configuring a plurality of data sources, the data sources including a data source name, a database type, a database corresponding driver, an address, a port, a username and a password and the like; and the configured data sources can be tested for connection. Source data sources and writing target data sources are dynamically managed, different database drivers are introduced to increase the supportability of different brands of databases of the system, so that the embodiment supports various relational databases with jdbc drivers, the system manages data sources through jdbc drivers, and obtains java Connection connection through the data sources to perform data query and writing.

[0107] In order to intuitively reflect the advantages of the multi-threaded data cache read-write parallel high-performance data extraction system, the following comparative experiments are performed:

[0108] The system of the embodiment is run on a hardware environment of 16 cores and 32G memory to open 5 parallel tasks and 1 parallel task, and a database migration tool of a domestic database manufacturer DreamV8 and a database migration product Kettler of Hitachi open source are taken for performance comparison.

[0109] As shown in Figures 6-9As shown, compared with the extraction of the same data table, the extraction speed of the system of the application is 203950 pieces per second under 5 task parallelism, and the extraction speed is 532540 pieces per second under 1 task parallelism, the extraction speed of the migration tool of the domestic database manufacturer Dameng database is 467560 pieces per second, and the extraction speed of the database migration product Kettler of Hitachi open source is 169480 pieces per second.

[0110] It can be seen that in the single task single thread mode: the extraction performance of the system of the application is 114% of the Dameng product, and 314% of the kettle; in the 5 task automatic monitoring parallel mode, the extraction performance of the system of the application is 436% of Dameng, and 1203% of kettle, and the extraction performance is much higher than the current mainstream data extraction product. And in the landing extraction use, the system of the application extracts 1.5 billion deposit details records for 2 hours and 27 minutes, and the extraction speed is 170000 pieces per second, and the extraction performance is high.

[0111] The technical features of the above-described embodiments can be combined arbitrarily, and in order to make the description simple, all possible combinations of the technical features in the above-described embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0112] The above-described embodiments only express several embodiments of the application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A multi-threaded data cache read and write parallel high-performance data extraction method, characterized in that: The multi-threaded data cache read and write parallel high-performance data extraction method includes: Establish an extraction plan, and establish one or more extraction tasks for each extraction plan; Set each extraction task in the extraction plan to be executed before or after extraction; Trigger the execution of an established extraction plan; Calculate the maximum number of parallel tasks based on the number of CPU cores, and start the extraction tasks in the triggered extraction plan based on the maximum number of parallel tasks. Each extraction task initially starts with 1 read thread and N write threads. Monitor the started extraction task. If the number of times the read thread in the extraction task continuously waits for writing to the cache exceeds the waiting threshold, and the number of idle CPU cores is not zero, start a new write thread for the extraction task, reset the number of times the read thread continuously waits for writing to the cache to zero, and continue monitoring until the extraction task is completed. The read thread segments the data to be read according to the number of data items, and determines whether the data segments are satisfied for writing into the memory cache; if satisfied, the data segments are cached, and the total cache segment count and the target table cache segment count are each increased by 1; otherwise, the read thread waits, and the number of times the read thread continuously waits for writing into the cache is increased by 1; and the determination of whether the data segments are satisfied for writing into the memory cache includes: when the total cache segment count does not reach the set maximum total number of cache segments DM1, and the target table cache segment count does not reach the set maximum number of cache segments DM2 for a single table, determining that the data segments are satisfied; otherwise, determining that the data segments are not satisfied; The write thread consumes the data slices in the memory cache, and at this time the total cache slice count and the target table cache slice count are both reduced by 1.

2. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 1, characterized in that: The calculation method of the maximum number of parallel tasks is: maximum number of parallel tasks = (number of CPU cores - number of system operation interaction cores) / (1 + N).

3. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 1, characterized in that: The maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2 are dynamically adjusted as follows: At preset intervals, determine whether the proportion of server memory occupied by the total cache shards exceeds the threshold. If not, do not adjust the maximum number of cache shards DM1 and the maximum number of cache shards per table DM2. Otherwise, adjust the maximum number of cache shards DM1 and the maximum number of cache shards per table DM2 as follows: ； ； ； Where, is the preset decrement step size.

4. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 1, characterized in that: It also includes configuring multiple data sources, which include: data source name, database type, database corresponding driver, address, port, user name and password.

5. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 4, characterized in that: The establishment of the extraction plan includes: establishing an extraction plan name, selecting a source data source, and selecting a target data source; Creating an extraction task includes setting the task name, the method for obtaining the source data, and the method for writing the target data.

6. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 1, characterized in that: The calculation method of the number of idle CPU cores is: number of idle CPU cores = number of CPU cores - total number of read threads - total number of write threads - number of system operation interaction cores.

7. A multi-threaded data cache read and write parallel high-performance data extraction system, characterized in that: The multi-threaded data cache read-write parallel high-performance data extraction system includes: The extraction plan module is used to establish an extraction plan and create one or more extraction tasks for each extraction plan; The extraction processing setting module is used to set each extraction task in the extraction plan to be executed before or after extraction; Plan execution module, used to trigger the execution of an established extraction plan; The task execution module is used to calculate the maximum number of parallel tasks based on the number of CPU cores, and start the extraction tasks in the triggered extraction plan based on the maximum number of parallel tasks. Each extraction task initially starts with 1 read thread and N write threads; The task monitoring module is used to monitor the started extraction tasks. If the number of times the read thread in the extraction task continuously waits for writing to the cache is greater than the waiting threshold, and the number of idle CPU cores is not zero, a new write thread for the extraction task is started, the number of times the read thread continuously waits for writing to the cache is cleared, and continuous monitoring is continued until the extraction task is completed. The read thread segments the data to be read according to the number of data items, and determines whether the data segments are satisfied for writing into the memory cache; if satisfied, the data segments are cached, and the total cache segment count and the target table cache segment count are each increased by 1; otherwise, the read thread waits, and the number of times the read thread continuously waits for writing into the cache is increased by 1; and the determination of whether the data segments are satisfied for writing into the memory cache includes: when the total cache segment count does not reach the set maximum total number of cache segments DM1, and the target table cache segment count does not reach the set maximum number of cache segments DM2 for a single table, determining that the data segments are satisfied; otherwise, determining that the data segments are not satisfied; The write thread consumes the data slices in the memory cache, and at this time the total cache slice count and the target table cache slice count are both reduced by 1.

Images