Multi-thread data cache read-write parallel high-performance data extraction method and system

Abstract

The invention belongs to the technical field of data extraction, and discloses a multi-thread data cache read-write parallel high-performance data extraction method and system. The method comprises the following steps: establishing extraction plans, and establishing one or more extraction tasks for each extraction plan; setting each extraction task in the extraction plan to be executed before extraction or executed after extraction; triggering an established extraction plan to execute; calculating the maximum parallel task number according to the CPU core number, and starting the extraction tasks in the triggered extraction plan based on the maximum parallel task number; and monitoring the started extraction task, if the number of times of continuously waiting for writing into the cache by a read thread in the extraction task is greater than a waiting threshold value and the number of idle CPU cores is not zero, starting a new write thread of the extraction task, resetting the number of times of continuously waiting for writing into the cache by the read thread, and continuously monitoring until the extraction task is completed. According to the method, high-performance, low-cost, centralized and easy-to-deploy-and-implement data extraction is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data extraction, and in particular relates to a multi-threaded data cache read-write parallel high-performance data extraction method and system. Background Art

[0002] Faced with massive data resources, how to extract valuable information efficiently and accurately has become an urgent problem to be solved in the current process of informatization towards dataization; enterprises and institutions often have multiple systems and different database products; the current mainstream data extraction tools often have drawbacks such as low efficiency and huge resource investment, and cannot meet the timeliness and economy requirements of modern enterprises and governments for data processing.

[0003] System operating costs, read-write synchronization speed, ease of use, support for different databases, task intervention, and monitorability are important considerations for selecting data synchronization software. The technologies used in current software technology data synchronization methods are mainly divided into the following five modes:

[0004] Database feature mode: connect through database features, read data through Select, and write data through Insert, such as data federation (Db2), DBlink (Oracle, DAMO, PostgreSql, MySQL), etc.; this mode is fast and low-cost, but it does not support the aggregation of complex and multi-type heterogeneous databases, internal read and write serialization, and there is no means to intervene and improve ultra-large synchronization tasks during execution.

[0005] Multi-task serial mode: multi-threaded read-then-write format, write-wait while reading, read-wait while writing. This mode has simple system logic and low cost. This mode is mostly used in government data centers for data development and aggregation. However, the multi-task serial mode has low server resource utilization and slow speed, and has no means to intervene and improve ultra-large synchronization tasks.

[0006] Multi-task parallel mode: Use multiple threads for parallel reading and writing. The reading thread writes data into the memory, and the reading service reads continuously. The writing thread takes data from the memory, and the writing thread writes continuously, such as IBM DataStage, open source Kettle and other products. This mode has high resource utilization, and commercial products are faster, but the cost is high, and the speed of open source products is average; the number of parallel tasks cannot be controlled, the memory usage cannot be controlled, and too many threads cause frequent switching of CPU time slices, resulting in reduced synchronization speed. There is no means to intervene and improve super-large synchronization tasks.

[0007] Distributed consumption mode: Service scheduling is performed through applications such as ZooKeeper, and multi-machine and multi-threaded data is used. One task is assigned to one machine, and one read task can be set to one or more write tasks. The machine that obtains the read task starts the read thread, and the read data is sharded and packaged according to a certain number of pieces, and the packaged data is sent to the MQ message middleware; the machine that obtains the write task starts the write thread, and obtains a data shard from the MQ message middleware to write the data to the target database. This mode supports a variety of different databases, has fast synchronization speed, and limited horizontal expansion of read and write capabilities, but requires the stacking of multiple server hardware, with high investment costs, complex structure, and difficulty in deployment, poor single-machine processing capabilities, and the scheduling service has the risk of brain splitting, and there is no means to intervene and improve super-large synchronization tasks.

[0008] Streaming synchronization mode: using database log analysis, often through log operation time point mark, first serial synchronization of stock data, and then use log analysis from the recorded time point, stream incremental processing, generally CDC for log data analysis, through Spark, Flink and other frameworks for streaming analysis logs, such as Apache SeaTunne system. This mode is mostly used in e-commerce user behavior analysis recommendation system, with the goal of processing incremental data, full data processing is slow, dependent on database archive logs, complex structure and difficult deployment, no intervention to improve the means of large synchronization tasks. Summary of the invention

[0009] One of the purposes of the present invention is to provide a multi-threaded data cache read and write parallel high-performance data extraction method to achieve high-performance, low-cost, centralized, and easy-to-deploy and implement data extraction.

[0010] To achieve the above object, the technical solution adopted by the present invention is:

[0011] A multi-threaded data cache read-write parallel high-performance data extraction method, the multi-threaded data cache read-write parallel high-performance data extraction method comprising:

[0012] Establish an extraction plan, and establish one or more extraction tasks for each extraction plan;

[0013] Set each extraction task in the extraction plan to be executed before extraction or after extraction;

[0014] Trigger the execution of an established extraction plan;

[0015] Calculate the maximum number of parallel tasks based on the number of CPU cores, and start the extraction tasks in the triggered extraction plan based on the maximum number of parallel tasks. Each extraction task initially starts with 1 read thread and N write threads.

[0016] Monitor the started extraction tasks. If the number of times the read thread in the extraction task waits continuously for the write cache is greater than the waiting threshold, and the number of idle CPU cores is not zero, start a new write thread for the extraction task, reset the number of times the read thread waits continuously for the write cache to zero, and continue monitoring until the extraction task is completed.

[0017] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0018] Preferably, the calculation method of the maximum number of parallel tasks is: maximum number of parallel tasks = (number of CPU cores - number of system operation interaction cores) / (1+N).

[0019] Preferably, the read thread segments the data to be read according to the number of data items, and determines whether the data segments are satisfied to be written into the memory cache; if satisfied, the data segments are cached, and the total cache segment count and the target table cache segment count are each increased by 1; otherwise, the read thread waits, and the number of times the read thread continuously waits to write into the cache is increased by 1;

[0020] The write thread consumes the data slices in the memory cache, and at this time, the total cache slice count and the target table cache slice count are each reduced by 1.

[0021] Preferably, the determining whether the data shards are written into the memory cache comprises:

[0022] When the total cache shard count does not reach the set maximum total number of cache shards DM1, and the target table cache shard count does not reach the set single-table maximum number of cache shards DM2, it is judged to be satisfied; otherwise, it is judged to be unsatisfied.

[0023] Preferably, the maximum total number of cache shards DM1 and the maximum number of cache shards for a single table DM2 are dynamically adjusted as follows:

[0024] At each preset time interval, determine whether the proportion of the total cache shards in the server memory exceeds the proportion threshold. If not, do not adjust the maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2; otherwise, adjust the maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2 as follows:

[0025]

[0026] Maximum number of cache shards for a single table DM2 = Maximum number of cache shards for a single table DM2-NJ

[0027] Where NJ is the preset decreasing step size.

[0028] Preferably, it also includes configuring multiple data sources, and the data sources include: data source name, database type, database corresponding driver, address, port, user name and password.

[0029] Preferably, the establishing of the extraction plan includes: establishing an extraction plan name, selecting a source data source, and selecting a target data source;

[0030] The establishment of the extraction task includes: setting the task name, the method for obtaining the extraction source data and the method for writing the target data.

[0031] Preferably, the method for calculating the number of idle CPU cores is: number of idle CPU cores = number of CPU cores - total number of read threads - total number of write threads - number of system operation interaction cores.

[0032] The second purpose of the present invention is to provide a multi-threaded data cache read and write parallel high-performance data extraction system, providing a system that does not rely on database archiving, has fast speed, low deployment cost, low structural complexity, controllable memory data volume, and automatically monitors and allocates CPU resources to improve extraction speed.

[0033] To achieve the above object, the technical solution adopted by the present invention is:

[0034] A multi-threaded data cache read-write parallel high-performance data extraction system, the multi-threaded data cache read-write parallel high-performance data extraction system comprising:

[0035] The extraction plan module is used to establish an extraction plan and to establish one or more extraction tasks for each extraction plan;

[0036] The extraction processing setting module is used to set each extraction task in the extraction plan to be executed before extraction or after extraction;

[0037] The plan execution module is used to trigger the execution of an established extraction plan;

[0038] The task execution module is used to calculate the maximum number of parallel tasks according to the number of CPU cores, and start the extraction tasks in the triggered extraction plan based on the maximum number of parallel tasks. Each extraction task initially starts with 1 read thread and N write threads;

[0039] The task monitoring module is used to monitor the started extraction tasks. If the number of times the read thread in the extraction task waits continuously for the write cache is greater than the waiting threshold, and the number of idle CPU cores is not zero, a new write thread for the extraction task is started, the number of times the read thread waits continuously for the write cache is cleared, and continuous monitoring is carried out until the extraction task is completed.

[0040] The present invention provides a multi-threaded data cache read and write parallel high-performance data extraction method and system, in order to improve the extraction speed, the parallel reading and writing threads are used to eliminate the serial waiting time of reading and writing; the reading thread writes the data into the memory cache, and the writing thread obtains the data from the memory cache, thereby reducing the network consumption of the writing and obtaining data time; multi-tasking and multi-threading improve the CPU resource utilization rate and reduce the total synchronization time of multiple tasks; the maximum number of parallel tasks is controlled according to the number of CPU cores of the server to ensure that the threads have the corresponding number of CPU cores, thereby eliminating the time consumption and memory loss of CPU time slice switching caused by frequent thread switching. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a multi-threaded data cache read-write parallel high-performance data extraction method of the present invention;

[0042] Figure 2 It is a schematic diagram of the task execution process of the present invention;

[0043] Figure 3 It is a schematic diagram of the task monitoring process of the present invention;

[0044] Figure 4 It is a first structural schematic diagram of a multi-threaded data cache read-write parallel high-performance data extraction system of the present invention;

[0045] Figure 5 A second structural schematic diagram of a multi-threaded data cache read-write parallel high-performance data extraction system of the present invention;

[0046] Figure 6 It is a schematic diagram of the extraction speed of the system of the present invention under 5-task parallelism;

[0047] Figure 7 It is a schematic diagram of the extraction speed of the system of the present invention under 1 task parallelism;

[0048] Figure 8 This is a schematic diagram of the extraction speed of the database migration tool of the domestic database manufacturer DAMO;

[0049] Fig. 9 A diagram showing the extraction speed of Kettler, Hitachi's open source database migration product. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0052] Example 1

[0053] In order to overcome the drawbacks of existing data extraction tools such as low efficiency and huge resource investment, this embodiment proposes a high-performance, low-cost, centralized, and easy-to-deploy and implement data extraction method, which is of great significance for improving data processing efficiency, reducing costs, and promoting data-driven decision-making.

[0054] like Figure 1 As shown, a multi-threaded data cache read and write parallel high-performance data extraction method of this embodiment includes the following steps:

[0055] (1) Establish an extraction plan and create one or more extraction tasks for each extraction plan.

[0056] Establish an extraction plan, including the name of the extraction plan, selecting the source data source, and selecting the target data source. After establishing the extraction plan, establish one or more extraction tasks. Establishing an extraction task includes setting the acquisition method of the source data in the source data source and the writing method and task name of the target data in the target data source. The above-mentioned acquisition method can be understood as the data items required for SQL query, and the writing method can be understood as the target table and field written by SQL.

[0057] For the normal execution of the task, when the amount of source data is greater than the data volume threshold, for example, the amount of source data is greater than 100 million, you can set Task Sharding to split a large extraction task into multiple parallel extraction tasks. It should be noted that the extraction tasks mentioned in the subsequent steps include the original extraction tasks and the sharded extraction tasks.

[0058] Task Sharding is a process of splitting a large task or data set into multiple smaller, independent subtasks or data fragments. These subtasks or data fragments can be run independently on different machines to speed up the overall processing speed, improve system throughput and scalability.

[0059] (2) Each extraction task in the extraction plan is set to be executed before or after extraction, that is, to set pre- and post-extraction tasks, such as clearing data, deleting indexes, creating indexes, etc., where post-extraction execution is the operation on the target table to be written.

[0060] (3) Trigger the execution of an established extraction plan. Obtain all extraction tasks in the extraction plan and start synchronous task execution. Furthermore, after the extraction plan is triggered, you can record and control the status of the extraction plan, as well as view the number of extraction tasks, extraction task execution records, etc.

[0061] (4) The maximum number of parallel tasks is calculated according to the number of CPU cores, and the extraction tasks in the triggered extraction plan are started based on the maximum number of parallel tasks. Each extraction task initially starts with 1 read thread and N write threads.

[0062] The number of CPU cores refers to the number of processing units integrated inside the central processing unit. Each processing unit is an independent computing engine that can execute instructions and process data. In order to set the effective maximum number of parallel tasks in this embodiment, the maximum number of parallel tasks is associated with the number of CPU cores, the maximum number of parallel tasks is confirmed according to the calculation of the number of CPU cores, and the extraction task is executed by queue to synchronize data extraction. The maximum number of parallel tasks enabled is related to the number of CPU cores. In order to improve the efficiency of task execution, each thread in the extraction task occupies an independent CPU to avoid waiting due to frequent switching of CPU time slices between threads, and a CPU is left for system operation interaction to ensure normal interactive execution. The calculation method of the maximum number of parallel tasks is: maximum number of parallel tasks = (number of CPU cores-number of system operation interaction cores) / (1+N). Each extraction task requires 1 read thread and N write threads. The number of system operation interaction cores defaults to 1; N is the initial setting value, which defaults to 2.

[0063] The CPU time slice is the time period that the operating system allocates to each process or thread for execution when scheduling multiple processes or threads. There are usually multiple processes or threads running in a computer system. In order to fairly and effectively manage the execution of these processes or threads, the operating system will adopt a strategy called "time slice scheduling". When switching from one process or thread to another, the operating system needs to perform a context switch. Context switching consumes a certain amount of CPU time and saves the status and data of the switched process or thread. Frequent switching of the CPU time slice occupancy of the process / thread will affect system performance. Therefore, this embodiment sets each thread to occupy an independent CPU, which can ensure system accessibility, reduce the frequent switching of CPU time slices between threads, and improve thread efficiency.

[0064] like Figure 2As shown in the figure, within the range of the maximum number of parallel tasks, each time an extraction task is started, the parallel task running count increases by 1. Each parallel task starts 1 read thread and N write threads by default; the data is cached through the singletone mode, where the read thread shards the source data to be processed according to the number of data items. After reading the data shards, the (synchronized) data synchronization addList method of the singletone mode controller is used to write to the memory cache, and the total cache shard count and the target table cache shard count are each increased by 1; the write thread consumes the memory cache data through the (synchronized) getList method of the singletone mode controller and writes it to the target database. At this time, the total cache shard count and the target table cache shard count are each reduced by 1. The singletone mode is a common design pattern that ensures that a class has only one instance and provides a global access point to obtain the instance.

[0065] When multiple threads are turned on, this embodiment introduces synchronized locks to synchronously control thread execution. Synchronized lock synchronization is an effective method to achieve mutual exclusion and synchronization of threads; it ensures that contention between threads will not cause deadlocks and errors, and is an effective control algorithm to avoid thread deadlocks. When the system has several threads A, B, and C, all of them need to use method f1. No matter which thread (for example, thread A) runs to method f1, it must check whether there are other threads B (or C, etc.) using method f1. If there are, A must wait until the thread that is using f1, such as B (or C), finishes running method f1 before thread A can use method f1; if not, A locks the call to method f1 and runs method f1 directly.

[0066] The extraction task entering parallel processing starts a read thread, and the read thread count is incremented by 1. The read thread uses the (synchronized) canAdd method of the singletone singleton mode controller to determine whether it is satisfied to write a data shard to the memory cache. If satisfied, the data shard is cached through the (synchronized) addList method of the singletone singleton mode controller, and the cached data is increased. Otherwise, after the read thread waits for M seconds, it judges again whether the data shard can be cached. The method for judging whether it is satisfied to write a data shard to the memory cache is as follows: when the total cached shard count has not reached the set maximum total cached shard count DM1, and the cached shard count of the target table has not reached the set maximum cached shard count DM2 for a single table, it is judged as satisfied; otherwise, it is judged as not satisfied. If the read thread has finished reading data, the status of the extraction task of the current thread's Task is set to [Read Complete], the read thread count is decremented by 1, and the read thread ends. When reading data, the read thread records the total amount of data read by the current thread and the thread start time, and after submitting the data cache, updates the recorded total amount of data read to the extraction task of the current thread's Task for monitoring access.

[0067] The extraction task entering parallel processing automatically starts a write thread. For each write thread started, the write thread count is incremented by 1. The write thread uses the (synchronized) getList method of the singletone singleton mode controller to obtain the data shard of the current task. If the data shard is obtained, it is written to the target database through the write method of the target data set when the task is established. If the data shard is not obtained, it further judges whether the status of the extraction task of the current thread's Task is [Read Complete] and whether the cached shard count of the target table is 0. If the status of the extraction task of the current thread's Task is [Read Complete] and the cached shard count of the target table is 0, the status of the extraction task of the current thread's Task is set to [Write Complete], the write thread count is decremented by 1, the parallel task running count is decremented by 1, and the current write thread ends. Otherwise, the write thread waits for X seconds ("X < M"), and then obtains the data shard of the current task for judgment. When writing data, the write thread records the total amount of data written by the current thread and the thread start time, and after writing the shard data, updates the total amount of data written to the extraction task of the current thread's Task for monitoring access. In this embodiment, M is defaulted to 1 and X is defaulted to 0.2.

[0068] The task queue judges that when the current number of parallel tasks is less than the maximum number of parallel tasks and there are new extraction tasks that have not been executed, a new extraction task is started for data synchronization, and the parallel task running count is incremented by 1.

[0069] (5) Monitor the started extraction task. If the number of times the read thread in the extraction task continuously waits for writing to the cache is greater than the waiting threshold, and the number of idle CPU cores is not zero, start a new write thread for the extraction task, reset the number of times the read thread continuously waits for writing to the cache, and continue monitoring until the extraction task is completed.

[0070] like Figure 3 As shown, data monitoring is performed on the extraction tasks that are running in the extraction plan, including the start time, running time, number of reads, number of writes, number to be written, extraction speed, etc.

[0071] This embodiment monitors and evaluates whether it is necessary to increase the write thread to increase the cache data consumption speed and improve the write speed. Specifically, if the read thread waits for M seconds, the number of times the read thread waits for writing to the cache is increased by 1. If the number of times the read thread waits for writing to the cache is greater than P times, the processing capacity of the current task write thread is insufficient, and the "need to increase the write thread need_new_wr" attribute in the current task attributes is set to 1, and the number of times the read thread waits for writing to the cache is set to 0. Usually P is a multiple of 10, and this embodiment defaults to 20.

[0072] In the extraction task monitoring stage, the timing is set by default to 1 second to loop and judge the parallel extraction tasks. If the task attribute "need to add write thread need_new_wr" is 1, it is judged that the extraction task needs to add write thread, and further judge whether the current number of idle CPU cores is 0. If it is not 0, a new write thread for the task is automatically opened, and the task "need to add write thread need_new_wr" attribute is set to 0; if it is 0, the new write thread is not opened, and the next monitoring is waited for to judge, and corresponding prompts can be given. This embodiment automatically adds write threads according to resources to eliminate data blockage and increase synchronization speed. The calculation method of the number of idle CPU cores is: number of idle CPU cores = number of CPU cores - total number of read threads - total number of write threads - number of system operation interaction cores.

[0073] To ensure stable and efficient task operation, it is necessary to use the cache scientifically and reasonably, and ensure that the cache occupies a reasonable range of system memory in the case of multi-tasking in parallel, so as to avoid the operating system using the disk for caching. This embodiment accurately controls the maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2. The dynamic adjustment method is as follows:

[0074] At preset intervals, for example, every 1 minute, determine whether the proportion of the total cache shards in the server memory exceeds the proportion threshold. The proportion threshold is set to 70% by default. If it does not exceed, the maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2 will not be adjusted; otherwise, the maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2 will be adjusted as follows:

[0075]

[0076] Maximum number of cache shards for a single table DM2 = Maximum number of cache shards for a single table DM2-NJ

[0077] Wherein, NJ is a preset decrement step size, which defaults to 10. When the maximum number of cache fragments DM2 for a single table is less than 20, the maximum number of cache fragments DM2 for a single table is no longer decremented. It should be noted that, in this embodiment, if the division is not exact after the division operation, a rounding down operation is performed.

[0078] This embodiment automatically monitors and allocates CPU resources to improve the extraction speed, monitors the speed of each parallel processing task thread, automatically identifies insufficient write processing tasks, and automatically increases write threads according to the number of idle CPU cores to improve the extraction speed. Furthermore, this embodiment also includes configuring multiple data sources, and the data source includes: data source name, database type, database corresponding driver, address, port, user name and password, etc., and the configured data source needs to be connected to the test. This embodiment supports various relational databases with jdbc drivers. The system manages the data source through the jdbc driver and obtains the javaConnection connection through the data source to query and write data.

[0079] Example 2

[0080] In order to overcome the drawbacks of existing data extraction tools such as low efficiency and huge resource investment, this embodiment provides a high-performance data extraction system based on adjustable CPU resources, supports multiple databases, has a simple structure and is easy to deploy, has low hardware cost, high-speed synchronization of data memory cache, and multi-task monitoring, which significantly reduces synchronization time.

[0081] like Figure 4 As shown, a multi-threaded data cache read-write parallel high-performance data extraction system of this embodiment includes the following modules:

[0082] A. Extraction plan module, used to establish an extraction plan and establish one or more extraction tasks for each extraction plan.

[0083] Establish an extraction plan, including the name of the extraction plan, selecting the source data source, and selecting the target data source. After establishing the extraction plan, establish one or more extraction tasks. Establishing an extraction task includes setting the acquisition method of the source data in the source data source and the writing method and task name of the target data in the target data source. The above-mentioned acquisition method can be understood as the data items required for SQL query, and the writing method can be understood as the target table and field written by SQL.

[0084] For the normal execution of the task, when the amount of source data is greater than the data volume threshold, for example, the amount of source data is greater than 100 million, you can set Task Sharding to split a large extraction task into multiple parallel extraction tasks. It should be noted that the extraction tasks mentioned in the subsequent steps include the original extraction tasks and the sharded extraction tasks.

[0085] Task Sharding is a process of splitting a large task or data set into multiple smaller, independent subtasks or data fragments. These subtasks or data fragments can be run independently on different machines to speed up the overall processing speed, improve system throughput and scalability.

[0086] B. The extraction processing setting module is used to set each extraction task in the extraction plan to be executed before or after extraction, that is, to set pre- and post-extraction tasks, such as clearing data, deleting indexes, creating indexes, etc., where post-extraction execution is the operation on the written target table.

[0087] C. Plan execution module, used to trigger the execution of an established extraction plan. Obtain all extraction tasks in the extraction plan and send them to the task execution module to start synchronous task execution. Furthermore, after the extraction plan is triggered, the status of the extraction plan can be recorded and controlled, and the number of extraction tasks and extraction task execution records can be viewed.

[0088] D. Task execution module, used to calculate the maximum number of parallel tasks according to the number of CPU cores, and start the extraction tasks in the triggered extraction plan based on the maximum number of parallel tasks. Each extraction task initially starts with 1 read thread and N write threads.

[0089] The number of CPU cores refers to the number of processing units integrated inside the central processing unit. Each processing unit is an independent computing engine that can execute instructions and process data. In order to set the effective maximum number of parallel tasks in this embodiment, the maximum number of parallel tasks is associated with the number of CPU cores, the maximum number of parallel tasks is confirmed according to the calculation of the number of CPU cores, and the extraction task is executed by queue to synchronize data extraction. The maximum number of parallel tasks enabled is related to the number of CPU cores. In order to improve the efficiency of task execution, each thread in the extraction task occupies an independent CPU to avoid waiting due to frequent switching of CPU time slices between threads, and a CPU is left for system operation interaction to ensure normal interactive execution. The calculation method of the maximum number of parallel tasks is: maximum number of parallel tasks = (number of CPU cores-number of system operation interaction cores) / (1+N). Each extraction task requires 1 read thread and N write threads. The initial value of the number of system operation interaction cores defaults to 1; N is the initial setting value, which defaults to 2.

[0090] The CPU time slice is the time period that the operating system allocates to each process or thread for execution when scheduling multiple processes or threads. There are usually multiple processes or threads running in a computer system. In order to fairly and effectively manage the execution of these processes or threads, the operating system will adopt a strategy called "time slice scheduling". When switching from one process or thread to another, the operating system needs to perform a context switch. Context switching consumes a certain amount of CPU time and saves the status and data of the switched process or thread. Frequent switching of the CPU time slice occupancy of the process / thread will affect system performance. Therefore, this embodiment sets each thread to occupy an independent CPU, which can ensure system accessibility, reduce the frequent switching of CPU time slices between threads, and improve thread efficiency.

[0091] like Figure 2 As shown in the figure, within the range of the maximum number of parallel tasks, each time an extraction task is started, the parallel task running count increases by 1. Each parallel task starts 1 read thread and N write threads by default; the data is cached through the singletone mode, where the read thread shards the source data to be processed according to the number of data items. After reading the data shards, the (synchronized) data synchronization addList method of the singletone mode controller is used to write to the memory cache, and the total cache shard count and the target table cache shard count are each increased by 1; the write thread consumes the memory cache data through the (synchronized) getList method of the singletone mode controller and writes it to the target database. At this time, the total cache shard count and the target table cache shard count are each reduced by 1. The singletone mode is a common design pattern that ensures that a class has only one instance and provides a global access point to obtain the instance.

[0092] When multithreading is enabled, this embodiment introduces synchronized locks to control thread execution. Synchronized lock synchronization is an effective method to achieve mutual exclusion and synchronization of threads; it ensures that contention between threads will not cause deadlocks and errors, and is an effective control algorithm to avoid thread deadlocks. When the system has several threads A, B, and C that all need to use method f1, no matter which thread (for example, thread A) runs to method f1, it must check whether there are other threads B (or C, etc.) using method f1. If there are, A must wait until the thread that is using f1, such as B (or C), finishes running method f1 before thread A can use method f1; if not, A locks the call to method f1 and runs method f1 directly.

[0093] The extraction task that enters parallel processing starts a read thread, and the read thread count is incremented by 1. The read thread uses the (synchronized) canAdd method of the singletone singleton pattern controller to determine whether it is satisfied to write a data shard to the memory cache. If satisfied, it caches the data shard through the (synchronized) addList method of the singletone singleton pattern controller and increases the cached data. Otherwise, after the read thread waits for M seconds, it determines again whether it can cache the data shard. The method for determining whether it is satisfied to write a data shard to the memory cache is as follows: when the total cached shard count has not reached the set maximum total cached shard count DM1, and the cached shard count of the target table has not reached the set maximum cached shard count DM2 for a single table, it is determined to be satisfied; otherwise, it is determined to be not satisfied. If the read thread has finished reading data, the status of the extraction task of the current thread's Task is set to [Read Complete], the read thread count is decremented by 1, and the read thread ends. When the read thread reads data, it simultaneously records the total amount of data read by the current thread and the thread start time, and after submitting the data cache, updates the recorded total amount of data read to the extraction task of the current thread's Task for monitoring access.

[0094] The extraction task that enters parallel processing automatically starts a write thread. For each write thread started, the write thread count is incremented by 1. The write thread uses the (synchronized) getList method of the singletone singleton pattern controller to obtain the data shard of the current task. If it gets the data shard, it writes it to the target database according to the write method of the target data set when the task is established. If it does not get the data shard, it further determines whether the status of the extraction task of the current thread's Task is [Read Complete] and whether the cached shard count of this target table is 0. If the status of the extraction task of the current thread's Task is [Read Complete] and the cached shard count of this target table is 0, the status of the extraction task of the current thread's Task is set to [Write Complete], the write thread count is decremented by 1, the parallel task running count is decremented by 1, and the current write thread ends. Otherwise, the write thread waits for X seconds ("X < M") and then gets the data shard of the current task again for judgment. When the write thread writes data, it simultaneously records the total amount of data written by the current thread and the thread start time, and after writing the shard data, updates the total amount of data written to the extraction task of the current thread's Task for monitoring access. In this embodiment, M is defaulted to 1 and X is defaulted to 0.2.

[0095] The task queue determines that the current number of parallel tasks is less than the maximum number of parallel tasks and there are new extraction tasks that have not been executed, then starts a new extraction task for data synchronization, and the parallel task running count is incremented by 1.

[0096] E. Task monitoring module, used to monitor the started extraction tasks. If the number of times the read thread in the extraction task waits continuously for writing to the cache is greater than the waiting threshold, and the number of idle CPU cores is not zero, a new write thread for the extraction task is started, the number of times the read thread waits continuously for writing to the cache is cleared, and continuous monitoring is carried out until the extraction task is completed.

[0097] like Figure 3 As shown, the task monitoring module monitors the data of the running extraction tasks in the startup extraction plan, including the start time, running time, number of reads, number of writes, number to be written, extraction speed, etc.

[0098] This embodiment monitors and evaluates whether it is necessary to increase the write thread to increase the cache data consumption speed and improve the write speed. Specifically, if the read thread waits for M seconds, the number of times the read thread waits for writing to the cache is increased by 1. If the number of times the read thread waits for writing to the cache is greater than P times, the processing capacity of the current task write thread is insufficient, and the "need to increase the write thread need_new_wr" attribute in the current task attributes is set to 1, and the number of times the read thread waits for writing to the cache is set to 0. Usually P is a multiple of 10, and this embodiment defaults to 20.

[0099] The task monitoring module is timed, for example, the default setting is 1 second loop to judge the parallel extraction tasks. If the task attribute "need to add write thread need_new_wr" is 1, it is judged that the extraction task needs to add write thread, and further judge whether the current number of idle CPU cores is 0. If it is not 0, a new write thread for the task is automatically opened, and the task "need to add write thread need_new_wr" attribute is set to 0; if it is 0, the new write thread is not opened, and the next monitoring is waited for to judge, and corresponding prompts can be given. This embodiment automatically adds write threads according to resources to eliminate data blockage and increase synchronization speed. The calculation method of the number of idle CPU cores is: number of idle CPU cores = number of CPU cores - total number of read threads - total number of write threads - number of system operation interaction cores.

[0100] To ensure stable and efficient task operation, it is necessary to use the cache scientifically and reasonably, and ensure that the cache occupies a reasonable range of system memory in the case of multi-tasking in parallel, so as to avoid the operating system using the disk for caching. This embodiment accurately controls the maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2. The dynamic adjustment method is as follows:

[0101] At preset intervals, for example, every 1 minute, determine whether the proportion of the total cache shards in the server memory exceeds the proportion threshold. The proportion threshold is set to 70% by default. If it does not exceed, the maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2 will not be adjusted; otherwise, the maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2 will be adjusted as follows:

[0102]

[0103] Maximum number of cache shards for a single table DM2 = Maximum number of cache shards for a single table DM2-NJ

[0104] Wherein, NJ is a preset decrement step size, which defaults to 10. When the maximum number of cache fragments DM2 for a single table is less than 20, the maximum number of cache fragments DM2 for a single table is no longer decremented. It should be noted that, in this embodiment, if the division is not exact after the division operation, a rounding down operation is performed.

[0105] The system logic structure of this embodiment is simple. It uses pure Java to develop the Web system. It does not rely on middleware services such as Zookeeper, MQ, Elasticsearch, etc., and is easy to deploy. It also automatically monitors and allocates CPU resources to improve the extraction speed, monitors the speed of each parallel processing task thread, automatically identifies insufficient write processing tasks, and automatically increases write threads according to the number of idle CPU cores to improve the extraction speed.

[0106] Further, such as Figure 5 As shown, this embodiment also includes a data source management module for configuring multiple data sources, including: data source name, database type, database corresponding driver, address, port, user name and password, etc.; and the configured data source can be connected and tested. Dynamically manage the source data source and write target data source, and increase the support of different brands of databases by introducing different database drivers, so that this embodiment supports various relational databases with jdbc drivers. The system manages the data source through the jdbc driver and obtains the java connection through the data source to query and write data.

[0107] In order to intuitively demonstrate the advantages of the multi-threaded data cache read-write parallel high-performance data extraction system of the present invention, the following comparative experiments are conducted:

[0108] The system of the present invention is run on a hardware environment with 16 cores and 32G memory to open 5 parallel tasks and 1 parallel task, and the database migration tool of the domestic database manufacturer DAMO V8 and the open source database migration product Kettler of Hitachi are used for performance comparison.

[0109] like Figure 6-Figure 9As shown, under the extraction comparison of the same data table, the extraction speed of the system of the present invention is 203,950 records / second under 5 task parallelism, and the extraction speed is 53,254 records / second under 1 task parallelism. The extraction speed of the database migration tool of the domestic database manufacturer DAMO is 46,756 records / second, and the extraction speed of Kettler, an open source database migration product of Hitachi, is 16,948 records / second.

[0110] It can be seen that in single-task single-thread mode: the extraction performance of the system of the present invention is 114% of Dameng's product and 314% of Kettle; in 5-task automatic monitoring parallel mode, the extraction performance of the system of the present invention is 436% of Dameng and 1203% of Kettle, which is much higher than the current mainstream data extraction products. And in the on-site extraction, the system of the present invention took 2 hours and 27 minutes to extract 1.5 billion deposit details, with an extraction speed of 170,000 records per second, and high extraction performance.

[0111] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A multi-threaded data cache read and write parallel high-performance data extraction method, characterized in that: The multi-threaded data cache read-write parallel high-performance data extraction method comprises: Establish an extraction plan, and establish one or more extraction tasks for each extraction plan; Set each extraction task in the extraction plan to be executed before extraction or after extraction; Trigger the execution of an established extraction plan; Calculate the maximum number of parallel tasks based on the number of CPU cores, and start the extraction tasks in the triggered extraction plan based on the maximum number of parallel tasks. Each extraction task initially starts with 1 read thread and N write threads. Monitor the started extraction tasks. If the number of times the read thread in the extraction task waits continuously for the write cache is greater than the waiting threshold, and the number of idle CPU cores is not zero, start a new write thread for the extraction task, reset the number of times the read thread waits continuously for the write cache to zero, and continue monitoring until the extraction task is completed.

2. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 1, characterized in that: The calculation method of the maximum number of parallel tasks is: maximum number of parallel tasks = (number of CPU cores - number of system operation interaction cores) / (1+N).

3. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 1, characterized in that: The read thread segments the data to be read according to the number of data items, and determines whether the data segments are written into the memory cache; if so, the data segments are cached, and the total cache segment count and the target table cache segment count are each increased by 1; Otherwise, the read thread waits, and the number of times the read thread waits continuously for the write cache increases by 1; The write thread consumes the data slices in the memory cache, and at this time, the total cache slice count and the target table cache slice count are each reduced by 1.

4. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 3, characterized in that: The determining whether the data fragment is written into the memory cache comprises: When the total cache shard count does not reach the set maximum total number of cache shards DM1, and the target table cache shard count does not reach the set single-table maximum number of cache shards DM2, it is judged to be satisfied; otherwise, it is judged to be unsatisfied.

5. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 4, characterized in that: The maximum total number of cache shards DM1 and the maximum number of cache shards for a single table DM2 are dynamically adjusted as follows: At each preset time interval, determine whether the proportion of the total cache shards in the server memory exceeds the proportion threshold. If not, do not adjust the maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2; otherwise, adjust the maximum total number of cache shards DM1 and the maximum number of cache shards per table DM2 as follows: Maximum number of cache shards for a single table DM2 = Maximum number of cache shards for a single table DM2 – NJ Where NJ is the preset decreasing step size.

6. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 1, characterized in that: It also includes configuring multiple data sources, and the data sources include: data source name, database type, database corresponding driver, address, port, user name and password.

7. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 6, characterized in that: The establishment of the extraction plan includes: establishing the name of the extraction plan, selecting the source data source and selecting the target data source; The establishment of the extraction task includes: setting the task name, the method for obtaining the extraction source data and the method for writing the target data.

8. The multi-threaded data cache read-write parallel high-performance data extraction method according to claim 1, characterized in that: The calculation method of the number of idle CPU cores is: number of idle CPU cores = number of CPU cores - total number of read threads - total number of write threads - number of system operation interaction cores.

9. A multi-threaded data cache read and write parallel high-performance data extraction system, characterized in that: The multi-threaded data cache read-write parallel high-performance data extraction system comprises: The extraction plan module is used to establish an extraction plan and to establish one or more extraction tasks for each extraction plan; The extraction processing setting module is used to set each extraction task in the extraction plan to be executed before extraction or after extraction; The plan execution module is used to trigger the execution of an established extraction plan; The task execution module is used to calculate the maximum number of parallel tasks according to the number of CPU cores, and start the extraction tasks in the triggered extraction plan based on the maximum number of parallel tasks. Each extraction task initially starts with 1 read thread and N write threads; The task monitoring module is used to monitor the started extraction tasks. If the number of times the read thread in the extraction task waits continuously for the write cache is greater than the waiting threshold, and the number of idle CPU cores is not zero, a new write thread for the extraction task is started, the number of times the read thread waits continuously for the write cache is cleared, and continuous monitoring is carried out until the extraction task is completed.

Images