A method for data migration

Abstract

The present invention discloses a method for data migration, which adopts a Spark computing engine to read and store data, repartitions the data in N partitions, and the data stored in the i-th partition after repartitioning are simultaneously and respectively input into the i-th buffer, and each buffer is implemented based on a blocking queue. After the repartitioning is completed, two threads are started for each partition, a producer thread and a consumer thread, and the producer thread traverses each piece of data in each partition and writes it into a blocking queue, and dynamically modifies the speed threshold according to the real-time use of database resources, and then controls the speed at which the producer thread writes data into the blocking queue according to the threshold, and while the producer thread writes the data into the blocking queue, the consumer thread obtains the data from the blocking queue and writes it into the database, thereby completing data synchronization. The present invention can improve the writing speed, thereby improving the migration speed.

Description

Technical Field

[0001] The invention relates to a data migration method and belongs to the technical field of big data processing. Background Art

[0002] The purpose of data migration is to match a more suitable storage environment for valuable data, so that it can serve customers more safely, reliably and effectively at every stage of its life cycle. All data relocation processes can be broadly referred to as data migration. Data must go through a life cycle of production, transmission, calculation, storage, archiving and destruction. Similarly, data-related devices need to cooperate with data to realize their value. The development of the Internet industry requires manufacturers to provide better data portability and interoperability.

[0003] The patent application with application number 201711158991.6 discloses a method and device for data migration, which relates to the field of e-commerce. The patent application loads a data migration component and reads the configuration information recorded in the configuration file; extracts the data to be migrated from the source database and imports it into the memory; runs the data migration logic in the data migration component and determines the target library table according to the configuration information; and allocates the data to be migrated to the target library table. The patent application can improve the efficiency of data migration and reduce costs. The patent application with application number 202110321312.2 discloses a data migration method, device, storage medium and platform, which relates to the field of big data processing technology. The method is applied to a distributed big data migration platform, including: loading the data to be migrated in the source database into the Hive data warehouse of the distributed big data migration platform; in the Hive data warehouse, performing data conversion on the data to be migrated by the Spark engine to generate target data; and migrating the target data from the Hive data warehouse to the target database. The patent application can quickly and efficiently migrate data from the source database to the target database, reducing the impact on system services during data migration.

[0004] Although the above two existing patent applications can effectively improve the efficiency of data migration, the impact of data migration on the database needs to be considered in the actual data migration process. In actual application scenarios, especially when using Spark (Spark refers to Apache Spark, which is a fast and general computing engine designed for large-scale data processing) for data migration, due to the long-term concurrent writing of large amounts of data, if it is not controlled, a large amount of IO resources of the database will be occupied for a long time, which will affect the performance of the database, resulting in a longer response time for database-related applications, thereby affecting the user experience, and even causing database downtime in severe cases. For this reason, data migration is generally carried out in a staggered manner, such as migrating data when the application system is not busy, such as in the early morning. However, this will cause the timeliness of the data to deteriorate and the migration efficiency to decrease. In addition, if it is found that the database performance is greatly affected during the migration process, in order not to affect the application system related to the database, the data migration task must be forcibly interrupted and then re-migrated, because some data has been migrated to the database, which will also lead to data inconsistency. Summary of the invention

[0005] The purpose of the present invention is to provide a method for data migration. During the data migration process, by real-time observation of database resource usage, different data transmission speeds are set according to different database resource usages; during data migration using the Spark computing engine, if it is found that the actual transmission speed of the current data is higher than the set speed, the writing speed is reduced; if it is found that the actual transmission speed of the current data is lower than the set speed, the writing speed is automatically increased to maintain the stability of the database and reduce the impact of data migration, so as to solve the problem of data inconsistency described in the background technology. This method can make full use of database resources, increase the writing speed in scenarios where the database resource usage rate is relatively low, thereby increasing the migration speed.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A method for data migration, which reads data using the Spark computing engine and stores it. The logical structure for storing data inside Spark is Rdd, and Rdd includes the 1st to the Nth partitions; repartition the data in the N partitions, and after repartitioning, the data stored in the ith partition is simultaneously input into the ith buffer respectively. Each buffer is implemented based on a blocking queue, where i = 1, 2, …, N; after repartitioning is completed, two threads are started for each partition, a producer thread and a consumer thread. The producer thread traverses each piece of data in each partition and writes it into the blocking queue, and dynamically modifies the threshold of the writing speed according to the real-time usage of database resources, and then controls the speed at which the producer thread writes data into the blocking queue according to the threshold. While the producer thread writes data into the blocking queue, the consumer thread reads data from the blocking queue and writes it into the database, thus completing data synchronization; the method for data migration includes the following steps:

[0008] Step 1) Traverse each piece of data in the ith partition simultaneously and add it to the ith set L i . If the number of data in L i reaches the preset number, or although the number of data in L i does not reach the preset number but the data in the ith partition has been traversed, then execute Step 2);

[0009] Step 2) The consumer thread and the producer thread proceed simultaneously. The consumer thread reads data from the blocking queue in real time and writes it into the ith database;

[0010] After the producer thread writes the data in L i into the blocking queue for the kth time, calculate the size △C k of the data written into the blocking queue for the kth time;

[0011] Calculate cp + △C k in real time. cp is the size of the data that has not been read by the consumer thread in the current blocking queue. When the consumer thread reads one piece of data from the blocking queue, update cp to cp = cp - size, where size is the size of the data read from the blocking queue each time; if cp + △C k > capacity, where capacity is the capacity of the blocking queue, the producer thread will be blocked; until cp + △C k < capacity, the data in L i will be written into the blocking queue. The total size C k of the data accumulated and written into the blocking queue at the kth time is C k = △C k-1 + C k-1is the total size of the data accumulated in the blocking queue at the (k - 1)-th time;

[0012] Step 3) Update the speed threshold speed after the k-th write to the blocking queue k , the method is as follows: Obtain the time t when the k-th producer thread finishes writing data to the blocking queue k , calculate the time interval1 = t k - TT from the time of the last query of the database's IO utilization rate (the IO utilization rate is the percentage of the sum of the disk processing read and write times within a certain period of time), where TT is the time of the last query of the database's IO utilization rate, and the initial value of TT is the start time of the producer thread;

[0013] If interval1 >= tt, then set TT = t k , where tt is the preset time interval for querying the database's IO utilization rate;

[0014] Obtain the database's IO utilization rate rate. If Y ≥ rate ≥ X, then speed k = speed k-1 ; If rate > Y, adjust the number of times N1 for the producer thread to decrease the data writing speed to the blocking queue by N1 = N1 + 1, the number of times N2 for the producer thread to increase the data writing speed to the blocking queue by N2 = 0, and speed k = speed k-1 - Z N1 * speed k-1 , where Z is in [0 - 1]; If speed k < minSpeed, then speed k = minSpeed; If rate < X, adjust N2 = N2 + 1, N1 = 0, then speed k = speed k-1 + Z N2 * speed k-1 , if speed k > maxSpeed, then speed k = maxSpeed; where X is the lower limit range of the IO utilization rate rate, Y is the upper limit range of the IO utilization rate rate, X is in [0 - 40], Y is in [60 - 100], minSpeed is the preset minimum speed for the producer thread to write data to the blocking queue, and maxSpeed is the preset maximum speed for the producer thread to write data to the blocking queue;

[0015] If interval1 < tt, then speed k = speed k-1 ;

[0016] Step 4) Perform speed measurement and calculate the speed measurement time interval interval2 = t k -T, if interval2>t, then go to step 5); otherwise k=k+1, then go to step 1); where T is the last speed measurement time, its initial value is the time when the producer thread is started, and t is the preset speed measurement time interval;

[0017] Step 5) Calculate the current actual writing speed: speed = (C k -C) / interval2, if speed>speed k , then go to step 6), otherwise go to step 7); where C is the size of the data accumulated in the blocking queue during the last speed measurement;

[0018] Step 6) Calculate the rest time st of the producer thread, st = speed*interval2 / speed k -interval2; if st is greater than 0, the producer thread starts to rest and stops writing data to the blocking queue. After st, the producer thread stops resting and continues writing data to the blocking queue, and goes to step 7);

[0019] Step 7) Set C = C k , T = t k , k=k+1, go to step 1);

[0020] If the data of each partition is written into the blocking queue by the producer thread, and the data in the blocking queue is all read by the consumer thread and written into the database, the entire data migration task is completed.

[0021] The purpose of the present invention can also be further achieved by the following technical measures:

[0022] Preferably, the data in the N partitions are repartitioned, and the algorithm used is Hash followed by modulus, so that the data of the original one partition is dispersed into multiple partitions.

[0023] Preferably, in step 3), X is 40, Y is 60, and Z is 0.5.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention dynamically modifies the speed threshold according to the real-time usage of database resources, and then controls the speed at which the producer thread writes data into the blocking queue according to the threshold. While the producer thread writes data into the blocking queue, the consumer thread obtains data from the blocking queue and writes it into the database, thereby completing data synchronization. The present invention maintains the stability of the database so that it is less affected by data migration, and can make full use of the resources of the database. In the scenario where the utilization rate of database resources is relatively low, the writing speed is increased, thereby increasing the migration speed. The present invention solves the problem of data inconsistency that often occurs during data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is the flowchart of how Spark reads data and writes it to the database;

[0026] Figure 2 This is a schematic diagram of the buffer for writing data. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0028] like Figure 1 As shown, the data migration method of the present invention adopts the Spark computing engine to read and store data. The logical structure of data stored in Spark is Rdd, and Rdd includes the 1st to Nth partitions; the data in the N partitions are repartitioned, and the data stored in the i-th partition after repartitioning are simultaneously and respectively input into the i-th buffer, and each buffer is implemented based on a blocking queue. After the repartitioning is completed, two threads will be started for each partition, a producer thread and a consumer thread. The producer thread traverses each data in each partition and writes it into the blocking queue, and dynamically modifies the speed threshold according to the real-time usage of database resources, and then controls the speed at which the producer thread writes data into the blocking queue according to the threshold. While the producer thread writes the data into the blocking queue, the consumer thread obtains the data from the blocking queue and writes it into the database, thereby completing the data synchronization; specifically as follows:

[0029] 1. Spark reads data from the Hive data warehouse. Hive is a data warehouse tool based on Hadoop. It is used to extract, transform, and load data. It is a mechanism that can store, query, and analyze large-scale data stored in Hadoop. The logical structure of Spark's internal data storage is Rdd. Rdd (Resilient Distributed Dataset) is called a resilient distributed dataset. It is the most basic data abstraction in Spark. It represents an immutable, partitionable collection whose elements can be calculated in parallel.

[0030] 2. Rdd consists of N partitions, each partition stores a part of the data. Generally, a partition corresponds to a physical data file. There is no rule for the data in each partition, so the data needs to be repartitioned according to business rules (different scenarios have different rules, and the rules depend on the business scenario) so that the partition data corresponds one-to-one with the database table.

[0031] 3. Repartitioning: Repartition the data according to business rules. Assume that the data in the Hive data warehouse consists of 100 files, and one file corresponds to one partition of Spark. After Spark reads, 100 partitions will be formed. Assume that the data in the Hive data warehouse needs to be migrated to 4 database tables according to the rules, and the data is split after the hash value of a certain field is modulo 4. Hash is a hash value, which transforms an input of any length (also called pre-image) into an output of fixed length through a hash algorithm. The output is the hash value hash. Therefore, the new partition ID = hash (the value of one or some fields in the data) modulo 4, and the partition IDs of 0, 1, 2, and 3 are obtained. The data of the original 100 partitions can be repartitioned into the new 4 partitions. That is, for the original 100 partitions, if the hash modulo value of each data in each partition is 0, it will be divided into the new partition 0, and the hash modulo value of 1 will be divided into the new partition 1, and so on. The data in the repartitioned partitions will correspond to the table one by one.

[0032] 4. After the repartitioning is completed, two threads will be started for each partition, one producer thread and one consumer thread. To increase the writing speed, the producer writes the repartitioned data into the buffer in batches of 1,000 records each, while the other consumer thread simultaneously reads the data in the buffer and writes the data into the table corresponding to the partition;

[0033] The buffer is implemented by blocking queues, such as Figure 2 As shown, the blocking queue has the following characteristics:

[0034] (1) Data is first-in, first-out;

[0035] (2) Data will be inserted into the tail of the blocking queue. Data is read from the head of the blocking queue. The data at the head of the blocking queue represents the longest time spent in the blocking queue, and the data at the tail of the blocking queue represents the shortest time spent in the blocking queue.

[0036] (3) The blocking queue will set a fixed capacity. The producer inserts data into the blocking queue, and the consumer reads data from the blocking queue. When the blocking queue is full, the producer will not be able to continue writing data to the blocking queue, and the producer will enter a waiting state. When the blocking queue is not full, the producer will continue to write data. Similarly, when the blocking queue is empty, the consumer will not be able to read data from the blocking queue and will wait. When there is data in the blocking queue, the consumer will continue to read data.

[0037] The specific data migration process is as follows:

[0038] Set the following variables:

[0039] k: The number of times the producer thread writes data into the blocking queue, initialize k = 0;

[0040] C k : The total size of the data written to the blocking queue by the producer thread for the kth time, initializing C k =0;

[0041] t0: producer thread start time;

[0042] C: The total size of data written into the blocking queue during the last speed measurement, C is initialized to 0;

[0043] T: the time of the last speed measurement, initialized to T = t0;

[0044] TT: The time when the database IO resource usage rate was last queried, initialize TT = t0;

[0045] Batch: the number of predictions, initialize batch = 1000;

[0046] j: traversal times, initialize j=0;

[0047] speed0: initial write speed to the blocking queue, initialized to 4M / sec;

[0048] maxSpeed: The maximum speed at which the producer thread writes data to the blocking queue. Initialize maxSpeed ​​= 8M / sec.

[0049] minSpeed: The minimum speed at which the producer thread writes data to the blocking queue. Initialize minSpeed ​​= 1M / sec.

[0050] cp: The size of the data in the blocking queue that has not been read by the consumer thread. Initialize cp = 0;

[0051] capacity: The capacity of the blocking queue. Initialize capacity = 8M;

[0052] t: The speed measurement time interval. Initialize t = 1 second;

[0053] tt: The time interval for querying the usage rate of database IO resources. Initialize tt = 60s;

[0054] N1: The number of times the writing speed of the producer thread to the blocking queue decreases. Initialize N1 = 0;

[0055] N2: The number of times the writing speed of the producer thread to the blocking queue increases. Initialize N2 = 0;

[0056] L i : A set that stores intermediate data.

[0057] Assume that there are a total of 10,000 pieces of data in the 0th partition.

[0058] Step 1. The producer thread starts to traverse each piece of data in the partition and saves the data to the set L i in. If the number in the set L i reaches batch, then enter the second step;

[0059] Step 2. Write to the blocking queue: If △C k + cp > capacity, △C k is the total size of the data in L i , then the producer thread will enter the waiting state until the consumer thread reads a part of the data from the blocking queue, making △C k + cp < capacity. The producer thread writes the data in L i to the blocking queue, calculates cp = cp + △C k , C = C + △C k .

[0060] Read data from the blocking queue: When the producer thread writes data to the blocking queue, the consumer thread will also read data from the blocking queue at the same time. After each time the consumer thread reads data, it first calculates the size size of the data read this time, writes the data to the database table corresponding to the partition, and calculates cp = cp - size.

[0061] Step 3 is how to dynamically adjust the writing speed of the producer thread to the blocking queue. Initialize N1 to 0. N1 represents the number of times the adjustment speed decreases, and N2 represents the number of times the adjustment speed increases, initialized to 0. There are three scenarios:

[0062] 1) If the IO usage is normal, the speed does not need to be adjusted. k =speed k-1 ;

[0063] 2) If the IO usage is too high, the speed needs to be reduced. Each time it is adjusted, N1=N1+1 and N2=0;

[0064] 3) If the IO utilization rate is too low, the speed needs to be increased, and each adjustment is made to N2=N2+1, while N1=0.

[0065] Step 3 Specific process: Query the database IO usage: The producer thread will L i After the data in is written into the blocking queue, the current system time t is obtained. k ;

[0066] 3.1 If t k -TT>tt, update TT=t k , and query the IO usage rate of the database where the table corresponding to the partition is located;

[0067] 3.1.1 If rate>=40% and rate<=60%, the database performance of the table corresponding to the partition is considered good, speed k =

[0068] speed k-1 ;

[0069] 3.1.2 If rate>60%, then speed k =speed k-1 –0.5 1 *speed k-1 ,0.5 1 is the coefficient of decrease, the first decrease is 0.5 1 , the second time is 0.5 2 , the third time is 0.5 3 The first drop is large, which quickly reduces the database IO usage and keeps the database performance unaffected. As the number of drops increases, the coefficient becomes smaller and smaller, and the drop also becomes smaller and smaller, preventing the speed from dropping too fast.

[0070] 3.1.3 If rate < 40%, then speed k =speed k-1 +0.5 1 *speed k-1 ,0.5 1 is the coefficient of rise, 0.5 for the first rise 1 , the second time is 0.5 2, the third time is 0.5 3 . The first upward amplitude is relatively large. As the number of upward times increases, the coefficient becomes smaller and smaller to prevent the speed from increasing too fast. If the coefficient does not become smaller, as the number of upward times increases, the speed may be too large, resulting in too high an IO usage rate of the database;

[0071] 3.1.4 Enter step 4;

[0072] 3.2 If t k -TT < tt, then enter step 4;

[0073] Step 4. Calculate the actual speed of writing data into the blocking queue:

[0074] If t k -T > t, calculate speed = (C k -C) / (t k -T), where C is the size of the data accumulated in the blocking queue during the previous speed measurement, and update T = t k ;

[0075] 4.1.1 If speed > speed k , calculate the rest time required for the producer thread: Assume speed = 6M / s, speed k = 4M / s, t k -T = 1.5s, then 6 * 1.5 / 4 - 1.5 = 0.75s. Finally, the producer thread needs to rest for 0.75s and will continue to work after 0.75 seconds.

[0076] 4. If t k -T < t, continue to traverse the data in the partition.

[0077] If all the data in the partition has been written into the queue by the producer thread and all the data in the queue has been read by the consumer thread, the task ends.

[0078] Except for the above embodiments, the present invention can also have other embodiments. Any technical solutions formed by equivalent replacement or equivalent transformation fall within the scope of protection required by the present invention.

Claims

1. A method for data migration, characterized in that: The Spark computing engine is used to read and store data. The logical structure of data storage in Spark is Rdd, which includes partitions 1 to N. The data in the N partitions are repartitioned. After repartitioning, the data stored in the i-th partition are simultaneously and respectively input into the i-th buffer. Each buffer is implemented based on a blocking queue, where i=1, 2, ..., N. After the repartitioning is completed, two threads are started for each partition, a producer thread and a consumer thread. The producer thread traverses each piece of data in each partition and writes it into the blocking queue. According to the real-time usage of database resources, the threshold of the writing speed is dynamically modified, and then the speed at which the producer thread writes data into the blocking queue is controlled according to the threshold. When the producer thread writes the data into the blocking queue, the consumer thread reads the data from the blocking queue and writes it into the database, thereby completing data synchronization. The data migration method includes the following steps: Step 1) Traverse each piece of data in the i-th partition at the same time and add it to the i-th set L i In, if L i The number of data in reaches the preset number, or although L i If the number of data in does not reach the preset number but the data in the i-th partition has been traversed, run step 2); Step 2) The consumer thread and the producer thread proceed simultaneously, and the consumer thread reads data from the blocking queue in real time and writes it to the i-th database; The producer thread sets L for the kth time i After the data in is written into the blocking queue, calculate the size of the data written into the blocking queue for the kth time △C k ; Real-time calculation of cp + △C k , where cp is the size of the data that has not been read by the consumer thread in the current blocking queue. When the consumer thread reads one piece of data from the blocking queue each time, cp is updated to cp = cp - size, and size is the size of the data read from the blocking queue each time; if cp + △C k > capacity, where capacity is the capacity of the blocking queue, the producer thread will be blocked; until cp + △C k < capacity, the data in L i will be written into the blocking queue. At the k-th time, the total size C k of the data accumulated and written into the blocking queue is C k = △C k-1 + C k-1 , where C k-1 is the total size of the data accumulated and written into the blocking queue at the (k - 1)-th time; Step 3) Update the speed threshold speed after the kth write to the blocking queue k , the method is as follows: Get the time t when the kth producer thread finishes writing data to the blocking queue k , calculate the time interval1=t since the last time the database IO usage was queried k -TT, TT is the time of the last query of the database IO usage rate, and the initial value of TT is the start time of the producer thread; If interval1>=tt, set TT=t k , tt is the preset time interval for querying the IO usage rate of the database; Obtain the I / O utilization rate rate of the database. If Y ≥ rate ≥ X, then speed k = speed k-1 ; If rate > Y, adjust the number of times N1 for the producer thread to decrease the data writing speed to the blocking queue by N1 = N1 + 1, and the number of times N2 for the producer thread to increase the data writing speed to the blocking queue is N2 = 0, speed k = speed k-1 - Z N1 * speed k-1 , where Z is in [0 - 1]; If speed k < minSpeed, then speed k = minSpeed; If rate < X, adjust N2 = N2 + 1, N1 = 0, then speed k = speed k-1 + Z N2 * speed k-1 , if speed k > maxSpeed, then speed k = maxSpeed; where X is the lower limit range of the I / O utilization rate rate, Y is the upper limit range of the I / O utilization rate rate, X is in [0 - 40], Y is in [60 - 100], minSpeed is the preset minimum speed for the producer thread to write data to the blocking queue, and maxSpeed is the preset maximum speed for the producer thread to write data to the blocking queue; If interval1 < tt, then speed k = speed k-1 ; Step 4) Perform speed measurement and calculate the speed measurement time interval interval2 = t k -T, if interval2>t, then go to step 5); otherwise k=k+1, then go to step 1); where T is the last speed measurement time, its initial value is the time when the producer thread is started, and t is the preset speed measurement time interval; Step 5) Calculate the current actual writing speed: speed = (C k -C) / interval2, if speed>speed k , then go to step 6), otherwise go to step 7); where C is the size of the data accumulated in the blocking queue during the last speed measurement; Step 6) Calculate the rest time st of the producer thread, st = speed*interval2 / speed k -interval2; if st is greater than 0, the producer thread starts to rest and stops writing data to the blocking queue. After st, the producer thread stops resting and continues writing data to the blocking queue, and goes to step 7); Step 7) Set C = C k , T = t k , k=k+1, go to step 1); If the data of each partition is written into the blocking queue by the producer thread, and the data in the blocking queue is read by the consumer thread and written into the database, the entire data migration task is completed; The data in the N partitions are repartitioned using a Hash followed by a modulus algorithm to distribute the data in the original one partition into multiple partitions.

2. A data migration method as claimed in claim 1, characterized in that: In the step 3), X is 40, Y is 60, and Z is 0.5.

Images