A data migration method and related apparatus
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- AGRICULTURAL BANK OF CHINA
- Filing Date
- 2022-10-19
- Publication Date
- 2026-07-07
AI Technical Summary
When migrating large amounts of data in an HBase cluster, using the put statement can cause memory spikes, affecting the stability of the target cluster's services. Existing technologies struggle to control the migration speed.
The data migration system dynamically adjusts the number of threads, combines flow control and data filtering, uses message queues to store data to be migrated, controls the data write speed, and ensures that the source cluster business is not affected.
Without affecting the source cluster's operations, improve data migration speed, alleviate storage pressure, and ensure the stability and data consistency of the target cluster.
Smart Images
Figure CN115480711B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a data migration method and related apparatus. Background Technology
[0002] HBase is a distributed, column-oriented open-source database. As a distributed storage system, HBase supports the storage of massive amounts of data. With the widespread application of big data, the need to migrate data stored in one cluster to other clusters is increasing. Due to the extremely large volume of data, data migration is particularly difficult.
[0003] In related technologies, the CopyTable statement is used to copy data from a table. This method of data migration is simple to perform, can be done without interrupting business operations, and supports data filtering and migration.
[0004] However, when the target cluster uses the HBase put statement to write data, the put statement first writes to HBase memory and then flushes it to disk. In the case of large-scale data migration and the inability to control the data migration speed, this can cause the HBase target cluster memory to spike, which will have a disastrous impact on the target cluster's business. Summary of the Invention
[0005] To address the aforementioned issues, this application provides a data migration method and related apparatus, which dynamically adjusts the data migration speed according to business conditions to avoid the impact of data migration on the source cluster business.
[0006] Based on this, the embodiments of this application disclose the following technical solutions:
[0007] On one hand, embodiments of this application provide a data migration system, which includes a data migration unit, and the data migration unit includes a thread manager, multiple data readers, and multiple data writers;
[0008] The thread manager is used to obtain the disk bandwidth of the source cluster; determine the number of threads to perform data migration based on the disk bandwidth, and create one or more threads based on the number of threads; wherein, the higher the disk bandwidth, the fewer the number of threads, and one thread corresponds to one data reader and one data writer;
[0009] The target data reader corresponding to the target thread is used to obtain the target row key from the message queue; obtain the target data to be migrated corresponding to the target row key from the source cluster according to the target row key, and send the target data to be migrated to the target data writer corresponding to the target thread; wherein, the message queue stores the row keys of the data to be migrated in the source cluster, and the target thread is one of the plurality of threads;
[0010] The target data writer is used to store the target data to be migrated into the target cluster.
[0011] Optionally, the data migration unit may further include multiple flow controllers;
[0012] The target data reader is specifically used to send the target data to be migrated to the target traffic controller corresponding to the target thread.
[0013] The target traffic controller is used to obtain the target traffic bandwidth of the target data to be migrated in the i-th time window; if the target traffic bandwidth is greater than the preset traffic bandwidth of the i-th time window, the target data to be migrated is sent to the target data writer after waiting for the target time.
[0014] Optionally, the target flow controller is further configured to:
[0015] The target time is determined based on the target bandwidth, the preset bandwidth of the i-th time window, and the duration of the i-th time window.
[0016] Optionally, the target flow controller is further configured to:
[0017] If the target traffic bandwidth is less than or equal to the preset traffic bandwidth, the preset traffic bandwidth of the (i+1)th time window is increased so that the target traffic controller can receive more of the target data to be migrated in the (i+1)th time window.
[0018] Optionally, the target flow controller is further configured to:
[0019] The preset traffic bandwidth for the (i+1)th time window is determined based on the target traffic bandwidth and the preset traffic bandwidth for the i-th time window.
[0020] Optionally, the data migration system further includes a data filtering unit, which includes a filtering filter and a filtering controller;
[0021] The filtering filter is used to obtain migration filtering conditions input by the user; generate the filtering rules required by HBase based on the migration filtering conditions; and send the filtering rules to the filtering controller.
[0022] The filtering controller is used to filter the data included in the source cluster according to the filtering rules to obtain the row key of the data to be migrated; and to store the row key of the data to be migrated into the message queue.
[0023] Optionally, the data migration system further includes a data inspection unit, which includes a first data reader, a second data reader, and a data comparator;
[0024] The first data reader is configured to obtain first data to be inspected from the source cluster based on the row key in the row key table, and send the first data to be inspected to the data comparator;
[0025] The second data reader is used to obtain the second data to be inspected from the target cluster according to the row key in the row key table, and send the second data to be inspected to the data comparator;
[0026] The data comparator is used to compare whether the first data to be checked and the second data to be checked are consistent based on the row key; if the first data to be checked and the second data to be checked are inconsistent, the row key of the first data to be checked is sent to the message queue.
[0027] On the other hand, this application provides a data migration method applied to a data migration system, the data migration system including a data migration unit, the data migration unit including a thread manager, at least one data reader and at least one data writer, the method including:
[0028] The thread manager obtains the disk bandwidth of the source cluster;
[0029] The thread manager determines the number of threads used to perform data migration based on the disk bandwidth; wherein, the higher the disk bandwidth, the fewer the number of threads.
[0030] The thread manager creates multiple threads based on the number of threads, and the number of multiple threads is equal to the number of threads. One thread corresponds to one data reader and one data writer.
[0031] The target data reader corresponding to the target thread obtains the target row key from the message queue, which stores the row keys of the data to be migrated in the source cluster;
[0032] The target data reader obtains the target data to be migrated corresponding to the target row key from the source cluster based on the target row key;
[0033] The target data reader sends the target data to be migrated to the target data writer corresponding to the target thread;
[0034] The target data writer stores the target data to be migrated into the target cluster.
[0035] On the other hand, this application provides a computer device, the device including a processor and a memory:
[0036] The memory is used to store program code and transmit the program code to the processor;
[0037] The processor is configured to execute the methods described above according to instructions in the program code.
[0038] On the other hand, this application provides a computer-readable storage medium for storing a computer program for performing the methods described above.
[0039] On the other hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described above.
[0040] The advantages of the above-mentioned technical solution in this application are:
[0041] As described in the above technical solution, the data migration system includes a data migration unit, which comprises a thread manager, at least one data reader, and at least one data writer. The thread manager obtains the disk bandwidth of the source cluster, determines the number of threads to perform the data migration based on the disk bandwidth, and creates one or more threads accordingly. Higher disk bandwidth requires fewer threads, thus dynamically adjusting the number of threads based on disk bandwidth to maximize data migration speed without affecting the business processing of the source cluster. The target thread corresponds to the target data reader and the target data writer. The target data reader retrieves the target row key from the message queue, obtains the target data to be migrated corresponding to the target row key from the source cluster based on the target row key, and sends the target data to be migrated to the target data writer. The target data writer stores the target data to be migrated in the target cluster. Since bank data is generally unstructured, each data entry occupies a significant amount of storage space. To alleviate storage pressure, the industry data to be migrated is stored in the message queue so that the target data reader can retrieve the target data to be migrated corresponding to the target row key from the source cluster based on the target row key. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 A schematic diagram of a data migration system provided in an embodiment of this application;
[0044] Figure 2 A schematic diagram of a data filtering unit provided in an embodiment of this application;
[0045] Figure 3 A schematic diagram of a data inspection unit provided in an embodiment of this application;
[0046] Figure 4 A flowchart illustrating a data migration method provided in this application embodiment;
[0047] Figure 5 This is a structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation
[0048] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.
[0049] The following is combined with Figure 1 This application describes a data migration system provided in an embodiment. See also... Figure 1 This figure is a schematic diagram of a data migration system provided in an embodiment of this application. The data migration system may include a data migration unit 100, which includes a thread manager 110, multiple data readers 120, and multiple data writers 130. These will be described in detail below.
[0050] Thread manager 110 is used to obtain the disk bandwidth of the source cluster 200; determine the number of threads to perform data migration based on the disk bandwidth, and create multiple threads based on the number of threads.
[0051] Because the source cluster handles different tasks at different times—more during the day and less at night—its disk bandwidth varies. To avoid excessive disk bandwidth consumption during data migration, which could impact the source cluster's processing, the thread manager obtains the source cluster's disk bandwidth and determines the number of threads needed to perform the data migration based on that bandwidth. Multiple threads are then created accordingly.
[0052] It's important to note that higher disk bandwidth in the source cluster results in fewer threads used for data migration, reducing the disk bandwidth consumed and thus minimizing its impact on the source cluster's operations. Conversely, lower disk bandwidth in the source cluster leads to more threads used for data migration, increasing the disk bandwidth consumed and thus improving migration speed. Therefore, by dynamically adjusting the number of threads based on disk bandwidth, data migration speed can be maximized without affecting the source cluster's business processes.
[0053] In this context, one thread corresponds to one data reader and one data writer, such as... Figure 1 It includes 4 data readers and 4 data writers, which can be used to build 4 threads. Taking the target thread among multiple threads as an example, after a target thread is created, a data reader will be selected from multiple data readers as the target data reader, and a data writer will be selected from multiple data writers as the target data writer. Thus, the target thread corresponds to one target data reader and one target data writer.
[0054] After creating a number of threads equal to the number of threads required, each thread can migrate data from the source cluster to the target cluster. The following explanation uses one of these threads, the target thread, as an example. The target thread corresponds to one target data reader and one target data writer. The target data reader is one of multiple data readers, and the target data writer is one of multiple data writers.
[0055] The target data reader is used to retrieve the target row key from message queue 400; retrieve the target data to be migrated corresponding to the target row key from source cluster 200 based on the target row key, and send the target data to be migrated to the target data writer.
[0056] The target data writer is used to store the target data to be migrated into the target cluster 300.
[0057] The message queue stores the row keys of the data to be migrated from the source cluster. Message queues are crucial components in distributed systems, storing data in a queue format. Messages are inserted from one side of the queue (producers) and read from the other side (consumers). This primarily addresses issues such as application coupling, asynchronous processing, and traffic shaping. In HBase, row keys are used to retrieve unique records in the table, with a one-to-one correspondence between row keys and stored data.
[0058] It should be noted that since bank data is generally unstructured, each piece of data occupies a lot of storage space. In order to alleviate storage pressure, the industry of the data to be migrated is stored in a message queue so that the target data reader can retrieve the target data to be migrated corresponding to the target row key from the source cluster based on the target row key. The target data to be migrated belongs to the data to be migrated and corresponds to the target industry.
[0059] It should be noted that both the source cluster and the target cluster are HBase clusters.
[0060] As described in the above technical solution, the data migration system includes a data migration unit, which comprises a thread manager, at least one data reader, and at least one data writer. The thread manager obtains the disk bandwidth of the source cluster, determines the number of threads to perform the data migration based on the disk bandwidth, and creates one or more threads accordingly. Higher disk bandwidth requires fewer threads, thus dynamically adjusting the number of threads based on disk bandwidth to maximize data migration speed without affecting the business processing of the source cluster. The target thread corresponds to the target data reader and the target data writer. The target data reader retrieves the target row key from the message queue, obtains the target data to be migrated corresponding to the target row key from the source cluster based on the target row key, and sends the target data to be migrated to the target data writer. The target data writer stores the target data to be migrated in the target cluster. Since bank data is generally unstructured, each data entry occupies a significant amount of storage space. To alleviate storage pressure, the industry data to be migrated is stored in the message queue so that the target data reader can retrieve the target data to be migrated corresponding to the target row key from the source cluster based on the target row key.
[0061] As one possible implementation, the data migration unit also includes multiple flow controllers, where one thread corresponds not only to one data reader and one data writer, but also to one flow controller. The following section combines... Figure 1 Let's continue with the explanation.
[0062] exist Figure 1 It includes four flow controllers (140), each corresponding to one thread. Continuing with the target thread as an example, the target thread not only corresponds to a target data reader and a target data writer, but also includes a target flow controller. The target data to be migrated is sent to the target data writer through the target flow controller. The target flow controller is used to control the flow of the target data during the migration process, that is, to control the write speed of the target data writer and the read speed of the target data reader. This will be explained in detail below.
[0063] After the target data reader sends the target data to be migrated to the target traffic controller, the target traffic controller obtains the target traffic bandwidth of the target data to be migrated in the i-th time window. If the target traffic bandwidth is greater than the preset traffic bandwidth of the i-th time window, the target data to be migrated is sent to the target data writer after waiting for the target time. This ensures that the traffic bandwidth in each time window is within a certain range, avoiding excessive peak traffic.
[0064] This application does not specifically limit the size of the target time. It can be set to a fixed value or a dynamically changing value. This application does not impose specific limitations on this, and those skilled in the art can set it according to actual needs. For example, the target time can be determined based on the target traffic bandwidth, the preset traffic bandwidth of the i-th time window, and the duration of the i-th time window.
[0065] For example, the target flow controller uses a fixed-time-window algorithm for flow control. Specifically, if the target flow bandwidth passing through the i-th time window T is P′, and assuming the preset flow bandwidth for the i-th time window is P, if P′ > P, then the current thread will no longer perform data migration, but will wait for the target time before sending the target data to be migrated to the target data writer. The target time can be expressed as follows:
[0066]
[0067] Therefore, when the target traffic bandwidth P′ is much larger than the preset traffic bandwidth P of the i-th time window, the target time can be made larger, and when the target traffic bandwidth P′ is much smaller than the preset traffic bandwidth P of the i-th time window, the target time can be made smaller, thereby dynamically adjusting the traffic during the migration process according to the actual situation of the target thread.
[0068] It should be noted that a preset traffic bandwidth can be set as the preset traffic bandwidth for the i-th time window. The embodiments of this application do not specifically limit the size of the preset traffic bandwidth, and those skilled in the art can set it according to actual needs.
[0069] As one possible implementation, if the target traffic bandwidth is less than or equal to the preset traffic bandwidth, the preset traffic bandwidth of the (i+1)th time window is increased so that the target traffic controller can receive more target data to be migrated in the (i+1)th time window.
[0070] For example, according to the target traffic bandwidth and the preset traffic bandwidth of the i-th time window, determine the preset traffic bandwidth of the (i + 1)-th time window. Continuing with the previous example, if the target traffic bandwidth P' is less than the preset traffic bandwidth P of the i-th time window, that is, P' < P, then the temporary bandwidth threshold that can pass within the (i + 1)-th time window set by the target thread can be expressed by the following formula:
[0071] P t = 2P - P'.
[0072] Where P t is the preset traffic bandwidth of the (i + 1)-th time window. Thus, by controlling the bandwidth thresholds of each window, the average traffic of the window can be in a balanced state.
[0073] As a possible implementation, the data migration system further includes a data screening unit, which will be described below in conjunction with Figure 2 for illustration.
[0074] Refer to Figure 2 , which is a schematic diagram of a data screening unit provided by an embodiment of the present application. The data screening unit includes a screening filter 210 and a screening controller 220.
[0075] Among them, the screening filter 210 is used to obtain the migration screening conditions input by the user; generate the screening rules required by HBase according to the migration screening conditions, and send the screening rules to the screening controller 220;
[0076] The screening controller 220 is used to screen the data included in the source cluster 200 according to the screening rules to obtain the row keys of the data to be migrated; store the row keys of the data to be migrated in the message queue 400.
[0077] Thus, the screening filter is used to accept the migration screening conditions input by the user, and according to the migration screening conditions input by the user, combine them into the screening rules required by HBase, and use the screening rules for the next HBase screening. The screening controller uses the MapReduce computing task to perform the scan of the HBase original cluster according to the screening rules of the previous step, and at the same time controls the number of Map (thread) tasks for the scan to avoid excessive business impact on the original cluster, and sends the screened qualified data row keys to the message queue.
[0078] As a possible implementation, the data migration system further includes a data inspection unit, which is used to check the consistency of the migrated data in the HBase source cluster and the HBase target cluster after the data migration is completed. It will be described below in conjunction with Figure 3 for illustration.
[0079] Refer to Figure 3The figure is a schematic diagram of a data inspection unit provided in an embodiment of this application. The data inspection unit includes a first data reader 310, a second data reader 320, and a data comparator 330.
[0080] The first data reader 310 is used to retrieve the first data to be checked from the source cluster based on the row key in the row key table, and send the first data to be checked to the data comparator 330. The row key table is constructed based on the row keys stored in the message queue.
[0081] The second data reader 320 is used to obtain the second data to be checked from the target cluster according to the row key in the row key table, and send the second data to be checked to the data comparator 330.
[0082] Data comparator 330 is used to compare whether the first data to be checked and the second data to be checked are consistent based on the row key; if the first data to be checked and the second data to be checked are inconsistent, the row key of the first data to be checked is sent to the message queue.
[0083] For example, the data comparator compares the column family, column identifier, and timestamp corresponding to each row key one by one. If the comparison is successful, a success result is returned; if inconsistent data is found, a failure result is returned, and the inconsistent row keys are stored in HDFS for later verification. HDFS is a distributed file storage system; HBase data is stored in HDFS in the form of HFiles.
[0084] In addition to the data migration system provided in this application, a data migration method is also provided, which is applied to the aforementioned... Figures 1-3 The data migration system shown is as follows: Figure 4 As shown, the data migration includes:
[0085] S401: The thread manager obtains the disk bandwidth of the source cluster.
[0086] S402: The thread manager determines the number of threads to perform data migration based on disk bandwidth.
[0087] Among these, the higher the disk bandwidth, the fewer the number of threads;
[0088] S403: The thread manager creates multiple threads based on the number of threads.
[0089] The number of multiple threads is equal to the number of threads, and one thread corresponds to one data reader and one data writer.
[0090] S404: The target data reader corresponding to the target thread retrieves the target row key from the message queue.
[0091] The target thread is one of multiple threads, and the message queue stores the row keys of the data to be migrated from the source cluster.
[0092] S405: The target data reader retrieves the target data to be migrated from the source cluster based on the target row key.
[0093] S406: The target data reader sends the target data to be migrated to the target data writer corresponding to the target thread.
[0094] S407: The target data writer stores the target data to be migrated into the target cluster.
[0095] As described in the above technical solution, the data migration system includes a data migration unit, which comprises a thread manager, at least one data reader, and at least one data writer. The thread manager obtains the disk bandwidth of the source cluster, determines the number of threads to perform the data migration based on the disk bandwidth, and creates one or more threads accordingly. Higher disk bandwidth requires fewer threads, thus dynamically adjusting the number of threads based on disk bandwidth to maximize data migration speed without affecting the business processing of the source cluster. The target thread corresponds to the target data reader and the target data writer. The target data reader retrieves the target row key from the message queue, obtains the target data to be migrated corresponding to the target row key from the source cluster based on the target row key, and sends the target data to be migrated to the target data writer. The target data writer stores the target data to be migrated in the target cluster. Since bank data is generally unstructured, each data entry occupies a significant amount of storage space. To alleviate storage pressure, the industry data to be migrated is stored in the message queue so that the target data reader can retrieve the target data to be migrated corresponding to the target row key from the source cluster based on the target row key.
[0096] As one possible implementation, the data migration unit further includes multiple flow controllers, wherein the target data reader sends the target data to be migrated to the target data writer corresponding to the target thread, including:
[0097] The target data reader sends the target data to be migrated to the target flow controller corresponding to the target thread;
[0098] The target traffic controller obtains the target traffic bandwidth of the target data to be migrated in the i-th time window; if the target traffic bandwidth is greater than the preset traffic bandwidth of the i-th time window, it waits for the target time and then sends the target data to be migrated to the target data writer.
[0099] As one possible implementation, the method further includes:
[0100] The target flow controller determines the target time based on the target flow bandwidth, the preset flow bandwidth of the i-th time window, and the duration of the i-th time window.
[0101] As one possible implementation, the method further includes:
[0102] If the target traffic bandwidth is less than or equal to the preset traffic bandwidth, the target traffic controller increases the preset traffic bandwidth in the (i+1)th time window so that the target traffic controller can receive more target data to be migrated in the (i+1)th time window.
[0103] As one possible implementation, the method further includes:
[0104] The target flow controller determines the preset flow bandwidth for the (i+1)th time window based on the target flow bandwidth and the preset flow bandwidth for the i-th time window.
[0105] As one possible implementation, the data migration system further includes a data filtering unit, which includes a filtering filter and a filtering controller; the method further includes:
[0106] The filter obtains the migration filtering conditions input by the user; generates the filtering rules required by HBase based on the migration filtering conditions, and sends the filtering rules to the filtering controller;
[0107] The filtering controller filters the data in the source cluster according to the filtering rules to obtain the row key of the data to be migrated; and stores the row key of the data to be migrated in the message queue.
[0108] As one possible implementation, the data migration system further includes a data inspection unit, which comprises a first data reader, a second data reader, and a data comparator; the method further includes:
[0109] The first data reader retrieves the first data to be checked from the source cluster based on the row key in the row key table, and sends the first data to be checked to the data comparator.
[0110] The second data reader retrieves the second data to be checked from the target cluster based on the row key in the row key table, and sends the second data to be checked to the data comparator.
[0111] The data comparator compares the first data to be checked with the second data to be checked based on the row key. If the first data to be checked and the second data to be checked are inconsistent, the row key of the first data to be checked is sent to the message queue.
[0112] This application also provides a computer device, see [link to relevant documentation] Figure 5 The figure illustrates a structural diagram of a computer device provided in an embodiment of this application, such as... Figure 5 As shown, the device includes a processor 5310 and a memory 520:
[0113] The memory 510 is used to store program code and transmit the program code to the processor;
[0114] The processor 520 is used to execute any of the data migration methods provided in the above embodiments according to the instructions in the program code.
[0115] This application provides a computer-readable storage medium for storing a computer program that executes any of the data migration methods provided in the above embodiments.
[0116] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data migration method provided in various optional implementations of the above aspects.
[0117] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.
[0118] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0119] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0120] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0121] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data migration system, characterized in that, The data migration system includes a data migration unit, which includes a thread manager, multiple data readers, and multiple data writers. The thread manager is used to obtain the disk bandwidth of the source cluster; The number of threads used to perform data migration is determined based on the disk bandwidth, and one or more threads are created based on the number of threads; wherein, the higher the disk bandwidth, the fewer the number of threads, and one thread corresponds to one data reader and one data writer; The target data reader corresponding to the target thread is used to obtain the target row key from the message queue; obtain the target data to be migrated corresponding to the target row key from the source cluster according to the target row key, and send the target data to be migrated to the target data writer corresponding to the target thread; wherein, the message queue stores the row keys of the data to be migrated in the source cluster, and the target thread is one of the plurality of threads; The target data writer is used to store the target data to be migrated into the target cluster; The data migration unit further includes multiple flow controllers; The target data reader is specifically used to send the target data to be migrated to the target traffic controller corresponding to the target thread. The target traffic controller is used to obtain the target traffic bandwidth of the target data to be migrated in the i-th time window; if the target traffic bandwidth is greater than the preset traffic bandwidth of the i-th time window, the target data to be migrated is sent to the target data writer after waiting for the target time. The target flow controller is also used for: The target time is determined based on the target bandwidth, the preset bandwidth of the i-th time window, and the duration of the i-th time window; The target flow controller is also used for: If the target traffic bandwidth is less than or equal to the preset traffic bandwidth, the preset traffic bandwidth of the (i+1)th time window is increased so that the target traffic controller can receive more of the target data to be migrated in the (i+1)th time window. The target flow controller is also used for: Based on the target bandwidth and the preset bandwidth of the i-th time window, the preset bandwidth of the (i+1)-th time window is determined; the target time is... P is the preset bandwidth for the i-th time window. The target traffic bandwidth passing through the i-th time window T; the preset traffic bandwidth for the (i+1)-th time window is... .
2. The system according to claim 1, characterized in that, The data migration system also includes a data filtering unit, which includes a filtering filter and a filtering controller. The filtering filter is used to obtain migration filtering conditions input by the user; generate the filtering rules required by HBase based on the migration filtering conditions; and send the filtering rules to the filtering controller. The filtering controller is used to filter the data included in the source cluster according to the filtering rules to obtain the row key of the data to be migrated; and to store the row key of the data to be migrated into the message queue.
3. The system according to claim 1, characterized in that, The data migration system further includes a data inspection unit, which includes a first data reader, a second data reader, and a data comparator. The first data reader is configured to obtain first data to be inspected from the source cluster based on the row key in the row key table, and send the first data to be inspected to the data comparator; The second data reader is used to obtain the second data to be inspected from the target cluster according to the row key in the row key table, and send the second data to be inspected to the data comparator; The data comparator is used to compare whether the first data to be checked and the second data to be checked are consistent based on the row key; if the first data to be checked and the second data to be checked are inconsistent, the row key of the first data to be checked is sent to the message queue.
4. A data migration method, characterized in that, The method is applied to a data migration system, the data migration system including a data migration unit, the data migration unit including a thread manager, at least one data reader, and at least one data writer, the method comprising: The thread manager obtains the disk bandwidth of the source cluster; The thread manager determines the number of threads used to perform data migration based on the disk bandwidth; wherein, the higher the disk bandwidth, the fewer the number of threads. The thread manager creates multiple threads based on the number of threads, and the number of multiple threads is equal to the number of threads. One thread corresponds to one data reader and one data writer. The target data reader corresponding to the target thread obtains the target row key from the message queue, which stores the row keys of the data to be migrated in the source cluster; The target data reader obtains the target data to be migrated corresponding to the target row key from the source cluster based on the target row key; The target data reader sends the target data to be migrated to the target data writer corresponding to the target thread; The target data writer stores the target data to be migrated into the target cluster; The data migration unit further includes multiple flow controllers; Send the target data to be migrated to the target traffic controller corresponding to the target thread; Obtain the target traffic bandwidth of the target data to be migrated in the i-th time window; if the target traffic bandwidth is greater than the preset traffic bandwidth of the i-th time window, send the target data to be migrated to the target data writer after waiting for the target time. The target time is determined based on the target bandwidth, the preset bandwidth of the i-th time window, and the duration of the i-th time window; If the target traffic bandwidth is less than or equal to the preset traffic bandwidth, the preset traffic bandwidth of the (i+1)th time window is increased so that the target traffic controller can receive more of the target data to be migrated in the (i+1)th time window. Based on the target bandwidth and the preset bandwidth of the i-th time window, the preset bandwidth of the (i+1)-th time window is determined; the target time is... P is the preset bandwidth for the i-th time window. The target traffic bandwidth passing through the i-th time window T; the preset traffic bandwidth for the (i+1)-th time window is... .
5. A computer device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the method of claim 4 according to the instructions in the program code.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method of claim 4.