Data batch migration method and device, equipment, medium and program product

Abstract

The invention provides a data batch migration method which can be applied to the technical field of big data. The method comprises the following steps: creating a thread pool for data batch migration by utilizing a dynamic multi-thread model; performing data slicing on the source data table based on the data volume of the source data table to obtain data batch information, and creating a batch manager with a synchronous lock mechanism; migrating the source data table into a temporary data table in batches by using threads in a thread pool based on the data batch information; the number of threads in the data batch migration process is dynamically adjusted through a dynamic multi-thread model, and the data batch migration process is synchronously controlled through a batch manager; under the condition that the batch manager detects that synchronization is completed, data consistency verification is conducted on the data in the temporary table, and then data batch migration of the source data table is completed. The invention further provides a data batch migration device and equipment, a medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, specifically to the field of big data management technology, and more specifically to a method, device, equipment, medium and program product for batch data migration. Background Art

[0002] Data migration refers to the process of selecting, preparing, extracting and converting data, and permanently transferring data from one computer storage system to another. In addition, verifying the consistency of the migrated data is also considered a part of the entire data migration process. Due to the rapid development of information technology, enterprises generate massive amounts of data during production or operation. How to store these massive amounts of data and, when necessary, efficiently migrate these massive amounts of data has become the focus of research for technicians.

[0003] When implementing data migration for massive amounts of data, existing technical solutions, such as data migration solutions based on direct copying and data migration solutions assisted by tools, often read the entire data set first and then write it, which requires more time and disk space and places high demands on device performance. In addition, the number of multi-threads used in the data migration process of existing data migration solutions is often fixed and cannot be flexibly changed according to the current network conditions, which leads to a waste of network resources. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a data batch migration method, apparatus, device, medium and program product for improving data migration efficiency.

[0005] According to a first aspect of the present disclosure, a method for batch data migration is provided, the method comprising:

[0006] Use the dynamic multi-threading model to create a thread pool for batch data migration;

[0007] Slice the source data table based on the data volume of the source data table, obtain data batch information, and create a batch manager with a synchronization lock mechanism;

[0008] Based on the data batch information, use the threads in the thread pool to migrate the source data table to the temporary data table in batches;

[0009] The dynamic multi-threading model is used to dynamically adjust the number of threads in the data batch migration process, and the batch manager is used to synchronously control the data batch migration process;

[0010] When the batch manager detects that the synchronization is complete, it performs a data consistency check on the data in the temporary table and completes the batch migration of the data in the source data table.

[0011] According to an embodiment of the present disclosure, the above-mentioned data batch migration method further includes:

[0012] Based on the preset weight coefficient, a dynamic multi-threading model is obtained by linearly fitting the system information used for data transmission and the attribute information of the source data table.

[0013] According to an embodiment of the present disclosure, the system information includes at least one of the following: processor quantity information, memory information, database information, and network status information;

[0014] The attribute information of the source data table includes at least one of the following: data volume, data structure information, data storage information, and data type information.

[0015] According to an embodiment of the present disclosure, the data slicing of the source data table based on the data volume of the source data table to obtain data batch information includes:

[0016] Calculate the total data migration batches in the data batch information according to the data volume of the source data table and the data volume of each migration batch set by the user;

[0017] The source data table stored in the database is sliced ​​using the database's auto-increment column mechanism to obtain data slicing results with identifiers in the data batch information.

[0018] According to an embodiment of the present disclosure, the data slicing of the source data table based on the data volume of the source data table to obtain data batch information also includes:

[0019] When the source data table does not have an auto-increment column, the database is used to create an auto-increment column for the source data table, and the source data table is sliced ​​using the auto-increment column mechanism to obtain a data slicing result with an identifier in the data batch information.

[0020] According to an embodiment of the present disclosure, the above-mentioned creation of a batch manager with a synchronization lock mechanism includes:

[0021] By parsing the data batch information, the total batch of data migration and the data slice results with identifiers are obtained;

[0022] Initialize the batch manager based on the data migration total batch and data slice results with identifiers, and open the synchronization lock of the batch manager.

[0023] According to an embodiment of the present disclosure, the above-mentioned method of dynamically adjusting the number of threads in the data batch migration process by using a dynamic multi-threading model includes:

[0024] According to the network status information, the dynamic multi-threading model is used to calculate the current number of threads required for the current data batch migration process, and the number of threads is compared with the number of threads in the thread pool to obtain a comparison result;

[0025] If the comparison result shows that the current number of threads is greater than the number of threads in the thread pool, the number of threads in the thread pool is dynamically increased, and the source data table is migrated to the temporary data table in batches using the newly added threads;

[0026] When the comparison result shows that the current number of threads is less than the number of threads in the thread pool, the number of threads in the thread pool is dynamically closed.

[0027] According to an embodiment of the present disclosure, the above-mentioned synchronous control of the data batch migration process by using a batch manager includes:

[0028] Read and write multiple data slice results in the data batch information corresponding to each thread in the thread pool in batches;

[0029] Use the batch manager to open the synchronization lock, and use the synchronization lock to synchronize the batch read operation and the batch write operation;

[0030] Use the batch manager to return the latest identifier of the data slicing result, and perform an increment operation on the latest identifier;

[0031] When the latest identifier after self-increment is equal to the total batch of data migration in the data batch information, the batch manager is used to end the synchronization control operation.

[0032] According to a second aspect of the present disclosure, a data batch migration device is provided, the device comprising:

[0033] A thread pool creation module is used to create a thread pool for batch data migration using a dynamic multi-threading model;

[0034] The data slicing and manager creation module is used to slice the source data table based on the data volume of the source data table, obtain data batch information, and create a batch manager with a synchronization lock mechanism;

[0035] The data batch migration module is used to migrate the source data table into the temporary data table in batches using threads in the thread pool based on the data batch information;

[0036] The thread dynamic adjustment and synchronization control module is used to dynamically adjust the number of threads in the data batch migration process using a dynamic multi-threading model, and to synchronously control the data batch migration process using a batch manager;

[0037] The data consistency check module is used to perform data consistency check on the data in the temporary table and complete the batch migration of the data in the source data table when the batch manager detects that the synchronization is completed.

[0038] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0039] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0040] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0041] The above-mentioned data batch migration method provided by the present disclosure utilizes a dynamic multi-thread model to dynamically adjust the number of threads used according to the actual system resource information and attribute information of the source data table during the data migration process, which can fully utilize system resources, effectively exert the functions of multiple processors, improve the transmission efficiency of data migration, and facilitate scheduling and improve concurrency. In addition, the above-mentioned data batch migration method provided by the present disclosure divides the data into small batches for processing, which can control the processing scale of each batch, thereby reducing memory usage and improving the overall input / output efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0043] Figure 1 is an application scenario diagram of the data batch migration method according to an embodiment of the present disclosure;

[0044] Figure 2 is a flow chart of a method for batch migration of data according to an embodiment of the present disclosure;

[0045] Figure 3 is a specific flow chart of dynamic multi-threaded data batch migration according to an embodiment of the present disclosure;

[0046] Figure 4 is a schematic diagram of the linkage between a dynamic multithreading model and a dynamic thread pool according to an embodiment of the present disclosure;

[0047] Figure 5 is a schematic diagram of a dynamic multi-threaded data batch migration process according to an embodiment of the present disclosure;

[0048] Figure 6 is a structural schematic diagram of a data batch migration device according to an embodiment of the present disclosure;

[0049] Figure 7 It is a block diagram of an electronic device suitable for implementing a method for batch data migration according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0051] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0052] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0053] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0054] In the field of big data technology, to migrate massive amounts of data from one computer system to another, the existing technology usually adopts direct copying or tool-assisted migration methods; among them, direct copying is to copy the data in the source database directly to the target database. This method is simple and fast, but it is necessary to ensure that the structure of the target database is consistent with the source database; while using tool copying is to use professional data migration tools to automatically complete data conversion and migration tasks. Use external storage devices, professional data migration software, network sharing or the data migration tools that come with the operating system to go online for dual writing, offline migration of historical data, switching reading, switching writing, switching everything to the new system, and stopping writing.

[0055] However, for large data sets, existing data migration solutions first read the entire data set and then write it, which requires more time and disk space and has high requirements on device performance. The existing multi-thread settings are all fixed numbers and will not flexibly change according to the current network conditions, wasting network resources.

[0056] In response to the technical problems in the prior art, the present disclosure provides a data batch migration method, device, equipment, medium and program product, which utilizes a dynamic multi-threaded model to dynamically adjust the number of threads in the data migration process, and can effectively solve the high software and hardware requirements for related systems or equipment in existing data migration solutions, and can make full use of network resources to efficiently complete data migration work.

[0057] It should be noted that in the technical solution of the present invention, the data involved are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0058] Figure 1 It is an application scenario diagram of the data batch migration method according to an embodiment of the present disclosure.

[0059] like Figure 1 As shown, the application scenario 100 according to this embodiment may include the field of big data technology or the field of database management technology. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0060] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0061] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0062] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0063] It should be noted that the data batch migration method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data batch migration device provided in the embodiment of the present disclosure can generally be set in the server 105. The data batch migration method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data batch migration device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0065] The following will be based on Figure 1 The scene described by Figure 2~Figure 5 The data batch migration method of the disclosed embodiment is described in detail.

[0066] Figure 2 The figure is a flow chart of a method for migrating data in batches according to an embodiment of the present disclosure.

[0067] like Figure 2 As shown, the data batch migration method of this embodiment includes operations S210 to S250.

[0068] In operation S210 , a thread pool for batch data migration is created using a dynamic multi-threading model.

[0069] The dynamic multi-threading model can dynamically adjust the number of threads used according to the system resources (or device resources) involved in the data migration process and the attribute information of the source table.

[0070] In operation S220 , data slicing is performed on the source data table based on the data volume of the source data table to obtain data batch information, and a batch manager with a synchronization lock mechanism is created.

[0071] During the data slicing process, the slicing results can be encoded using the auto-increment column of the database system to obtain a unique identifier for each slice.

[0072] Those skilled in the art may perform slicing in other ways, such as slicing according to the time attribute of the data in the source data table, or slicing according to the characteristics of the data in the source data table.

[0073] In operation S230 , based on the data batch information, the source data table is migrated to the temporary data table in batches using threads in the thread pool.

[0074] Each thread in the thread pool can be responsible for data migration of one or more data slice results.

[0075] In operation S240 , the number of threads in the data batch migration process is dynamically adjusted using a dynamic multi-threading model, and the data batch migration process is synchronously controlled using a batch manager.

[0076] The number of threads used in the current data migration process is determined based on the calculation results of the dynamic multi-threading model, and the batch manager is used to monitor the deadlock status that may be encountered in the data migration process.

[0077] In operation S250 , when the batch manager detects that the synchronization is completed, a data consistency check is performed on the data in the temporary table to complete the batch migration of the data in the source data table.

[0078] Operation S250 is used to verify whether the data migrated to the temporary table is consistent with the data in the source data table. If the data consistency check passes, the data migration is completed; if the data consistency check fails, the data that fails the consistency check is re-migrated.

[0079] The data consistency check involved in operation S250 can reduce the possibility of data errors and inconsistencies, thereby improving data quality and reliability.

[0080] The above-mentioned data batch migration method provided by the present disclosure utilizes a dynamic multi-thread model to dynamically adjust the number of threads used according to the actual system resource information and attribute information of the source data table during the data migration process, which can fully utilize system resources, effectively exert the functions of multiple processors, improve the transmission efficiency of data migration, and facilitate scheduling and improve concurrency. In addition, the above-mentioned data batch migration method provided by the present disclosure divides the data into small batches for processing, which can control the processing scale of each batch, thereby reducing memory usage and improving the overall input / output efficiency of the system.

[0081] The following is a specific embodiment and combined with the attached Figure 3The data batch migration method provided by the present disclosure is further explained in detail.

[0082] Figure 3 It is a specific flow chart of dynamic multi-threaded data batch migration according to an embodiment of the present disclosure.

[0083] like Figure 3 As shown, first, a dynamic multi-threaded model is established; second, the source data table is divided into batches; third, a multi-threaded task is created, that is, a dynamic thread pool and a batch manager are created; finally, data consistency verification is performed to complete the batch migration of the data of the entire source data table.

[0084] Among them, during the data batch migration initialization process, data batching mainly includes obtaining the size s of the source data table, that is, the amount of data in the source data table; setting the number of rows n that need to be synchronized each time; and calculating the total batch i of data migration based on the size s of the source data table and the amount of data in each batch n.

[0085] In the batch manager creation process, the batch manager is set according to the obtained total batch of data migration and the data batch ID in the current data migration process.

[0086] In the process of creating a multi-threaded synchronization task, it is necessary to set the total batch of data migration, obtain the latest data batch ID at any time, and judge the synchronization process.

[0087] According to an embodiment of the present disclosure, the above-mentioned data batch migration method also includes: based on a preset weight coefficient, a dynamic multi-threaded model is obtained by linearly fitting the system information used for data transmission and the attribute information of the source data table.

[0088] In the process of establishing the above dynamic multi-threaded model, the linear regression method can be used to construct the model, and those skilled in the art can also construct other types of dynamic multi-threaded models that are substantially the same as those disclosed in the present invention in other ways.

[0089] The dynamic multi-threading model disclosed in the present invention is used to solve the problem of insufficient utilization of system resources (or device resources) or resource overload during data migration, and can make full use of relevant resources.

[0090] According to an embodiment of the present disclosure, the above-mentioned system information includes at least one of the following: processor quantity information, memory information, database information and network status information; wherein, the attribute information of the source data table includes at least one of the following: data volume, data structure information, data storage information and data type information.

[0091] The process of establishing a dynamic multi-threaded model disclosed in the present invention is further described in detail below through specific embodiments.

[0092] First, obtain the software and hardware resource information used in the data migration process and the attribute information of the source data table (such as data type, data characteristics, etc.).

[0093] Secondly, based on the acquired information, a dynamic multi-threaded model for data transmission is established using the linear regression method, as shown in formula (1):

[0094] (1),

[0095] in, Indicates Preset weight coefficients, Indicates the optimal number of threads for the current data migration process. The value of is dynamically adjusted according to the network status during data migration. Represents the floor function.

[0096] Finally, the dynamic multi-threaded model established by formula (1) outputs the optimal number of threads in real time, which is used to perform dynamic and batch data migration on the source data table.

[0097] It should be noted that the above-mentioned system information or attribute information of source data is optional, and those skilled in the art can set the system information and attribute information of source data used according to actual needs. Similarly, those skilled in the art can establish other models for dynamically adjusting the number of threads using linear or nonlinear regression methods based on the technical inspiration given by the present disclosure.

[0098] According to an embodiment of the present disclosure, the above-mentioned data slicing of the source data table based on the data volume of the source data table to obtain the data batch information includes: calculating the total data migration batches in the data batch information according to the data volume of the source data table and the data volume of each migration batch set by the user; and utilizing the database's auto-increment column mechanism to perform data slicing on the source data table stored in the database to obtain the data slicing results with identifiers in the data batch information.

[0099] In the embodiments of the present disclosure, data slicing can improve the efficiency of data migration; in addition, data slicing can also improve the efficiency of data consistency verification. Those skilled in the art can also adopt other data slicing methods based on the technical inspiration given by the present disclosure.

[0100] According to an embodiment of the present disclosure, the above-mentioned data slicing of the source data table based on the data volume of the source data table to obtain data batch information also includes: when the source data table does not have an auto-increment column, using the database to create an auto-increment column of the source data table, and using the auto-increment column mechanism to slice the source data table, to obtain data slicing results with identifiers in the data batch information.

[0101] The data slicing process provided by the present disclosure is further described in detail below through specific embodiments.

[0102] In an embodiment of the present disclosure, data slicing can migrate the source data table in batches. For example, the source data table has a data volume of size s. The data is sliced ​​through the database auto-increment column to determine the size n of each batch. The total batches that need to be synchronized are calculated i=s / n and rounded up. It is determined whether the source data table has an auto-increment column Z_ID. If no auto-increment column is created, the auto-increment column is used as the division mark. The value range of the auto-increment column Z_ID of the Jth batch of data is [(J-1)n+1,Jn], and the i-th column of data is the last batch, and the value range is [(i-1)n+1,s].

[0103] According to an embodiment of the present disclosure, the above-mentioned creation of a batch manager with a synchronization lock mechanism includes: obtaining the total batch of data migration and the data slicing results with identifiers by parsing the data batch information; initializing the batch manager based on the total batch of data migration and the data slicing results with identifiers, and turning on the synchronization lock of the batch manager.

[0104] By creating a batch manager with a synchronous lock mechanism, thread safety can be guaranteed and deadlock problems caused by multiple threads accessing the same data slice can be prevented; at the same time, a batch manager with a synchronous lock mechanism can also ensure data consistency during data migration.

[0105] According to an embodiment of the present disclosure, the above-mentioned use of a dynamic multi-threading model to dynamically adjust the number of threads in the data batch migration process includes: according to network status information, using a dynamic multi-threading model to calculate the current number of threads required for the current data batch migration process, and comparing the number of threads with the number of threads in the thread pool to obtain a comparison result; when the comparison result is that the current number of threads is greater than the number of threads in the thread pool, dynamically adding threads to the thread pool, and using the newly added threads to migrate the source data table in batches to the temporary data table; when the comparison result is that the current number of threads is less than the number of threads in the thread pool, dynamically closing the number of threads in the thread pool.

[0106] The following is a specific embodiment and combined with the attached Figure 4 The dynamic adjustment process of threads in the embodiments of the present disclosure is further described in detail.

[0107] Figure 4 It is a schematic diagram of the linkage between the dynamic multi-thread model and the dynamic thread pool according to an embodiment of the present disclosure.

[0108] like Figure 4As shown, the number of threads K is output through the dynamic multi-threaded model. During the data migration process, the number of threads in the thread pool will be dynamically adjusted with the network status, that is, the current thread status is judged and adjusted to prepare for batch synchronization tasks. When K threads are full, the batch data will no longer be read. When there are idle threads, batch data will be obtained from the batch data again until all batch data are executed. That is, the batch ID is the total synchronization batch i, and the dynamic thread pool is stopped.

[0109] According to an embodiment of the present disclosure, the above-mentioned use of a batch manager to synchronously control the data batch migration process includes: batch reading and batch writing of multiple data slice results in the data batch information corresponding to each thread in the thread pool; using the batch manager to open a synchronization lock, and using the synchronization lock to synchronously control the batch reading operation and the batch writing operation; using the batch manager to return the latest identifier of the data slice result, and auto-incrementing the latest identifier; when the latest identifier after auto-increment is equal to the total batch of data migration in the data batch information, using the batch manager to end the synchronization control operation.

[0110] The following is a specific embodiment and combined with the attached Figure 5 The role and working principle of the batch manager involved in the present disclosure are further explained in detail.

[0111] Figure 5 It is a schematic diagram of a dynamic multi-threaded data batch migration process according to an embodiment of the present disclosure.

[0112] Create a batch manager and enable synchronization lock, such as Figure 5 As shown, the batch manager initializes the batch ID to 0 and the total synchronization batch to i. Each thread reads and writes in batches to complete the migration of the current batch data and determines whether the current batch is completed. If not, the data of the current batch is synchronized from the original table to the temporary table. If completed, the latest batch ID is returned. After the synchronization is completed, the batch ID is changed to batch ID+1. When the latest batch ID is the total synchronization batch i, the synchronization ends.

[0113] Based on the above-mentioned data batch migration method, the present disclosure also provides a data batch migration device. Figure 6 The device is described in detail.

[0114] Figure 6 It is a structural schematic diagram of a data batch migration device according to an embodiment of the present disclosure.

[0115] like Figure 6As shown, the above-mentioned data batch migration device 600 includes a model thread pool creation module 610, a data slice and manager creation module 620, a data batch migration module 630, a thread dynamic adjustment and synchronization control module 640 and a data consistency verification module 650.

[0116] The thread pool creation module 610 is used to create a thread pool for batch data migration using a dynamic multi-threading model; the thread pool creation module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0117] The data slicing and manager creation module 620 is used to slice the source data table based on the data volume of the source data table, obtain data batch information, and create a batch manager with a synchronization lock mechanism; the data slicing and manager creation module 620 can be used to execute the operation S220 described above, which will not be repeated here.

[0118] The data batch migration module 630 is used to migrate the source data table to the temporary data table in batches based on the data batch information using threads in the thread pool; the data batch migration module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0119] The thread dynamic adjustment and synchronization control module 640 is used to dynamically adjust the number of threads in the data batch migration process using a dynamic multi-threading model, and use a batch manager to synchronize and control the data batch migration process; the thread dynamic adjustment and synchronization control module 640 can be used to execute the operation S240 described above, which will not be repeated here.

[0120] The data consistency check module 650 is used to perform data consistency check on the data in the temporary table and complete the batch migration of the data in the source data table when the batch manager detects that the synchronization is completed. The data consistency check module 650 can be used to perform the operation S250 described above, which will not be described in detail here.

[0121] According to an embodiment of the present disclosure, any multiple modules of the thread pool creation module 610, the data slice and manager creation module 620, the data batch migration module 630, the thread dynamic adjustment and synchronization control module 640, and the data consistency check module 650 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the thread pool creation module 610, the data slice and manager creation module 620, the data batch migration module 630, the thread dynamic adjustment and synchronization control module 640, and the data consistency check module 650 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, at least one of the thread pool creation module 610, the data slicing and manager creation module 620, the data batch migration module 630, the thread dynamic adjustment and synchronization control module 640, and the data consistency verification module 650 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is executed.

[0122] The data batch migration device 600 further includes a dynamic multi-thread model building module 660. The dynamic multi-thread model building module 660 is used to obtain a dynamic multi-thread model by linearly fitting the system information for data transmission and the attribute information of the source data table based on a preset weight coefficient.

[0123] According to an embodiment of the present disclosure, the data slice and manager creation module 620 includes a total batch calculation unit and a first data slice unit.

[0124] The total batch calculation unit is used to calculate the total batch of data migration in the data batch information according to the data volume of the source data table and the data volume of each migration batch set by the user.

[0125] The first data slicing unit is used to perform data slicing on the source data table stored in the database by using the database's auto-increment column mechanism, and obtain a data slicing result with an identifier in the data batch information.

[0126] According to an embodiment of the present disclosure, the data slice and manager creation module 620 further includes a second data slice unit.

[0127] Among them, the second data slicing unit is used to create an auto-increment column of the source data table using the database when the source data table does not have an auto-increment column, and use the auto-increment column mechanism to slice the source data table to obtain a data slicing result with an identifier in the data batch information.

[0128] According to an embodiment of the present disclosure, the data slice and manager creation module 620 further includes a parsing unit and an initialization unit.

[0129] The parsing unit is used to parse the data batch information to obtain the total batch of data migration and the data slicing results with identifiers.

[0130] The initialization unit is used to initialize the batch manager based on the total batch of data migration and the data slice results with identifiers, and open the synchronization lock of the batch manager.

[0131] According to an embodiment of the present disclosure, the thread dynamic adjustment and synchronization control module 640 includes a comparison and judgment unit, a first processing unit and a second processing unit.

[0132] The comparison and judgment unit is used to calculate the current number of threads required for the current data batch migration process using a dynamic multi-threading model according to network status information, and compare the number of threads with the number of threads in the thread pool to obtain a comparison result.

[0133] The first processing unit is used to dynamically increase the number of threads in the thread pool when the comparison result shows that the current number of threads is greater than the number of threads in the thread pool, and migrate the source data table to the temporary data table in batches by using the newly added threads.

[0134] The second processing unit is used to dynamically close the number of threads in the thread pool when the comparison result shows that the number of current threads is less than the number of threads in the thread pool.

[0135] According to an embodiment of the present disclosure, the thread dynamic adjustment and synchronization control module 640 further includes a batch reading and writing unit, a synchronization control unit, an auto-increment unit, and a judgment unit.

[0136] The batch reading and writing unit is used to read and write in batches the multiple data slice results in the data batch information corresponding to each thread in the thread pool.

[0137] The synchronization control unit is used to open the synchronization lock by using the batch manager, and to synchronously control the batch reading operation and the batch writing operation by using the synchronization lock.

[0138] The auto-increment unit is used to use the batch manager to return the latest identifier of the data slicing result and perform an auto-increment operation on the latest identifier.

[0139] The judgment unit is used to terminate the synchronization control operation by using the batch manager when the latest identifier after self-increment is equal to the total batch of data migration in the data batch information.

[0140] Figure 7 It is a block diagram of an electronic device suitable for implementing a method for batch data migration according to an embodiment of the present disclosure.

[0141] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0142] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0143] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0144] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0145] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0146] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data batch migration method provided by the embodiment of the present disclosure.

[0147] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 701. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0148] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0149] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0150] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0151] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0152] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0153] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for batch migration of data, characterized in that: The method comprises: Using the dynamic multi-threading model to create a thread pool for batch data migration; Slice the source data table based on the data volume of the source data table to obtain data batch information, and create a batch manager with a synchronization lock mechanism; Based on the data batch information, the source data table is migrated into a temporary data table in batches by using threads in the thread pool; The dynamic multi-thread model is used to dynamically adjust the number of threads in the data batch migration process, and the batch manager is used to synchronously control the data batch migration process; When the batch manager detects that the synchronization is completed, a data consistency check is performed on the data in the temporary table to complete the batch migration of the data in the source data table.

2. The method according to claim 1, further comprising: Based on the preset weight coefficient, the dynamic multi-threading model is obtained by linearly fitting the system information used for data transmission and the attribute information of the source data table.

3. The method according to claim 2, characterized in that The system information includes at least one of the following: processor quantity information, memory information, database information, and network status information; The attribute information of the source data table includes at least one of the following: data volume, data structure information, data storage information and data type information.

4. The method according to claim 1, characterized in that The source data table is sliced ​​based on the amount of data in the source data table to obtain data batch information including: Calculate the total batch of data migration in the data batch information according to the data volume of the source data table and the data volume of each migration batch set by the user; The source data table stored in the database is sliced ​​using the database's auto-increment column mechanism to obtain a data slicing result with an identifier in the data batch information.

5. The method according to claim 4, characterized in that Also includes: When the source data table does not have an auto-increment column, the database is used to create an auto-increment column for the source data table, and the source data table is sliced ​​using the auto-increment column mechanism to obtain a data slicing result with an identifier in the data batch information.

6. The method according to claim 1, characterized in that Creating a batch manager with a synchronization lock mechanism involves: By parsing the data batch information, a total batch of data migration and a data slice result with an identifier are obtained; The batch manager is initialized based on the total batch of data migration and the data slicing result with the identifier, and the synchronization lock of the batch manager is opened.

7. The method according to claim 1, characterized in that Dynamically adjusting the number of threads in the data batch migration process using the dynamic multi-threading model includes: According to the network status information, the dynamic multi-threading model is used to calculate the current number of threads required for the current data batch migration process, and the number of threads is compared with the number of threads in the thread pool to obtain a comparison result; If the comparison result shows that the current number of threads is greater than the number of threads in the thread pool, the number of threads in the thread pool is dynamically increased, and the source data table is migrated to the temporary data table in batches using the newly increased threads; When the comparison result shows that the current number of threads is less than the number of threads in the thread pool, the number of threads in the thread pool is dynamically closed.

8. The method according to any one of claims 1 to 7, characterized in that Using the batch manager to synchronously control the data batch migration process includes: Read and write in batches the multiple data slice results in the data batch information corresponding to each thread in the thread pool; Using the batch manager to open a synchronization lock, and using the synchronization lock to synchronously control batch reading operations and batch writing operations; Using the batch manager to return the latest identifier of the data slicing result, and performing a self-increment operation on the latest identifier; When the latest identifier after self-increment is equal to the total batch of data migration in the data batch information, the batch manager is used to end the synchronization control operation.

9. A data batch migration device, characterized in that: The device comprises: A thread pool creation module, used to create a thread pool for batch data migration using the dynamic multi-threading model; A data slicing and manager creation module, used to slice the source data table based on the data volume of the source data table, obtain data batch information, and create a batch manager with a synchronization lock mechanism; A data batch migration module, used to migrate the source data table into a temporary data table in batches by using threads in the thread pool based on the data batch information; A thread dynamic adjustment and synchronization control module, used to dynamically adjust the number of threads in the data batch migration process using the dynamic multi-threading model, and to synchronously control the data batch migration process using the batch manager; The data consistency check module is used to perform data consistency check on the data in the temporary table and complete the batch migration of the data in the source data table when the batch manager detects that the synchronization is completed.

10. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images