Data migration method and device, computer device, and storage medium

Abstract

The application provides a data migration method and device, computer equipment and a storage medium. The method comprises the following steps: in response to a data migration request for to-be-migrated data, determining a migration data volume of the to-be-migrated data; determining a target reading strategy according to the migration data volume; reading the to-be-migrated data into a local memory for storage according to the target reading strategy; and dynamically scheduling an idle sub-thread to migrate the to-be-migrated data in the local memory to a remote database by using the idle sub-thread. The method can effectively improve the data migration efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data migration method and device, computer equipment and storage medium. BACKGROUND

[0002] With the development of database technology, the application of database is more and more widely. Large database is the necessary support for the normal operation of key business system, and its importance is self-evident. Therefore, importing local data into the database is an essential step in system upgrade or machine learning model development.

[0003] However, the existing data migration technology will expose many problems in the face of tens of thousands of data or even hundreds of millions of data, which directly affects the safety and stability of data migration, especially when using single-thread scheme, the read data is also a single insertion, which leads to slow migration speed and easy fault interruption.

[0004] Therefore, the existing data migration technology has the technical problem of low data migration efficiency due to unreasonable migration strategy. SUMMARY

[0005] The purpose of the present application is to provide a data migration method, device, computer equipment and storage medium to improve the data migration strategy and improve the data migration efficiency.

[0006] In a first aspect, the present application provides a data migration method, comprising:

[0007] In response to a data migration request for the to-be-migrated data, determining a migration data volume of the to-be-migrated data;

[0008] According to the migration data volume, determining a target reading strategy;

[0009] According to the target reading strategy, reading the to-be-migrated data into the local memory for storage;

[0010] Dynamically scheduling idle sub-threads to migrate the to-be-migrated data in the local memory to the remote database by using the idle sub-threads.

[0011] In some embodiments of the present application, according to the migration data volume, the target reading strategy is determined, comprising: obtaining the idle memory volume of the local memory; dividing the idle memory volume based on a preset division ratio to obtain a target idle memory volume; and determining the target reading strategy according to the target idle memory volume and the migration data volume.

[0012] In some embodiments of this application, a target reading strategy is determined based on the target amount of free memory and the amount of migrated data, including: if the target amount of free memory is greater than the amount of migrated data, a preset one-time reading strategy is determined as the target reading strategy; if the target amount of free memory is less than or equal to the amount of migrated data, a preset line-by-line reading strategy is determined as the target reading strategy.

[0013] In some embodiments of this application, the target reading strategy includes a line-by-line reading strategy. After reading the data to be migrated into local memory for storage according to the target reading strategy, the strategy further includes: determining the data to be migrated that has been stored in local memory as read data; when the read data reaches a preset reading cutoff amount, performing a step of dynamically scheduling idle sub-threads for the read data; and reading the remaining data to be migrated that has reached the reading cutoff amount into local memory in batches according to the target reading strategy, so as to wait for the remote migration of idle sub-threads.

[0014] In some embodiments of this application, dynamically scheduling idle sub-threads to migrate data to be migrated from local memory to a remote database includes: obtaining the status information of each sub-thread in the thread pool through the main thread; extracting at least one sub-thread whose status information is idle as an idle sub-thread; and using the idle sub-threads to migrate the data to be migrated from local memory to the remote database in batches according to a preset target migration batch.

[0015] In some embodiments of this application, before using idle sub-threads to migrate data in batches from local memory to a remote database according to a preset target migration batch, the method further includes: determining a first migration batch and a second migration batch to maximize the data migration volume, wherein the difference between the first migration batch and the second migration batch is a preset gradient value, and the second migration batch is greater than the first migration batch; calculating the total amount of migrated data obtained by the idle sub-threads migrating data according to the first migration batch within a preset time period, as the first total migration data volume; and calculating the total amount of migrated data obtained by the idle sub-threads migrating data according to the second migration batch within a preset time period, as the second total migration data volume; if the second total migration data volume is less than the first total migration data volume, then determining the first migration batch as the target migration batch; if the second total migration data volume is greater than or equal to the first total migration data volume, then determining the second migration batch as the target migration batch.

[0016] In some embodiments of this application, the data migration method further includes: obtaining the migration result of the data to be migrated after at least one batch migration; if the migration result includes migration failure, then using a binary recursive algorithm to perform anomaly analysis on the data to be migrated that failed to migrate, and filtering out abnormal data and normal data; and using an idle sub-thread to migrate the normal data to a remote database.

[0017] In some embodiments of the present application, if the migration result contains migration failure, a binary recursive algorithm is used to perform abnormal analysis on the to-be-migrated data of migration failure, and abnormal data and normal data are screened out, including: if the migration result contains migration failure, based on the binary recursive algorithm, the to-be-migrated data is divided into a first array and a second array; the idle sub-threads are dynamically scheduled to use the idle sub-threads to obtain the migration result of the first array and the second array, and to determine the array whose migration result is migration failure as a target array; the binary grouping abnormal analysis operation for the target array is repeatedly performed until the abnormal data and the normal data are screened out.

[0018] In a second aspect, the present application provides a data migration device, comprising:

[0019] A request response module is configured to respond to a data migration request for to-be-migrated data, and determine a migration data volume of the to-be-migrated data.

[0020] A strategy determination module is configured to determine a target reading strategy according to the migration data volume.

[0021] A data migration module is configured to read the to-be-migrated data into a local memory according to the target reading strategy.

[0022] A dynamic scheduling module is configured to dynamically schedule idle sub-threads to use the idle sub-threads to migrate the to-be-migrated data in the local memory to a remote database.

[0023] In a third aspect, the present application further provides a computer device, comprising:

[0024] One or more processors;

[0025] A memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the data migration method of the first aspect.

[0026] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program. The computer program is loaded by a processor to perform the steps in the data migration method.

[0027] In a fifth aspect, the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method provided in the first aspect.

[0028] The data migration method, device, computer device and storage medium, the server determines the migration data amount of the to-be-migrated data by responding to the data migration request for the to-be-migrated data, determines the target reading strategy according to the migration data amount, reads the to-be-migrated data into the local memory according to the target reading strategy, and stores the to-be-migrated data in the local memory, and then dynamically schedules the idle sub-threads to migrate the to-be-migrated data in the local memory to the remote database by using the idle sub-threads. In this way, the appropriate data reading strategy is intelligently selected by analyzing the migration data amount, the data migration strategy is improved, and the data migration efficiency is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0030] Figure 1 The scene schematic diagram of the data migration method in the embodiments of the present application;

[0031] Figure 2 The flowchart of the data migration method in the embodiments of the present application;

[0032] Figure 3 The dynamic scheduling flowchart of the data migration method in the embodiments of the present application;

[0033] Figure 4 The structure schematic diagram of the data migration device in the embodiments of the present application;

[0034] Figure 5 The structure schematic diagram of the computer device in the embodiments of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0036] In the description of the present application, the terms "first", "second" are only for descriptive purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise explicitly specified.

[0037] In the description of the present application, the term "for example" is used to indicate "as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily interpreted as more preferred or more advantageous than other embodiments. The following description is given in order to enable any person skilled in the art to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that those skilled in the art can recognize that the present application can be implemented without using these specific details. In other examples, well-known structures and processes will not be described in detail in order to avoid unnecessary details making the description of the present application obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope consistent with the principles and features disclosed in the present application.

[0038] In the embodiments of the present application, the data migration method provided by the embodiments of the present application can be applied to the data migration system as shown in Figure 1 The data migration system includes a terminal 102 and a server 104. The terminal 102 can be a device that includes receiving and transmitting hardware, i.e. a device with receiving and transmitting hardware capable of performing bidirectional communication on a bidirectional communication link. Such a device can include a cellular or other communication device with a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The terminal 102 can be a desktop terminal or a mobile terminal, and the terminal 102 can also be one of a mobile phone, a tablet computer, and a notebook computer. The server 104 can be a stand-alone server, or a server network or a server cluster composed of servers, including but not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing. In addition, the terminal 102 and the server 104 establish a communication connection through a network, and the network can be any one of a wide area network, a local area network, and a metropolitan area network.

[0039] Those skilled in the art can understand that Figure 1 The application environment shown in the above Figure 1More or less devices are shown in the figure. For example, Figure 1 Only one server is shown in the figure. It can be understood that the data migration system can also include one or more other devices, which are not limited here. In addition, the data migration system can also include a memory for storing data, such as storing data to be migrated.

[0040] It should be noted that Figure 1 The scenario diagram of the data migration system shown is only an example, and the data migration system and scenario described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, as the data migration system evolves and new business scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0041] Referring to Figure 2 The embodiments of the present application provide a data migration method, and the embodiments mainly take the server 104 in the above Figure 1 as an example to illustrate the method, which includes steps S201 to S204, and the details are as follows:

[0042] S201, in response to a data migration request for data to be migrated, determining a migration data amount of the data to be migrated.

[0043] The data to be migrated can be offline data or online data selected based on any business requirement and to be migrated from one place to another. The place includes but is not limited to a server.

[0044] The migration data amount can be the total data amount of the data to be migrated, for example, the migration data amount of the data to be migrated is 10KB, 10MB, 10GB, etc., which is not limited in the embodiments of the present application.

[0045] In a specific implementation, before the server 104 starts a data migration task, the data to be migrated needs to be obtained, and the instruction indicating the server 104 to obtain the data to be migrated can be a data migration request, but the embodiments of the present application are not limited to the source of the data migration request, that is, the data migration request can be from a task timing start mechanism pre-stored locally in the server 104, or from other devices. That is, after the server 104 receives the data migration request sent by the local or other devices, the server 104 can obtain the migration data amount of the data to be migrated in response to the data to be migrated indicated by the request, so as to use the migration data amount as a basis for subsequent processing.

[0046] It should be noted that the migration data amount of the to-be-migrated data can be acquired when the server 104 receives the data migration request, or can be acquired again after the server 104 receives the request, and the specific embodiment of the present application is not limited.

[0047] S202, determining a target reading strategy according to the migration data amount.

[0048] The target reading strategy includes but is not limited to a one-time reading strategy and a line-by-line reading strategy. The one-time reading strategy is also called "readfile", and the line-by-line reading strategy is called "readline". The readfile method is to read all data into the memory and then parse the data into a string. The readline method is to output the data and parse it into a string when a line break signal is encountered while reading the data, and then continue to read the next data. Finally, the data is uploaded to another server to realize migration from one place to another.

[0049] In a specific implementation, the to-be-migrated data can be data stored in a file, and therefore the above description of reading data can be replaced by reading a file stream. Therefore, the one-time reading strategy is to read all data in the file to the memory without interruption, and the line-by-line reading strategy is to read the data in the file line by line. It can be understood that the above reading strategies can be implemented through respective API interfaces.

[0050] For example, a text file is divided into 100 lines, and the total data amount is "1MB". Assuming that the content of each line is the same, the data amount of each line is "0.01MB". If the readfile method is used, "1MB" of text content is acquired at a time, and if the readline method is used, "0.01MB" of text content is output each time. In general, the text content is divided into 100 times of output.

[0051] In an embodiment, the step includes: acquiring a free memory amount of a local memory; dividing the free memory amount based on a preset division ratio to obtain a target free memory amount; and determining the target reading strategy according to the target free memory amount and the migration data amount.

[0052] The local memory can be a virtual memory space used by the current program. "Local" refers to the physical memory, which is directly related to the size of the disk, and "memory" refers to the virtual swap space. For example, the local memory is "8GB".

[0053] The amount of free memory can be the virtual memory space that is not occupied by programs, that is, the size of the free and available virtual memory space. For example, if the local memory is "8GB" and the used memory is "2GB", then the amount of free memory is "6GB".

[0054] The division ratio can be an influencing factor used to determine the target reading strategy. For example, the division ratio is a value greater than or equal to "0.5".

[0055] In specific implementation, after obtaining the migration data volume, server 104 can further obtain the amount of free memory in local memory, and then divide the free memory volume based on the partitioning ratio to obtain the target free memory volume after partitioning, denoted as "S". This target reading strategy is then determined in conjunction with the analysis of the migration data volume. It should be noted that the purpose of this application embodiment in analyzing the target free memory volume and the migration data volume to determine the target reading strategy is to analyze whether the target free memory volume can be completely integrated into the migration data volume, and whether the analysis result is positive or negative affects the application of the target reading strategy.

[0056] For example, if the partition ratio is "0.5" and the amount of free memory is "6GB", then the target amount of free memory is "S = 3GB". The steps for determining the read strategy involved in this embodiment will be described in detail below.

[0057] In one embodiment, determining a target read strategy based on the target amount of free memory and the amount of migrated data includes: if the target amount of free memory is greater than the amount of migrated data, then determining a preset one-time read strategy as the target read strategy; if the target amount of free memory is less than or equal to the amount of migrated data, then determining a preset line-by-line read strategy as the target read strategy.

[0058] In specific implementation, the method for determining the target reading strategy proposed in this application embodiment depends on the size of the target free memory and the amount of data to be migrated. If the target free memory is greater than the amount of data to be migrated, it means that the virtual memory space in local memory not occupied by the program is sufficient to store the data to be migrated. In this case, a one-time reading strategy is determined as the target reading strategy, and the server 104 can read all the data to be migrated to local storage at once. Conversely, if the target free memory is less than or equal to the amount of data to be migrated, it means that more than half of the virtual memory space in local memory is occupied by the program and is insufficient to store the data to be migrated. In this case, a line-by-line reading strategy is determined as the target reading strategy, and the server 104 can read all the data to be migrated to local storage line by line.

[0059] For example, if the division ratio is "0.5" and the amount of free memory is "170MB", the target amount of free memory is "S=85MB". At this time, if the amount of data to be migrated is "80MB", denoted as "size", the target amount of free memory is greater than the amount of data to be migrated (S>size), and the target read strategy is the one-time read strategy (readfile method). If the amount of data to be migrated is "86MB", the target amount of free memory is less than the amount of data to be migrated (S≤size), and the target read strategy is the line-by-line read strategy (readline method).

[0060] S203, according to the target read strategy, reading the data to be migrated into the local memory for storage.

[0061] In a specific implementation, the data to be migrated can be data that meets the subsequent processing requirements as soon as it is obtained, or can be obtained after a certain preprocessing method, including but not limited to: deduplication, error checking, format conversion, etc., and the specific embodiments of the present application are not limited.

[0062] Specifically, if the data to be migrated is obtained by requesting other devices, it can include but is not limited to one of the following ways: 1. In a normal network structure, the server 104 receives the data to be migrated from the terminal 102 or other cloud devices that have established a network connection in response to the data migration request; 2. In a pre-set blockchain network, the server 104 can synchronize the data to be migrated from other terminal nodes or server nodes in response to the data migration request. The blockchain network can be a public chain, a private chain, etc.; 3. In a pre-set tree structure, the server 104 requests attribute information from the upper server in response to the data migration request, or polls the data to be migrated from the lower server. In this way, the data migration efficiency can be intelligently improved.

[0063] In one embodiment, the target read strategy includes a line-by-line read strategy, and after this step, it further includes: determining the data to be migrated stored in the local memory as read data; when the read data reaches a pre-set read cutoff amount, performing the step of dynamically scheduling the idle sub-thread for the read data; and according to the target read strategy, reading the remaining data to be migrated that reaches the read cutoff amount into the local memory in batches to wait for the remote migration of the idle sub-thread.

[0064] Wherein, the read cutoff amount is a key data proposed by the embodiments of the present application, which can be denoted as "M", and its role is to control the batch migration of the data to be migrated, to ensure the safety and stability of data migration, and to improve the data migration efficiency.

[0065] In a specific implementation, when the read data (collectively referred to as "read data") reaches a preset read cutoff amount "M", the server 104 can perform the step of dynamically scheduling an idle sub-thread for the read data, that is, when the server 104 reads the data to be migrated up to the read cutoff amount "M" according to the row-by-row reading strategy, the idle sub-thread can be dynamically scheduled to migrate the read data to the remote database using the idle sub-thread. During the migration process, the remaining data to be migrated that has not been read can continue to be read, and the reading and migration of the data are synchronized, thereby effectively improving the data migration efficiency.

[0066] Further, if the read data is not completely migrated to the remote database, that is, the migration is blocked, the reading of the remaining data can be controlled to stop, and the previous batch of data can be smoothly migrated, and then the reading of the remaining data can be controlled to continue. In addition, a batch of data that has completed migration (whether successful or failed) in the local memory is cleaned up to ensure the running speed and the safety of the memory. It should be noted that the cleaning here only deletes the batch of data in the local memory to release the memory space, and the initial source file of the batch of data is still retained, that is, there is no data loss problem.

[0067] In addition, the migration success and the migration failure can be determined by receiving the feedback instruction of the remote database. The migration success means that all the data corresponding to a batch "M" are migrated to the remote database without any remaining data, and the migration failure means that the data corresponding to a batch "M" cannot be completely migrated to the remote database, and there is part of the abnormal data that cannot be migrated. The processing of the abnormal data will be described in detail in subsequent embodiments.

[0068] S204, dynamically scheduling an idle sub-thread to migrate the data to be migrated in the local memory to the remote database using the idle sub-thread.

[0069] In the embodiments of the present application, a multi-thread architecture is provided, including a main thread and at least one sub-thread. The main thread can communicate with each sub-thread through postMessage (a commonly used function in an application programming interface (API), used to put a message into a message queue). Each sub-thread has its own state value, "0" representing idle and "1" representing busy. The idle sub-thread can be a sub-thread with a state value of "0".

[0070] In a specific implementation, the main thread synchronizes the state of the sub-thread through communication (postMessage). All data migration tasks are scheduled by the main thread. When the data to be migrated is read through a file and passed to the main thread, the main thread checks whether there is an idle sub-thread (with a state value of "0"). If there is, the data migration task is assigned to the idle sub-thread, and the idle sub-thread is used to implement the data migration task for the data to be migrated.

[0071] In one embodiment, the step comprises: obtaining state information of each sub-thread in the thread pool by the main thread; extracting at least one sub-thread with the state information as an idle state as an idle sub-thread; and migrating the batch of to-be-migrated data in the local memory to the remote database by using the idle sub-thread and according to a preset target migration batch.

[0072] The thread pool is a form of multi-thread processing, and tasks are added to a queue during processing, and then the tasks are automatically started after the threads are created.

[0073] The state information can include the state values "0" and "1" described above, "0" indicating idle and "1" indicating busy.

[0074] The target migration batch can be a migration batch that maximizes the amount of migrated data. For example, when the migration batch is "600MB", the total amount of migrated data within 10 seconds reaches "10GB", and when the migration batch is "700MB", the total amount of migrated data within 10 seconds is only "7GB", the target migration batch can be preset to "600MB". The effect of the target migration batch can be understood as follows: if the target migration batch is "20MB" and the amount of migrated data is "100MB", the to-be-migrated data can be divided into "5 batches" and migrated to the remote database.

[0075] In a specific implementation, the server 104 can count the state information of each sub-thread and filter out the sub-threads with the state value "0" as candidate sub-threads. Then, according to the amount of migrated data of the to-be-migrated data, at least one candidate sub-thread that can execute the current data migration task is selected as an idle sub-thread. After receiving the to-be-migrated data, the idle sub-thread can start data migration, migrate the batch of to-be-migrated data to the remote database according to the preset target migration batch "M value", and notify the main thread that it is in a busy state (state value "1").

[0076] In addition, if the main thread finds that all sub-threads are in a busy state (state value "1") during the task distribution process, it will first store a batch of data (such as "600MB"), and if a row-by-row reading strategy is used, it will pause the reading program for the remaining data until all the read data in the previous batch in the local memory are migrated and the thread pool releases an idle sub-thread (state value "0").

[0077] It should be noted that the above description can be referred to Figure 3If the main thread dispatches a data migration task to a busy child thread (with a state value of "1"), the busy child thread will return the dispatched data to be migrated to the main thread and inform the main thread that it refuses to receive new tasks. Until all the current data is uploaded, the child thread will not notify the main thread again that it is in an idle state (with a state value of 0) and can accept new tasks again. In addition, the number of idle child threads can be determined according to the number of CPU cores of the server 104, for example, 8 idle child threads can be selected for an 8-core CPU to execute data migration tasks.

[0078] In one embodiment, before the data to be migrated in the local memory is batch migrated to the remote database by using the idle child threads and according to the preset target migration batch, the method further comprises: determining a first migration batch and a second migration batch for maximizing the data migration amount, the difference between the first migration batch and the second migration batch being a preset gradient value, and the second migration batch being greater than the first migration batch; counting the total migration data amount obtained by the idle child threads in a preset time period according to the first migration batch as a first total migration data amount; and counting the total migration data amount obtained by the idle child threads in the preset time period according to the second migration batch as a second total migration data amount; if the second total migration data amount is less than the first total migration data amount, determining the first migration batch as the target migration batch; and if the second total migration data amount is greater than or equal to the first total migration data amount, determining the second migration batch as the target migration batch.

[0079] The preset gradient value can be a preset migration proportion gradient value, for example, if the preset gradient value is "100", the first migration batch can be "600" and the second migration batch can be "700".

[0080] In a specific implementation, because the greater the value of the target migration batch, the longer the data migration time, that is, the data migration efficiency cannot be effectively improved, the value of the target migration batch cannot be simply set to the maximum value, but needs to be optimized to maximize the total migration data amount in a short time under the influence of the target migration batch, and then the most appropriate batch migration of the data to be migrated is performed according to the optimal target migration batch, so as to avoid the failure of most data migration due to individual data migration anomaly and / or the long migration time due to the limited thread bearing capacity, and finally effectively improve the data migration efficiency and stability. For this purpose, the embodiment proposes an optimal scheme for the target migration batch, that is, comparing the sizes of the total migration data amounts in 10 seconds under different migration batches (M values).

[0081] For example, the first migration batch is set to "600MB", the preset gradient value is "100", the second migration batch can be "700MB", if the total amount of data migrated within 10 seconds by the second migration batch (M2=700MB) is "7GB", the total amount of data migrated within 10 seconds by the first migration batch (M1=600MB) is "10GB", and the target migration batch is "600MB".

[0082] In one embodiment, the data migration method further comprises: obtaining a migration result of the to-be-migrated data after at least one batch migration; if the migration result contains migration failure, using a binary recursive algorithm to perform abnormal analysis on the to-be-migrated data with migration failure, and screening out abnormal data and normal data; and using idle sub-threads to migrate the normal data to the remote database.

[0083] The binary recursive algorithm is also called binary search algorithm, and the binary search is also called dichotomy search. The binary search has the advantages of less comparison times, fast search speed, and good average performance. However, the binary search requires that the to-be-searched table be an ordered table, and it is difficult to insert and delete.

[0084] In one embodiment, if the migration result contains migration failure, the binary recursive algorithm is used to perform abnormal analysis on the to-be-migrated data with migration failure, and to screen out abnormal data and normal data, which comprises: if the migration result contains migration failure, the to-be-migrated data is divided into a first array and a second array based on the binary recursive algorithm; idle sub-threads are dynamically scheduled to obtain the migration results of the first array and the second array by using the idle sub-threads, and to determine the array with migration failure in the migration results as a target array; and the binary group abnormal analysis operation for the target array is repeatedly performed until the abnormal data and the normal data are screened out.

[0085] In a specific implementation, when migration occurs abnormally, the current to-be-migrated data can be cut into two arrays from the middle, and the cut arrays are recursively called to the current function. If one half of the cut arrays is normally uploaded and the other half is abnormal, the abnormal part is again cut into two arrays from the middle for recursion until all abnormal data is separated out and all normal data is uploaded. In this way, the abnormal data can be efficiently locked, and the data migration efficiency is not affected by individual data abnormality.

[0086] The data migration method in the above embodiments, the server determines the migration data amount of the to-be-migrated data by responding to the data migration request for the to-be-migrated data, determines the target reading strategy according to the migration data amount, reads the to-be-migrated data into the local memory for storage according to the target reading strategy, and dynamically schedules the idle sub-threads to migrate the to-be-migrated data in the local memory to the remote database by using the idle sub-threads. In this way, the appropriate data reading strategy is intelligently selected by analyzing the migration data amount, the data migration strategy is improved, and the data migration efficiency is effectively improved.

[0087] It should be understood that, although Figure 2 The steps in the flowchart of the method are displayed in sequence according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps in the method can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0088] In order to better implement the data migration method provided by the embodiments of the present application, on the basis of the data migration method proposed in the embodiments of the present application, the data migration device is also provided in the embodiments of the present application, as shown in Figure 4 The data migration device 400 includes:

[0089] The request response module 410 is configured to respond to the data migration request for the to-be-migrated data, and determine the migration data amount of the to-be-migrated data.

[0090] The strategy determination module 420 is configured to determine the target reading strategy according to the migration data amount.

[0091] The data migration module 430 is configured to read the to-be-migrated data into the local memory for storage according to the target reading strategy.

[0092] The dynamic scheduling module 440 is configured to dynamically schedule the idle sub-threads to migrate the to-be-migrated data in the local memory to the remote database by using the idle sub-threads.

[0093] In one embodiment, the strategy determination module 420 is further configured to obtain the idle memory amount of the local memory, divide the idle memory amount based on a preset division ratio to obtain a target idle memory amount, and determine the target reading strategy according to the target idle memory amount and the migration data amount.

[0094] In one embodiment, the policy determining module 420 is further configured to determine a preset one-time read policy as the target read policy if the target free memory amount is greater than the migration data amount, and determine a preset row-by-row read policy as the target read policy if the target free memory amount is less than or equal to the migration data amount.

[0095] In one embodiment, the target read policy comprises a row-by-row read policy, and the data migration module 430 is further configured to determine the to-be-migrated data stored in the local memory as read data, perform the step of dynamically scheduling the idle sub-thread for the read data when the read data reaches a preset read cutoff amount, and read the remaining to-be-migrated data reaching the read cutoff amount into the local memory in batches according to the target read policy to wait for remote migration by the idle sub-thread.

[0096] In one embodiment, the dynamic scheduling module 440 is further configured to acquire state information of each sub-thread in the thread pool by the main thread, extract at least one sub-thread with the state information being an idle state as an idle sub-thread, and use the idle sub-thread to batch migrate the to-be-migrated data in the local memory to the remote database according to a preset target migration batch.

[0097] In one embodiment, the data migration apparatus 400 further comprises a migration amount optimizing module configured to determine a first migration batch and a second migration batch for maximizing a data migration amount, a difference between the first migration batch and the second migration batch being a preset gradient value, the second migration batch being greater than the first migration batch, count a total migration data amount obtained by the idle sub-thread in a preset time period according to the first migration batch as a first total migration data amount, count a total migration data amount obtained by the idle sub-thread in the preset time period according to the second migration batch as a second total migration data amount, determine the first migration batch as the target migration batch if the second total migration data amount is less than the first total migration data amount, and determine the second migration batch as the target migration batch if the second total migration data amount is greater than or equal to the first total migration data amount.

[0098] In one embodiment, the data migration apparatus 400 further comprises an exception handling module configured to acquire a migration result of the to-be-migrated data after at least one batch migration, use a binary recursive algorithm to perform exception analysis on the to-be-migrated data with a migration failure if the migration result contains the migration failure, filter out normal data and abnormal data, and use the idle sub-thread to migrate the normal data to the remote database.

[0099] In an embodiment, the exception processing module is further configured to, if the migration result contains migration failure, perform exception analysis on the to-be-migrated data that fails in migration by using a binary recursive algorithm, and filter out abnormal data and normal data, including: if the migration result contains migration failure, dividing the to-be-migrated data into a first array and a second array based on the binary recursive algorithm; dynamically scheduling an idle sub-thread to obtain a migration result of the first array and the second array by using the idle sub-thread, and determining an array whose migration result is migration failure as a target array; repeatedly performing binary grouping exception analysis operation on the target array until abnormal data and normal data are filtered out.

[0100] In the above embodiments, the server determines the migration data volume of the to-be-migrated data by responding to the data migration request for the to-be-migrated data, determines the target reading strategy according to the migration data volume, reads the to-be-migrated data into the local memory according to the target reading strategy, and stores the to-be-migrated data in the local memory, and then dynamically schedules the idle sub-thread to migrate the to-be-migrated data in the local memory to the remote database by using the idle sub-thread. In this way, the appropriate data reading strategy is intelligently selected by analyzing the migration data volume, the data migration strategy is improved, and the data migration efficiency is effectively improved.

[0101] It should be noted that the specific limitations of the data migration device can be referred to the limitations of the data migration method in the above, which will not be repeated here. Each module in the above data migration device can be realized by software, hardware and their combinations in whole or in part. The above modules can be embedded in or independent of the processor in the electronic device in hardware form, or can be stored in the memory in the electronic device in the form of software, so as to be called and executed by the processor to perform the operations corresponding to the above modules.

[0102] In some embodiments of the present application, the data migration device 400 can be realized in the form of a computer program, which can run on the computer device as shown in Figure 5 The memory of the computer device can store various program modules constituting the data migration device 400, such as the request response module 410, the strategy determination module 420, the data migration module 430 and the dynamic scheduling module 440 as shown in Figure 4 The computer program constituted by each program module makes the processor execute the steps in the data migration method of each embodiment of the present application described in the specification. For example, Figure 5 The computer device as shown in Figure 4The request response module 410 in the data migration apparatus 400 shown performs step S201. The computer device can perform step S202 through the policy determination module 420. The computer device can perform step S203 through the data migration module 430. The computer device can perform step S204 through the dynamic scheduling module 440. The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with external computer devices through network connections. The computer program is executed by the processor to implement a data migration method.

[0103] Those skilled in the art can understand that, Figure 4 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0104] In some embodiments of the present application, a computer device is provided, including one or more processors, a memory, and one or more application programs, wherein the one or more application programs are stored in the memory and configured to execute the steps of the data migration method by the processor. The steps of the data migration method can be the steps of the data migration method in each of the above embodiments.

[0105] In some embodiments of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is loaded by the processor, so that the processor executes the steps of the data migration method. The steps of the data migration method can be the steps of the data migration method in each of the above embodiments.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0107] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0108] The above describes in detail a data migration method, device, computer device and storage medium provided by the embodiments of the present application. The principle and implementation mode of the present application are described by applying specific examples. The above embodiment description is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A data migration method, characterized by, The method comprises the following steps: in response to a data migration request for to-be-migrated data, determining a migration data amount of the to-be-migrated data; determining a target reading strategy according to the migration data amount; reading the to-be-migrated data into a local memory according to the target reading strategy; determining a first migration batch and a second migration batch for maximizing the data migration amount, the difference between the first migration batch and the second migration batch being a preset gradient value, and the second migration batch being greater than the first migration batch; statistically determining a total migration data amount of the to-be-migrated data migrated by the idle sub-threads according to the first migration batch within a preset time period as a first total migration data amount; statistically determining a total migration data amount of the to-be-migrated data migrated by the idle sub-threads according to the second migration batch within the preset time period as a second total migration data amount; if the second total migration data amount is less than the first total migration data amount, determining the first migration batch as a target migration batch; if the second total migration data amount is greater than or equal to the first total migration data amount, determining the second migration batch as the target migration batch; dynamically scheduling idle sub-threads to migrate the to-be-migrated data in the local memory to a remote database according to the preset target migration batch. The method further comprises the following steps:

2. The method of claim 1, wherein, obtaining a free memory amount of the local memory; dividing the free memory amount according to a preset division ratio to obtain a target free memory amount; determining the target reading strategy according to the target free memory amount and the migration data amount. The method further comprises the following steps:

3. The method of claim 2, wherein, if the target free memory amount is greater than the migration data amount, determining a preset one-time reading strategy as the target reading strategy; if the target free memory amount is less than or equal to the migration data amount, determining a preset row-by-row reading strategy as the target reading strategy. The target reading strategy comprises a row-by-row reading strategy, and the method further comprises the following steps:

4. The method of claim 1, wherein, determining to-be-migrated data stored in the local memory as read data; when the read data reaches a preset reading cutoff amount, performing the step of dynamically scheduling idle sub-threads on the read data; and reading the to-be-migrated data that reaches the reading cutoff amount in batches into the local memory according to the target reading strategy to wait for remote migration by the idle sub-threads. The method further comprises the following steps:

5. The method of claim 1, wherein, obtaining state information of each sub-thread in a thread pool by a main thread; extracting at least one sub-thread in an idle state from the state information as the idle sub-threads. ​ The idle sub-threads are utilized to batch migrate the to-be-migrated data in the local memory to the remote database according to a preset target migration batch.

6. The method of any one of claims 1 to 5, wherein, Further comprising: obtaining a migration result of the to-be-migrated data after at least one batch migration; if the migration result contains migration failure, using a binary recursive algorithm to perform abnormal analysis on the to-be-migrated data of migration failure, and screening out abnormal data and normal data; using the idle sub-threads to migrate the normal data to the remote database.

7. The method of claim 6, wherein, If the migration result contains migration failure, the binary recursive algorithm is used to divide the to-be-migrated data into a first array and a second array. The idle sub-threads are dynamically scheduled to utilize the idle sub-threads to obtain the migration result of the first array and the second array, and determine the array of which the migration result is migration failure as a target array. The binary grouping abnormal analysis operation for the target array is repeatedly performed until the abnormal data and the normal data are screened out. Comprising:

8. A data migration apparatus, characterized by comprising: a request response module configured to respond to a data migration request for to-be-migrated data, and determine a migration data amount of the to-be-migrated data; a strategy determination module configured to determine a target reading strategy according to the migration data amount; a data migration module configured to read the to-be-migrated data into a local memory according to the target reading strategy; a migration amount optimization module configured to determine a first migration batch and a second migration batch for maximizing a data migration amount, a difference between the first migration batch and the second migration batch being a preset gradient value, and the second migration batch being greater than the first migration batch; a first total migration data amount is obtained by counting a total migration data amount of the idle sub-threads in a preset time period according to the first migration batch; a second total migration data amount is obtained by counting a total migration data amount of the idle sub-threads in the preset time period according to the second migration batch; if the second total migration data amount is less than the first total migration data amount, the first migration batch is determined as a target migration batch; if the second total migration data amount is greater than or equal to the first total migration data amount, the second migration batch is determined as the target migration batch; a dynamic scheduling module configured to dynamically schedule idle sub-threads to utilize the idle sub-threads to batch migrate the to-be-migrated data in the local memory to the remote database according to a preset target migration batch. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the data migration method of any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, ​

Images