Database data migration method and device, computer equipment and readable storage medium

Abstract

The invention relates to the technical field of computers, in particular to a database data migration method and device, computer equipment and a computer readable storage medium. The database data migration method comprises the steps of obtaining data information of target data in a local database; based on a plurality of migration models and the data information, the estimated migration time of the target data in each migration model is calculated, and the migration models are determined at least based on the data volume correlation parameters and the transmission efficiency correlation parameters; determining the shortest estimated migration time in the plurality of estimated migration times; determining the migration model corresponding to the shortest estimated migration time as a target migration model; and migrating the target data to the cloud by using the target migration model. According to the embodiment of the invention, the shortest pre-estimated migration time is obtained by calculating the pre-estimated migration time of the target data in each migration model, and the target data is migrated to the cloud by using the migration model corresponding to the shortest pre-estimated migration time, so that the data transmission efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a database data migration method, device, computer equipment and computer-readable storage medium. Background Art

[0002] The advent of the cloud computing era has enabled a variety of applications and services to be deployed in the cloud. Among them, cloud databases are an extremely important part of cloud computing technology. Cloud database deployment in the cloud can not only serve as a disaster recovery for important systems, but its resource elasticity can also improve service performance. However, as a bridge between local databases and cloud databases, data migration services must not only meet the correctness of migration, but also its performance is crucial.

[0003] Currently, the performance optimization direction of data migration services is mainly to improve the data extraction and writing rate, that is, to improve the performance of database drivers and systems. However, there is little optimization for the scheduling of multiple batches of tasks, and most of them are directly executed according to the principle of first come first served. At the same time, when selecting task machines, the currently idle machines are directly selected for execution. This scheduling method is likely to cause large performance fluctuations due to the order in which tasks arrive, and if the tasks with smaller amounts of database data to be migrated arrive first for scheduling, they may occupy machines with superior performance, resulting in the overall task execution time being extended. Summary of the invention

[0004] Based on this, it is necessary to provide a database data migration method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve data migration efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for migrating database data, the method comprising:

[0006] Get the data information of the target data in the local database;

[0007] Based on a plurality of migration models and the data information, calculating an estimated migration time of the target data in each of the migration models, the migration model being determined based on at least a parameter associated with the data volume and a parameter associated with the transmission efficiency;

[0008] Determining the shortest estimated migration time among the plurality of estimated migration times;

[0009] Determining the migration model corresponding to the shortest estimated migration time as the target migration model;

[0010] The target data is migrated to the cloud using the target migration model.

[0011] In one embodiment, the calculating, based on the plurality of migration models and the data information, the estimated migration time of the target data in each migration model includes:

[0012] Sampling the data information to obtain sampled data;

[0013] Using the sample data and a plurality of the migration models, a plurality of the estimated migration times are determined.

[0014] In one embodiment, the target data includes m databases, each of the databases includes n data tables, and m and n are positive integers;

[0015] The step of sampling the data information to obtain sampled data includes:

[0016] Extracting a database of a first target ratio from the m databases;

[0017] Extracting a data table of a second target ratio from each of the n data tables extracted from the database;

[0018] The data in the extracted data table is used as the sampling data.

[0019] In one embodiment, the first target ratio is the same as the second target ratio.

[0020] In one embodiment, the step of calculating the estimated migration time of the target data in each migration model based on the plurality of migration models and the data information includes:

[0021] Determine the specifications of the data migration system;

[0022] The step of calculating the estimated migration time of the target data in each migration model based on the plurality of migration models and the data information includes:

[0023] Based on the data information, determining the number of instructions;

[0024] Determining instruction time based on the instruction quantity and the specification parameters;

[0025] Determining a transmission time based on the data information and the specification parameters;

[0026] The estimated migration time is calculated based on the command time and the transmission time.

[0027] In one embodiment, the specification parameters of the data migration system include at least one of an average frequency, an average number of cores, and an average bandwidth of the data migration system.

[0028] In one embodiment, the step of migrating the target data to the cloud using the target migration model includes:

[0029] Using the target migration model, sorting the target data and generating a task queue;

[0030] Based on the task queue, the target data is migrated to the cloud.

[0031] In a second aspect, the present application further provides a database data migration device, the device comprising:

[0032] An acquisition module is used to obtain data information of target data in a local database;

[0033] a calculation module, configured to calculate an estimated migration time of the target data in each migration model based on a plurality of migration models and the data information, wherein the migration model is determined based on at least a parameter associated with the data volume and a parameter associated with the transmission efficiency;

[0034] A first determining module, configured to determine the shortest estimated migration time among the plurality of estimated migration times;

[0035] A second determination module, configured to determine the migration model corresponding to the shortest estimated migration time as a target migration model;

[0036] A migration module is used to migrate the target data to the cloud using the target migration model.

[0037] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0038] Get the data information of the target data in the local database;

[0039] Based on a plurality of migration models and the data information, calculating an estimated migration time of the target data in each of the migration models, the migration model being determined based on at least a parameter associated with the data volume and a parameter associated with the transmission efficiency;

[0040] Determining the shortest estimated migration time among the multiple estimated migration times;

[0041] Determining the migration model corresponding to the shortest estimated migration time as the target migration model;

[0042] The target data is migrated to the cloud using the target migration model.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0044] Get the data information of the target data in the local database;

[0045] Based on a plurality of migration models and the data information, calculating an estimated migration time of the target data in each of the migration models, the migration model being determined based on at least a parameter associated with the data volume and a parameter associated with the transmission efficiency;

[0046] Determining the shortest estimated migration time among the multiple estimated migration times;

[0047] Determining the migration model corresponding to the shortest estimated migration time as the target migration model;

[0048] The target data is migrated to the cloud using the target migration model.

[0049] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0050] Get the data information of the target data in the local database;

[0051] Based on a plurality of migration models and the data information, calculating an estimated migration time of the target data in each of the migration models, the migration model being determined based on at least a parameter associated with the data volume and a parameter associated with the transmission efficiency;

[0052] Determining the shortest estimated migration time among the multiple estimated migration times;

[0053] Determining the migration model corresponding to the shortest estimated migration time as the target migration model;

[0054] The target data is migrated to the cloud using the target migration model.

[0055] The above-mentioned database data migration method, apparatus, computer device, computer-readable storage medium and computer program product calculate the estimated migration time of the target data in each migration model to obtain the shortest estimated migration time, and use the migration model corresponding to the shortest estimated migration time to migrate the target data to the cloud, thereby improving data transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 A schematic diagram of the process of database data migration method provided in the first embodiment;

[0058] Figure 2 A schematic diagram of the flow of a database data migration method provided in the second embodiment;

[0059] Figure 3 A schematic diagram of a flow chart of a database data migration method provided in a third embodiment;

[0060] Figure 4 A schematic diagram of a flow chart of a database data migration method provided in a fourth embodiment;

[0061] Figure 5 A schematic diagram of a flow chart of a database data migration method provided in a fifth embodiment;

[0062] Figure 6 A schematic diagram of a data source provided in an embodiment;

[0063] Figure 7 A schematic diagram of machine specifications provided in one embodiment;

[0064] Figure 8 A schematic diagram of a flow chart of a database data migration method provided in a sixth embodiment;

[0065] Fig. 9 A schematic diagram of a task scheduling framework provided in one embodiment;

[0066] Fig.10 A structural block diagram of a database data migration device in one embodiment;

[0067] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] The database data migration method provided in this application can be applied to an environment where a terminal and a server interact. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0070] In one embodiment, Figure 1 As shown, a method for database data migration is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The method for database data migration in this embodiment includes the following steps:

[0071] Step S100: Acquire data information of target data in the local database.

[0072] Step S300: Based on a plurality of migration models and data information, an estimated migration time of target data in each migration model is calculated, wherein the migration model is determined based on at least a parameter associated with data volume and a parameter associated with transmission efficiency.

[0073] Step S400: determining the shortest estimated migration time among multiple estimated migration times.

[0074] Step S500: determining the migration model corresponding to the shortest estimated migration time as the target migration model.

[0075] Step S600: Migrate target data to the cloud using the target migration model.

[0076] In step S100, the specific scope of the target data may be determined. As an example, the files, databases or data sets included in the target data may be determined. The target data may be stored locally, and the target data needs to be uploaded to the cloud.

[0077] In step S300, the migration time can be evaluated by different migration models to find the optimal migration solution. As an example, at this time, the data information of the target data can be brought into different migration models and solved to obtain the estimated migration time of each migration model.

[0078] The migration model may include at least a data volume parameter and a transmission efficiency parameter. As an example, the data volume parameter may be used to represent the number of migration instructions for target data, and the transmission efficiency parameter may be used to represent the bandwidth, transmission protocol, etc. of the migration machine.

[0079] Of course, this embodiment does not limit the specific form and specific number of migration models. As an example, the migration model can consider two stages: task sorting and machine selection. On the one hand, task sorting can include rules such as the priority of the largest number of library tables, the priority of the largest amount of data, and the priority of the largest average amount of library table data. In the priority of the largest number of library tables, the priority of migrating the largest number of library tables can reduce the overhead of frequently establishing database connections in subsequent tasks. In the priority of the largest amount of data, the priority of migrating the largest amount of data can maximize the use of bandwidth and reduce the overall delay caused by waiting for the completion of the transmission of large tables. In the priority of the largest average amount of library table data, the priority of migrating the largest average library table can ensure that there will be no obvious resource bottlenecks during the migration process. On the other hand, machine selection can include rules such as the earliest available time priority, the shortest execution time priority, and the maximum idle computing amount priority. In the earliest available time priority, by selecting the earliest available machine, it can be ensured that the migration task can start as soon as possible, thereby shortening the overall migration cycle. In the priority of the shortest execution time, by selecting the machine that can complete the task the fastest, the migration progress can be accelerated. In the maximum idle computing capacity priority, by selecting the machine with the maximum idle computing capacity, it can be ensured that all machines can be used evenly, which helps to avoid the situation where some machines are overloaded while other machines are idle. The migration model can combine multiple rules of the above two factors to obtain multiple migration models. Of course, the above stages and rules are only for exemplary description. In actual embodiments, other stages and rules can also be set.

[0080] In step S400, the shortest estimated migration time may be determined among all the estimated migration times. As an example, at this time, all the estimated migration times may be sorted in ascending order, and the shortest estimated migration time may be obtained.

[0081] In step S500, the migration model corresponding to the shortest estimated migration time is determined as the target migration model. Of course, it can also be ensured that the target migration model meets the actual migration requirements.

[0082] In step S600, the target migration model can be used to perform data migration operations to migrate the target data to the cloud. During this process, the target data can also be segmented, compressed, transmitted, etc. according to the requirements of the target migration model. In addition, the migration process is monitored to ensure that the data integrity and transmission speed meet expectations. Furthermore, after the migration is completed, necessary verification and testing can be performed to ensure that the target data can be correctly accessed and used in the cloud.

[0083] This embodiment constructs a mathematical model and constraints for the data migration task orchestration optimization problem, a rule-based heuristic algorithm framework, and heuristic rules for different stages. By calculating the estimated migration time of the target data in each migration model, the shortest estimated migration time is obtained, and the migration model corresponding to the shortest estimated migration time is used to migrate the target data to the cloud, thereby improving data transmission efficiency.

[0084] Further, the inventor believes that, for data migration tasks, the more database tables are migrated, the more instructions need to be executed. If most of the tables of the database to be migrated are tables with a small amount of data, most of the time when the program is executed will be consumed in establishing a connection with the database and executing data processing. If there are fewer tables to be migrated, but the amount of data is large, most of the time for program execution is waiting for network I / O. Both of these situations will affect the execution time of the program. Therefore, in order to balance the relationship between the two, the library to be migrated can be verified by using rules such as the largest number of database tables, the largest amount of data, and the largest average amount of database table data, so as to find the most suitable rules and determine the execution order of the tasks. Moreover, choosing a suitable machine to execute tasks will also affect the final completion time of all tasks. Under the premise that the machine resources can meet the task execution, if the machine with the earliest available time is selected, the utilization rate of the machine can be maximized. Selecting the machine with the shortest execution time can reduce the waiting time of the task; and selecting the maximum idle computing amount first (defined as the product of the time the core of the machine lasts when the computing task is not running and the core frequency) can achieve load balancing of all machines to a certain extent. Therefore, the rules of earliest available time first, shortest execution time first, maximum idle computing amount first, etc. will produce different effects for machines of different specifications and different databases. This embodiment provides examples of migration models of different combinations, so that a more accurate target migration model can be obtained, thereby improving the data transmission efficiency. This embodiment decomposes the task scheduling into several different stages (task sorting and machine selection), and calls different rules at different stages, so that this embodiment can obtain a better solution to the non-deterministic problem (NP-complete problem) of this polynomial complexity to a certain extent, thereby achieving better results and reducing the full data transmission time.

[0085] In one embodiment, Figure 2 As shown, step S300 includes:

[0086] Step S310: sampling the data information to obtain sampled data.

[0087] Step S320: using the sample data and multiple migration models to determine multiple estimated migration times.

[0088] In step S310 to step S320, representative sample data may be extracted from the target data, so that the sample data may be used in subsequent steps to estimate the migration time of the entire data set, thereby speeding up the calculation efficiency of the estimated migration time of the target data.

[0089] As an example, when sampling data information, random sampling, systematic sampling, stratified sampling, etc. can be used. When obtaining the sampled data, the sample size, data format, etc. of the sampled data can be recorded. Afterwards, in each migration model, the estimated migration time is calculated based on the sampled data.

[0090] Specifically, the target data includes m databases, each database includes n data tables, and m and n are positive integers. As an example, the target data may include 3000 databases, each database may include 200 data tables. The above data are only examples. In actual embodiments, the number of databases and data tables is not limited to the above data. In addition, Figure 3 As shown, step S310 may include:

[0091] Step S311: extracting a database with a first target ratio from the m databases.

[0092] Step S312: extracting a data table with a second target ratio from each of the n data tables extracted from the database.

[0093] Step S313: Use the data in the extracted data table as sampling data.

[0094] In step S311 to step S313, as an example, the first target ratio can be 1%, 5%, 10%, 50%, etc. The second target ratio can be 0.5%, 2%, 30%, 70%, etc. This embodiment does not limit the specific values ​​of the first target ratio and the second target ratio. Further, the first target ratio and the second target ratio can be the same. For example, the first target ratio and the second target ratio can both be 10%, etc. In addition, for example, the database of the first target ratio can also be a database of the top 10% or top 20% of the data volume, etc. The data table of the second target ratio can also be a data table of the top 5% or top 30% of importance, etc.

[0095] After extracting the data table of the second target ratio, the data in all the extracted data tables may be integrated together to form the final sampling data.

[0096] In this embodiment, sample data is obtained by sampling, and the sample data is used to estimate the time required for different migration models to migrate the entire data set, which helps to better understand the performance of different migration models and efficiently select the optimal migration model. In addition, by sampling the migration library, data with certain representativeness is selected, and different rules at different stages are tested to select a migration rule combination that is more suitable for the current database. Compared with the fixed scheduling steps of other technologies, it is more customized to meet the needs of different databases and has universal applicability.

[0097] In one embodiment, Figure 4 As shown, before step S300, the following steps are included:

[0098] Step S200: Determine specification parameters of the data migration system.

[0099] Accordingly, step S300 includes:

[0100] Step S330: Determine the number of instructions based on the data information.

[0101] Step S331: Determine the instruction time based on the instruction quantity and specification parameters.

[0102] Step S332: Determine the transmission time based on the data information and specification parameters.

[0103] Step S333: Calculate the estimated migration time based on the command time and the transmission time.

[0104] In step S200 , the specification parameters of the data migration system include at least one of an average frequency, an average number of cores, and an average bandwidth of the data migration system.

[0105] In step S330, for the data migration task, the more database (target data) tables are migrated, the more instructions are required to be executed. As an example, in this step, the types of instructions that may be required (e.g., data reading, data conversion, data writing, etc.) can be determined based on the data information, and then the number of each instruction type can be determined, and finally the number of instructions can be obtained.

[0106] In step S331, during the data migration task, it is necessary to wait for network I / O, which will affect the execution time of data migration. Therefore, the instruction time can be determined by the instruction quantity and specification parameters.

[0107] In step S332, the parameters of the data migration system such as average frequency, average number of cores, average bandwidth, etc. may affect the transmission time. As an example, in this step, the network transmission time may be estimated by dividing the data information (data volume) by the average bandwidth.

[0108] In step S333 , the estimated migration time may be the sum of the time for determining each instruction, the time for executing the instruction, and the time for data transmission.

[0109] In this embodiment, it is necessary to estimate the time required for the task to run based on the resources of the selected machine and the amount of data in the database to be migrated by the task. Specifically, this embodiment uses the average frequency and core number of all machines and the average bandwidth to estimate the execution time of the task, so that the task execution status can be estimated comprehensively.

[0110] In one embodiment, Figure 5 As shown, step S600 includes:

[0111] Step S610: Use the target migration model to sort the target data and generate a task queue.

[0112] Step S620: Migrate target data to the cloud based on the task queue.

[0113] In step S610, the target data may be sorted and grouped to generate an ordered task queue. As an example, the target data may be sorted according to one or more sorting criteria, such as data size, data type, data importance, dependencies between data, etc. Then, the target data is sorted and grouped according to the determined sorting criteria. The sorted and grouped data is then converted into a series of migration tasks, and these tasks are added to the task queue in a certain order (such as priority, timestamp, etc.).

[0114] In step S620, the target data can be migrated to the cloud, provided that the cloud storage system is ready and has sufficient storage space to receive the migrated data. As an example, at this point, the first task can be taken out from the task queue, and the data can be migrated to the cloud according to the parameters and configuration of the target migration model. Of course, steps such as data compression, encryption, and transmission may need to be processed during the migration process. After the migration task is completed, the task is deleted from the task queue, and the next task is taken out to continue execution. This process is repeated until all tasks in the task queue are migrated.

[0115] In addition, the execution of the migration task can be monitored in real time, including the amount of data migrated, the migration speed, the remaining time, etc. After the migration is completed, the migrated data can also be verified and tested to ensure the integrity and correctness of the data. As an example, the migrated data can be verified by comparing the data before and after the migration, checking the readability and accessibility of the data, etc.

[0116] In this embodiment, the target data is migrated to the cloud by forming a task queue, ensuring that the data can be smoothly and efficiently migrated to the cloud, improving the migration efficiency, and ensuring the smooth completion of the migration task. Moreover, by sorting the tasks, the first-come-first-served strategy in most transmission services can be avoided. The transmission time fluctuation caused by the order of task creation and the amount of task transmission data can be more convenient for users to evaluate the transmission time.

[0117] The following is an exemplary description of the application of the database data migration method provided in this application. Figure 6 and Figure 7 The assumed data source and machine specifications are shown as examples. Figure 8 The specific process of the database data migration method is exemplified.

[0118] The migration model can be:

[0119]

[0120] Where n represents the total number of tasks, μ k represents the number of machines of the kth specification, x i；k,l Indicates whether task i is executed in the lth machine of the kth specification, τ i represents the number of instructions for the i-th task, represents the processor frequency of the lth machine of the kth specification, W i represents the amount of data to be migrated by task i, represents the uplink bandwidth of the lth machine of the kth specification, represents the downlink bandwidth of the lth machine of the kth specification. In addition, when executing a task, the remaining resources of the machine must also meet the task requirements, that is, in represents the remaining resources of the lth machine of the kth specification, is a resource vector, and a total of m resources are required.

[0121] Each task can only be executed on one machine, so the following conditions must also be met:

[0122]

[0123] The task scheduling framework proposed in this embodiment is as follows Fig. 9 As shown in the figure, the task scheduler is responsible for arranging the machines for task execution, the machine agent is responsible for managing and distributing tasks to machines of different specifications, and also obtains the current resource status of the machine through the resource script and feeds it back to the scheduler. The task sequencer is responsible for sorting all tasks and determining a scheduling order. The console and the machine agent connect the source library (local) and the target library (cloud).

[0124] Since a single migration task can only migrate one data source, when sampling the library to be migrated (target data), each data source can be taken as a whole, and a simple random sampling can be performed on each data source according to the proportion of the number of its libraries in the total number of libraries of all data sources. A total of 10% of the databases in the total library are selected, and then 10% of the tables in each library are randomly sampled to form the library to be migrated (sampled data) during the experiment. That is, the first target ratio and the second target ratio are both 10%.

[0125] When calculating the task running time, Calculate the execution time, where τ i Directly related to the number of database tables, W i Directly related to the amount of data.

[0126] For the migration model, you can use the three task sorting rules and three machine selection rules mentioned above to form different orchestration algorithms. Sample some tables in the database to be migrated for migration experiments, compare the migration time required for different combinations, and select the most suitable rule combination for this database to complete the migration of the entire database.

[0127] See also Figure 6 , shows an example of migrating four data sources, with the amount of data in each table evenly distributed. Figure 7 , showing the machine resources as an example. There is one machine for each specification, and the machine core frequency is 2.1GHz. Figure 8 First, we sample the databases to be migrated. We select 300 databases for data source 1 and 20 tables for each database. We do the same for data sources 2, 3, and 4 and calculate the estimated execution time of each task. For simplicity, we take (number of databases × number of tables × 10 14 ) is the number of instructions that the task needs to execute, which is divided by the average computing power of the machine (here 12×10 9 ) can be used to get the time required to execute the instruction. The network transmission time can be obtained by dividing the data volume by the bandwidth, and finally the task execution time can be obtained. Since each task can only migrate one data source, 4 migration tasks will be created for the 4 databases to be migrated. According to the task sorting rules mentioned above, migration experiments are carried out in the database sample according to different rules. A total of 3×3=9 migrations are carried out to find The smallest migration method is to apply these two rules to the migration of all databases in the four data sources to complete all full migrations.

[0128] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0129] Based on the same inventive concept, the embodiment of the present application also provides a database data migration device for implementing the database data migration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more database data migration device embodiments provided below can refer to the limitations of the database data migration method above, and will not be repeated here.

[0130] In an exemplary embodiment, Fig.10 As shown, a database data migration device is provided, including: an acquisition module, a calculation module, a first determination module, a second determination module and a migration module, wherein:

[0131] The acquisition module is used to obtain data information of target data in the local database.

[0132] A calculation module is used to calculate the estimated migration time of the target data in each migration model based on multiple migration models and the data information, and the migration model is determined based on at least a data volume-related parameter and a transmission efficiency-related parameter.

[0133] The first determining module is used to determine the shortest estimated migration time among the multiple estimated migration times.

[0134] The second determining module is used to determine the migration model corresponding to the shortest estimated migration time as the target migration model.

[0135] A migration module is used to migrate the target data to the cloud using the target migration model.

[0136] In one embodiment, the calculation module is used to sample data information to obtain sampled data; and to determine multiple estimated migration times using the sampled data and multiple migration models.

[0137] In one embodiment, the target data includes m databases, each database includes n data tables, and m and n are positive integers; the calculation module is used to extract a database with a first target ratio from the m databases; extract a data table with a second target ratio from the n data tables of each extracted database; and use the data in the extracted data table as sampling data.

[0138] In one embodiment, the first target ratio is the same as the second target ratio.

[0139] In one embodiment, the database data migration device includes a third determination module for determining specification parameters of the data migration system. The calculation module is used to determine the number of instructions based on the data information; determine the instruction time based on the number of instructions and the specification parameters; determine the transmission time based on the data information and the specification parameters; and calculate the estimated migration time based on the instruction time and the transmission time.

[0140] In one embodiment, the specification parameter of the data migration system includes at least one of an average frequency, an average number of cores, and an average bandwidth of the data migration system.

[0141] In one embodiment, the migration module is further used to use the target migration model to sort the target data and generate a task queue; based on the task queue, the target data is migrated to the cloud.

[0142] Each module in the above-mentioned database data migration device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules. In addition, the database data migration device can also include other modules for executing other steps or methods mentioned in this application. For example, the database data migration device can also include a first module. A second module, etc.

[0143] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.11As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data migration data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a database data migration method is implemented.

[0144] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0145] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0146] Step S100: Acquire data information of target data in the local database.

[0147] Step S300: Based on a plurality of migration models and data information, an estimated migration time of target data in each migration model is calculated, wherein the migration model is determined based on at least a parameter associated with data volume and a parameter associated with transmission efficiency.

[0148] Step S400: determining the shortest estimated migration time among multiple estimated migration times.

[0149] Step S500: determining the migration model corresponding to the shortest estimated migration time as the target migration model.

[0150] Step S600: Migrate target data to the cloud using the target migration model.

[0151] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0152] Step S310: sampling the data information to obtain sampled data.

[0153] Step S320: using the sample data and multiple migration models to determine multiple estimated migration times.

[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0155] Step S311: extracting a database with a first target ratio from the m databases.

[0156] Step S312: extracting a data table with a second target ratio from each of the n data tables extracted from the database.

[0157] Step S313: Use the data in the extracted data table as sampling data.

[0158] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0159] Step S200: Determine specification parameters of the data migration system.

[0160] Step S330: Determine the number of instructions based on the data information.

[0161] Step S331: Determine the instruction time based on the instruction quantity and specification parameters.

[0162] Step S332: Determine the transmission time based on the data information and specification parameters.

[0163] Step S333: Calculate the estimated migration time based on the command time and the transmission time.

[0164] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0165] Step S610: Use the target migration model to sort the target data and generate a task queue.

[0166] Step S620: Migrate target data to the cloud based on the task queue.

[0167] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0168] Step S100: Acquire data information of target data in the local database.

[0169] Step S300: Based on a plurality of migration models and data information, an estimated migration time of target data in each migration model is calculated, wherein the migration model is determined based on at least a parameter associated with data volume and a parameter associated with transmission efficiency.

[0170] Step S400: determining the shortest estimated migration time among multiple estimated migration times.

[0171] Step S500: determining the migration model corresponding to the shortest estimated migration time as the target migration model.

[0172] Step S600: Migrate target data to the cloud using the target migration model.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0174] Step S310: sampling the data information to obtain sampled data.

[0175] Step S320: using the sample data and multiple migration models to determine multiple estimated migration times.

[0176] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0177] Step S311: extracting a database with a first target ratio from the m databases.

[0178] Step S312: extracting a data table with a second target ratio from each of the n data tables extracted from the database.

[0179] Step S313: Use the data in the extracted data table as sampling data.

[0180] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0181] Step S200: Determine specification parameters of the data migration system.

[0182] Step S330: Determine the number of instructions based on the data information.

[0183] Step S331: Determine the instruction time based on the instruction quantity and specification parameters.

[0184] Step S332: Determine the transmission time based on the data information and specification parameters.

[0185] Step S333: Calculate the estimated migration time based on the command time and the transmission time.

[0186] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0187] Step S610: Use the target migration model to sort the target data and generate a task queue.

[0188] Step S620: Migrate target data to the cloud based on the task queue.

[0189] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0190] Step S100: Acquire data information of target data in the local database.

[0191] Step S300: Based on a plurality of migration models and data information, an estimated migration time of target data in each migration model is calculated, wherein the migration model is determined based on at least a parameter associated with data volume and a parameter associated with transmission efficiency.

[0192] Step S400: determining the shortest estimated migration time among multiple estimated migration times.

[0193] Step S500: determining the migration model corresponding to the shortest estimated migration time as the target migration model.

[0194] Step S600: Migrate target data to the cloud using the target migration model.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] Step S310: sampling the data information to obtain sampled data.

[0197] Step S320: using the sample data and multiple migration models to determine multiple estimated migration times.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0199] Step S311: extracting a database with a first target ratio from the m databases.

[0200] Step S312: extracting a data table with a second target ratio from each of the n data tables extracted from the database.

[0201] Step S313: Use the data in the extracted data table as sampling data.

[0202] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0203] Step S200: Determine specification parameters of the data migration system.

[0204] Step S330: Determine the number of instructions based on the data information.

[0205] Step S331: Determine the instruction time based on the instruction quantity and specification parameters.

[0206] Step S332: Determine the transmission time based on the data information and specification parameters.

[0207] Step S333: Calculate the estimated migration time based on the command time and the transmission time.

[0208] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0209] Step S610: Use the target migration model to sort the target data and generate a task queue.

[0210] Step S620: Migrate target data to the cloud based on the task queue.

[0211] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0212] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0213] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A database data migration method, characterized in that: The method comprises: Get the data information of the target data in the local database; Based on a plurality of migration models and the data information, calculating an estimated migration time of the target data in each of the migration models, the migration model being determined based on at least a parameter associated with the data volume and a parameter associated with the transmission efficiency; Determining the shortest estimated migration time among the plurality of estimated migration times; Determining the migration model corresponding to the shortest estimated migration time as the target migration model; The target data is migrated to the cloud using the target migration model.

2. The database data migration method according to claim 1, characterized in that: The step of calculating the estimated migration time of the target data in each migration model based on the plurality of migration models and the data information includes: Sampling the data information to obtain sampled data; Using the sample data and a plurality of the migration models, a plurality of the estimated migration times are determined.

3. The database data migration method according to claim 2, characterized in that: The target data includes m databases, each of which includes n data tables, where m and n are positive integers; The step of sampling the data information to obtain sampled data includes: Extracting a database of a first target ratio from the m databases; Extracting a data table of a second target ratio from each of the n data tables extracted from the database; The data in the extracted data table is used as the sampling data.

4. The database data migration method according to claim 3, characterized in that: The first target ratio is the same as the second target ratio.

5. The database data migration method according to claim 1, characterized in that: The step of calculating the estimated migration time of the target data in each migration model based on the plurality of migration models and the data information includes: Determine the specifications of the data migration system; The step of calculating the estimated migration time of the target data in each migration model based on the plurality of migration models and the data information includes: Based on the data information, determining the number of instructions; Determining instruction time based on the instruction quantity and the specification parameters; Determining a transmission time based on the data information and the specification parameters; The estimated migration time is calculated based on the command time and the transmission time.

6. The database data migration method according to claim 5, characterized in that: The specification parameters of the data migration system include at least one of an average frequency, an average number of cores, and an average bandwidth of the data migration system.

7. The database data migration method according to claim 1, characterized in that: The using the target migration model to migrate the target data to the cloud includes: Using the target migration model, sorting the target data and generating a task queue; Based on the task queue, the target data is migrated to the cloud.

8. A database data migration device, characterized in that: The device comprises: An acquisition module, used to acquire data information of target data, wherein the target data includes local data; A calculation module, configured to calculate an estimated migration time of the target data in each migration model based on a plurality of migration models and the data information, wherein the migration model at least includes a data volume parameter and a transmission efficiency parameter; A first determining module, configured to determine the shortest estimated migration time among the plurality of estimated migration times; A second determination module, configured to determine the migration model corresponding to the shortest estimated migration time as a target migration model; A migration module is used to migrate the target data to the cloud using the target migration model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images