Data migration method and device, electronic equipment and storage medium

Abstract

The invention discloses a data migration method and device, electronic equipment and a storage medium, and belongs to the technical field of databases, in the method, for a distributed database deployed to a computing power network, when it is determined that a data migration condition is met according to service representation data of the database, multiple pieces of to-be-migrated second data are screened from the database, and the data migration condition is determined to be met; and according to the access requirements of the multiple pieces of second data and the network resource information of the database, by taking minimization of access delay, load balancing and minimization of network resource consumption of data migration as targets, a migration strategy of the multiple pieces of second data is determined, and data migration is carried out, so that when the data access service provided by the database is not too good, the migration strategy of the multiple pieces of second data is determined. According to the method, the migration strategy can be determined by combining the access requirement of the to-be-migrated data and the network resource information of the database and comprehensively considering the multiple targets of minimum access delay, load balancing and minimum network resource consumption of data migration, the dynamic change characteristics of a computing power network are better met, and the data access performance of the database can be well improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data migration method, device, electronic device and storage medium. Background Art

[0002] For distributed databases (referred to as databases for short), data storage strategy is one of the key technologies to ensure data availability.

[0003] At present, for databases deployed on the traditional Internet, the number of data centers is relatively small, the network between data centers is relatively stable, and the network delay between different data centers is not much different. Storing data based on load balancing strategies can ensure that the database can provide good data access services. However, for databases deployed on the computing power network, the number of data centers has increased sharply, and the network delay between different data centers has also varied greatly. The data storage strategy in related technologies can no longer guarantee that the database can provide good data access services. Summary of the invention

[0004] The embodiments of the present application provide a data migration method, device, electronic device and storage medium to solve the problem in the related art that it is difficult for a database deployed in a computing network to provide good data access services.

[0005] In a first aspect, an embodiment of the present application provides a data migration method, including:

[0006] Determining whether the database meets the data migration condition according to the service characterization data of the database, wherein the database is a distributed database that provides data access services through a computing power network;

[0007] If it is determined that the data migration condition is met, selecting a plurality of second data to be migrated from the plurality of first data in the database according to the access log of the database;

[0008] Determine, according to the access requirements of the plurality of second data and the network resource information of the database, a migration strategy for the plurality of second data with the goal of minimizing access delay, load balancing, and minimizing network resource consumption for data migration;

[0009] A migration instruction is sent to the database according to the migration strategy to migrate the plurality of second data.

[0010] In some embodiments, the service characterization data includes the load of each computing node and the access delay of each application in the database, and determining whether the database meets the data migration condition according to the service characterization data of the database includes:

[0011] Determine the load of each data center according to the load of each computing node and the corresponding relationship between each computing node and the data center of the database, and determine the load balancing degree of the database according to the load of each data center;

[0012] Determining an average access delay of the database according to the access delay of each application and the total number of applications served by the database;

[0013] If the load balancing degree reaches a preset load upper limit, and / or the average access delay reaches a preset delay upper limit, it is determined that the data migration condition is met.

[0014] In some embodiments, selecting the plurality of second data to be migrated from the plurality of first data in the database according to the access log of the database includes:

[0015] Obtaining, from the access log, access times of multiple pieces of first data in M ​​historical periods;

[0016] Inputting the number of accesses of the plurality of first data in M ​​historical cycles into a pre-trained long short-term memory (LSTM) network to obtain the number of accesses of the plurality of first data in N future cycles, where M is an integer greater than N;

[0017] The plurality of second data are selected from the plurality of first data according to the number of accesses to the plurality of first data in N future cycles.

[0018] In some embodiments, according to the access requirements of the plurality of second data and the network resource information of the database, with the goal of minimizing access delay, load balancing, and minimizing network resource consumption of data migration, determining the migration strategy of the plurality of second data includes:

[0019] According to the access requirements of the plurality of second data and the preset selection rule, determining L correspondences between the plurality of second data and the plurality of data centers of the database, where L is an integer greater than 1;

[0020] Determine the loss value of the corresponding relationship according to the access delay of each application after migration, the load of each data center, and the network resources consumed by the migration according to the corresponding relationship;

[0021] The preset selection rule is updated to execute the step of determining L types of correspondences between the plurality of second data and the plurality of data centers of the database according to the access requirements of the plurality of second data and the preset selection rule;

[0022] Until the cycle stop condition is met, the corresponding relationship between the storage nodes and the second data in each data center is determined according to the corresponding relationship with the minimum loss value and the data storage rule of a single data center to obtain the migration strategy.

[0023] In some embodiments, according to the access requirements of the plurality of second data and according to a preset selection rule, determining L types of correspondences between the plurality of second data and the plurality of data centers of the database includes:

[0024] Perform the following steps L times:

[0025] Determine the probability of the target data being migrated to the data center according to the attraction factor, heuristic factor and random factor of each data center for the target data, wherein the target data is the data with the i-th highest access count among the plurality of second data, i is 1 initially, the attraction factor is used to characterize the probability of the data center being selected by the second data other than the target data, the heuristic factor is used to characterize the degree to which the data center satisfies the access requirements of the target data, and the random factor is randomly generated;

[0026] Determining the data center corresponding to the target data according to the probability of the target data being migrated to each data center;

[0027] Update i to i+1, and execute the step of determining the probability of migrating the target data to the data center based on the attraction factor, inspiration factor and random factor of each data center for the target data, until i is greater than C, and a corresponding relationship is obtained, where C is the total number of the multiple second data.

[0028] In some embodiments, updating the preset selection rule includes:

[0029] For each data center, if the server corresponding to the application to which the target data belongs is a server in the data center, the attraction factor of the data center for the target data is updated using a preset attenuation coefficient and a minimum loss value;

[0030] If the server corresponding to the application to which the target data belongs is not a server in the data center, the attenuation coefficient is used to update the attraction factor of the data center for the target data.

[0031] In a second aspect, an embodiment of the present application provides a data migration device, including:

[0032] A judgment module, used to determine whether the database meets the data migration conditions according to the service characterization data of the database, wherein the database is a distributed database that provides data access services through a computing power network;

[0033] a screening module, configured to screen the plurality of second data to be migrated from the plurality of first data in the database according to the access log of the database if it is determined that the data migration condition is met;

[0034] A determination module, configured to determine a migration strategy for the plurality of second data according to access requirements of the plurality of second data and network resource information of the database, with the goal of minimizing access delay, load balancing, and minimizing network resource consumption for data migration;

[0035] A sending module is used to send a migration instruction to the database according to the migration strategy to migrate the plurality of second data.

[0036] In some embodiments, the service characterization data includes the load of each computing node in the database and the access delay of each application, and the judgment module is specifically used to:

[0037] Determine the load of each data center according to the load of each computing node and the corresponding relationship between each computing node and the data center of the database, and determine the load balancing degree of the database according to the load of each data center;

[0038] Determining an average access delay of the database according to the access delay of each application and the total number of applications served by the database;

[0039] If the load balancing degree reaches a preset load upper limit, and / or the average access delay reaches a preset delay upper limit, it is determined that the data migration condition is met.

[0040] In some embodiments, the screening module is specifically configured to:

[0041] Obtaining, from the access log, access times of multiple pieces of first data in M ​​historical periods;

[0042] Inputting the number of accesses of the plurality of first data in M ​​historical cycles into a pre-trained long short-term memory (LSTM) network to obtain the number of accesses of the plurality of first data in N future cycles, where M is an integer greater than N;

[0043] The plurality of second data are selected from the plurality of first data according to the number of accesses to the plurality of first data in N future cycles.

[0044] In some embodiments, the determination module is specifically configured to:

[0045] According to the access requirements of the plurality of second data and the preset selection rule, determining L correspondences between the plurality of second data and the plurality of data centers of the database, where L is an integer greater than 1;

[0046] Determine the loss value of the corresponding relationship according to the access delay of each application after migration, the load of each data center, and the network resources consumed by the migration according to the corresponding relationship;

[0047] The preset selection rule is updated to execute the step of determining L types of correspondences between the plurality of second data and the plurality of data centers of the database according to the access requirements of the plurality of second data and the preset selection rule;

[0048] Until the cycle stop condition is met, the corresponding relationship between the storage nodes and the second data in each data center is determined according to the corresponding relationship with the minimum loss value and the data storage rule of a single data center to obtain the migration strategy.

[0049] In some embodiments, when determining L types of correspondences between the plurality of second data and the plurality of data centers of the database according to the access requirements of the plurality of second data and the preset selection rules, the determination module is specifically used to:

[0050] Perform the following steps L times:

[0051] Determine the probability of the target data being migrated to the data center according to the attraction factor, heuristic factor and random factor of each data center for the target data, wherein the target data is the data with the i-th highest access count among the plurality of second data, i is 1 initially, the attraction factor is used to characterize the probability of the data center being selected by the second data other than the target data, the heuristic factor is used to characterize the degree to which the data center satisfies the access requirements of the target data, and the random factor is randomly generated;

[0052] Determining the data center corresponding to the target data according to the probability of the target data being migrated to each data center;

[0053] Update i to i+1, and execute the step of determining the probability of migrating the target data to the data center based on the attraction factor, inspiration factor and random factor of each data center for the target data, until i is greater than C, and a corresponding relationship is obtained, where C is the total number of the multiple second data.

[0054] In some embodiments, when updating the preset selection rule, the determination module is specifically configured to:

[0055] For each data center, if the server corresponding to the application to which the target data belongs is a server in the data center, the attraction factor of the data center for the target data is updated using a preset attenuation coefficient and a minimum loss value;

[0056] If the server corresponding to the application to which the target data belongs is not a server in the data center, the attenuation coefficient is used to update the attraction factor of the data center for the target data.

[0057] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein:

[0058] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform any of the above-mentioned data migration methods.

[0059] In a fourth aspect, an embodiment of the present application provides a storage medium. When a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can perform the above-mentioned data migration method.

[0060] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned data migration methods when executed by a processor.

[0061] In the embodiment of the present application, for the distributed database (hereinafter referred to as the database) deployed to the computing power network, when it is determined that the data migration conditions are met according to the service characterization data of the database, multiple second data to be migrated are screened from the database, and according to the access requirements of the multiple second data and the network resource information of the database, the migration strategy of the multiple second data is determined and the data migration is performed with the goal of minimizing access delay, load balancing and minimizing network resource consumption of data migration. In this way, when the data access service provided by the database is not very good, the access requirements of the data to be migrated and the network resource information of the database can be combined to comprehensively consider the multiple goals of minimizing access delay, load balancing and minimizing network resource consumption of data migration to determine the migration strategy, which is more in line with the dynamic change characteristics of the computing power network, so it can ensure good data storage services, and the data migration cost is relatively low. In addition, after the migration, the load balancing degree of the database is better, and the data access delay is also lower, so the data access performance of the database can also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0063] Figure 1 A schematic diagram of data migration provided in an embodiment of the present application;

[0064] Figure 2A flowchart of a data migration method provided in an embodiment of the present application;

[0065] Figure 3 A flowchart for selecting data to be migrated provided in an embodiment of the present application;

[0066] Figure 4 A flowchart for determining a migration strategy provided in an embodiment of the present application;

[0067] Figure 5 A schematic diagram of the structure of a TiTB database provided in an embodiment of the present application;

[0068] Figure 6 A flowchart of another data migration method provided in an embodiment of the present application;

[0069] Figure 7 A schematic diagram of the structure of a data migration device provided in an embodiment of the present application;

[0070] Figure 8 A schematic diagram of the hardware structure of an electronic device for implementing a data migration method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to solve the problem in the related art that it is difficult for a database deployed in a computing network to provide good data access services, an embodiment of the present application provides a data migration method, device, electronic device and storage medium.

[0072] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0073] In order to facilitate the understanding of this application, the technical terms involved in this application are:

[0074] The computing power network refers to the China Computing Power Network, which is a technology that effectively allocates basic resources such as computing and storage between the cloud, edge, and end through network means. It aims to aggregate the idle computing power of the entire network into unified computing resources through network connections, and manage and schedule them to provide support for applications and services in various scenarios;

[0075] TiDB is an open source distributed relational database with features such as horizontal expansion and contraction, financial-grade high availability, and real-time hybrid transactional / analytical processing (HTAP).

[0076] The solution of the embodiment of the present application is applicable to a distributed database (referred to as database) deployed on a computing network. The database is composed of multiple geographically distributed storage nodes. These storage nodes are connected through a high-speed network to achieve cross-domain data computing and storage. In order to improve the efficiency of data access and load balancing, data needs to be reasonably migrated.

[0077] See also Figure 1 , Figure 1 A schematic diagram of data migration provided in an embodiment of the present application, wherein R1, R2, R3...R C Indicates C pieces of data to be migrated in the database, DC1, DC2, DC3…DC M Represents the M data centers (DC) of the database. Each data center is a server cluster. DC1 corresponds to storage nodes 1 to k, and DC2 corresponds to storage nodes k+1 to s…DC M Corresponding to storage nodes p~q, migration means moving R1, R2, R3…R C Stored in DC1, DC2, DC3…DC M These data centers correspond to storage nodes 1~q.

[0078] In addition, it should be noted that databases generally have copy strategies such as 3 copies, 5 copies, etc. Taking 3 copies as an example, 3 copies means that each data has 2 copies. A piece of data and its 2 copies are generally stored in the same data center to ensure the validity of the backup. In addition, when data and a copy of the data or two copies of the same data are stored in the same data center, they can be stored on storage nodes corresponding to different servers in the data center.

[0079] refer to Figure 2 , Figure 2 A flowchart of a data migration method provided in an embodiment of the present application includes the following steps.

[0080] In step 201, it is determined whether the database meets the data migration conditions based on the service characterization data of the database, and the database is a distributed database that provides data access services through a computing power network.

[0081] In some embodiments, the service characterization data of the database includes the load of each computing node and the access delay of each application.

[0082] Then, on the one hand, the load of each data center can be determined based on the load of each computing node and the corresponding relationship between each computing node and the data center of the database. For example, the load of each computing node corresponding to each data center is summed up, and the sum result is used as the load of this data center. Then, based on the load of each data center, the degree of load balancing of the database can be determined.

[0083] For example, the load balancing degree of the database is determined according to the following formula: Cost w :

[0084]

[0085] Where M represents the total number of data centers in the database, and W n Indicates the initial (i.e. before migration) nth data center DC n of load.

[0086] On the other hand, the average access delay of the database can be determined based on the access delay of each application and the total number of applications served by the database. For example, if the number of applications served by the database is N, then the average access delay Cost of applications accessing data on the computing network is R (i.e., the average access delay of the database) can be expressed as:

[0087]

[0088] Among them, RT i represents the i-th application S of the database service at the initial stage (i.e. before migration) i In practical applications, the initial application S i The average delay of multiple accesses to the data in the database is taken as RT i .

[0089] Furthermore, if the database load balancing degree Cost w Reaching the preset load limit and / or the average access delay Cost R If the preset delay limit is reached, it means that the database service needs to be improved, that is, it is determined that the database meets the data migration conditions. The preset load limit and the preset delay limit can be determined by relevant personnel according to actual needs and will not be repeated here.

[0090] In step 202, if it is determined that the data migration condition is met, multiple pieces of second data to be migrated are screened from multiple pieces of first data in the database according to the access log of the database.

[0091] In specific implementation, Figure 3 The process shown selects data to be migrated (ie, multiple pieces of second data), and includes the following steps.

[0092] In step 221, the number of accesses to multiple pieces of first data in M ​​historical cycles is obtained from the access log.

[0093] For example, extract the number of accesses to each first data in the access log in the past 7×24 hours. If the number of accesses to a first data in some hours is missing, it can be filled directly with the number of accesses in the previous hour, or with the average number of accesses to this first data in each hour;

[0094] In step 2022, the number of accesses to the plurality of first data in M ​​historical periods is input into a pre-trained LSTM network to obtain the number of accesses to the plurality of first data in N future periods, where M is an integer greater than N.

[0095] For example, to predict the number of visits to the first data in the next 1×24 hours, the sequence consisting of the number of visits to each piece of first data in the past 7×24 hours is input into the LSTM network to obtain the number of visits to the first data in the next 1×24 hours.

[0096] In step 2023, a plurality of second data are selected from the plurality of first data according to the number of accesses to the plurality of first data in N future cycles.

[0097] In specific implementation, the future access times of each first data piece can be determined based on the access times of the first data piece in the next 1×24 hours. For example, the average access times of the first data piece in the next 1×24 hours can be used as the future access times of the first data piece. For another example, the maximum access times of the first data piece in the next 1×24 hours can be used as the future access times of the first data piece. Then, the first data piece whose future access times are higher than the preset times can be determined as the second data, thereby obtaining multiple second data pieces.

[0098] In step 203, based on the access requirements of the plurality of second data and the network resource information of the database, a migration strategy for the plurality of second data is determined with the goal of minimizing access delay, load balancing, and minimizing network resource consumption of data migration.

[0099] The access requirements for the plurality of second data include access popularity and / or access delay, and the network resource information of the database includes the network bandwidth and / or network delay between the data centers in the database.

[0100] In specific implementation, the migration strategy of multiple second data can be determined by particle optimization algorithm, see Figure 4 , including the following steps.

[0101] In step 2031, according to the access requirements of the plurality of second data and in accordance with preset selection rules, L types of correspondences between the plurality of second data and the plurality of data centers of the database are determined, where L is an integer greater than 1.

[0102] Generally, L corresponding relationships can be obtained by performing spatial search through L search objects (ie, ants), wherein each search object obtains one corresponding relationship according to the following steps.

[0103] Step 1: Determine the probability of migrating the target data to the data center based on the attraction factor, inspiration factor and random factor of each data center for the target data. The target data is the second data with the i-th highest access count among multiple second data, and i is 1 initially.

[0104] Generally, multiple pieces of second data are sorted in descending order of access times before data migration is performed. Therefore, the second data with the i-th highest access times is also the i-th second data. In this way, the corresponding data center is determined preferentially for the second data with the highest access times, that is, the access needs of the data with the highest access popularity are preferentially met, which can improve the overall access performance of the database.

[0105] In practical applications, the attraction factor is used to characterize the probability that a data center is selected by the second data other than the target data. Initially, the attraction factor is M is the number of data centers, which will be updated later according to preset rules. The heuristic factor is used to characterize the degree to which the data center satisfies the access requirements of the target data, and can be determined based on the number of accesses to the target data and the geographical distance between the server of the application to which the target data belongs and the data center. The random factor is a randomly generated decimal, which is used to characterize the randomness of the target data center selection.

[0106] For example, the probability P of placing the i-th second data (i.e., target data) in the j-th data center is determined according to the following formula: ij :

[0107]

[0108] in:

[0109] τ ij represents the attraction factor of the jth data center to the i-th second data, τ ik It represents the attraction factor of the k-th data center to the i-th second data. Initially, M is the number of data centers, which will be updated later according to the preset rules;

[0110] η ij It represents the heuristic factor of the j-th data center for the i-th second data, for example, F i represents the number of accesses to the second data of the ith item, D ij represents the geographical distance between the server of the application to which the i-th second data belongs and the j-th data center, which can reflect the access delay after migrating the i-th second data to the j-th data center. Similarly, ηik represents the inspiration factor of the k-th data center for the i-th second data, D ik The geographical distance between the server of the application to which the i-th second data belongs and the k-th data center can reflect the access delay after the i-th second data is migrated to the k-th data center;

[0111] γ ij It represents the random factor when the search object selects the jth data center for the i-th second data. It is a randomly generated decimal. It should be noted that γ ij The random factors are generated randomly at each calculation. The random factors for selecting the jth data center for the i-th second data at different times for the same search object are generally different. The random factors for selecting the jth data center for the i-th second data for different search objects are generally different. In this way, the search ability of the search object can be enhanced.

[0112] α and β are predetermined coefficients.

[0113] Step 2: Determine the data center corresponding to the target data based on the probability of the target data being migrated to each data center.

[0114] For example, the data center with the highest probability may be selected as the data center corresponding to the target data.

[0115] Step 3: Update i to i+1, and repeat step 1 until i is greater than C, to obtain a corresponding relationship between multiple pieces of second data and multiple data centers, where C is the total number of pieces of second data.

[0116] In step 2032, the loss value of the corresponding relationship is determined according to the access delay of each application after migration, the load of each data center, and the network resources consumed by the migration according to the corresponding relationship.

[0117] In specific implementation, for each correspondence between the plurality of second data obtained in this round and the plurality of data centers, the loss value may be determined according to the following formula:

[0118] F=(ω1*Cost R ′+ω2*Cost w ′+(1-ω1-ω2)*Cost m ′),

[0119] Among them, ω1 and ω2 are preset weights, both are decimals, Cost R ′ represents the average access delay of the database after migration according to this correspondence, Cost w ′ represents the load balancing degree of the database after migration according to this corresponding relationship, Cost m′ represents the size of network resources consumed by migration according to this correspondence (also called migration cost);

[0120] For example, Cost can be determined according to the following formula: R ′:

[0121]

[0122] Among them, RT i ′ represents the i-th application S after migration according to this correspondence i It should be noted that although no data migration is performed, the access delay of the i-th application S after migration can be i The geographical distribution of the accessed data in the database is used to estimate the i-th application S i Access delay RT i ′, N is the total number of applications;

[0123] For example, Cost can be determined according to the following formula: w ′:

[0124]

[0125] Where M represents the number of data centers, W n ′ represents the nth data center DC after migration according to this correspondence n It should be noted that although no data migration is performed, the nth data center DC after migration can be used to n The storage space usage and bandwidth usage of the nth data center DC after migration are estimated n Load;

[0126] For example, Cost can be determined according to the following formula: m ′:

[0127]

[0128] Where C represents the total number of second data, C mi ′ represents the transfer of a second data from data center D i Migrate to Data Center D m The size of network resources required (i.e., migration cost), S i is the data size of the second data, BW mi Data Center D i To Data Center D m The effective network bandwidth, T delay Data Center D i To Data Center D m The network delay between .

[0129] In step 2033, determine whether the loop stop condition is met, if not, go to step 2034, if so, go to step 2035.

[0130] The loop stop condition is when the number of loops reaches a preset value.

[0131] In step 2034 , the preset selection rule is updated and the process returns to step 2031 .

[0132] In order to prevent the search object from falling into a local optimal solution, the attraction factor can be attenuated using the attenuation coefficient ρ to encourage the exploration of new data centers corresponding to each second data, where 0<ρ<1. In addition, in order to reduce data access latency, the global attraction factor is optimized. When the server corresponding to the application to which a second data belongs is a server in the data center, the attenuation coefficient can be appropriately increased to optimize the algorithm performance.

[0133] To this end, for each data center, if the server corresponding to the application to which the target data belongs is a server in the data center, the attraction factor of the data center for the target data is updated using the preset attenuation coefficient and minimum loss value, for example, τ ij =(1-ρ)×τ ij +ρ×F(X*), where F(X*) refers to the minimum loss value; if the server corresponding to the application to which the target data belongs is not a server in the data center, the attenuation coefficient can be directly used to update the attraction factor of the data center for the target data, for example, τ ij =(1-ρ)×τ ij .

[0134] In step 2035, the correspondence between the storage nodes and the second data in each data center is determined based on the correspondence with the minimum loss value and the data storage rule of a single data center to obtain a migration strategy.

[0135] Among them, the data storage rules of a single data center include load balancing, each second data and its copy are not stored on the same server, etc.

[0136] That is, after determining which second data corresponds to each data center, the storage nodes corresponding to these second data in this data center are determined according to the data storage rules of a single data center, and the storage nodes corresponding to all second data are determined, and the migration strategy is obtained.

[0137] In step 204, a migration instruction is sent to the database according to the migration strategy to migrate the plurality of second data.

[0138] That is, according to the storage nodes corresponding to the plurality of second data, an SQL statement for triggering data migration is generated, and the migration of the plurality of second data can be completed by sending the SQL statement to the database for execution.

[0139] The following takes the TiTB database deployed to the example network as an example to introduce the solution of the embodiment of the present application.

[0140] First, let's briefly introduce the TiTB database. The TiTB database consists of three core modules: PD Server, TiDB Server, and TiKV Server, which are responsible for metadata management of the TiDB cluster, SQL layer calculation, and data storage.

[0141] See also Figure 5 , Figure 5 A structural diagram of a TiTB database provided for an embodiment of the present application includes a PD cluster (Cluster), a TiDB Cluster and a TiKV Cluster, wherein the PD Cluster includes multiple PD Servers, and the multiple PDServers are provided by multiple management nodes; the TiDB Cluster includes multiple TiDB Servers, and the multiple TiDB Servers are provided by multiple computing nodes; the TiKV Cluster includes multiple TiKV Servers, and the multiple TiKV Servers are provided by multiple storage nodes.

[0142] Next, the process of determining the data migration strategy is introduced.

[0143] See also Figure 6 , Figure 6 A flowchart of another data migration method provided in an embodiment of the present application includes the following steps.

[0144] In step 601, it is determined whether the database meets the data migration condition according to the service representation data of the database obtained.

[0145] The service characterization data of the database may include the load of each computing node and the access delay of each application.

[0146] In specific implementation, the loads of the computing nodes corresponding to the nth data center can be summed up, and the sum result can be used as the load W of the nth data center. n Then, determine the load balancing degree of the database according to the following formula: Cost w :

[0147]

[0148] Where M represents the total number of data centers in the database, and W n Indicates the initial (i.e. before migration) nth data center DC n of load.

[0149] The average access delay Cost of the database can be determined according to the following formula: R :

[0150]

[0151] Among them, RT i represents the i-th application S of the database service at the initial stage (i.e. before migration) i The access delay can be applied to the initial i The average delay of multiple accesses to the data in the database is taken as RT i , N is the number of applications served by the database.

[0152] If the database load balancing level Cost w Reaching the preset load limit or average access delay Cost R When the preset delay upper limit is reached, it is determined that the database meets the data migration conditions, wherein the preset load upper limit and the preset delay upper limit can be determined by relevant personnel according to actual needs, which will not be repeated here.

[0153] In step 602, when the database meets the data migration conditions, the LSTM model is used to predict the number of data accesses.

[0154] Step 1. Data preparation.

[0155] 1) Data collection.

[0156] Collect logs: extract data access records from the database log management system such as ELK Stack;

[0157] Extract fields from the log: The extracted fields include access time (timestamp), user ID (user_id), replica ID (replica_id) (data ID), and access count (access_count).

[0158] 2) Data preprocessing.

[0159] Time serialization: load data into a DataFrame, sort by timestamp, and convert timestamp to a usable data time DateTime format;

[0160] Data aggregation: Use the pandas library to group multiple first data by data identifier and hour and aggregate the number of visits to generate the number of visits for each first data in each hour;

[0161] Missing value processing: If the number of visits to any first data in a certain hour is missing, it can be directly filled with the number of visits to the first data in the previous hour, or the average number of visits to the first data in each hour can be used to fill it;

[0162] Normalization operation: Use MinMaxScaler to normalize the access count to the interval [0,1].

[0163] 3) Build a dataset.

[0164] Define the time window: Select a sliding window and use the number of visits in the previous 168 hours (7 days) to predict the number of visits in the next 24 hours (one day).

[0165] Feature and label generation: Generate feature set (X) and label set (y), where X contains data for each hour of the previous 7 days and y is the number of visits predicted for the next 24 hours; use numpy to construct a three-dimensional array suitable for LSTM input (number of samples, time steps, number of features).

[0166] Step 2: Model design and training.

[0167] 1) Design an LSTM network.

[0168] Network structure: Use TensorFlow or Keras to build the model. The first layer of LSTM has 64 neurons. The return sequence is true (return_sequences = True) to stack the second layer. The second layer of LSTM has 32 neurons. A Dropout layer is added to prevent overfitting (such as a dropout rate of 0.2).

[0169] Output layer: Dense layer with 24 nodes (one prediction per hour), and linear activation function.

[0170] 2) Model training.

[0171] Model segmentation: Use train_test_split to divide the data set into training set and validation set, such as 80% as training set and 20% as validation set;

[0172] Model compilation: The loss function can use mean square error (MSE); the optimizer can use Adam, and the initial learning rate can be set to 0.001;

[0173] Training model: train for 20 epochs, batch size (batch_size) is set to 64; use EarlyStopping callback, the amount to be monitored is the loss value (monitor = 'val_loss'), patience = 3, that is, after EarlyStopping is activated (if it is found that the loss has not decreased compared with the previous epoch training), stop training after patience epochs.

[0174] Step 3: Model evaluation.

[0175] Test set evaluation: Use the test set to evaluate model performance and calculate loss and accuracy through model.evaluate();

[0176] Error analysis: Calculate the mean square error (MSE), mean absolute error (MAE), and root mean square error (RMSE); draw a comparison chart between the predicted value and the true value to more intuitively evaluate the model effect.

[0177] Step 4: Deployment and prediction.

[0178] 1) Model deployment.

[0179] Export and deploy: Save the model in HDF5 or TensorFlow Serving format and deploy it on a local server;

[0180] Interface creation: Create an Application Program Interface (API) (such as using Flask or FastAPI) to allow external systems to send access records and receive prediction results.

[0181] 2) Implement the forecast.

[0182] Real-time data acquisition: A scheduled task (such as every hour) obtains the latest access data, transmits it to the model through the API, and obtains the prediction results. The prediction results include the number of visits to multiple first data in the next 1×24 hours.

[0183] Step 5: Monitoring and maintenance.

[0184] 1) Monitoring.

[0185] Real-time tracking: Use dashboards (such as Grafana) to monitor the accuracy of model predictions and generate logs to track anomalies or growing errors.

[0186] 2) Model update.

[0187] Timely updates: Regularly update training data to include the latest access patterns, and retrain the model every month or quarter to keep it up to date;

[0188] Test and Iterate: After evaluating the new model in a test environment, gradually migrate it to the production environment.

[0189] In this way, the TiDB database can perceive future access patterns in advance, further optimize data migration strategies, and realize intelligent management of the TiDB database.

[0190] In step 603, a plurality of first data pieces having access times higher than a preset number are selected as a plurality of second data pieces to be migrated.

[0191] For example, the average number of accesses to each first data piece in the next 1×24 hours is taken as the future access number of the first data piece, and then the first data piece with a future access number higher than a preset number is determined as the second data, thereby obtaining multiple pieces of data to be migrated.

[0192] In step 604, with the goal of minimizing access delay, load balancing, and minimizing network resource consumption for data migration, the corresponding relationship between the plurality of second data and the plurality of data centers of the database is determined by using an ant colony optimization algorithm.

[0193] In specific implementation, the data center corresponding to each second data can be determined in descending order of the number of accesses. In this way, the access requirements of hot data can be met first, thereby improving the overall access performance of the TiDB database.

[0194] Next, the process of determining the correspondence between the plurality of second data and the plurality of data centers by using the ant colony optimization algorithm is introduced.

[0195] Step 1: Determine L corresponding relationships between a plurality of second data and a plurality of data centers through L search objects (ie, L ants).

[0196] In practical applications, each search object can determine the probability P of placing the i-th second data (i.e., target data) in the j-th data center according to the following formula: ij :

[0197]

[0198] in:

[0199] τ ij represents the attraction factor of the jth data center to the i-th second data, τ ik It represents the attraction factor of the k-th data center to the i-th second data. Initially, M is the number of data centers, which will be updated later according to the preset rules;

[0200] η ijIt represents the heuristic factor of the j-th data center for the i-th second data, for example, F i represents the number of accesses to the second data of the ith item, D ij represents the geographical distance between the server of the application to which the i-th second data belongs and the j-th data center, which can reflect the access delay after migrating the i-th second data to the j-th data center. Similarly, η ik represents the inspiration factor of the k-th data center for the i-th second data, D ik The geographical distance between the server of the application to which the i-th second data belongs and the k-th data center can reflect the access delay after the i-th second data is migrated to the k-th data center;

[0201] γ ij It represents the random factor when the search object selects the jth data center for the i-th second data. It is a randomly generated decimal. It should be noted that γ ij The random factors are generated randomly at each calculation. The random factors for selecting the jth data center for the i-th second data at different times for the same search object are generally different. The random factors for selecting the jth data center for the i-th second data for different search objects are generally different. In this way, the search ability of the search object can be enhanced.

[0202] α and β are predetermined coefficients.

[0203] Afterwards, the data center with the highest probability may be selected as the data center corresponding to the i-th second data.

[0204] Through the above method, corresponding data centers can be selected for all second data to obtain a correspondence between multiple second data and multiple data centers. L search objects are executed in parallel to obtain L types of correspondence between multiple second data and multiple data centers.

[0205] Step 2: Determine the loss value corresponding to each of the L corresponding relationships through the loss function. The loss value is used to characterize the access delay, load balancing degree and network resource consumption of data migration.

[0206] For each correspondence between the multiple pieces of second data and the multiple data centers obtained in this round, the loss value can be determined according to the following formula:

[0207] F=(ω1*Cost R ′+ω2*Cost w ′+(1-ω1-ω2)*Cost m ′),

[0208] Among them, ω1 and ω2 are preset weights, both are decimals, CostR ′ represents the average access delay of the database after migration according to this correspondence, Cost w ′ represents the load balancing degree of the database after migration according to this corresponding relationship, Cost m ′ represents the size of network resources consumed by migration according to this correspondence (also called migration cost);

[0209] For example, Cost can be determined according to the following formula: R ′:

[0210]

[0211] Among them, RT i ′ represents the i-th application S after migration according to this correspondence i It should be noted that although no data migration is performed, the access delay of the i-th application S after migration can be i The geographical distribution of the accessed data in the database is used to estimate the i-th application S i Access delay RT i ′, N is the total number of applications;

[0212] For example, Cost can be determined according to the following formula: w ′:

[0213]

[0214] Where M represents the number of data centers, W n ′ represents the nth data center DC after migration according to this correspondence n It should be noted that although no data migration is performed, the nth data center DC after migration can be used to n The storage space usage and bandwidth usage of the nth data center DC after migration are estimated n Load;

[0215] For example, Cost can be determined according to the following formula: m ′:

[0216]

[0217] Where C represents the total number of second data, C mi ′ represents the transfer of a second data from data center D i Migrate to Data Center D m The size of network resources required (i.e., migration cost), S i is the data size of the second data, BW mi Data Center D i To Data Center Dm The effective network bandwidth, T delay Data Center D i To Data Center D m The network delay between .

[0218] Step 3: Take the corresponding relationship with the smallest loss value among the determined corresponding relationships as the candidate corresponding relationship X*, and update τ ij and τ ik .

[0219] For example, for the jth data center, if the server corresponding to the application to which the i-th second data belongs is a server in the jth data center, then according to the formula τ ij =(1-ρ)×τ ij +ρ×F(X*) versus τ ij Update, F(X*) refers to the loss value of the alternative corresponding relationship X*, that is, the minimum loss value. If the server corresponding to the application to which the i-th second data belongs is not the server in the j-th data center, then according to the formula τ ij =(1-ρ)×τ ij For τ ij Update is performed, wherein ρ is a preset decay coefficient used to characterize the forgetting speed of selecting the data center for each second data.

[0220] τ ik The same update rule applies to the k-th data center. If the server corresponding to the application to which the i-th second data belongs is a server in the k-th data center, then according to the formula τ ik =(1-ρ)×τ ik +ρ×F(X*) versus τ ik Update, F(X*) refers to the loss value of the alternative corresponding relationship X*, that is, the minimum loss value. If the server corresponding to the application to which the i-th second data belongs is not the server in the k-th data center, then according to the formula τ ik =(1-ρ)×τ ik For τ ik Update is performed, wherein ρ is a preset decay coefficient used to characterize the forgetting speed of selecting the data center for each second data.

[0221] Step 4, return to step 1, until the number of cycles reaches the set value, the candidate corresponding relationship X* is determined as the corresponding relationship between the plurality of second data and the plurality of data centers.

[0222] In complex network environments and scenarios with changing data access requirements, multi-objective optimization algorithms can demonstrate better robustness and adaptability. Multi-objective optimization algorithms can comprehensively consider multiple performance indicators and provide a global optimal solution. By introducing the ant colony optimization algorithm, a good balance can be achieved between multiple objectives, significantly improving the overall performance of the TiDB database.

[0223] In step 605, according to the determined correspondence and the data storage rule of a single data center, the correspondence between the TiKV nodes and the second data in each data center is determined to obtain a migration strategy.

[0224] For example, if you decide to migrate 10,000 pieces of second data to a data center, and assume that this data center corresponds to 20 TiKV nodes, you can distribute the 10,000 pieces of second data on these 20 TiKV nodes according to load balancing and the rule that each piece of second data and its copy are not stored on the same server.

[0225] In step 606, a migration instruction is sent to the TiDB node according to the migration strategy to trigger the PD node to migrate the multiple second data.

[0226] In actual applications, the corresponding SQL statements can be generated according to the Sql placement characteristics of the TiDB database, and the placement locations of the second data in the TiKV cluster can be configured through SQL statements to complete data migration.

[0227] Specifically, for each piece of second data, obtain the label (Lable) of the TiKV node corresponding to the second data, then generate an SQL statement for adding the Lable of the TiKV node to the second data, send the SQL statement to the TiDB node, and then add the Lable of the TiKV node to the second data. Subsequently, after the PD node detects that the Lable of the second data is different from the Lable of the TiKV node where it is located, it will migrate the second data to the TiKV node with this Lable for storage based on the Lable of the second data.

[0228] In this way, by dynamically generating and executing SQL statements, it is possible to quickly respond to changes in load and access patterns in the cluster, optimizing the resource scheduling and utilization efficiency of the TiDB database.

[0229] The technical effects of the embodiments of the present application are as follows:

[0230] (1) Significantly reduce data access latency: By dynamically adjusting the data storage location, data can be placed on the storage node closest to the user or data demand point, reducing data transmission time and thus significantly reducing data access latency;

[0231] (2) Improve system load balancing: Use the model to predict the number of future data accesses for each data point, and combine it with the ant colony optimization algorithm to achieve load balancing. This can more evenly distribute the database load and avoid overloading of individual storage nodes. This is conducive to maintaining load balance in a dynamic environment such as the computing power network, ensuring efficient operation of the database.

[0232] (3) Reduce the cost of replica migration: Through multi-objective optimization, it is possible to optimize data storage while reducing the network overhead caused by data migration. The database's network resources are used more efficiently, which can reduce the delay and bandwidth usage caused by frequent data migration, making the database more stable and energy-efficient.

[0233] (4) Enhance the adaptability and intelligence of the database: By predicting the number of data accesses through the model, it is possible to perceive future access patterns in advance and adjust the data storage strategy in real time, making the database more adaptable and intelligent, thereby maintaining good performance under the ever-changing network environment and user needs.

[0234] Based on the same technical concept, the embodiment of the present application also provides a data migration device. The principle of solving the problem by the data migration device is similar to that of the above-mentioned data migration method. Therefore, the implementation of the data migration device can refer to the implementation of the data migration method, and the repeated parts will not be repeated.

[0235] Figure 7 A schematic diagram of a data migration device provided in an embodiment of the present application includes:

[0236] A judgment module 701 is used to determine whether the database meets the data migration condition according to the service characterization data of the database, wherein the database is a distributed database that provides data access services through a computing power network;

[0237] A screening module 702 is configured to screen a plurality of second data to be migrated from a plurality of first data in the database according to an access log of the database if it is determined that the data migration condition is met;

[0238] A determination module 703, configured to determine a migration strategy for the plurality of second data according to access requirements of the plurality of second data and network resource information of the database, with the goal of minimizing access delay, load balancing, and minimizing network resource consumption for data migration;

[0239] The sending module 704 is configured to send a migration instruction to the database according to the migration strategy, so as to migrate the plurality of second data.

[0240] In some embodiments, the service characterization data includes the load of each computing node in the database and the access delay of each application. The judgment module 701 is specifically used to:

[0241] Determine the load of each data center according to the load of each computing node and the corresponding relationship between each computing node and the data center of the database, and determine the load balancing degree of the database according to the load of each data center;

[0242] Determining an average access delay of the database according to the access delay of each application and the total number of applications served by the database;

[0243] If the load balancing degree reaches a preset load upper limit, and / or the average access delay reaches a preset delay upper limit, it is determined that the data migration condition is met.

[0244] In some embodiments, the screening module 702 is specifically configured to:

[0245] Obtaining, from the access log, access times of multiple pieces of first data in M ​​historical periods;

[0246] Inputting the number of accesses of the plurality of first data in M ​​historical cycles into a pre-trained long short-term memory (LSTM) network to obtain the number of accesses of the plurality of first data in N future cycles, where M is an integer greater than N;

[0247] The plurality of second data are selected from the plurality of first data according to the number of accesses to the plurality of first data in N future cycles.

[0248] In some embodiments, the determination module 703 is specifically configured to:

[0249] According to the access requirements of the plurality of second data and the preset selection rule, determining L correspondences between the plurality of second data and the plurality of data centers of the database, where L is an integer greater than 1;

[0250] Determine the loss value of the corresponding relationship according to the access delay of each application after migration, the load of each data center, and the network resources consumed by the migration according to the corresponding relationship;

[0251] The preset selection rule is updated to execute the step of determining L types of correspondences between the plurality of second data and the plurality of data centers of the database according to the access requirements of the plurality of second data and the preset selection rule;

[0252] Until the cycle stop condition is met, the corresponding relationship between the storage nodes and the second data in each data center is determined according to the corresponding relationship with the minimum loss value and the data storage rule of a single data center to obtain the migration strategy.

[0253] In some embodiments, when determining L types of correspondences between the plurality of second data and the plurality of data centers of the database according to the access requirements of the plurality of second data and the preset selection rule, the determination module 703 is specifically used to:

[0254] Perform the following steps L times:

[0255] Determine the probability of the target data being migrated to the data center according to the attraction factor, heuristic factor and random factor of each data center for the target data, wherein the target data is the data with the i-th highest access count among the plurality of second data, i is 1 initially, the attraction factor is used to characterize the probability of the data center being selected by the second data other than the target data, the heuristic factor is used to characterize the degree to which the data center satisfies the access requirements of the target data, and the random factor is randomly generated;

[0256] Determining the data center corresponding to the target data according to the probability of the target data being migrated to each data center;

[0257] Update i to i+1, and execute the step of determining the probability of migrating the target data to the data center based on the attraction factor, inspiration factor and random factor of each data center for the target data, until i is greater than C, and a corresponding relationship is obtained, where C is the total number of the multiple second data.

[0258] In some embodiments, when updating the preset selection rule, the determination module 703 is specifically configured to:

[0259] For each data center, if the server corresponding to the application to which the target data belongs is a server in the data center, the attraction factor of the data center for the target data is updated using a preset attenuation coefficient and a minimum loss value;

[0260] If the server corresponding to the application to which the target data belongs is not a server in the data center, the attenuation coefficient is used to update the attraction factor of the data center for the target data.

[0261] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional modules in the embodiments of the present application may be integrated in a processor, or may exist physically separately, or two or more modules may be integrated in one module. The coupling between the modules may be achieved through some interfaces, which are usually electrical communication interfaces, but it is not excluded that they may be mechanical interfaces or other forms of interfaces. Therefore, the modules described as separate components may or may not be physically separated, and may be located in one place or distributed to different locations of the same or different devices. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0262] After introducing the data migration method and device according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0263] Refer to the following Figure 8 The electronic device 130 implemented according to this embodiment of the present application is described. Figure 8 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0264] like Figure 8 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0265] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0266] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0267] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0268] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0269] In an exemplary embodiment, a storage medium is also provided, and when a computer program in the storage medium is executed by a processor of an electronic device, the electronic device can perform the above-mentioned data migration method. Optionally, the storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0270] In an exemplary embodiment, the electronic device of the present application may include at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor may execute the steps of any data migration method provided in the embodiments of the present application.

[0271] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed by an electronic device, the electronic device can implement any exemplary method provided in the present application.

[0272] Furthermore, the computer program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0273] The program product for data migration in the embodiment of the present application may adopt a CD-ROM and include program code, and may be run on a computing device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0274] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0275] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

[0276] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network such as a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0277] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0278] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0279] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0280] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0281] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0282] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0283] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0284] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application also includes these modifications and variations.

Claims

1. A data migration method, characterized in that: include: Determining whether the database meets the data migration condition according to the service characterization data of the database, wherein the database is a distributed database that provides data access services through a computing power network; If it is determined that the data migration condition is met, selecting a plurality of second data to be migrated from the plurality of first data in the database according to the access log of the database; Determine, according to the access requirements of the plurality of second data and the network resource information of the database, a migration strategy for the plurality of second data with the goal of minimizing access delay, load balancing, and minimizing network resource consumption for data migration; A migration instruction is sent to the database according to the migration strategy to migrate the plurality of second data.

2. The method according to claim 1, characterized in that The service characterization data includes the load of each computing node and the access delay of each application in the database. Determining whether the database meets the data migration condition according to the service characterization data of the database includes: Determine the load of each data center according to the load of each computing node and the corresponding relationship between each computing node and the data center of the database, and determine the load balancing degree of the database according to the load of each data center; Determining an average access delay of the database according to the access delay of each application and the total number of applications served by the database; If the load balancing degree reaches a preset load upper limit, and / or the average access delay reaches a preset delay upper limit, it is determined that the data migration condition is met.

3. The method according to claim 1, characterized in that Filtering a plurality of second data to be migrated from a plurality of first data in the database according to the access log of the database includes: Obtaining, from the access log, access times of multiple pieces of first data in M ​​historical periods; Inputting the number of accesses of the plurality of first data in M ​​historical cycles into a pre-trained long short-term memory (LSTM) network to obtain the number of accesses of the plurality of first data in N future cycles, where M is an integer greater than N; The plurality of second data are selected from the plurality of first data according to the number of accesses to the plurality of first data in N future cycles.

4. The method according to any one of claims 1 to 3, characterized in that: According to the access requirements of the plurality of second data and the network resource information of the database, with the goal of minimizing access delay, load balancing and minimizing network resource consumption of data migration, determining a migration strategy for the plurality of second data, including: According to the access requirements of the plurality of second data and the preset selection rule, determining L correspondences between the plurality of second data and the plurality of data centers of the database, where L is an integer greater than 1; Determine the loss value of the corresponding relationship according to the access delay of each application after migration, the load of each data center, and the network resources consumed by the migration according to the corresponding relationship; The preset selection rule is updated to execute the step of determining L types of correspondences between the plurality of second data and the plurality of data centers of the database according to the access requirements of the plurality of second data and the preset selection rule; Until the cycle stop condition is met, the corresponding relationship between the storage nodes and the second data in each data center is determined according to the corresponding relationship with the minimum loss value and the data storage rule of a single data center to obtain the migration strategy.

5. The method according to claim 4, characterized in that According to the access requirements of the plurality of second data and according to the preset selection rule, L corresponding relationships between the plurality of second data and the plurality of data centers of the database are determined, including: Perform the following steps L times: Determine the probability of the target data being migrated to the data center according to the attraction factor, heuristic factor and random factor of each data center for the target data, wherein the target data is the data with the i-th highest access count among the plurality of second data, i is 1 initially, the attraction factor is used to characterize the probability of the data center being selected by the second data other than the target data, the heuristic factor is used to characterize the degree to which the data center satisfies the access requirements of the target data, and the random factor is randomly generated; Determining the data center corresponding to the target data according to the probability of the target data being migrated to each data center; Update i to i+1, and execute the step of determining the probability of migrating the target data to the data center based on the attraction factor, inspiration factor and random factor of each data center for the target data, until i is greater than C, and a corresponding relationship is obtained, where C is the total number of the multiple second data.

6. The method according to claim 5, characterized in that Updating the preset selection rule includes: For each data center, if the server corresponding to the application to which the target data belongs is a server in the data center, the attraction factor of the data center for the target data is updated using a preset attenuation coefficient and a minimum loss value; If the server corresponding to the application to which the target data belongs is not a server in the data center, the attenuation coefficient is used to update the attraction factor of the data center for the target data.

7. A data migration device, characterized in that: include: A judgment module, used to determine whether the database meets the data migration conditions according to the service characterization data of the database, wherein the database is a distributed database that provides data access services through a computing power network; a screening module, configured to screen the plurality of second data to be migrated from the plurality of first data in the database according to the access log of the database if it is determined that the data migration condition is met; A determination module, configured to determine a migration strategy for the plurality of second data according to access requirements of the plurality of second data and network resource information of the database, with the goal of minimizing access delay, load balancing, and minimizing network resource consumption for data migration; A sending module is used to send a migration instruction to the database according to the migration strategy to migrate the plurality of second data.

8. An electronic device, characterized in that: include: at least one processor, and a memory communicatively connected to the at least one processor, wherein: The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: When the computer program in the storage medium is executed by a processor of an electronic device, the electronic device can execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images