Data migration method and device, equipment, storage medium and computer program product

Abstract

The invention discloses a data migration method and device, equipment, a storage medium and a computer program product, and relates to the technical field of data migration, and the method comprises the steps: responding to a node splitting request, carrying out the statistics of a fragmentation factor corresponding to each piece of data in a preset to-be-split node, and obtaining a fragmentation factor range statistical result; dividing data in the to-be-split node into two groups according to a fragment factor range statistical result to obtain to-be-migrated data; and migrating the to-be-migrated data from the to-be-split node to the target node through the management node. In the application, when the load of the to-be-split node reaches the load threshold, node splitting and data migration are carried out according to the data fragmentation factor in the split node, so that the system can adapt to the distribution condition of the data along with the actual use of the data. Therefore, the problem that the storage and processing pressure is unbalanced due to the fact that an existing data storage method cannot be dynamically adjusted is solved, and the method is more suitable for application scenes of mass data and high-concurrency access.

Description

Technical Field

[0001] The present application relates to the technical field of data migration, and in particular to a data migration method, apparatus, device, storage medium and computer program product. Background Art

[0002] With the development of business and the expansion of user base, the number of requests and data that the system needs to process is increasing. The performance bottleneck of the traditional single database architecture is increasingly prominent when facing massive data and high concurrent access. A common optimization method is to introduce sharding database optimization means, which is to shard data according to the value range of a certain field and divide the data into multiple databases or data tables. For example, user data with user IDs from 1 to 1000 is stored in the first database, and user data with user IDs from 1001 to 2000 is stored in the second database, thereby reducing the pressure on the single database.

[0003] However, if the amount of data grows rapidly in certain ranges and slowly in other ranges, the storage pressure on the shards in these rapidly growing ranges will increase, resulting in data skew. For example, if user activity in a certain user ID range is abnormally active, the corresponding few shards will be overloaded, resulting in an imbalance in storage and processing pressure, affecting the overall performance of the system.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present application is to provide a data migration method, device, equipment, storage medium and computer program product, aiming to solve the technical problem that the existing data storage method cannot be dynamically adjusted, resulting in unbalanced storage and processing pressure.

[0006] To achieve the above objectives, the present application proposes a data migration method, which includes:

[0007] In response to a node splitting request, statistics are collected on the sharding factors corresponding to each piece of data in a preset node to be split, and a sharding factor range statistical result is obtained. The node splitting request is triggered when the load of the node to be split reaches a preset load threshold.

[0008] According to the sharding factor range statistics, the data in the node to be split is divided into two groups to obtain the data to be migrated;

[0009] The to-be-migrated data is migrated from the to-be-split node to a preset target node through a preset management node.

[0010] In one embodiment, the step of dividing the data in the node to be split into two groups according to the sharding factor range statistics, and obtaining the data to be migrated includes:

[0011] Determine a splitting threshold value according to the statistical result of the fragmentation factor range;

[0012] According to the split boundary value, the data in the node to be split is divided into two groups to obtain first split data and second split data, wherein the first split data is data corresponding to a sharding factor greater than the split boundary value, and the second split data is data corresponding to a sharding factor less than or equal to the split boundary value;

[0013] Compare the data amounts of the first split data and the second split data to obtain a data amount comparison result;

[0014] When the data volume comparison result indicates that the data volume of the first split data is smaller than the data volume of the second split data, the first split data is used as the data to be migrated;

[0015] or,

[0016] When the data volume comparison result indicates that the data volume of the second split data is smaller than the data volume of the first split data, the second split data is used as the data to be migrated.

[0017] In one embodiment, the step of performing statistics on the sharding factors corresponding to each piece of data in the preset node to be split in response to the node splitting request, and obtaining the sharding factor range statistics result also includes:

[0018] In response to a preset timed detection task, the management node performs load monitoring on a preset data node group to obtain a load monitoring result, wherein the data node group includes a plurality of data nodes;

[0019] When the load monitoring result shows that the load of any data node in the data node group reaches the load threshold, the data node is defined as the node to be split, and the node splitting request is triggered.

[0020] In one embodiment, the data to be migrated includes a plurality of data to be called, and after the step of migrating the data to be migrated from the node to be split to a preset target node through a preset management node, the method further includes:

[0021] Receiving a data call request corresponding to any of the data to be called;

[0022] Performing data query on the target node according to the data call request to obtain a query result;

[0023] When the query result indicates that the data to be called is not found in the target node, a data query is performed on the node to be split according to the data calling request.

[0024] In one embodiment, after the step of migrating the to-be-migrated data from the to-be-split node to a preset target node through a preset management node, the process further includes:

[0025] By means of the management node, the target node is updated from a normal state to a migrating state, and a first data node change result is generated;

[0026] Broadcast the first data node change result to related applications.

[0027] In one embodiment, after the step of migrating the to-be-migrated data from the to-be-split node to a preset target node through a preset management node, the step further includes:

[0028] When it is detected that all the data to be migrated have been migrated to the target node, the target node is updated from a migrating state to a normal state through the management node, and a second data node change result is generated;

[0029] Broadcast the second data node change result to related applications.

[0030] In addition, to achieve the above-mentioned purpose, the present application also proposes a data migration device, which includes:

[0031] A sharding factor statistics module is used to, in response to a node splitting request, perform statistics on the sharding factors corresponding to each piece of data in a preset node to be split, and obtain sharding factor range statistics results. The node splitting request is triggered when the load of the node to be split reaches a preset load threshold;

[0032] A node splitting module, used to divide the data in the node to be split into two groups according to the statistical result of the sharding factor range, and obtain the data to be migrated;

[0033] The data migration module is used to migrate the data to be migrated from the node to be split to a preset target node through a preset management node.

[0034] In addition, to achieve the above objectives, the present application also proposes a data migration device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data migration method described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data migration method described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data migration method described above are implemented.

[0037] The present application provides a data migration method. When the load of the node to be split reaches a preset load threshold, node splitting and data migration are performed according to the data sharding factor in the split node, so that the system can adapt to the distribution of data as the data is actually used. Thus, the technical problem that the existing data storage method cannot be dynamically adjusted, resulting in uneven storage and processing pressure, is solved, and it is more suitable for application scenarios with massive data and high concurrent access. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A flowchart of the first embodiment of the data migration method of this application is provided;

[0041] Figure 2 A flowchart of the second embodiment of the data migration method of this application is provided;

[0042] Figure 3 A schematic diagram of the structure of the data migration system provided in Embodiment 1 or 2 of the present application;

[0043] Figure 4 A brief flowchart of the data migration method provided in Embodiment 1 or 2 of the present application;

[0044] Figure 5 A schematic diagram of data node data volume monitoring of the data migration method provided in Embodiment 1 or 2 of the present application;

[0045] Figure 6 A schematic diagram of node splitting of the data migration method provided in Embodiment 1 or 2 of the present application;

[0046] Figure 7 A schematic diagram of updating the distribution relationship at the start of data migration of the data migration method provided in Embodiment 1 or 2 of the present application;

[0047] Figure 8 A data migration diagram of the data migration method provided in Embodiment 1 or 2 of the present application;

[0048] Fig. 9 A schematic diagram of updating the distribution relationship of data migration completion of the data migration method provided in Embodiment 1 or 2 of the present application;

[0049] Fig.10 This is a schematic diagram of the module structure of the data migration device according to an embodiment of the present application;

[0050] Fig.11 A schematic diagram of the device structure of the hardware operating environment involved in the data migration method in the embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solutions of the embodiments of this application are:

[0055] In response to a node splitting request, statistics are collected on the sharding factors corresponding to each piece of data in a preset node to be split, and a sharding factor range statistical result is obtained. The node splitting request is triggered when the load of the node to be split reaches a preset load threshold.

[0056] According to the sharding factor range statistics, the data in the node to be split is divided into two groups to obtain the data to be migrated;

[0057] The to-be-migrated data is migrated from the to-be-split node to a preset target node through a preset management node.

[0058] In the existing technology, with the development of business and the expansion of user base, the number of requests and data that the system needs to process is increasing day by day. The performance bottleneck of the traditional single database architecture is increasingly prominent when facing massive data and high concurrent access. A common optimization method is to introduce sharding database optimization means, which is to shard according to the value range of a certain field and divide the data into multiple databases or data tables. For example, user data with user IDs from 1 to 1000 is stored in the first database, and user data with user IDs from 1001 to 2000 is stored in the second database, thereby reducing the pressure on the single database.

[0059] However, if the amount of data grows rapidly in certain ranges and slowly in other ranges, the storage pressure on the shards in these rapidly growing ranges will increase, resulting in data skew. For example, if user activity in a certain user ID range is abnormally active, the corresponding few shards will be overloaded, resulting in an imbalance in storage and processing pressure, affecting the overall performance of the system.

[0060] This application provides a solution. When the load of the node to be split reaches a preset load threshold, node splitting and data migration are performed according to the data sharding factor in the split node, so that the system can adapt to the distribution of data as the data is actually used. This solves the technical problem that the existing data storage method cannot be dynamically adjusted, resulting in uneven storage and processing pressure, and is more suitable for application scenarios with massive data and high concurrent access.

[0061] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data migration system, etc. The following takes the data migration system as an example to illustrate this embodiment and the following embodiments.

[0062] Based on this, the present application embodiment provides a data migration method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the data migration method of the present application.

[0063] In this embodiment, the data migration method includes steps S10 to S30:

[0064] Step S10, in response to a node splitting request, statistics are collected on the sharding factors corresponding to each piece of data in a preset node to be split, and a sharding factor range statistics result is obtained, wherein the node splitting request is triggered when the load of the node to be split reaches a preset load threshold;

[0065] In response to the node splitting request, the data migration system counts the sharding factors corresponding to each piece of data in the preset node to be split, and obtains the sharding factor range statistics. The node splitting request is triggered when the load of the node to be split reaches a preset load threshold.

[0066] It should be noted that the node splitting request is a signal indicating that the load of the node to be split has reached the preset threshold and a splitting operation is required, which is used to trigger the data splitting and migration process to reduce the node load; the node to be split refers to the node whose load has reached the preset threshold and needs to perform data splitting; the data sharding factor is a value generated based on certain characteristics of the data. For a specific piece of data, this value is unique and is used to guide decisions on data splitting and migration to ensure the rationality of data distribution; the sharding factor range statistical result refers to the result obtained after statistical analysis of the data sharding factor, which is used to guide data splitting, provide specific information on data distribution, and help determine the strategy and goals of data splitting.

[0067] Step S20, dividing the data in the node to be split into two groups according to the sharding factor range statistics result, to obtain the data to be migrated;

[0068] Step S30: Migrate the data to be migrated from the node to be split to a preset target node through a preset management node.

[0069] The data migration system divides the data in the node to be split into two groups according to the sharding factor range statistics to obtain the data to be migrated. The sharding factor range statistics provide detailed information on the data distribution. This information is used to guide the data splitting process to ensure that the data can be evenly distributed to different nodes. Then, the data to be migrated is migrated from the node to be split to the preset target node through the preset management node. The management node plays the role of coordinator in this process and is responsible for monitoring, scheduling and executing data migration tasks. Data migration usually involves data replication and synchronization to ensure the consistency of data between the source node and the target node.

[0070] In a feasible implementation manner, after step S30, steps S401 to S402 may also be included:

[0071] Step S401, updating the target node from a normal state to a migrating state through the management node, and generating a first data node change result;

[0072] Step S402: broadcast the first data node change result to related applications.

[0073] During the data migration process, the state of the target node needs to be updated to reflect its current working status. The management node is responsible for monitoring and updating these states to ensure that other parts of the system can recognize and adapt to these changes. The data migration system updates the target node from a normal state to a migrating state through the management node, and generates the first data node change result. The state update usually involves modifying database records, in-memory data structures, or information in the distributed cache to ensure that all related components can access the latest node state. Then the management node acts as a publisher to publish events of data node state changes (i.e., the first data node change result), and related applications act as subscribers to listen to and respond to these events. The broadcast mechanism ensures that all related applications can receive information about data node state changes so that they can make corresponding adjustments.

[0074] In this implementation, the state update and broadcast mechanism can solve the problem of state synchronization in the distributed system, ensuring that all components can access the latest state information; by timely notifying the relevant application node state changes, the system's adaptability to changes can be improved, and errors caused by inconsistent states can be reduced; by reducing service interruptions caused by state changes, the end-user experience can be improved. As a result, the system can effectively manage and propagate changes in node states, ensuring that all relevant components can respond to these changes in a timely manner, thereby improving the stability and responsiveness of the system.

[0075] In another feasible implementation manner, after step S30, steps S403 to S404 may also be included:

[0076] Step S403, when it is detected that all the data to be migrated are migrated to the target node, the target node is updated from a migrating state to a normal state through the management node, and a second data node change result is generated;

[0077] Step S404: broadcast the change result of the second data node to related applications.

[0078] Once all the data to be migrated has been successfully migrated to the target node, the system needs to confirm the completion of this process. When it is detected that all the data to be migrated has been migrated to the target node, the data migration system updates the target node from the migrating state to the normal state through the management node, and generates a second data node change result. The management node is responsible for monitoring the data migration process, and after confirming that all data has been successfully migrated, it performs a status update operation, and then broadcasts the second data node change result to the relevant applications. The management node acts as the publisher of the event, and propagates the result of the data node state change (the second data node change result) to the relevant applications in the system through a certain communication mechanism, thereby synchronizing the status of the relevant applications. This communication mechanism can be direct network communication, message queue, event bus or any other form of publish / subscribe system.

[0079] In this implementation, the state synchronization problem in the distributed system can be solved through the state update and event broadcast mechanism, ensuring that all components can access the latest state information; by timely notifying the relevant applications of node state changes, the system's adaptability to changes can be improved, and errors caused by inconsistent states can be reduced; by ensuring that all relevant applications can respond to changes in node states, data inconsistency problems can be avoided and data reliability can be improved. As a result, the system can effectively broadcast changes in data node states to all relevant applications, ensuring that they can respond to these changes in a timely manner, thereby improving the stability and responsiveness of the system.

[0080] This embodiment provides a data migration method. When the load of the node to be split reaches a preset load threshold, node splitting and data migration are performed according to the data sharding factor in the split node, so that the system can adapt to the distribution of data as the data is actually used. This solves the technical problem that the existing data storage method cannot be dynamically adjusted, resulting in uneven storage and processing pressure, and is more suitable for application scenarios with massive data and high concurrent access.

[0081] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 , step S20 also includes steps S201 to S205:

[0082] Step S201, determining a splitting threshold value according to the sharding factor range statistical result;

[0083] Step S202: According to the split boundary value, the data in the node to be split is divided into two groups to obtain first split data and second split data, wherein the first split data is data corresponding to a sharding factor greater than the split boundary value, and the second split data is data corresponding to a sharding factor less than or equal to the split boundary value;

[0084] Step S203, comparing the data amounts of the first split data and the second split data to obtain a data amount comparison result;

[0085] Step S204: when the data volume comparison result indicates that the data volume of the first split data is smaller than the data volume of the second split data, the first split data is used as the data to be migrated;

[0086] or,

[0087] Step S205: When the data volume comparison result indicates that the data volume of the second split data is smaller than the data volume of the first split data, the second split data is used as the data to be migrated.

[0088] The data migration system is based on a data splitting strategy, and according to the statistical results of the sharding factor range, determines a splitting boundary value, and according to the splitting boundary value, splits the data in the node to be split to obtain the first splitting data and the second splitting data, wherein the first splitting data is the data belonging to the sharding factor greater than the splitting boundary value, and the second splitting data is the data belonging to the sharding factor less than or equal to the splitting boundary value, wherein the sharding factor may include the size, type, access frequency, business logic, etc. of the data, and the splitting strategy may involve algorithms, such as range-based splitting, hash-based splitting, or list-based splitting, to ensure that the data can be evenly or according to specific rules. The data volume of the first splitting data and the second splitting data are compared based on the data volume measurement and comparison algorithm to obtain the data volume comparison result. The data volume measurement and comparison algorithm involves measuring or calculating the size of the first split data and the second split data, which can be done in a variety of ways, such as counting the number of records, calculating storage space, or evaluating the business value of data. The comparison process may involve simple arithmetic operations, such as comparing two values, or more complex analysis, such as considering the dimension and complexity of the data. When the data volume comparison result is expressed as the data volume of the first split data is less than the data volume of the second split data, the first split data is used as the data to be migrated. When the data volume comparison result is expressed as the data volume of the second split data is less than the data volume of the first split data, the second split data is used as the data to be migrated. The purpose is to ensure that the system load is more balanced after the data migration, and avoid migrating too large a data set to a single node, which may cause new load imbalance.

[0089] In a feasible implementation manner, before step S10, steps S501 to S502 may also be included:

[0090] Step S501, in response to a preset timed detection task, load monitoring is performed on a preset data node group through the management node to obtain a load monitoring result, wherein the data node group includes a plurality of data nodes;

[0091] Step S502: When the load monitoring result shows that the load of any data node in the data node group reaches the load threshold, the data node is defined as the node to be split, and the node splitting request is triggered.

[0092] The data migration system responds to the preset timed detection task, and the management node automatically performs the load monitoring task at the preset time interval (for example, every minute, every hour), collects the performance data of all data nodes in the preset data node group, performs load monitoring on the data node group, and obtains the load monitoring result. Load monitoring can be achieved in a variety of ways, including but not limited to monitoring key performance indicators such as CPU usage, memory usage, disk I / O, network traffic, etc., and then when the load monitoring result shows that the load of any data node in the data node group reaches the load threshold, the data node is defined as a node to be split, and the node splitting request is made. The load threshold is a pre-set performance indicator used to identify when a node becomes overloaded and needs to be split to reduce the load.

[0093] In this implementation, the node load can be effectively managed through regular monitoring and automatic response mechanisms to avoid problems caused by excessive load; the automatically triggered node splitting request helps maintain the stability of the system and prevents system crashes caused by overloading of a single node; reasonable node splitting can improve the overall performance of the system and optimize resource utilization. Thus, the system can regularly monitor the load of the node and automatically trigger the splitting operation when the load is too high, thereby achieving load balancing and maintaining system performance.

[0094] In a feasible implementation manner, the data to be migrated includes a plurality of data to be called, and after step S30, steps S601 to S603 may also be included:

[0095] Step S601, receiving a data call request corresponding to any of the data to be called;

[0096] Step S602, performing data query on the target node according to the data call request to obtain a query result;

[0097] Step S603: when the query result indicates that the data to be called is not found in the target node, a data query is performed on the node to be split according to the data call request.

[0098] The data migration system receives a data call request corresponding to any data to be called. The data call request may be sent through API call, message queue, database query or other communication protocol. Then, based on the data retrieval mechanism, the target node is queried according to the data call request to obtain the query result. The management node initiates a query operation to the target node to retrieve the requested data based on the received data call request. The data query can be executed through SQL query, NoSQL query, API call or other data access methods, depending on the format and system architecture of the target node data storage. Finally, based on the data query fallback mechanism, when the query result indicates that the data to be called is not found in the target node, the system will automatically fall back to the original node to be split for further data query. This mechanism ensures that the data is accessible even during the data migration process, ensuring that the data call request can be met.

[0099] In this implementation, by flexibly switching queries between the target node and the node to be split, the continuity problem of data access during data migration is solved, ensuring that even if part of the data is being migrated, the system can still respond to data call requests, improving the availability of the system, and during the data migration process, the consistency of data is ensured through the fallback query mechanism, avoiding data loss or inconsistency. As a result, the system can flexibly handle data call requests, ensuring the availability and continuity of data even during the data migration process.

[0100] In this embodiment, by reasonably splitting the data and selecting a smaller data set for migration, the load on a single node can be effectively reduced to avoid overload. Data splitting and migration help to more effectively manage large-scale data, improve data accessibility and processing speed, and optimize data distribution. The system can be more easily expanded to adapt to the growing amount of data and user needs. Thus, the system can effectively reasonably split the data in the node to be split according to the statistical results of the sharding factor range, and select a smaller data set as the data to be migrated according to the data volume comparison results, thereby achieving load balancing and optimizing resource allocation.

[0101] For example, to help understand the implementation process of the data migration method obtained by combining the first embodiment and the second embodiment, please refer to Figure 3 to Figure 9 , Figure 3 A schematic diagram of the structure of the data migration system provided in Embodiment 1 or 2 of the present application, Figure 4 A brief flowchart of the data migration method provided in Embodiment 1 or 2 of the present application is shown in FIG. Figure 5 A schematic diagram of data node data volume monitoring of the data migration method provided in Embodiment 1 or 2 of the present application, Figure 6 A schematic diagram of node splitting of the data migration method provided in Embodiment 1 or 2 of the present application, Figure 7 A schematic diagram of updating the distribution relationship of the data migration start of the data migration method provided in Embodiment 1 or 2 of the present application, Figure 8 A data migration diagram of the data migration method provided in Embodiment 1 or 2 of the present application, Fig. 9 A schematic diagram of updating the distribution relationship of data migration completion of the data migration method provided in Embodiment 1 or 2 of the present application, specifically:

[0102] The system consists of a management node and multiple data nodes. The data node is responsible for storing data. The data is divided according to specific rules, and each data node stores a part of the data. The management node is responsible for monitoring the load of each node and dynamically adjusting the sharding factor range of each data node according to the actual data distribution.

[0103] The management node will regularly monitor the load of each data node. When it detects that the load of a data node exceeds the threshold, it will trigger the split of the data node and migrate part of the data in the node to a new node. After the split starts and is completed, the management node will broadcast the changes in the sharding factor and the update of the node status to all applications, so that the applications can update their caches to ensure that the data nodes can be correctly located during queries. During the migration process, data access operations may need to access the original node and the new node until the data migration is completed. Taking the node to be split as data node A and the target node as data node C as an example, the specific implementation is as follows:

[0104] (1) Monitor node load.

[0105] In this embodiment, there is a management node that regularly monitors the amount of data on each data node. When the amount of data stored on a data node reaches a threshold, it triggers a shard split, which means that the data on the data node is divided into two parts according to the shard factor. The node stores one of the data and the other is migrated to a new node for storage. Figure 5 For example, the management node detects that the amount of data in data node A is too much, and decides to split node A into shards, splitting the shard factor that the node is responsible for into two parts, 0-1 and 2-4, each stored by one node.

[0106] (2) Dynamically select the splitting method.

[0107] When splitting a shard, the shard factor is split into two parts. The data volumes corresponding to these two parts are usually inconsistent. In order to minimize the workload of data migration, the part with larger data volume needs to be left in the original node.

[0108] like Figure 6As shown in the figure, in order to facilitate the statistics of the amount of data within each sharding factor range, the sharding factor of the data will be stored in the node as a data field when the data is stored. When the shard is split, the amount of data corresponding to the two sharding factor ranges can be directly counted, that is, there are 17 data in the sharding factor range of 0-1, and 10 data in the sharding factor range of 2-4. It is decided that the original node A stores the data with a sharding factor of 0-1, and the new node C stores the data with a sharding factor of 2-4. The data with a sharding factor of 2-4 in node A needs to be migrated to the new node.

[0109] (3) Broadcast the split start message.

[0110] The management node will broadcast the changes of the data node to all applications. Since the data has not yet been migrated from the original node to the new node, the data in the new node may still be in the original node. The new node status is in migration. If the data is not found in the node, it is also necessary to query the original node, that is, the previous node or the next node. Figure 7 As shown, node C will take over the data with a sharding factor of 2-4 in the original node A, so the state of node C is L at this time, and the data may be in node C or in node A.

[0111] (4)Data migration.

[0112] The management node starts to migrate data one by one. Since the data stores the sharding factor of the data, the data to be migrated can be found directly without traversing all the data in the original node and calculating the sharding factor. During the migration process, the management node will migrate the data to the new node one by one. Both nodes will provide services to the outside world. However, when the corresponding data is not found in the new node, it may be that the data has not been migrated yet and is still in the original node, so it is necessary to visit the original node again.

[0113] like Figure 8 As shown in the figure, when querying data b, since the data has been migrated, it can be found directly in node C. However, for data c, since the data has not been migrated, it cannot be found when querying in node C now, and it is necessary to query in node A again.

[0114] This data migration method minimizes the migration dimension, so that the storage system only locks the single data being migrated, without affecting the normal access and writing of other data under the shard, avoiding impact on the business system.

[0115] (5) Broadcast the split completion message.

[0116] After all data migration is completed, the management node will broadcast again to update the status of the new node from migration to normal, and the split is completed.

[0117] Compared with the prior art, the data updating method described in this embodiment has the following advantages:

[0118] (1) By continuously monitoring the load of each node and automatically adjusting the range of node sharding factors, the system can adapt to changes in data distribution and dynamically adjust node load.

[0119] (2) Break down data migration operations into operations on a single piece of data so that services can continue to be provided to the outside world during the migration process.

[0120] (3) Dynamically select the node splitting direction based on the data distribution within the node, effectively reducing the workload of data migration and the impact of the migration process on the system.

[0121] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data migration method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0122] This application also provides a data migration device, please refer to Fig.10 , the data migration device comprises:

[0123] A sharding factor statistics module 10 is used to, in response to a node splitting request, perform statistics on the sharding factors corresponding to each piece of data in a preset node to be split, and obtain a sharding factor range statistics result. The node splitting request is triggered when the load of the node to be split reaches a preset load threshold;

[0124] A node splitting module 20 is used to divide the data in the node to be split into two groups according to the sharding factor range statistical result to obtain data to be migrated;

[0125] The data migration module 30 is used to migrate the data to be migrated from the node to be split to a preset target node through a preset management node.

[0126] Optionally, the node splitting module 20 is further configured to:

[0127] Determine a splitting threshold value according to the statistical result of the fragmentation factor range;

[0128] According to the split boundary value, the data in the node to be split is divided into two groups to obtain first split data and second split data, wherein the first split data is data corresponding to a sharding factor greater than the split boundary value, and the second split data is data corresponding to a sharding factor less than or equal to the split boundary value;

[0129] Compare the data amounts of the first split data and the second split data to obtain a data amount comparison result;

[0130] When the data volume comparison result indicates that the data volume of the first split data is smaller than the data volume of the second split data, the first split data is used as the data to be migrated;

[0131] or,

[0132] When the data volume comparison result indicates that the data volume of the second split data is smaller than the data volume of the first split data, the second split data is used as the data to be migrated.

[0133] Optionally, the fragmentation factor statistics module 10 is further used for:

[0134] In response to a preset timed detection task, the management node performs load monitoring on a preset data node group to obtain a load monitoring result, wherein the data node group includes a plurality of data nodes;

[0135] When the load monitoring result shows that the load of any data node in the data node group reaches the load threshold, the data node is defined as the node to be split, and the node splitting request is triggered.

[0136] Optionally, the data to be migrated includes a plurality of data to be called, and the data migration module 30 is further used for:

[0137] Receiving a data call request corresponding to any of the data to be called;

[0138] Performing data query on the target node according to the data call request to obtain a query result;

[0139] When the query result indicates that the data to be called is not found in the target node, a data query is performed on the node to be split according to the data calling request.

[0140] Optionally, the data migration module 30 is further used for:

[0141] By means of the management node, the target node is updated from a normal state to a migrating state, and a first data node change result is generated;

[0142] Broadcast the first data node change result to related applications.

[0143] Optionally, the data migration module 30 is further used for:

[0144] When it is detected that all the data to be migrated have been migrated to the target node, the target node is updated from a migrating state to a normal state through the management node, and a second data node change result is generated;

[0145] Broadcast the second data node change result to related applications.

[0146] The data migration device provided by the present application adopts the data migration method in the above embodiment, which can solve the technical problem that the existing data storage method cannot be adjusted dynamically, resulting in unbalanced storage and processing pressure. Compared with the prior art, the beneficial effects of the data migration device provided by the present application are the same as the beneficial effects of the data migration method provided by the above embodiment, and the other technical features in the data migration device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0147] The present application provides a data migration device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data migration method in the above-mentioned embodiment one.

[0148] Reference below Fig.11 , which shows a schematic diagram of the structure of a data migration device suitable for implementing an embodiment of the present application. The data migration device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.11 The data migration device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0149] like Fig.11As shown, the data migration device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data migration device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data migration device to communicate with other devices wirelessly or by wire to exchange data. Although the data migration device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0150] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0151] The data migration device provided by the present application adopts the data migration method in the above embodiment, which can solve the technical problem that the existing data storage method cannot be adjusted dynamically, resulting in unbalanced storage and processing pressure. Compared with the prior art, the beneficial effects of the data migration device provided by the present application are the same as the beneficial effects of the data migration method provided by the above embodiment, and the other technical features in the data migration device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0152] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0154] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data migration method in the above-mentioned embodiment.

[0155] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0156] The computer-readable storage medium may be included in the data migration device; or may exist independently without being assembled into the data migration device.

[0157] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the data migration device, the data migration device:

[0158] In response to a node splitting request, statistics are collected on the sharding factors corresponding to each piece of data in a preset node to be split, and a sharding factor range statistical result is obtained. The node splitting request is triggered when the load of the node to be split reaches a preset load threshold.

[0159] According to the sharding factor range statistics, the data in the node to be split is divided into two groups to obtain the data to be migrated;

[0160] The to-be-migrated data is migrated from the to-be-split node to a preset target node through a preset management node.

[0161] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] The modules involved in the embodiments of the present application may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0164] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data migration method, and can solve the technical problem that the existing data storage method cannot be dynamically adjusted, resulting in unbalanced storage and processing pressure. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the data migration method provided by the above-mentioned embodiment, and will not be repeated here.

[0165] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data migration method when executed by a processor.

[0166] The computer program product provided by the present application can solve the technical problem that the existing data storage method cannot be dynamically adjusted, resulting in uneven storage and processing pressure. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the data migration method provided by the above embodiment, which will not be repeated here.

[0167] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data migration method, characterized in that: The method comprises: In response to a node splitting request, statistics are collected on the sharding factors corresponding to each piece of data in a preset node to be split, and a sharding factor range statistical result is obtained. The node splitting request is triggered when the load of the node to be split reaches a preset load threshold. According to the sharding factor range statistics, the data in the node to be split is divided into two groups to obtain the data to be migrated; The to-be-migrated data is migrated from the to-be-split node to a preset target node through a preset management node.

2. The method according to claim 1, characterized in that The step of dividing the data in the node to be split into two groups according to the sharding factor range statistics to obtain the data to be migrated comprises: Determine a splitting threshold value according to the statistical result of the fragmentation factor range; According to the split boundary value, the data in the node to be split is divided into two groups to obtain first split data and second split data, wherein the first split data is data corresponding to a sharding factor greater than the split boundary value, and the second split data is data corresponding to a sharding factor less than or equal to the split boundary value; Compare the data amounts of the first split data and the second split data to obtain a data amount comparison result; When the data volume comparison result indicates that the data volume of the first split data is smaller than the data volume of the second split data, the first split data is used as the data to be migrated; or, When the data volume comparison result indicates that the data volume of the second split data is smaller than the data volume of the first split data, the second split data is used as the data to be migrated.

3. The method according to claim 1, characterized in that Before the step of performing statistics on the sharding factors corresponding to each piece of data in the preset node to be split in response to the node splitting request and obtaining the statistical result of the sharding factor range, the step also includes: In response to a preset timed detection task, the management node performs load monitoring on a preset data node group to obtain a load monitoring result, wherein the data node group includes a plurality of data nodes; When the load monitoring result shows that the load of any data node in the data node group reaches the load threshold, the data node is defined as the node to be split, and the node splitting request is triggered.

4. The method according to claim 1, characterized in that The data to be migrated includes a plurality of data to be called. After the step of migrating the data to be migrated from the node to be split to the preset target node through the preset management node, the method further includes: Receiving a data call request corresponding to any of the data to be called; Performing data query on the target node according to the data call request to obtain a query result; When the query result indicates that the data to be called is not found in the target node, a data query is performed on the node to be split according to the data calling request.

5. The method according to claim 1, characterized in that After the step of migrating the data to be migrated from the node to be split to the preset target node through the preset management node, the method further includes: By means of the management node, the target node is updated from a normal state to a migrating state, and a first data node change result is generated; Broadcast the first data node change result to related applications.

6. The method according to claim 5, characterized in that After the step of migrating the data to be migrated from the node to be split to the preset target node through the preset management node, the method further includes: When it is detected that all the data to be migrated have been migrated to the target node, the target node is updated from a migrating state to a normal state through the management node, and a second data node change result is generated; Broadcast the second data node change result to related applications.

7. A data migration device, characterized in that: The device comprises: A sharding factor statistics module is used to, in response to a node splitting request, perform statistics on the sharding factors corresponding to each piece of data in a preset node to be split, and obtain sharding factor range statistics results. The node splitting request is triggered when the load of the node to be split reaches a preset load threshold; A node splitting module, used to divide the data in the node to be split into two groups according to the statistical result of the sharding factor range, and obtain the data to be migrated; The data migration module is used to migrate the data to be migrated from the node to be split to a preset target node through a preset management node.

8. A data migration device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data migration method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 6 are implemented.

Images