Online fragment recombination method and related equipment

Abstract

The invention provides an online fragment restructuring method and related equipment, and relates to the technical field of computers.The method comprises the steps that an online fragment restructuring request is received, a target fragment is determined, a migration task is created in the target fragment, and a first coroutine and a second coroutine are created in the target fragment according to the incremental copying process in the migration task, the first coroutine and the second coroutine are executed asynchronously and concurrently, a binary log event in the source fragment is obtained in the first coroutine and written into a relay log of the target fragment, and data in the relay log is written into the target fragment through the second coroutine. The problem of shutdown migration is solved, shutdown processing is not needed, usability and efficiency are improved, dependence on manual operation is reduced, and the scene application range is widened.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to an online sharding reorganization method and related equipment. BACKGROUND

[0002] With the development of business, the amount of data in the database will change continuously. Users may need to expand or shrink the database nodes according to the size of the data amount. When new nodes need to be added to improve the processing capacity of the system, the original data needs to be redistributed to the new nodes to avoid high node load, thereby affecting the query and write performance. Conversely, when the nodes need to be reduced to reduce costs, the data on these nodes also needs to be migrated to other nodes to improve resource utilization.

[0003] In the related art, the processing mode of expansion and shrinkage is stoppage migration, that is, stopping the write service, exporting data through other tools, splitting data according to new sharding rules, and restoring the service after importing to the new shards.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The purpose of the present disclosure is to provide an online sharding reorganization method and related equipment, which at least partially solves the problem of stoppage maintenance for expansion and shrinkage in the prior art and the limitation of scene application.

[0006] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, an online sharding reorganization method is provided, the method comprising:

[0008] receiving an online sharding reorganization request, determining a target shard, and creating a migration task in the target shard;

[0009] for an incremental replication process in the migration task, creating a first coroutine and a second coroutine in the target shard; the first coroutine and the second coroutine are asynchronously and concurrently executed;

[0010] writing, by the first coroutine, binary log events in a source shard to a relay log in the target shard;

[0011] writing, by the second coroutine, data in the relay log to the target shard.

[0012] In a possible embodiment, the online sharding reorganization request comprises an expansion request and a shrinkage request;

[0013] When the online shard reorganization request is an expansion request, the target shard is multiple;

[0014] When the online shard reorganization request is a shrink request, the source shard is multiple.

[0015] In a possible embodiment, the method further comprises:

[0016] When the data replication delay time is less than the first preset time period, the routing information of the read operation is modified from the source shard to the target shard; the data replication time is the current time, and the remaining time for replicating the incremental data in one source shard to one target shard.

[0017] In a possible embodiment, the method further comprises:

[0018] Stopping the write operation in the source shard;

[0019] Obtaining a first data site of the source shard and a second data site of the target shard;

[0020] Judging whether the second data site catches up with the first data site within a second preset time period;

[0021] If yes, the write operation is allowed to be performed in the target shard, and the routing information of the write operation is modified from the source shard to the target shard;

[0022] If no, a rollback operation is performed, and the write operation is continued to be performed in the source shard before the write operation in the source shard is stopped.

[0023] In a possible embodiment, the method further comprises:

[0024] According to the task parameter of the online shard reorganization request, the to-be-copied table in the source shard is persisted and written into a replication state table in a meta database table;

[0025] The to-be-copied table in the meta database table is copied to the target shard, and a record of the to-be-copied table is deleted in the replication state table every time one to-be-copied table is copied.

[0026] In a possible embodiment, the method further comprises:

[0027] Through the first coroutine, every time a binary log event is written in the relay log in the source shard, data in the relay log is written in the target shard through the second coroutine; based on a pre-set condition variable synchronization primitive, the first coroutine and the second coroutine are asynchronously and concurrently executed.

[0028] In a possible embodiment, the method further comprises:

[0029] A shard reorganization completion request is received;

[0030] Source shard information is cleaned up;

[0031] deleting data of the target shard in the metabase.

[0032] According to still another aspect of the present disclosure, there is provided an online shard reorganization system, comprising a management module, a controller, an executor and a topology server;

[0033] The management module receives an online shard reorganization request, determines a target shard, and creates a migration task in the target shard;

[0034] The controller creates a first coroutine and a second coroutine in the target shard for an incremental replication process in the migration task; the first coroutine and the second coroutine are asynchronously and concurrently executed;

[0035] The controller invokes the executor, and the executor writes binary log events in the source shard to a relay log in the target shard through the first coroutine;

[0036] The executor writes data in the relay log to the target shard through the second coroutine;

[0037] The topology server is configured to store metadata of the source shard and the target shard.

[0038] In a possible embodiment, the executor is further configured to:

[0039] When a data replication delay time is less than a first preset time period, the routing information of the read operation is modified from the source shard to the target shard; the data replication time is a current time, and the remaining time for replicating incremental data in one source shard to one target shard.

[0040] In a possible embodiment, the executor is further configured to:

[0041] stop executing the write operation in the source shard;

[0042] obtain a first data site of the source shard and a second data site of the target shard;

[0043] determine whether the second data site catches up with the first data site within a second preset time period;

[0044] if yes, allow executing the write operation in the target shard, and modify the routing information of the write operation from the source shard to the target shard;

[0045] if no, perform a rollback operation, and roll back to before stopping executing the write operation in the source shard, and continue to execute the write operation in the source shard.

[0046] In a possible embodiment, the executor is further configured to:

[0047] According to the task parameters of the online shard reorganization request issued by the controller, the to-be-copied table in the source shard is persisted and written into the replication state table in the metabase table;

[0048] The to-be-copied table in the metabase table is copied into the target shard, and the record of the to-be-copied table in the replication state table is deleted after copying each to-be-copied table.

[0049] In a possible embodiment, the executor is further configured to:

[0050] Through the first coroutine, each time a binary log event is written into the relay log in the source shard, the data in the relay log is written into the target shard through the second coroutine; based on the pre-set condition variable synchronization primitive, the first coroutine and the second coroutine are asynchronously and concurrently executed.

[0051] In a possible embodiment, the management module is further configured to:

[0052] A shard reorganization completion request is received, and the source shard information is cleaned up; the data of the target shard in the metabase is deleted.

[0053] According to still another aspect of the present disclosure, there is provided an online shard reorganization device, comprising:

[0054] A management unit is configured to receive an online shard reorganization request, determine a target shard, and create a migration task in the target shard;

[0055] A control unit is configured to create a first coroutine and a second coroutine in the target shard for the incremental replication process in the migration task; the first coroutine and the second coroutine are asynchronously and concurrently executed;

[0056] A first execution unit is configured to acquire, through the first coroutine, a binary log event in the source shard to write into a relay log in the target shard;

[0057] A second execution unit is configured to write, through the second coroutine, data in the relay log into the target shard.

[0058] According to still another aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method of any one of the first aspect via execution of the executable instructions.

[0059] According to still another aspect of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method of any one of the first aspect.

[0060] According to still another aspect of the present disclosure, a computer program product or computer program is also provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method in the first aspect above.

[0061] The online shard reorganization method and related device provided by the embodiments of the present disclosure relate to the technical field of computers, and the method comprises the following steps: receiving an online shard reorganization request, determining a target shard, creating a migration task in the target shard, creating a first coroutine and a second coroutine in the target shard for an incremental replication process in the migration task, the first coroutine and the second coroutine are asynchronously and concurrently executed, writing binary log events in a source shard into a relay log in the target shard through the first coroutine, and writing data in the relay log into the target shard through the second coroutine. The problem of downtime migration is solved, and the downtime processing is not required, thereby improving the availability and efficiency, reducing the dependence on manual operation, and improving the scene application range.

[0062] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0063] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0064] Figure 1 A structural schematic diagram of an online shard reorganization system in an embodiment of the present disclosure is shown;

[0065] Figure 2 A schematic diagram of capacity expansion of an online shard reorganization system in an embodiment of the present disclosure is shown;

[0066] Figure 3 A schematic diagram of capacity reduction of an online shard reorganization system in an embodiment of the present disclosure is shown;

[0067] Figure 4 A flowchart of an online shard reorganization method in an embodiment of the present disclosure is shown;

[0068] Figure 5 A flowchart of a data replication process in an embodiment of the present disclosure is shown;

[0069] Figure 6A flowchart illustrating a read-write operation switching method in an embodiment of the present disclosure is shown.

[0070] Figure 7 A structural diagram of an online shard reorganization device in an embodiment of the present disclosure is shown.

[0071] Figure 8 A structural diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0072] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive concept to those skilled in the art. Like reference numerals may be used to refer to like elements throughout.

[0073] In addition, the accompanying drawings are included to provide a thorough understanding of the present disclosure and are not intended to be exhaustive or to limit the present disclosure to the precise outlines described. The same or similar components may be designated by the same or similar reference numerals throughout the drawings, and thus repeated description may be omitted. Some of the block diagrams shown in the drawings are functional entities, and thus do not necessarily have to correspond to physically or logically independent entities. The functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0074] As the business develops, the amount of data in the database will change. Users may need to expand or shrink the database nodes according to the size of the data amount. When new nodes need to be added to improve the processing capacity of the system, the original data needs to be redistributed to the new nodes to avoid excessive node load and thus affect the query and write performance; conversely, when the nodes need to be reduced to reduce costs, the data on these nodes also needs to be migrated to other nodes to improve resource utilization.

[0075] In related technologies, the processing method of expansion and shrinkage is stop-migration, that is, stopping the write service, exporting data through other tools, splitting data according to the new shard rule, restoring the service after importing to the new shard.

[0076] The above scheme has the following problems: stop-migration, which cannot meet the high availability requirement. Manual complexity is high, and shard migration involves data replication, routing switching, consistency verification, and other steps. Manual operation is prone to errors and low in efficiency, and lacks flexibility: the shard topology is fixed, and it is difficult to dynamically adapt to changes in business requirements, the applicable scenarios are limited, and it is only suitable for systems with small data volume and allowing downtime maintenance, such as internal management systems, and is not suitable for financial and e-commerce scenarios.

[0077] Based on this, the embodiment of the disclosure provides an online shard reorganization method and related equipment, which relates to the technical field of computers. The method comprises: receiving an online shard reorganization request, determining a target shard, creating a migration task in the target shard, creating a first coroutine and a second coroutine in the target shard for an incremental replication process in the migration task, the first coroutine and the second coroutine being asynchronously and concurrently executed, writing binary log events in a source shard into a relay log of the target shard through the first coroutine, and writing data in the relay log into the target shard through the second coroutine. The problem of downtime migration is solved, and the dependence on manual operation is reduced, the availability and efficiency are improved, and the scope of application is improved. The application is not aware of the downtime, the overall operation is simple, and the application is not aware of the downtime.

[0078] Figure 1 The structure of an online shard reorganization system in the embodiment of the disclosure is shown in FIG. 1. Figure 1 As shown in FIG. 1, the online shard reorganization system 100 comprises a management module 101, a controller 102, an executor 103, and a topology server 104.

[0079] The management module 101 is configured to receive a request of a user and create a task, and distribute the task to the controller 102.

[0080] The controller 102 is configured to receive a request of a user from the management module 101 and execute through the executor 103.

[0081] The executor 103 is configured to synchronize the stock and incremental data of a source shard and write into a target shard.

[0082] The topology server 104 is configured to store metadata, wherein the metadata comprises routing planning and other information, such as routing information for performing read and write operations, information for stopping read and write operations, and information for allowing read and write operations.

[0083] The management module 101 receives an online shard reorganization request, determines a target shard, and creates a migration task in the target shard. The controller 102 creates a first coroutine and a second coroutine in the target shard for an incremental replication process in the migration task; the first coroutine and the second coroutine are asynchronously and concurrently executed; the controller 102 calls the executor 103, the executor 103 writes binary log events in a source shard into a relay log of the target shard through the first coroutine; the executor 103 writes data in the relay log into the target shard through the second coroutine; and the topology server 104 is configured to store metadata of the source shard and the target shard.

[0084] Through the above system, flexible expansion can be performed, dynamic adjustment of shard topology can be supported, and business changes can be adapted.

[0085] In a possible embodiment,Figure 1 The system in the embodiment of the present disclosure supports scaling up of the shards and also supports scaling down of the shards.

[0086] Figure 2 A schematic diagram of the online shard reorganization system in the embodiment of the present disclosure is shown in FIG. 1, which includes two target shards, target shard 1 and target shard 2, and one source shard. Figure 2 A schematic diagram of the online shard reorganization system in the embodiment of the present disclosure is shown in FIG. 1, which includes two target shards, target shard 1 and target shard 2, and one source shard.

[0087] The request of the user is received by the management module 101, and based on the parameters of the target shard carried in the user request, the target shard is created, and the topology information of the shard is saved in the topology server 103, Figure 2 The management module 101, the controller 102 and the executor 103 in the embodiment of the present disclosure, and the calling mode between the source shard and the target shard, are taken as an example in the form of remote procedure call (RPC).

[0088] The controller 102 issues a task to the executor 103, and the executor 103 migrates data from the instance of the source shard to the instances of the target shard 1 and the target shard 2. The instance is taken as an example of a mysql instance.

[0089] The management module 101 can obtain the read-write topology (topo) information in the topology server 104.

[0090] Figure 3 A schematic diagram of the online shard reorganization system in the embodiment of the present disclosure is shown in FIG. 1, which includes two target shards, target shard 1 and target shard 2, and one source shard. Figure 3 A schematic diagram of the online shard reorganization system in the embodiment of the present disclosure is shown in FIG. 1, which includes two target shards, target shard 1 and target shard 2, and one source shard.

[0091] It should be noted that the source shard in the embodiment of the present disclosure can include multiple, for example, when scaling up, two source shards are scaled up to 6 target shards, and when scaling down, 10 source shards are scaled down to 3 target shards.

[0092] The embodiment of the present disclosure is described for the method, which is specifically described as follows.

[0093] Figure 4 A flowchart of an online shard reorganization method in the embodiment of the present disclosure is shown in FIG. 1, which can be applied to an electronic device, and can also be applied to the online shard reorganization system in the embodiment of the present disclosure. Figure 1 A flowchart of an online shard reorganization method in the embodiment of the present disclosure is shown in FIG. 1, which can be applied to an electronic device, and can also be applied to the online shard reorganization system in the embodiment of the present disclosure. Figure 4 A flowchart of an online shard reorganization method in the embodiment of the present disclosure is shown in FIG. 1, which can be applied to an electronic device, and can also be applied to the online shard reorganization system in the embodiment of the present disclosure.

[0094] S402: Receive an online shard reorganization request, determine a target shard, and create a migration task in the target shard.

[0095] In a possible embodiment, the parameter of the target shard is included in the online shard reorganization request, and after the management module receives the shard reorganization request, a migration task is created on the target shard through a remote procedure call according to the parameter of the target shard.

[0096] The creation of the migration task on the target shard can include adding a task record of the migration task in a virtual replication task table (vreplication) of a metadata database table.

[0097] S404: For the incremental replication process in the migration task, a first coroutine and a second coroutine are created in the target shard, and the first coroutine and the second coroutine are asynchronously and concurrently executed.

[0098] In a possible embodiment, the management module issues the migration task to a controller of the target shard, and the controller creates the first coroutine and the second coroutine in the target shard for the incremental replication process in the migration task, and replicates the incremental data.

[0099] S406: The binary log event in the source shard is obtained through the first coroutine and written into a relay log of the target shard.

[0100] S408: The data in the relay log is written into the target shard through the second coroutine.

[0101] In a possible embodiment, the executor writes the binary log event in the source shard into the relay log of the target shard through the first coroutine, and writes the data in the relay log into the target shard through the second coroutine. The first coroutine and the second coroutine are asynchronously and concurrently executed.

[0102] In a possible embodiment, a condition variable (sync.Cond) synchronization primitive can be set to asynchronously and concurrently execute the first coroutine and the second coroutine. Through the first coroutine, a binary log event is written into the relay log of the source shard, and through the second coroutine, the data in the relay log is written into the target shard.

[0103] Synchronization through the sync.Cond synchronization primitive can improve the concurrency capability and migration efficiency of the system.

[0104] In a possible embodiment, the online shard reorganization request includes a capacity expansion request and a capacity reduction request.

[0105] In a possible embodiment, when the online shard reorganization request is the capacity expansion request, the data in the source shard is synchronously replicated into multiple target shards, and the target shards are multiple.

[0106] In a possible embodiment, when the online shard reorganization request is a shrink request, data in the plurality of source shards is synchronously copied into one target shard, and the source shards are the plurality of shards.

[0107] In a possible embodiment, the source shard is an original existing database node, the source shard is a shard that needs to migrate data out, the target shard is a shard that needs to migrate data in, and the target shard is a new database node.

[0108] In a possible embodiment, for the inventory replication process in the migration task, the following steps can be performed: the executor writes the to-be-copied tables in the source shard into a copy state table in a meta database table according to the task parameters of the online shard reorganization request issued by the controller, copies the to-be-copied tables in the meta database table to the target shard, and deletes the record of each to-be-copied table in the copy state table after copying one to-be-copied table. Through the above manner, the inventory copying process is completed. In the embodiment of the present disclosure, the inventory replication process and the incremental replication process can be completed independently or sequentially, and are not limited.

[0109] Figure 5 A flowchart of a data replication process in the embodiment of the present disclosure is shown as follows: Figure 5 The flowchart includes the following steps:

[0110] S502: receiving an online shard reorganization request, determining a target shard, and creating a migration task in the target shard.

[0111] S504: writing the to-be-copied tables in the source shard into a copy state table in a meta database table according to the task parameters of the online shard reorganization request.

[0112] S506: copying the to-be-copied tables in the meta database table to the target shard, deleting the record of each to-be-copied table in the copy state table after copying one to-be-copied table, and completing the inventory replication process.

[0113] In a possible embodiment, when the to-be-copied tables are completely copied, it indicates that the inventory replication process is completed, and the incremental replication process is entered.

[0114] S508: for the incremental replication process in the migration task, creating a first coroutine and a second coroutine in the target shard, and the first coroutine and the second coroutine are asynchronously and concurrently executed.

[0115] S510: acquiring, through the first coroutine, binary log events in the source shard and writing the binary log events into a relay log in the target shard.

[0116] S512: writing, through the second coroutine, data in the relay log into the target shard.

[0117] Through the above manner in the embodiments of the present disclosure, the shard structure of the database cluster is dynamically adjusted without shutdown, the number of shards is increased or reduced, and the needs of data growth or performance optimization are met in a manner of online shard reorganization. During the shard processing, no perception is applied, and the shard can be completed by simply inputting an online shard reorganization request command.

[0118] In a possible embodiment, after receiving the command of context cancellation, the replication process of the two coroutines is stopped, and the incremental replication process is ended.

[0119] In a possible embodiment, during the incremental replication process, the read-write traffic can be continuously served, and the switching of the read-write traffic operation is completed. The following steps can be performed, including: when the data replication time is less than the first preset time period, the routing information of the read operation is modified from the source shard to the target shard, the data replication time is the remaining time from the current time when the incremental data in a source shard is replicated to a target shard.

[0120] In a possible embodiment, the user traffic switching operation request sent by the user can be received, and the switching of the read-write operation is triggered.

[0121] In a possible embodiment, the data replication time can be obtained according to the third preset time period, and it is determined whether the data replication time is less than the first preset time period and whether the switching condition of the read-write operation is met, so as to trigger the switching of the read-write operation.

[0122] Figure 6 A flowchart of the switching of the read-write operation in the embodiments of the present disclosure is shown. Taking receiving the switching operation request of the user as an example, as shown in FIG. 6, the following steps are included. Figure 6

[0123] S602: The user traffic switching operation request sent by the user is received.

[0124] S604: It is determined whether the data replication time is less than the first preset time period. If yes, S606 is performed; if no, the process is ended.

[0125] The data replication time is the remaining time from the current time when the incremental data in a source shard is replicated to a target shard.

[0126] In a possible embodiment, when the capacity is expanded, the source shard is expanded into multiple target shards, and the data replication time may be different when the incremental data in the source shard is replicated to different target shards. The data replication time of each target shard is determined separately, and the switching of the read-write operation is initiated separately.

[0127] ​If the data replication time is less than the first preset time period, it indicates that the incremental data in the source shard needs a relatively short time to be replicated to the target shard A, and the switching condition for initiating the write operation is met, and thus the write operation can be initiated.

[0128] In a possible embodiment, when the capacity is shrunk, a plurality of source shards are shrunk into one target shard, and the data replication time in different source shards may be different when incremental data is replicated to the target shard. The data replication time is determined for each source shard separately, and the switching of the read-write operation is initiated separately.

[0129] If the data replication time is less than the first preset time period, it indicates that the incremental data in the source shard A needs a relatively short time to be replicated to the target shard, and the switching condition for initiating the read-write operation is met, and thus the read-write operation can be initiated.

[0130] In a possible embodiment, if the data replication time is greater than the first preset time period, it indicates that the switching condition for initiating the read-write operation is not met, the switching process of the read-write operation is ended, the next switching request of the read-write operation is waited for, or the data replication time is obtained according to the third preset time period, and it is determined whether the switching condition for initiating the read-write operation is met, and the read-write operation is still completed in the source shard.

[0131] S606: The routing information of the read operation is modified from the source shard to the target shard.

[0132] S608: The write operation in the source shard is stopped.

[0133] S610: A first data point of the source shard and a second data point of the target shard are obtained.

[0134] S612: It is determined whether the second data point catches up with the first data point within a second preset time period; if yes, S614 is performed; if no, S616 is performed.

[0135] S614: The write operation in the target shard is allowed, and the routing information of the write operation is modified from the source shard to the target shard.

[0136] S616: A rollback operation is performed, and the write operation in the source shard before the write operation in the source shard is stopped is continued.

[0137] In a possible embodiment, within the second preset time period, the second data point (GTID) does not catch up with the first data point, which indicates that the data synchronization between the source shard and the target shard is not completed, and the write operation in the source shard needs to be continued, and thus a rollback operation is performed, and the write operation in the source shard before the write operation in the source shard is stopped is continued.

[0138] In a possible embodiment, the steps of S606, S608, S614 can be completed by modifying information in the topo server.

[0139] In a possible embodiment, the management module receives a sharding reorganization completion request sent by a user, cleans up source shard information in the topo server, and deletes data of the target shard in the metadata database. Deleting data in the metadata database can include data involved in the migration task.

[0140] According to the same inventive concept as the method embodiments described above, the embodiments of the present application also provide an online sharding reorganization device. Figure 7 FIG. 7 shows a structural schematic diagram of an online sharding reorganization device provided by an embodiment of the present application.

[0141] The device 70 includes: a management unit 701 configured to receive an online sharding reorganization request, determine a target shard, and create a migration task in the target shard; a control unit 702 configured to create a first coroutine and a second coroutine in the target shard for an incremental replication process in the migration task; the first coroutine and the second coroutine are asynchronously and concurrently executed; a first execution unit 703 configured to write binary log events in a source shard to a relay log in the target shard through the first coroutine; and a second execution unit 704 configured to write data in the relay log to the target shard through the second coroutine.

[0142] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be specifically implemented as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system".

[0143] The electronic device 800 according to this embodiment of the present application will be described below with reference to Figure 8 FIG. 8. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0144] As shown in Figure 8 FIG. 8, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 can include, but are not limited to, the at least one processing unit 810 described above, the at least one storage unit 820 described above, and a bus 830 connecting different system components, including the storage unit 820 and the processing unit 810.

[0145] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention.

[0146] Storage unit 820 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 8201 and / or cache memory 8202, and may further include a read-only memory (ROM) 8203.

[0147] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0148] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0149] Electronic device 800 can also communicate with one or more external devices 840 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0150] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) execute the methods according to the embodiments of the present disclosure.

[0151] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, which stores a program product capable of implementing the above-mentioned method of the present disclosure. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps according to various example embodiments of the present disclosure described in the above-mentioned “example method” section of the present disclosure when the program product is run on the terminal device.

[0152] A program product for implementing the above-mentioned method according to the embodiments of the present disclosure is described, which can adopt a portable compact disc read-only memory (CD-ROM) and includes program codes, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in the present document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or apparatus.

[0153] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0154] A computer readable signal medium can include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that can be involved in

[0155] The code can be transmitted in any coded or modular form, including via wireless, wire line, optical cable, RF, or any suitable combination of the foregoing.

[0156] The program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0157] It should be noted that, although several modules or units for a device for action execution are mentioned in the foregoing detailed description, such a division is not mandatory. Indeed, features and functionalities of two or more modules or units described above can be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, features and functionalities of one module or unit described above can be further divided into multiple modules or units.

[0158] Moreover, although the various steps of the methods of the present disclosure are described in a particular order in the figures, this is not required or implied, nor is it necessary to perform all of the steps shown in order to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into a single step, a single step can be broken into multiple steps, and / or the like.

[0159] Those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware, through the above description of the embodiments. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0160] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure that come within the scope of the following claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

Claims

1. An online fragment reassembly method, characterized by, The method comprises: receiving an online shard reorganization request, determining a target shard, and creating a migration task in the target shard; for an incremental replication process in the migration task, creating a first coroutine and a second coroutine in the target shard; the first coroutine and the second coroutine are asynchronously and concurrently executed; through the first coroutine, a binary log event in a source shard is written into a relay log of the target shard; through the second coroutine, data in the relay log is written into the target shard.

2. The method of claim 1, wherein, The online shard reorganization request comprises: a capacity expansion request and a capacity reduction request; when the online shard reorganization request is the capacity expansion request, the target shard is multiple; when the online shard reorganization request is the capacity reduction request, the source shard is multiple.

3. The method of claim 1, wherein, The method further comprises: when the data replication delay time is less than a first preset time period, the routing information of the read operation is modified from the source shard to the target shard; the data replication time is the current time, and the remaining time for incremental data replication from one source shard to one target shard.

4. The method according to claim 1 or 3, characterized in that, The method further comprises: stopping the execution of the write operation in the source shard; obtaining a first data site of the source shard and a second data site of the target shard; determining whether the second data site catches up with the first data site within a second preset time period; if yes, allowing the execution of the write operation in the target shard, and modifying the routing information of the write operation from the source shard to the target shard; if no, performing a rollback operation, rolling back to before the stop of the execution of the write operation in the source shard, and continuing to execute the write operation in the source shard.

5. The method of claim 1, wherein, The method further comprises: according to the task parameters of the online shard reorganization request, persistently writing the to-be-copied tables in the source shard into a replication state table in a meta database table; copying the to-be-copied tables in the meta database table to the target shard, and deleting the record of each to-be-copied table in the replication state table after copying one to-be-copied table.

6. The method of claim 1, wherein, The method further comprises: through the first coroutine, when one binary log event is written into the relay log from the source shard, the data in the relay log is written into the target shard through the second coroutine; based on a pre-set condition variable synchronization primitive, the first coroutine and the second coroutine are asynchronously and concurrently executed.

7. The method of claim 1, wherein, The method further comprises: receiving a shard reorganization completion request; cleaning up the source shard information; deleting the data of the target shard in the meta database.

8. An online fragment reassembly system, characterized by, It comprises: a management module, a controller, an executor, and a topology server; the management module receives an online shard reorganization request, determines a target shard, and creates a migration task in the target shard; the controller creates a first coroutine and a second coroutine in the target shard for an incremental replication process in the migration task; the first coroutine and the second coroutine are asynchronously and concurrently executed; the controller invokes the executor, and the executor writes a binary log event in a source shard into a relay log of the target shard through the first coroutine; the executor writes data in the relay log into the target shard through the second coroutine; The topology server is configured to store metadata of the source shard and the target shard.

9. An online slice reassembly apparatus, characterized by, The method comprises the following steps: a management unit configured to receive an online shard reorganization request, determine a target shard, and create a migration task in the target shard; a control unit configured to create a first coroutine and a second coroutine in the target shard for an incremental replication process in the migration task; the first coroutine and the second coroutine are asynchronously and concurrently executed; a first execution unit configured to write binary log events in the source shard to a relay log in the target shard through the first coroutine; a second execution unit configured to write data in the relay log to the target shard through the second coroutine.

10. An electronic device, comprising: The method comprises the following steps: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute the method according to any one of claims 1-7 by executing the executable instructions.

11. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the method according to any one of claims 1-7.

12. A computer program product, comprising: A computer program or instructions, wherein the computer program or instructions are executed by a processor to implement the method according to any one of claims 1-7.

Images