Data storage and expansion method, device and storage medium

Abstract

The present application discloses a data storage expansion method, device and storage medium. The data storage expansion method comprises: obtaining data sharding information corresponding to an original data storage end and an expanded data storage end, synchronizing the full amount of data stored in the original data storage end to the expanded data storage end, the expanded data storage end determining, based on the data sharding information, data in the expanded data storage end that are duplicated with data content to be stored in the original data storage end, obtaining first duplicate data, and terminating the life cycle of the first duplicate data; the original data storage end determining, based on the data sharding information, data in the original data storage end that are duplicated with data content to be stored in the expanded data storage end, obtaining second duplicate data, and terminating the life cycle of the second duplicate data. By deleting duplicate data in a manner of terminating the life cycle of duplicate data, data splitting is avoided, expansion operation is simplified, expansion time is shortened, and expansion efficiency of the data storage system is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data storage expansion method, device and storage medium. Background Art

[0002] In a distributed storage system, due to the continuous growth of data, in order to increase the capacity of the system, it is necessary to ensure that the system has good scalability, that is, a capacity expansion mechanism is needed.

[0003] Currently, commonly used expansion methods include: adding new data nodes, gathering the data of all existing data nodes in the system, splitting the gathered data, and distributing it to all data nodes (including new data nodes and existing data nodes); or, adding new data nodes, splitting the data on some of the existing data nodes in the system, and then distributing the split data to the new data nodes.

[0004] However, when the above method is used to expand the capacity of the data node cluster, the data rows need to be split, which is difficult to implement, the splitting process is time-consuming, and data loss or confusion is more likely to occur. Summary of the invention

[0005] In order to solve the above technical problems, the present application at least provides a data storage expansion method, device and storage medium.

[0006] In a first aspect, the present application provides a data storage expansion method, which is applied to an original data storage end, and the method comprises: in response to an expansion request, obtaining data sharding information corresponding to the original data storage end and the expanded data storage end; wherein the data sharding information is used to indicate data content that needs to be stored in the original data storage end and data content that needs to be stored in the expanded data storage end, which are obtained by dividing the data stored in the original data storage end; synchronizing the entire amount of data stored in the original data storage end to the expanded data storage end, so that the expanded data storage end determines, based on the data sharding information, data that is duplicated with the data content that needs to be stored in the original data storage end, obtains first duplicate data, ends the life cycle of the first duplicate data, and deletes the first duplicate data; and determines, based on the data sharding information, data that is duplicated with the data content that needs to be stored in the original data storage end, obtains second duplicate data, ends the life cycle of the second duplicate data, and deletes the second duplicate data.

[0007] In one embodiment, the original data storage end includes a master node and a slave node for data storage, and the slave node is used to back up the data of the master node; the data stored in the original data storage end is synchronized in full to the expanded data storage end, including: suspending the data backup service of the slave node in the original data storage end to back up the data of the master node; and copying the data stored in the slave node in the original data storage end to the expanded data storage end.

[0008] In one embodiment, the method also includes: in response to the suspension of the data backup service of the slave node in the original data storage end, the master node in the original data storage end generates a data change log based on the received data change operation; if it is detected that the full replication of the data of the slave node in the original data storage end is completed, the master node in the original data storage end sends the data change log to the slave node in the original data storage end for data change synchronization, and starts the data backup service of the slave node in the original data storage end; and, the master node in the original data storage end sends the data change log to the expanded data storage end for data change synchronization.

[0009] In one embodiment, the original data storage end uses the storage buckets in the object storage system to store data; obtains the data sharding information corresponding to the original data storage end and the expanded data storage end, including: obtaining the data information of each storage bucket in the original data storage end, the data information includes the data growth rate, and / or data growth amount, and / or total data amount of the storage bucket within a preset time period; based on the data information of each storage bucket, determines the storage buckets that need to be divided into the expanded data storage end and the storage buckets retained in the original data storage end, and obtains the data sharding information.

[0010] In one embodiment, the original data storage end uses the storage buckets in the object storage system to store data; obtains the data sharding information corresponding to the original data storage end and the expanded data storage end, including: counting the number of data change operations of each storage bucket in the original data storage end within a preset time period, and sorting each storage bucket in ascending order according to the number of data change operations of each storage bucket to obtain a sorting result; allocating a preset number of storage buckets with high rankings in the sorting result to the expanded data storage end, and retaining the remaining storage buckets to the original data storage end; based on the storage buckets allocated to the expanded data storage end, generating the data sharding information corresponding to the expanded data storage end, and, based on the storage buckets retained to the original data storage end, generating the data sharding information corresponding to the original data storage end.

[0011] In one embodiment, the method also includes: switching the access entry of the original data storage end to the data storage proxy service, wherein the data storage proxy service routes data operation requests matching the data content stored in the original data storage end to the original data storage end based on the data sharding information.

[0012] A second aspect of the present application provides a data storage expansion method, which is applied to an expanded data storage end, and the method includes: in response to an expansion request, obtaining data sharding information corresponding to the original data storage end and the expanded data storage end; wherein the data sharding information is used to indicate data content that needs to be stored in the original data storage end and data content that needs to be stored in the expanded data storage end obtained by dividing the data stored in the original data storage end; receiving all data stored in the synchronized original data storage end; wherein the original data storage end determines, based on the data sharding information, data in the original data storage end that is duplicated with the data content that needs to be stored in the expanded data storage end, obtains second duplicate data, ends the life cycle of the second duplicate data, and deletes the second duplicate data; and determines, based on the data sharding information, data in the expanded data storage end that is duplicated with the data content that needs to be stored in the original data storage end, obtains first duplicate data, ends the life cycle of the first duplicate data, and deletes the first duplicate data.

[0013] In one embodiment, the method also includes: switching the access entry of the expanded data storage end to the data storage proxy service, wherein the data storage proxy service routes data operation requests matching the data content stored in the expanded data storage end to the expanded data storage end based on the data sharding information.

[0014] According to a third aspect of the present application, there is provided a data storage expansion device deployed at an original data storage end, the device comprising: a first information acquisition module, for obtaining data sharding information corresponding to the original data storage end and the expanded data storage end in response to an expansion request; wherein the data sharding information is used to indicate data content to be stored in the original data storage end and data content to be stored in the expanded data storage end obtained by dividing the data stored in the original data storage end; a data synchronization module, for synchronizing the full amount of data stored in the original data storage end to the expanded data storage end, so that the expanded data storage end determines, based on the data sharding information, data that is duplicated with the data content to be stored in the original data storage end, obtains first duplicate data, ends the life cycle of the first duplicate data, and deletes the first duplicate data; a first data deletion module, for determining, based on the data sharding information, data that is duplicated with the data content to be stored in the original data storage end, obtains second duplicate data, ends the life cycle of the second duplicate data, and deletes the second duplicate data.

[0015] A fourth aspect of the present application provides a data storage expansion device deployed at an expanded data storage end, the device comprising: a second information acquisition module, for obtaining data sharding information corresponding to the original data storage end and the expanded data storage end in response to an expansion request; wherein the data sharding information is used to indicate data content to be stored in the original data storage end and data content to be stored in the expanded data storage end obtained by dividing the data stored in the original data storage end; a data receiving module, for receiving all data stored in the synchronized original data storage end; wherein the original data storage end determines, based on the data sharding information, data in the original data storage end that is duplicated with the data content to be stored in the expanded data storage end, obtains second duplicate data, ends the life cycle of the second duplicate data, and deletes the second duplicate data; a second data deletion module, for determining, based on the data sharding information, data in the expanded data storage end that is duplicated with the data content to be stored in the original data storage end, obtains first duplicate data, ends the life cycle of the first duplicate data, and deletes the first duplicate data.

[0016] A fifth aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned data storage expansion method.

[0017] A sixth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned data storage expansion method when the program instructions are executed by a processor.

[0018] The above scheme obtains the data content that needs to be stored in the original data storage end and the data content that needs to be stored in the expanded data storage end by acquiring the data sharding information corresponding to the original data storage end and the expanded data storage end, and synchronizes the data stored in the original data storage end to the expanded data storage end in full. The expanded data storage end determines the data in the expanded data storage end that is duplicated with the data content that needs to be stored in the original data storage end based on the data sharding information, obtains the first duplicate data, ends the life cycle of the first duplicate data, and deletes the first duplicate data; and the original data storage end determines the data in the original data storage end that is duplicated with the data content that needs to be stored in the expanded data storage end based on the data sharding information, obtains the second duplicate data, ends the life cycle of the second duplicate data, and deletes the second duplicate data. The duplicate data is directly deleted by ending the life cycle of the duplicate data, thereby avoiding data splitting, simplifying the expansion operation, shortening the expansion time, and improving the expansion efficiency of the data storage system.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0021] Figure 1 is a schematic diagram of a solution implementation environment shown in an exemplary embodiment of the present application;

[0022] Figure 2 is a flow chart of a data storage capacity expansion method shown in an exemplary embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an object storage system shown in an exemplary embodiment of the present application;

[0024] Figure 4 is a schematic diagram of data synchronization shown in an exemplary embodiment of the present application;

[0025] Figure 5 is a schematic diagram of a routing of a data operation request shown in an exemplary embodiment of the present application;

[0026] Figure 6 is a flow chart of a data storage capacity expansion method shown in another exemplary embodiment of the present application;

[0027] Figure 7 is a block diagram of a data storage expansion device deployed at an original data storage end shown in an exemplary embodiment of the present application;

[0028] Figure 8 is a block diagram of a data storage expansion device deployed at an expansion data storage end, shown in an exemplary embodiment of the present application;

[0029] Fig. 9 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;

[0030] Fig.10 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0031] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0032] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0033] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0034] The data storage capacity expansion method provided in the embodiment of the present application is described below.

[0035] Please refer to Figure 1 , Figure 1 is a schematic diagram of a solution implementation environment shown in an exemplary embodiment of the present application, such as Figure 1 As shown, multiple servers 110 form a server cluster, which can perform distributed storage of data. The terminal 120 communicates with the server cluster composed of servers 110 through the network to write or read data. One or more servers 110 in the server cluster correspond to the original data storage end and the expanded data storage end of the present application.

[0036] The terminal 120 includes but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices.

[0037] See also Figure 2 , Figure 2 is a flowchart of a data storage expansion method shown in an exemplary embodiment of the present application. The data storage expansion method can be applied to Figure 1 In the implementation environment shown, the data storage expansion method is specifically executed by the original data storage end, such as Figure 2 As shown, the data storage capacity expansion method at least includes steps S210 to S230, which are described in detail as follows:

[0038] Step S210: In response to the expansion request, obtain data sharding information corresponding to the original data storage end and the expanded data storage end; wherein the data sharding information is used to indicate the data content to be stored in the original data storage end and the data content to be stored in the expanded data storage end obtained by dividing the data stored in the original data storage end.

[0039] Among them, the capacity expansion request can be actively triggered by the user, such as the user sending a capacity expansion request to the cluster through the terminal; the capacity expansion request can also be automatically generated after detecting that the current data scale of the cluster has reached the upper limit. This application does not limit the generation method of the capacity expansion request.

[0040] Take the object storage system as an example for explanation. Figure 3 , Figure 3 A schematic diagram of an object storage system is shown as an exemplary embodiment of the present application. Figure 3 As shown in the figure, the object storage system consists of a metadata server (MetaDataServer, MDS) and a storage node pool. The storage node pool contains multiple data storage nodes. The metadata server is responsible for managing metadata and processing read and write requests. The metadata mainly includes object storage bucket information, object name information, and the location of data storage nodes where objects are distributed. Among them, the storage bucket is a container for organizing objects, and the data storage node is responsible for storing specific object data. The client is responsible for sending read and write requests to the object storage system to read and write metadata and object data.

[0041] For example, when a user uses a client to store object data, he first requests the metadata server to allocate capacity, persistent object name, object size, object distribution data storage node location and other metadata information, and returns a list of object distribution nodes to the client. The client requests the data storage node and writes the actual object data into the data storage node.

[0042] A master-slave architecture can be used in the metadata server to store metadata. When the master-slave architecture is used in scenarios with smaller data scale, the scale of metadata that can be stored is limited by the single-machine hardware resources used for metadata storage. As the business scale increases, the metadata scale reaches its upper limit. In order to continue to increase the cluster scale, it is necessary to expand the metadata server and expand the metadata service from the master-slave architecture to the metadata cluster mode. At this time, the original metadata server belongs to the original data storage end, and the newly added metadata server belongs to the expanded data storage end.

[0043] Optionally, one metadata service may be deployed corresponding to one physical server, that is, one physical server may correspond to one data storage terminal, so as to unify the cluster deployment architecture, facilitate subsequent cluster maintenance, and reduce cluster maintenance costs.

[0044] Of course, in addition to the data storage scenarios exemplified above, the present invention can also be applied to other data storage scenarios, which is not limited in the present application.

[0045] The data stored in the original data storage end is divided to obtain the data content that the original data storage end needs to store and the data content that the expanded data storage end needs to store, that is, the data sharding information corresponding to the original data storage end and the expanded data storage end is obtained.

[0046] Also take the object storage system as an example. If the metadata stored in the original data storage end includes bucket 1 and bucket 2, the data stored in the original data storage end is divided, and the data sharding information obtained is that the data content that the original data storage end needs to store is bucket 1, and the data content that the expanded data storage end needs to store is bucket 2.

[0047] Step S220: Synchronize all the data stored in the original data storage end to the expanded data storage end, so that the expanded data storage end can determine the data in the expanded data storage end that is duplicate of the data content that needs to be stored in the original data storage end based on the data sharding information, obtain the first duplicate data, end the life cycle of the first duplicate data, and delete the first duplicate data.

[0048] Directly synchronize the full amount of data stored in the original data storage end to the expanded data storage end, so that the synchronized original data storage end and the expanded data storage end both contain the same full amount of data.

[0049] Then, for the expanded data storage end, according to the data sharding information, the data in the expanded data storage end that duplicates the data content to be stored in the original data storage end is determined, the first duplicate data is obtained, and the life cycle of the first duplicate data is ended to delete the first duplicate data.

[0050] For example, if the metadata stored in the original data storage end includes bucket 1 and bucket 2, the data stored in the original data storage end is divided, and the data sharding information is obtained that the data content that the original data storage end needs to store is bucket 1, and the data content that the expanded data storage end needs to store is bucket 2, then the first duplicate data in the expanded data storage end is bucket 1, and the life cycle of the metadata in bucket 1 is ended to delete the metadata in bucket 1.

[0051] It should be noted that the number of original data storage terminals and expanded data storage terminals in this application can be one or more. For the original data storage terminal, it can be that after detecting that the current data size of the data storage terminal has reached the upper limit, the data storage terminal is used as the original data storage terminal; or the data storage terminal selected by the user is used as the original data storage terminal; for the expanded data storage terminal, it can be that the newly built data storage terminal is used as the expanded data storage terminal, or the data storage terminal whose current data size is less than the lower limit is used as the original data storage terminal. This application does not limit the number of original data storage terminals and expanded data storage terminals, and the determination method.

[0052] Step S230: Based on the data sharding information, determine the data in the original data storage end that duplicates the data content to be stored in the expanded data storage end, obtain the second duplicate data, end the life cycle of the second duplicate data, and delete the second duplicate data.

[0053] Similarly, for the original data storage end, based on the data sharding information, determine the data in the original data storage end that is duplicated with the data content that needs to be stored in the expanded data storage end, obtain the second duplicate data, end the life cycle of the second duplicate data, and delete the second duplicate data.

[0054] For example, if the metadata stored in the original data storage end includes bucket 1 and bucket 2, the data stored in the original data storage end is divided, and the data sharding information obtained is that the data content that the original data storage end needs to store is bucket 1, and the data content that the expanded data storage end needs to store is bucket 2. Then the second duplicate data in the original data storage end is bucket 2, and the life cycle of the metadata in bucket 2 is ended to delete the metadata in bucket 2.

[0055] Exemplarily, the end time of the life cycle of the first duplicate data in the expanded data storage end and the second duplicate data in the original data storage end is modified to the current time, so that expired data is cleaned up according to the life cycle management service, thereby deleting duplicate data.

[0056] Compared with the prior art method that requires data to be split before expansion, the present application first synchronizes all data in full, and then determines the data content that does not need to be stored based on the data content that needs to be stored divided by each data storage end, obtains duplicate data, and directly deletes duplicate data by ending the duplicate data life cycle. Since the data splitting is avoided, the expansion operation is simplified, the expansion time is shortened, and the expansion efficiency of the data storage system is improved.

[0057] Next, some embodiments of the present application are described in detail.

[0058] In some embodiments, the original data storage end includes a master node and a slave node for data storage, and the slave node is used to back up the data of the master node; in step S220, the full amount of data stored in the original data storage end is synchronized to the expanded data storage end, including: suspending the data backup service of the slave node in the original data storage end to back up the data of the master node; and copying the full amount of data stored in the slave node in the original data storage end to the expanded data storage end.

[0059] It should be noted that the data storage end can be implemented in a one-master-multiple-slave or one-master-one-slave manner, and the original data storage end and the expanded data storage end can adopt the same master-slave architecture, that is, the expanded data storage end also includes a master node and a slave node for data storage.

[0060] When storing data, the slave node is used to back up the data of the master node to improve the system's fault tolerance.

[0061] After receiving the expansion request, it is necessary to suspend the data backup service of the slave node in the original data storage end to back up the data of the master node in order to perform the data synchronization task. Specifically, all the data stored in the slave node in the original data storage end is copied to the expanded data storage end.

[0062] In some embodiments, in response to the suspension of the data backup service of the slave node in the original data storage end, the master node in the original data storage end generates a data change log based on the received data change operation; if it is detected that the full replication of the data of the slave node in the original data storage end is completed, the master node in the original data storage end sends the data change log to the slave node in the original data storage end for data change synchronization, and starts the data backup service of the slave node in the original data storage end; and, the master node in the original data storage end sends the data change log to the expanded data storage end for data change synchronization.

[0063] Because the master node in the original data storage end detects that the data backup service of the slave node is suspended, the master node automatically retains all data change operations during data synchronization and generates a data change log.

[0064] Then, after detecting that the full copy of the data from the slave node in the original data storage end is completed, the master node in the original data storage end sends the data change log to the slave node in the original data storage end for data change synchronization, and starts the data backup service of the slave node in the original data storage end to ensure the data consistency between the master node and the slave node; and the master node in the original data storage end sends the data change log to the expanded data storage end for data change synchronization to ensure the data consistency between the original data storage end and the expanded data storage end.

[0065] Take the object storage system as an example for explanation. Figure 4 , Figure 4 A schematic diagram of data synchronization is shown for an exemplary embodiment of the present application. Figure 4As shown, the original data storage end contains master node A1, slave node A2 and slave node A3, which correspond to storage bucket 1 and storage bucket 2. The expansion request indicates that there is currently a newly added expanded data storage end, and the expanded data storage end contains master node B1, slave node B2 and slave node B3. The synchronization steps between the original data storage end and the expanded data storage end include: suspending the data backup service of slave node A3; master node A1 detects that the data backup service of slave node A3 is suspended, records the received data change operation, and generates a data change log; after the data backup service of slave node A3 is suspended, the metadata of storage bucket 1 and storage bucket 2 of slave node A3 are fully copied to master node B1, and data backup services are performed on slave nodes B2 and B3; after the metadata of slave node A3 is fully copied, At this time, the data in slave node A3, master node B1, slave node B2 and slave node B3 are the same, and slave node A3 restarts the data backup service; master node A1 detects that the data backup service of slave node A3 is restarted, and sends the recorded data change log to slave node A3 and master node B1 respectively, so that slave node A3 and master node B1 synchronize data changes according to the data change log. After slave node B2 and slave node B3 detect that there is a data change in master node B1, they also change the data through the data backup service. Finally, master node A1, slave node A2, slave node A3, master node B1, slave node B2 and slave node B3 all contain the same data content.

[0066] If the data sharding information obtained is that the data content to be stored in the original data storage end is bucket 1, and the data content to be stored in the expanded data storage end is bucket 2, then the data in bucket 2 in the master node A1, slave node A2, and slave node A3 is redundant data, i.e., the second duplicate data, and the data in bucket 1 in the master node B1, slave node B2, and slave node B3 is redundant data, i.e., the first duplicate data, and the expiration time of the life cycle of bucket 2 is adjusted in the master node A1, slave node A2, and slave node A3 to the current time, or the time before the current time, so as to clean up the data in bucket 2 in the master node A1, slave node A2, and slave node A3 according to the life cycle management service; and, the expiration time of the life cycle of bucket 1 is adjusted in the master node B1, slave node B2, and slave node B3 to the current time, or the time before the current time, so as to clean up the data in bucket 1 in the master node B1, slave node B2, and slave node B3 according to the life cycle management service.

[0067] Finally, the master node A1, slave node A2, and slave node A3 in the original data storage end only contain the metadata of bucket 1, and the master node B1, slave node B2, and slave node B3 in the expanded data storage end only contain the metadata of bucket 2, completing the data storage expansion.

[0068] In some implementations, the access entry of the original data storage end is switched to the data storage proxy service, wherein the data storage proxy service routes data operation requests matching the data content stored in the original data storage end to the original data storage end based on the data sharding information. Also, the access entry of the expanded data storage end is switched to the data storage proxy service, wherein the data storage proxy service routes data operation requests matching the data content stored in the expanded data storage end to the expanded data storage end based on the data sharding information.

[0069] Since the data storage capacity is expanded, there are at least two data storage terminals, so a data storage proxy service is required to perform routing management of data operation requests.

[0070] For example, see Figure 5 , Figure 5 A schematic diagram of a routing of a data operation request is shown for an exemplary embodiment of the present application. Figure 5 As shown, the data storage proxy service routes the data operation request belonging to bucket 1 to the original data storage end, and routes the data operation request belonging to bucket 2 to the expanded data storage end.

[0071] It should be noted that Figure 4 and Figure 5 The following is an example using one original data storage terminal and one expanded data storage terminal. In actual application scenarios, there may be more original data storage terminals and expanded data storage terminals. The expansion method is similar and will not be described in detail here.

[0072] In some implementations, the data stored in the original data storage end may be evenly divided according to the number of existing data storage ends (including the original data storage end and the expanded data storage end) to obtain the data content that needs to be stored in the original data storage end and the data content that needs to be stored in the expanded data storage end; it may also be possible to flexibly determine the data content that needs to be stored in the original data storage end and the data content that needs to be stored in the expanded data storage end according to the specific attributes of the data stored in the original data storage end.

[0073] Take the example of using the bucket in the object storage system to store data on the original data storage end to illustrate:

[0074] Embodiment 1 obtains data information of each storage bucket in the original data storage end, the data information includes the data growth rate, and / or data growth amount, and / or total data amount of the storage bucket within a preset time period; based on the data information of each storage bucket, determines the storage bucket that needs to be divided to the expanded data storage end and the storage bucket that is retained to the original data storage end, and obtains data sharding information.

[0075] For example, according to the data growth rate of the bucket and the current total amount of data, the total amount of data in the bucket in the future time period is estimated, and based on the estimated total amount of data in each bucket and the number of existing data storage ends (including the original data storage end and the expanded data storage end), multiple buckets are divided into multiple groups, the number of groups is equal to the number of existing data storage ends, and the difference between the sum of the estimated total amount of data in the buckets of each group is less than a preset threshold. Then, each group of buckets is divided respectively to the existing data storage end, that is, respectively to the original data storage end and the expanded data storage end, to obtain the buckets corresponding to the original data storage end and the expanded data storage end, respectively.

[0076] For another example, a load score is performed on each storage bucket based on the data growth rate, and / or data growth amount, and / or total data amount of each storage bucket within a preset time period. The greater the data growth rate, and / or the data growth amount, and / or the total data amount, the higher the load score. With the goal that the sum of the load scores of all storage buckets of a single data storage end does not exceed a preset load threshold, multiple storage buckets are divided into multiple groups, and the number of groups is equal to the number of existing data storage ends. Then, each group of storage buckets is respectively divided to the existing data storage end, that is, respectively to the original data storage end and the expanded data storage end, to obtain the storage buckets corresponding to the original data storage end and the expanded data storage end, respectively.

[0077] Through the above embodiments, the load between each data storage end can be balanced, thereby improving the cluster stability.

[0078] Embodiment 2 counts the number of data change operations on each bucket in the original data storage end within a preset time period, and sorts each bucket in ascending order according to the number of data change operations on each bucket to obtain a sorting result; allocates a preset number of buckets with the top rankings in the sorting result to the expanded data storage end, and retains the remaining buckets to the original data storage end; based on the buckets allocated to the expanded data storage end, generates data sharding information corresponding to the expanded data storage end, and, based on the buckets retained to the original data storage end, generates data sharding information corresponding to the original data storage end.

[0079] Since it is necessary to record data change operations during the data synchronization process, the present application counts the number of data change operations in each bucket in the original data storage end within a preset time period when determining the storage buckets that need to be stored in each data storage end, and determines the storage buckets allocated to the original data storage end and the expanded data storage end based on the number of data change operations in each bucket.

[0080] Specifically, the storage buckets with a small number of data change operations are allocated to the expanded data storage end, and the storage buckets with a large number of data change operations are retained to the original data storage end, so as to reduce the number of data change operations that need to be synchronized later and improve the expansion efficiency.

[0081] For example, each bucket is sorted in ascending order according to the number of data change operations on each bucket to obtain a sorting result; a preset number of buckets with the highest ranking in the sorting result are allocated to the expanded data storage end, and the remaining buckets are retained to the original data storage end.

[0082] For another example, the storage buckets whose number of data change operations is less than a preset number threshold are allocated to the expanded data storage end, and the storage buckets whose number of data change operations is not less than the preset number threshold are retained to the original data storage end.

[0083] It should be noted that the execution entity of the above-mentioned division of the data stored in the original data storage end can be the original data storage end, or the expanded data storage end, or other computing ends, and this application does not limit this.

[0084] See also Figure 6 , Figure 6 is a flowchart of a data storage expansion method shown in another exemplary embodiment of the present application, and the data storage expansion method is specifically performed by the expansion data storage end, such as Figure 6 As shown, the data storage capacity expansion method at least includes steps S610 to S630, which are described in detail as follows:

[0085] Step S610: In response to the expansion request, obtain data sharding information corresponding to the original data storage end and the expanded data storage end; wherein the data sharding information is used to indicate the data content to be stored in the original data storage end and the data content to be stored in the expanded data storage end obtained by dividing the data stored in the original data storage end.

[0086] Step S620: Receive all data stored in the synchronized original data storage end; wherein, the original data storage end determines, based on the data sharding information, the data in the original data storage end that is duplicate of the data content that needs to be stored in the expanded data storage end, obtains the second duplicate data, ends the life cycle of the second duplicate data, and deletes the second duplicate data.

[0087] Step S630: based on the data sharding information, determine the data in the expanded data storage end that duplicates the data content to be stored in the original data storage end, obtain the first duplicate data, end the life cycle of the first duplicate data, and delete the first duplicate data.

[0088] The execution steps of the expanded data storage end are similar to those of the original data storage end. Please refer to the specific instructions of the relevant steps of the original data storage end for details. This application will not go into details here.

[0089] The data storage expansion method provided by the present application obtains data sharding information corresponding to the original data storage end and the expanded data storage end to obtain data content that needs to be stored in the original data storage end and data content that needs to be stored in the expanded data storage end, and synchronizes the data stored in the original data storage end to the expanded data storage end in full. The expanded data storage end determines, based on the data sharding information, data in the expanded data storage end that is duplicated with the data content that needs to be stored in the original data storage end, obtains first duplicate data, and ends the life cycle of the first duplicate data to delete the first duplicate data; and the original data storage end determines, based on the data sharding information, data in the original data storage end that is duplicated with the data content that needs to be stored in the expanded data storage end, obtains second duplicate data, and ends the life cycle of the second duplicate data to delete the second duplicate data. The duplicate data is directly deleted by ending the life cycle of the duplicate data, thereby avoiding data splitting, simplifying the expansion operation, shortening the expansion time, and improving the expansion efficiency of the data storage system.

[0090] Figure 7 is a block diagram of a data storage expansion device deployed at an original data storage end, as shown in an exemplary embodiment of the present application. Figure 7 As shown, the exemplary data storage expansion device 700 deployed at the original data storage end includes:

[0091] The first information acquisition module 710 is used to obtain data sharding information corresponding to the original data storage end and the expanded data storage end in response to the expansion request; wherein the data sharding information is used to indicate the data content to be stored in the original data storage end and the data content to be stored in the expanded data storage end obtained by dividing the data stored in the original data storage end;

[0092] The data synchronization module 720 is used to synchronize the full amount of data stored in the original data storage end to the expanded data storage end, so that the expanded data storage end determines the data in the expanded data storage end that is duplicated with the data content to be stored in the original data storage end based on the data sharding information, obtains the first duplicate data, ends the life cycle of the first duplicate data, and deletes the first duplicate data;

[0093] The first data deletion module 730 is used to determine the data in the original data storage end that is duplicated with the data content to be stored in the expanded data storage end based on the data sharding information, obtain the second duplicate data, end the life cycle of the second duplicate data, and delete the second duplicate data.

[0094] Figure 8is a block diagram of a data storage expansion device deployed at an expansion data storage end, as shown in an exemplary embodiment of the present application. Figure 8 As shown, the exemplary data storage expansion device 800 deployed at the expansion data storage end includes:

[0095] The second information acquisition module 810 is used to obtain data sharding information corresponding to the original data storage end and the expanded data storage end in response to the expansion request; wherein the data sharding information is used to indicate the data content to be stored in the original data storage end and the data content to be stored in the expanded data storage end obtained by dividing the data stored in the original data storage end;

[0096] The data receiving module 820 is used to receive all data stored in the synchronized original data storage end; wherein the original data storage end determines, based on the data sharding information, data in the original data storage end that is duplicated with the data content to be stored in the expanded data storage end, obtains second duplicate data, and ends the life cycle of the second duplicate data to delete the second duplicate data;

[0097] The second data deletion module 830 is used to determine the data in the expanded data storage end that is duplicated with the data content to be stored in the original data storage end based on the data sharding information, obtain the first duplicate data, end the life cycle of the first duplicate data, and delete the first duplicate data.

[0098] It should be noted that the data storage expansion device provided in the above embodiment and the data storage expansion method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In actual applications, the data storage expansion device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0099] See also Fig. 9 , Fig. 9 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 900 includes a memory 901 and a processor 902, and the processor 902 is used to execute program instructions stored in the memory 901 to implement the steps in any of the above data storage expansion method embodiments. In a specific implementation scenario, the electronic device 900 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 900 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0100] Specifically, the processor 902 is used to control itself and the memory 901 to implement the steps in any of the above-mentioned data storage expansion method embodiments. The processor 902 can also be called a central processing unit (CPU). The processor 902 may be an integrated circuit chip with signal processing capabilities. The processor 902 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 902 can be implemented by an integrated circuit chip.

[0101] See also Fig.10 , Fig.10 The computer-readable storage medium 1000 stores program instructions 1010 that can be executed by a processor, and the program instructions 1010 are used to implement the steps in any of the above data storage expansion method embodiments.

[0102] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0103] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0104] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0105] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

Claims

1. A data storage capacity expansion method, characterized in that: Applied to an original data storage end, the original data storage end uses a storage bucket in an object storage system to store data, and the method includes: In response to the expansion request, the number of data change operations of each storage bucket in the original data storage end within a preset time period is counted, and according to the number of data change operations of each storage bucket, the storage buckets allocated to the original data storage end and the expanded data storage end are determined, and data sharding information corresponding to the original data storage end and the expanded data storage end is obtained; wherein the data sharding information is used to indicate the data content to be stored in the original data storage end and the data content to be stored in the expanded data storage end obtained by dividing the data stored in the original data storage end; Synchronize the entire amount of data stored in the original data storage end to the expanded data storage end, so that the expanded data storage end determines, based on the data sharding information, data in the expanded data storage end that is duplicated with the data that needs to be stored in the original data storage end, obtains first duplicate data, and ends the life cycle of the first duplicate data to delete the first duplicate data; Based on the data sharding information, determine the data in the original data storage end that duplicates the data content that needs to be stored in the expanded data storage end, obtain second duplicate data, end the life cycle of the second duplicate data, and delete the second duplicate data.

2. The method according to claim 1, characterized in that The original data storage end includes a master node and a slave node for data storage, and the slave node is used to back up the data of the master node; and synchronizing the full amount of data stored in the original data storage end to the expanded data storage end includes: Suspending the data backup service of backing up the data of the master node from the slave node in the original data storage end; The data stored in the slave node in the original data storage end is copied in full to the expanded data storage end.

3. The method according to claim 2, characterized in that The method further comprises: In response to the suspension of the data backup service of the slave node in the original data storage end, the master node in the original data storage end generates a data change log based on the received data change operation; If it is detected that the data of the slave node in the original data storage end is fully copied, the master node in the original data storage end sends the data change log to the slave node in the original data storage end for data change synchronization, and starts the data backup service of the slave node in the original data storage end; and The master node in the original data storage end sends the data change log to the expanded data storage end for data change synchronization.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire data information of each storage bucket in the original data storage end, wherein the data information includes a data growth rate, a data growth amount, and / or a total amount of data in the storage bucket within a preset time period; Based on the data information of each storage bucket, the storage buckets that need to be divided into the expanded data storage end and the storage buckets that are retained in the original data storage end are determined to obtain data sharding information.

5. The method according to claim 1, characterized in that: The counting of the number of data change operations of each storage bucket in the original data storage end within a preset time period, determining the storage buckets to which the original data storage end and the expanded data storage end are allocated according to the number of data change operations of each storage bucket, and obtaining data sharding information corresponding to the original data storage end and the expanded data storage end, includes: Counting the number of data change operations of each storage bucket in the original data storage end within a preset time period, and sorting each storage bucket in ascending order according to the number of data change operations of each storage bucket to obtain a sorting result; Allocate a preset number of storage buckets that are ranked high in the sorting results to the expanded data storage end, and retain the remaining storage buckets to the original data storage end; Based on the storage buckets allocated to the expanded data storage end, data sharding information corresponding to the expanded data storage end is generated; and based on the storage buckets reserved to the original data storage end, data sharding information corresponding to the original data storage end is generated.

6. The method according to claim 1, characterized in that The method further comprises: The access entry of the original data storage end is switched to the data storage proxy service, wherein the data storage proxy service routes data operation requests matching the data content stored in the original data storage end to the original data storage end based on the data sharding information.

7. A data storage capacity expansion method, characterized in that: Applied to expanding the data storage end, the method includes: In response to the expansion request, data sharding information corresponding to the original data storage end and the expanded data storage end is obtained; wherein the original data storage end uses the storage buckets in the object storage system to store data, the original data storage end counts the number of data change operations of each storage bucket in the original data storage end within a preset time period, and according to the number of data change operations of each storage bucket, determines the storage buckets allocated to the original data storage end and the expanded data storage end, and obtains the data sharding information corresponding to the original data storage end and the expanded data storage end, wherein the data sharding information is used to indicate the data content to be stored in the original data storage end and the data content to be stored in the expanded data storage end obtained by dividing the data stored in the original data storage end; Receive all data stored in the synchronized original data storage end; wherein, the original data storage end determines, based on the data sharding information, data in the original data storage end that is duplicated with the data content to be stored in the expanded data storage end, obtains second duplicate data, and ends the life cycle of the second duplicate data to delete the second duplicate data; Based on the data sharding information, determine the data in the expanded data storage end that duplicates the data content that needs to be stored in the original data storage end, obtain first duplicate data, end the life cycle of the first duplicate data, and delete the first duplicate data.

8. The method according to claim 7, characterized in that The method further comprises: The access entry of the expanded data storage end is switched to the data storage proxy service, wherein the data storage proxy service routes data operation requests matching the data content stored in the expanded data storage end to the expanded data storage end based on the data sharding information.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 8.

Images