Object storage system expansion method, device, system and computer equipment

Abstract

The embodiment of the present application relates to the technical field of data storage, and discloses an object storage system expansion method, which comprises the following steps: determining whether expansion is needed according to the current capacity of an object storage system; when expansion is needed, deploying one or more expansion data clusters for the data subsystem and / or the metadata subsystem; and registering the cluster information of the expansion data cluster to an access gateway layer and saving the cluster information into the metadata subsystem for calling by a data center. In the above manner, the embodiment of the present application realizes expansion of the object storage system in the form of independent clusters, and controls the size of a single cluster.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data storage technology, specifically to an object storage system expansion method, apparatus, object storage system, computer equipment, and computer-readable storage medium. Background Technology

[0002] Object storage systems are distributed storage systems that use objects as storage units. They are used to store unstructured data and are widely used in public cloud scenarios. In public cloud environments, challenges often arise such as multi-tenancy, cross-regional operations, uneven capacity demands, and the need for elastic resource scaling. Supporting massive user engagement with object storage requires that its capacity be horizontally scalable.

[0003] Existing object storage systems typically scale up by adding nodes to the cluster. In a single-bucket scaling solution, objects within a bucket are distributed across different underlying object storage clusters. When performing bucket-level operations, multiple underlying object storage clusters need to be accessed simultaneously. For example, executing a bucket list requires accessing all underlying storage clusters. Furthermore, if one underlying storage cluster becomes inaccessible, it may affect the functionality of some object storage buckets. The centralized metadata center can also become a performance bottleneck, and scaling it results in a large single cluster size, high overhead for heartbeat detection between nodes, and data imbalance. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide an object storage system expansion method, apparatus, object storage system, computer device and computer-readable storage medium to solve the problems of large single cluster size, large heartbeat detection overhead between nodes in the cluster and data imbalance that exist in the prior art when expansion is carried out.

[0005] According to one aspect of the present invention, a method for expanding an object storage system is provided, the method comprising:

[0006] Determine whether expansion is needed based on the current capacity of the object storage system;

[0007] When expansion is required, one or more expansion data clusters shall be deployed for the data subsystem and / or the metadata subsystem.

[0008] The cluster information of the expanded data cluster is registered to the access gateway layer and saved in the metadata subsystem for use by the data center.

[0009] In one alternative approach, when expansion is required, deploying one or more expanded data clusters for the data subsystem and / or the metadata subsystem includes:

[0010] When it is determined that the data subsystem needs to be expanded, the number of data centers is determined;

[0011] The number of data clusters to be expanded is determined based on the number of data centers.

[0012] Deploy the aforementioned number of expanded data clusters;

[0013] The step of registering the cluster information of the expanded data cluster to the access gateway layer and saving it to the metadata subsystem for data center access includes:

[0014] The cluster information of the expanded data cluster is sent to the access gateway layer so that the access gateway layer can register the expanded data cluster based on the cluster information.

[0015] The access gateway layer saves the cluster information of the expanded data cluster to the metadata subsystem.

[0016] In one alternative approach, deploying the number of expanded data clusters includes:

[0017] Create storage pools for storage in each of the aforementioned expanded data clusters;

[0018] After saving the cluster information of the expanded data cluster to the metadata subsystem through the access gateway layer, the method further includes:

[0019] When an existing bucket needs to use the expanded data cluster, the information of the storage pool in the expanded data cluster is set as the redundancy type parameter of the bucket, so that the newly created bucket can call the expanded data cluster.

[0020] In one alternative approach, when expansion is required, deploying one or more expanded data clusters for the data subsystem and / or the metadata subsystem includes:

[0021] When it is determined that the metadata subsystem needs to be expanded, one or more metadata clusters are deployed as the expansion data clusters; the preset size of the metadata cluster is the sum of the sizes of multiple data centers;

[0022] The step of registering the cluster information of the expanded data cluster to the access gateway layer and saving it to the metadata subsystem for data center access includes:

[0023] The access domain name of the metadata cluster is configured in the global resource information of the access gateway layer and persisted to the default metadata cluster for use by various data centers.

[0024] In one alternative approach, the default metadata cluster is used to store persistent global resource information for access to the access gateway layer; the metadata cluster is used to store object index information.

[0025] In an alternative approach, the method further includes:

[0026] Obtain the remaining capacity information of each of the data subsystems and the metadata subsystems;

[0027] Based on the remaining capacity information, capacity ratings are performed on each of the data subsystems and each of the metadata subsystems to obtain rating information;

[0028] The data subsystem and the metadata subsystem are scheduled according to the rating information.

[0029] According to another aspect of the present invention, an object storage system expansion device is provided, the device comprising:

[0030] The determination module is used to determine whether expansion is needed based on the current capacity of the object storage system;

[0031] The deployment module is used to deploy one or more expanded data clusters for the data subsystem and / or the metadata subsystem when expansion is required;

[0032] The registration module is used to register the cluster information of the expanded data cluster to the access gateway layer and save it to the metadata subsystem for the data center to access.

[0033] According to another aspect of the present invention, an object storage system is provided, including a data subsystem for storing data of the object and a metadata subsystem for storing metadata of the object, and further including the object storage system expansion device.

[0034] According to another aspect of the present invention, a computer device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0035] The memory is used to store at least one executable instruction that causes the processor to perform the operation of the object storage system expansion method.

[0036] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the operation of the object storage system expansion method.

[0037] This invention determines whether expansion is needed based on the current capacity of the object storage system. When expansion is required, one or more expansion data clusters are deployed to the data subsystem and / or the metadata subsystem. The cluster information of the expansion data clusters is saved to the metadata subsystem for data center access. This allows for expansion of the object storage system as independent clusters, controlling the size of individual clusters and minimizing the blast radius. Furthermore, each expansion is performed with a fixed-size cluster, eliminating the need for data balancing during expansion.

[0038] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0039] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0040] Figure 1 A schematic diagram of the object storage system structure provided in an embodiment of the present invention is shown;

[0041] Figure 2 A flowchart illustrating the object storage system expansion method provided in an embodiment of the present invention is shown;

[0042] Figure 3 This diagram illustrates the expansion of an object storage system provided in an embodiment of the present invention.

[0043] Figure 4 A schematic diagram of the middle bucket in the object storage system expansion method provided in an embodiment of the present invention is shown;

[0044] Figure 5 A schematic diagram of the structure of the object storage system expansion device provided in an embodiment of the present invention is shown;

[0045] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0046] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0047] First, the names or terms in the embodiments of this invention will be explained accordingly:

[0048] In an object storage system, the system that stores object data is called the data subsystem, and the data subsystem expands the data cluster in the form of DataSets.

[0049] The data cluster corresponding to the data subsystem refers to a cluster based on Ceph's native architecture, primarily a RADOS cluster, which can provide storage services through librados.

[0050] Expanding in the form of a DataSet refers to deploying a separate data cluster.

[0051] First, the application environment (object storage system) of this embodiment of the invention will be introduced. The object storage system of this embodiment includes multiple buckets for storing objects; each bucket corresponds to a data subsystem for storing data of the objects and a metadata subsystem for storing metadata of the objects. For example... Figure 1 As shown, specifically, the object storage system of this embodiment includes multiple data centers, a load balancing scheduling layer, multiple access gateway layers, a metadata subsystem, and a data subsystem. From another perspective, the object storage system includes a data plane and a control plane. The data plane includes a load balancing scheduling layer, an access gateway layer, and a data layer; the control plane mainly includes an automatic cluster management module, an observability module, an inspection module, and a resource orchestration module. For object storage system expansion, the core is to expand the data layer. The data layer is divided into a data subsystem and a metadata subsystem, which are independent of each other and implemented differently. The data subsystem is implemented using Ceph's native RADOS, and the metadata subsystem is implemented using the Scylladb database. The data subsystem and metadata subsystem register with the access gateway layer and save their configurations in the metadata subsystem, which is then scheduled and used by the access gateway layer. The data centers transmit data with their respective data subsystems and metadata subsystems through the access gateways corresponding to the access gateway layers.

[0052] Figure 2 A flowchart of an object storage system expansion method provided in an embodiment of the present invention is shown. This method is executed by a computer device. The computer device can be a server device, a distributed device, a cloud storage device, etc., and can specifically be applied to the aforementioned object storage system; this embodiment of the present invention does not impose specific limitations. Figure 2 As shown, the method includes the following steps:

[0053] Step 110: Determine whether expansion is needed based on the current capacity of the object storage system.

[0054] The data subsystem can be a separate subsystem comprising multiple data clusters (DataSet), and the metadata subsystem comprises multiple metadata clusters (MetaSet). Each data center corresponds to one or more data clusters (DataSet) within the data subsystem. The metadata subsystem is deployed across data centers, therefore multiple data centers correspond to multiple metadata clusters (MetaSet). Therefore, when determining whether to expand the object storage system based on its current capacity, for the data subsystem, expansion is determined based on the current capacity of the data clusters corresponding to each data center. For the metadata subsystem, expansion is determined based on the current capacity of each metadata cluster within the metadata subsystem itself.

[0055] Step 120: When expansion is required, deploy one or more expanded data clusters for the data subsystem and / or the metadata subsystem.

[0056] When it is determined that the data subsystem needs to be expanded, the number of data centers is determined, and the number of expansion data clusters is determined based on the number of data centers; the number of expansion data clusters is then deployed. The data subsystem is expanded using fixed-size expansion data clusters. Since each data center in the data subsystem is deployed independently, this fixed size can be the size corresponding to one data center. Specifically, as... Figure 3 As shown, expanding the data subsystem first requires deploying an expanded data storage cluster. Since public cloud object storage typically uses a cross-datacenter deployment approach, it's necessary to deploy one expanded data storage cluster (DataSet) in each of multiple data centers simultaneously. For example... Figure 3 As shown, DataSet4, DataSet5, and DataSet6 are deployed in three data centers respectively. Storage pools are created in each of the expanded data clusters for storage. Within each DataSet, several storage pools are created to provide storage services; by default, one node is created as one pool.

[0057] When it is determined that the metadata subsystem needs to be expanded, one or more metadata clusters are deployed as the expansion data clusters. The access domain names of the metadata clusters are configured in the global resource information of the access gateway layer and persisted to the default metadata cluster for use by various data centers. Since the metadata subsystem is deployed across data centers, the size of each expansion data cluster is the sum of the sizes of all data centers when expanding the metadata subsystem. Specifically, expanding the metadata subsystem requires deploying new cross-DC (data center) metadata clusters. Figure 3 As shown, the newly expanded metadata cluster is MetaSet2.

[0058] Step 130: Register the cluster information of the expanded data cluster to the access gateway layer and save it to the metadata subsystem for the data center to access.

[0059] After the data subsystem expansion is completed, the cluster information of the expanded data cluster is sent to the access gateway layer. This allows the access gateway layer to register the expanded data cluster based on the cluster information and save the cluster information to the metadata subsystem. After deployment, the newly deployed data cluster is registered to the access gateway layer. Each expanded data cluster (DataSet) is a Ceph cluster, and each DataSet has at least one MON node, typically three. Therefore, information for three MON nodes (mon_host) needs to be provided. This information is then configured in the access gateway layer's global resource information (regionset). After configuration, the access gateway layer saves the global resource information (regionset) to the default metadata cluster (default_MetaSet). Once the cluster information of the expanded data storage cluster in the data subsystem is registered, the data center can use the new storage space of the expanded data storage cluster. Figure 4 As shown, when an existing bucket needs to use the expanded data cluster, the information of the storage pool in the expanded data cluster is set as the bucket's redundancy type parameter. This allows the existing bucket to call the expanded data cluster to write newly added objects from the existing bucket into the expanded data cluster. For example, for the existing bucket `bucket1`, modifying the storage pool `pool` used in the existing bucket's `redundancytype` allows selection of the pool created in the new expanded data cluster `DataSet`. For newly created buckets, a new redundancy type parameter `redundancytype` can be directly specified, specifying the storage pool `pool` in the expanded data cluster `DataSet` configured in the new redundancy type parameter `redundancytype`. The metadata subsystem includes a default metadata cluster `default_MetaSet`, which stores persistent global resource information for access by the access gateway layer. In an object storage system, there is only one default metadata cluster (default_MetaSet), which stores two parts of information. The first part is user information, bucket information, and global resource configuration information, which is used to persist global resource information for access by the access gateway layer. The second part is the object index information, which is still stored together with the default metadata cluster in the smallest scale.

[0060] After the metadata subsystem is expanded, the access domain name of the metadata cluster is configured in the global resource information of the access gateway layer and persisted to the default metadata cluster for use by various data centers. Since the metadata cluster is deployed across 3DCs, to provide proximity access within each DC, the access gateway layer needs the newly expanded metadata cluster MetaSet2 to provide an access domain name endpoint in each DC. After obtaining all the access domain name endpoints, the access gateway layer configures them into the indexset (non-empty index set) parameter under each data center in the global resource information regionset of the access gateway layer, enabling each data center to access the newly expanded metadata cluster MetaSet2 through its corresponding access domain name. For the metadata subsystem, after registration, the storage space of the newly expanded metadata cluster MetaSet2 can be used. Specifically, when creating bucket2, specifying the bucket's indexset attribute and selecting metaset2_endpoint allows the index metadata of objects uploaded to bucket2 to be stored in the MetaSet2 cluster.

[0061] The object storage system was expanded using the above methods. After expansion, this embodiment of the invention further obtains the remaining capacity information of each data subsystem and each metadata subsystem; based on the remaining capacity information, each data subsystem and each metadata subsystem is rated to obtain rating information; and the data subsystems and metadata subsystems are scheduled according to the rating information. The rating information is determined based on the remaining capacity of each data cluster and metadata cluster. Data clusters and metadata clusters with higher remaining capacity are prioritized for scheduling.

[0062] This invention determines whether expansion is needed based on the current capacity of the object storage system. When expansion is required, one or more expansion data clusters are deployed to the data subsystem and / or the metadata subsystem. The cluster information of the expansion data clusters is saved to the metadata subsystem for data center access. This allows for expansion of the object storage system as independent clusters, controlling the size of individual clusters and minimizing the blast radius. Furthermore, each expansion is performed with a fixed-size cluster, eliminating the need for data balancing during expansion.

[0063] Figure 5 A schematic diagram of the structure of the object storage system expansion device provided in an embodiment of the present invention is shown.

[0064] like Figure 5 As shown, the device 300 includes:

[0065] The determination module 310 is used to determine whether expansion is needed based on the current capacity of the object storage system;

[0066] Deployment module 320 is used to deploy one or more expanded data clusters to the data subsystem and / or the metadata subsystem when expansion is required;

[0067] The registration module 330 is used to register the cluster information of the expanded data cluster to the access gateway layer and save it to the metadata subsystem for the data center to call.

[0068] In one alternative approach, when expansion is required, deploying one or more expanded data clusters for the data subsystem and / or the metadata subsystem includes:

[0069] When it is determined that the data subsystem needs to be expanded, the number of data centers is determined;

[0070] The number of data clusters to be expanded is determined based on the number of data centers.

[0071] Deploy the aforementioned number of expanded data clusters;

[0072] The step of registering the cluster information of the expanded data cluster to the access gateway layer and saving it to the metadata subsystem for data center access includes:

[0073] The cluster information of the expanded data cluster is sent to the access gateway layer so that the access gateway layer can register the expanded data cluster based on the cluster information.

[0074] The access gateway layer saves the cluster information of the expanded data cluster to the metadata subsystem.

[0075] In one alternative approach, deploying the number of expanded data clusters includes:

[0076] Create storage pools for storage in each of the aforementioned expanded data clusters;

[0077] After saving the cluster information of the expanded data cluster to the metadata subsystem through the access gateway layer, the method further includes:

[0078] When an existing bucket needs to use the expanded data cluster, the information of the storage pool in the expanded data cluster is set as the redundancy type parameter of the bucket, so that the newly created bucket can call the expanded data cluster.

[0079] In one alternative approach, when expansion is required, deploying one or more expanded data clusters for the data subsystem and / or the metadata subsystem includes:

[0080] When it is determined that the metadata subsystem needs to be expanded, one or more metadata clusters are deployed as the expansion data clusters; the preset size of the metadata cluster is the sum of the sizes of multiple data centers;

[0081] The step of registering the cluster information of the expanded data cluster to the access gateway layer and saving it to the metadata subsystem for data center access includes:

[0082] The access domain name of the metadata cluster is configured in the global resource information of the access gateway layer and persisted to the default metadata cluster for use by various data centers.

[0083] In one alternative approach, the default metadata cluster is used to store persistent global resource information for access to the access gateway layer; the metadata cluster is used to store object index information.

[0084] In an alternative approach, the method further includes:

[0085] Obtain the remaining capacity information of each of the data subsystems and the metadata subsystems;

[0086] Based on the remaining capacity information, capacity ratings are performed on each of the data subsystems and each of the metadata subsystems to obtain rating information;

[0087] The data subsystem and the metadata subsystem are scheduled according to the rating information.

[0088] The working process of the object storage system expansion device in this embodiment of the invention is largely the same as the specific steps of the above method embodiment, and will not be repeated here.

[0089] This invention determines whether expansion is needed based on the current capacity of the object storage system. When expansion is required, one or more expansion data clusters are deployed to the data subsystem and / or the metadata subsystem. The cluster information of the expansion data clusters is saved to the metadata subsystem for data center access. This allows for expansion of the object storage system as independent clusters, controlling the size of individual clusters and minimizing the blast radius. Furthermore, each expansion is performed with a fixed-size cluster, eliminating the need for data balancing during expansion.

[0090] This invention also provides an object storage system, including a data subsystem for storing data of the object and a metadata subsystem for storing metadata of the object, and further including the object storage system expansion device described above.

[0091] Figure 6The diagram shows a structural schematic of a computer device provided in an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computer device.

[0092] like Figure 6 As shown, the computer device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0093] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other network elements such as clients or other servers. The processor 402 executes program 410, specifically performing the relevant steps described in the embodiment of the object storage system expansion method.

[0094] Specifically, program 410 may include program code, which includes computer-executable instructions.

[0095] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0096] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0097] Specifically, program 410 can be called by processor 402 to cause the computer device to perform the following operations:

[0098] Determine whether expansion is needed based on the current capacity of the object storage system;

[0099] When expansion is required, one or more expansion data clusters shall be deployed for the data subsystem and / or the metadata subsystem.

[0100] The cluster information of the expanded data cluster is registered to the access gateway layer and saved in the metadata subsystem for use by the data center.

[0101] In one alternative approach, when expansion is required, deploying one or more expanded data clusters for the data subsystem and / or the metadata subsystem includes:

[0102] When it is determined that the data subsystem needs to be expanded, the number of data centers is determined;

[0103] The number of data clusters to be expanded is determined based on the number of data centers.

[0104] Deploy the aforementioned number of expanded data clusters;

[0105] The step of registering the cluster information of the expanded data cluster to the access gateway layer and saving it to the metadata subsystem for data center access includes:

[0106] The cluster information of the expanded data cluster is sent to the access gateway layer so that the access gateway layer can register the expanded data cluster based on the cluster information.

[0107] The access gateway layer saves the cluster information of the expanded data cluster to the metadata subsystem.

[0108] In one alternative approach, deploying the number of expanded data clusters includes:

[0109] Create storage pools for storage in each of the aforementioned expanded data clusters;

[0110] After saving the cluster information of the expanded data cluster to the metadata subsystem through the access gateway layer, the method further includes:

[0111] When an existing bucket needs to use the expanded data cluster, the information of the storage pool in the expanded data cluster is set as the redundancy type parameter of the bucket, so that the newly created bucket can call the expanded data cluster.

[0112] In one alternative approach, when expansion is required, deploying one or more expanded data clusters for the data subsystem and / or the metadata subsystem includes:

[0113] When it is determined that the metadata subsystem needs to be expanded, one or more metadata clusters are deployed as the expansion data clusters; the preset size of the metadata cluster is the sum of the sizes of multiple data centers;

[0114] The step of registering the cluster information of the expanded data cluster to the access gateway layer and saving it to the metadata subsystem for data center access includes:

[0115] The access domain name of the metadata cluster is configured in the global resource information of the access gateway layer and persisted to the default metadata cluster for use by various data centers.

[0116] In one alternative approach, the default metadata cluster is used to store persistent global resource information for access to the access gateway layer; the metadata cluster is used to store object index information.

[0117] In an alternative approach, the method further includes:

[0118] Obtain the remaining capacity information of each of the data subsystems and the metadata subsystems;

[0119] Based on the remaining capacity information, capacity ratings are performed on each of the data subsystems and each of the metadata subsystems to obtain rating information;

[0120] The data subsystem and the metadata subsystem are scheduled according to the rating information.

[0121] This invention determines whether expansion is needed based on the current capacity of the object storage system. When expansion is required, one or more expansion data clusters are deployed to the data subsystem and / or the metadata subsystem. The cluster information of the expansion data clusters is saved to the metadata subsystem for data center access. This allows for expansion of the object storage system as independent clusters, controlling the size of individual clusters and minimizing the blast radius. Furthermore, each expansion is performed with a fixed-size cluster, eliminating the need for data balancing during expansion.

[0122] This invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on a computer device, causes the computer device to perform the object storage system expansion method described in any of the above method embodiments.

[0123] Executable instructions can be used to cause a computer device to perform the following operations:

[0124] Determine whether expansion is needed based on the current capacity of the object storage system;

[0125] When expansion is required, one or more expansion data clusters shall be deployed for the data subsystem and / or the metadata subsystem.

[0126] The cluster information of the expanded data cluster is registered to the access gateway layer and saved in the metadata subsystem for use by the data center.

[0127] In one alternative approach, when expansion is required, deploying one or more expanded data clusters for the data subsystem and / or the metadata subsystem includes:

[0128] When it is determined that the data subsystem needs to be expanded, the number of data centers is determined;

[0129] The number of data clusters to be expanded is determined based on the number of data centers.

[0130] Deploy the aforementioned number of expanded data clusters;

[0131] The step of registering the cluster information of the expanded data cluster to the access gateway layer and saving it to the metadata subsystem for data center access includes:

[0132] The cluster information of the expanded data cluster is sent to the access gateway layer so that the access gateway layer can register the expanded data cluster based on the cluster information.

[0133] The access gateway layer saves the cluster information of the expanded data cluster to the metadata subsystem.

[0134] In one alternative approach, deploying the number of expanded data clusters includes:

[0135] Create storage pools for storage in each of the aforementioned expanded data clusters;

[0136] After saving the cluster information of the expanded data cluster to the metadata subsystem through the access gateway layer, the method further includes:

[0137] When an existing bucket needs to use the expanded data cluster, the information of the storage pool in the expanded data cluster is set as the redundancy type parameter of the bucket, so that the newly created bucket can call the expanded data cluster.

[0138] In one alternative approach, when expansion is required, deploying one or more expanded data clusters for the data subsystem and / or the metadata subsystem includes:

[0139] When it is determined that the metadata subsystem needs to be expanded, one or more metadata clusters are deployed as the expansion data clusters; the preset size of the metadata cluster is the sum of the sizes of multiple data centers;

[0140] The step of registering the cluster information of the expanded data cluster to the access gateway layer and saving it to the metadata subsystem for data center access includes:

[0141] The access domain name of the metadata cluster is configured in the global resource information of the access gateway layer and persisted to the default metadata cluster for use by various data centers.

[0142] In one alternative approach, the default metadata cluster is used to store persistent global resource information for access to the access gateway layer; the metadata cluster is used to store object index information.

[0143] In an alternative approach, the method further includes:

[0144] Obtain the remaining capacity information of each of the data subsystems and the metadata subsystems;

[0145] Based on the remaining capacity information, capacity ratings are performed on each of the data subsystems and each of the metadata subsystems to obtain rating information;

[0146] The data subsystem and the metadata subsystem are scheduled according to the rating information.

[0147] This invention determines whether expansion is needed based on the current capacity of the object storage system. When expansion is required, one or more expansion data clusters are deployed to the data subsystem and / or the metadata subsystem. The cluster information of the expansion data clusters is saved to the metadata subsystem for data center access. This allows for expansion of the object storage system as independent clusters, controlling the size of individual clusters and minimizing the blast radius. Furthermore, each expansion is performed with a fixed-size cluster, eliminating the need for data balancing during expansion.

[0148] This invention provides an object storage system expansion device for performing the above-described object storage system expansion method.

[0149] This invention provides a computer program that can be invoked by a processor to cause a computer device to execute the object storage system expansion method in any of the above method embodiments.

[0150] This invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed on a computer, cause the computer to perform the object storage system expansion method described in any of the above method embodiments.

[0151] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0152] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0153] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0154] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0155] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for expanding the capacity of an object storage system, characterized in that, The method includes: Determine whether expansion is needed based on the current capacity of the object storage system; When expansion is required, one or more expansion data clusters are deployed as independent clusters for the data subsystem and metadata subsystem. The data subsystem is expanded using fixed-size expansion data clusters, while the metadata subsystem is deployed across data centers. When expanding the metadata subsystem, the size of each expansion data cluster is the sum of the sizes of the multiple data centers. Specifically, when it is determined that expansion of the data subsystem is needed, the number of data centers is determined, and the number of expansion data clusters is determined based on the number of data centers, and the specified number of expansion data clusters are deployed. When it is determined that expansion of the metadata subsystem is needed, one or more metadata clusters are deployed as expansion data clusters. The cluster information of the expanded data cluster is registered to the access gateway layer and saved in the metadata subsystem for the data center to access. The expansion is carried out in the form of an independent cluster, and data balancing is not required.

2. The method according to claim 1, characterized in that, The step of registering the cluster information of the expanded data cluster to the access gateway layer and saving it to the metadata subsystem for data center access includes: When it is determined that the data subsystem needs to be expanded, the cluster information of the expanded data cluster is sent to the access gateway layer so that the access gateway layer can register the expanded data cluster based on the cluster information. The access gateway layer saves the cluster information of the expanded data cluster to the metadata subsystem.

3. The method according to claim 2, characterized in that, The deployment of the aforementioned number of expanded data clusters includes: Create storage pools for storage in each of the aforementioned expanded data clusters; After saving the cluster information of the expanded data cluster to the metadata subsystem through the access gateway layer, the method further includes: When an existing bucket needs to use the expanded data cluster, the information of the storage pool in the expanded data cluster is set as the redundancy type parameter of the bucket, so that the newly created bucket can call the expanded data cluster.

4. The method according to claim 1, characterized in that, The step of registering the cluster information of the expanded data cluster to the access gateway layer and saving it to the metadata subsystem for data center access includes: When it is determined that the metadata subsystem needs to be expanded, the access domain name of the metadata cluster is configured in the global resource information of the access gateway layer and persisted to the default metadata cluster for use by various data centers.

5. The method according to claim 4, characterized in that, The default metadata cluster is used to store persistent global resource information for access gateway layer; the metadata cluster is used to store object index information.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Obtain the remaining capacity information of each of the data subsystems and the metadata subsystems; Based on the remaining capacity information, capacity ratings are performed on each of the data subsystems and each of the metadata subsystems to obtain rating information; The data subsystem and the metadata subsystem are scheduled according to the rating information.

7. An object storage system expansion device, characterized in that, The device includes: The determination module is used to determine whether expansion is needed based on the current capacity of the object storage system; The deployment module is used to deploy one or more expansion data clusters as independent clusters for the data subsystem and metadata subsystem when expansion is required. The data subsystem is expanded using fixed-size expansion data clusters, while the metadata subsystem is deployed across data centers. When expanding the metadata subsystem, the size of each expansion data cluster is the sum of the sizes of the multiple data centers. Specifically, when it is determined that expansion of the data subsystem is needed, the number of data centers is determined, and the number of expansion data clusters is determined based on the number of data centers, and the required number of expansion data clusters is deployed. When it is determined that expansion of the metadata subsystem is needed, one or more metadata clusters are deployed as the expansion data clusters. The registration module is used to register the cluster information of the expanded data cluster to the access gateway layer and save it to the metadata subsystem for the data center to call. The expansion is carried out in the form of an independent cluster, and the expansion does not require data balancing.

8. An object storage system, characterized in that, The object storage system includes a data subsystem for storing data of the object and a metadata subsystem for storing metadata of the object, and also includes the object storage system expansion device as described in claim 7.

9. A computer device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the object storage system expansion method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the operation of the object storage system expansion method as described in any one of claims 1-6.

Images