Cluster expansion method, device, storage medium and electronic device

Abstract

A cluster expansion method, device, storage medium and electronic device. The method comprises: when cluster expansion is triggered, determining the target shard that needs to be migrated on the currently started historical physical host (101); starting a new physical host, and starting a node instance for consuming the target shard on the new physical host (102); setting the original node instance on the original historical physical host that originally consumed the target shard and provided external services as the master node instance, and setting the newly started node instance on the new physical host as the slave node instance (103); when the consumption progress of the slave node instance is consistent with that of the master node instance, taking the master node instance offline, and promoting the slave node instance to provide external services to the master node instance (104).

Description

Technical Field

[0001] The embodiments of the present application relate to the field of network technology, and in particular to a cluster expansion method, device, storage medium, and electronic device. Background Art

[0002] One of the more important issues in distributed data storage solutions is how to achieve smooth cluster expansion. A more specific demand scenario is that the business needs to expand capacity before the peak period to cope with the growth of data storage scale during the peak period. In traditional distributed data storage solutions, data sharding uses a hash algorithm. When expanding capacity, the server cluster needs to be offline, and then the data in all servers needs to be traversed, and the hash algorithm needs to be used again to determine the server address before migration can be performed. Summary of the invention

[0003] The present application provides a cluster expansion method, device, storage medium and electronic device, which can achieve smooth expansion of a distributed data storage cluster.

[0004] In a first aspect, the present application provides a cluster expansion method, which is applied to a distributed data storage cluster, wherein multiple slots of the data storage cluster are associated with a shard of Kafka, and a node instance in the data storage cluster consumes a shard of Kafka, and the cluster expansion method includes:

[0005] When cluster expansion is triggered, determine the target shards that need to be migrated on the currently started historical physical host;

[0006] Starting a new physical host, and starting a node instance for consuming the target shard on the new physical host;

[0007] The original node instance on the original historical physical host that originally consumed the target shard and provided external services is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance;

[0008] When the consumption progress of the slave node instance is consistent with that of the master node instance, the master node instance is taken offline, and the slave node instance is promoted to the master node instance to provide external services.

[0009] In a second aspect, the present application further provides a cluster expansion device, which is applied to a distributed data storage cluster, wherein multiple slots of the data storage cluster are associated with a shard of Kafka, and a node instance in the data storage cluster consumes a shard of Kafka, and the cluster expansion device includes:

[0010] The determination module is used to determine the target shards that need to be migrated on the currently started historical physical host when cluster expansion is triggered;

[0011] A startup module, used to start a new physical host and start a node instance for consuming the target slice on the new physical host;

[0012] A partitioning module, used to set the original node instance on the original historical physical host that originally consumed the target shard and provided services to the outside as the master node instance, and set the newly started node instance on the new physical host as the slave node instance;

[0013] The switching module, when the consumption progress of the slave node instance is consistent with that of the master node instance, takes the master node instance offline and promotes the slave node instance to the master node instance to provide external services.

[0014] In a third aspect, the present application further provides a storage medium, which is applied to a distributed data storage cluster, wherein multiple slots of the data storage cluster are associated with a shard of Kafka, and a node instance in the data storage cluster consumes a shard of Kafka, and a computer program is stored thereon, and when the computer program is executed on a computer, the computer executes:

[0015] When cluster expansion is triggered, determine the target shards that need to be migrated on the currently started historical physical host;

[0016] Starting a new physical host, and starting a node instance for consuming the target shard on the new physical host;

[0017] The original node instance on the original historical physical host that originally consumed the target shard and provided external services is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance;

[0018] When the consumption progress of the slave node instance is consistent with that of the master node instance, the master node instance is taken offline, and the slave node instance is promoted to the master node instance to provide external services.

[0019] In a fourth aspect, the present application further provides an electronic device, which is applied to a distributed data storage cluster, wherein multiple slots of the data storage cluster are associated with a shard of Kafka, and a node instance in the data storage cluster consumes a shard in Kafka, and the electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and the processor is used to execute by calling the computer program stored in the memory:

[0020] When cluster expansion is triggered, determine the target shards that need to be migrated on the currently started historical physical host;

[0021] Starting a new physical host, and starting a node instance for consuming the target shard on the new physical host;

[0022] The original node instance on the original historical physical host that originally consumed the target shard and provided external services is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance;

[0023] When the consumption progress of the slave node instance is consistent with that of the master node instance, the master node instance is taken offline, and the slave node instance is promoted to the master node instance to provide external services. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The technical solution and beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0025] Figure 1 It is a flow chart of a cluster expansion method provided in an embodiment of the present application.

[0026] Figure 2 This is another flow chart of the cluster expansion method provided in an embodiment of the present application.

[0027] Figure 3 This is an example diagram of a data storage cluster provided in an embodiment of the present application.

[0028] Figure 4 This is another example diagram of a data storage cluster provided in an embodiment of the present application.

[0029] Figure 5 This is another example diagram of a data storage cluster provided in an embodiment of the present application.

[0030] Figure 6 It is a structural diagram of a cluster capacity expansion device provided in an embodiment of the present application.

[0031] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] Please refer to the drawings, where the same component symbols represent the same components. The principle of the present application is illustrated by implementing it in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application, which should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0033] Please refer to Figure 1 , Figure 1This is a first flow diagram of the cluster expansion method provided in an embodiment of the present application. The cluster expansion method is applied to a distributed data storage cluster, where multiple slots of the data storage cluster are associated with a shard of Kafka, and a node instance in the data storage cluster consumes a shard of Kafka. The process of the cluster expansion method may include:

[0034] In 101, when cluster expansion is triggered, the target shards that need to be migrated on the currently started historical physical host are determined.

[0035] It should be noted that a distributed data storage cluster aggregates the storage space in multiple physical hosts into a storage pool that can provide a unified access interface and management interface to the application server. Applications can transparently access and utilize the storage space on all physical hosts through this access interface, which can give full play to the performance of the physical hosts and the utilization of storage space. Data will be stored and read from multiple physical hosts according to certain rules to obtain higher concurrent access performance.

[0036] Compared with ordinary distributed data storage clusters, the distributed data storage cluster provided in the present application is a distributed key-value data storage cluster, which includes a preset number of slots and uses a hash algorithm (such as CRC16 algorithm) to map each key value to each specific slot.

[0037] In this application, the data storage cluster uses Kafka as a cache. The data is first written to a topic in Kafka. Each Kafka shard stores data in multiple slots. Then the data storage cluster consumes data from Kafka and writes it to the data storage cluster. In the data storage cluster, multiple process instances are started on a physical host, and one process instance is responsible for consuming a shard of Kafka.

[0038] Based on the data storage cluster provided by the present application, the present application provides a cluster capacity expansion method applied to the data storage cluster.

[0039] When cluster expansion is triggered, the electronic device records the currently started physical host in the data storage cluster as a historical physical host, and determines the target shard that needs to be migrated on the currently started historical physical host according to a preset data migration strategy.

[0040] It should be noted that this application does not impose any specific restrictions on the method of triggering cluster expansion, for example, it can be triggered manually by a user or automatically by an electronic device. In addition, this application does not impose any specific restrictions on the configuration of data migration strategies, which can be configured by ordinary technicians in this field according to actual needs. For example, some shards consumed on a historical physical host can be randomly selected as target shards that need to be migrated.

[0041] In 102, a new physical host is started, and a node instance for consuming the target shard is started on the new physical host.

[0042] After determining the target shard to be migrated, the electronic device starts a physical host that has not yet been started in the data storage cluster. To distinguish it from the previously started historical physical host, the newly started physical host is recorded as a new physical host.

[0043] After completing the startup of the new physical host, the electronic device starts a corresponding number of node instances for consuming the target shards on the new physical host according to the determined number of target shards.

[0044] For example, assuming that the number of target shards is two, namely shard A and shard B, since a node instance in this application is only responsible for consuming one shard, the electronic device will start two node instances accordingly on the new physical host, one of which is responsible for consuming shard A and the other is responsible for consuming shard B.

[0045] In 103, the original node instance on the original historical physical host that originally consumes the target shard and provides external services is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance.

[0046] In the embodiment of the present application, the historical physical host that originally consumes the target shard and provides services to the outside is recorded as the original historical physical host. Since there are two node instances on two different physical hosts consuming the same shard in Kafka at this time, in order for the data storage cluster to provide services to the outside normally, the original node instance on the original historical physical host is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance.

[0047] Among them, the original node instance set as the master node instance continues to provide services to the outside, while the slave node does not provide services to the outside.

[0048] In 104, when the consumption progress of the slave node instance is consistent with that of the master node instance, the master node instance is taken offline, and the slave node instance is promoted to the master node instance to provide external services.

[0049] When the consumption progress of the slave node instance is consistent with that of the master node instance, the master node instance on the original historical physical host can be taken offline, and the slave node instance on the new physical host can be promoted to the master node instance to continue to provide services to the outside world. At this point, the smooth expansion of the data storage cluster is completed, and the business is unaware of the entire expansion process. As a result, the readability and writability of the entire data storage system are improved, and the high availability and serviceability of the data storage system are guaranteed. At the same time, the entire expansion process is clear and can also reduce the difficulty of implementing the data storage system and improve development efficiency.

[0050] As can be seen from the above, multiple slots of the data storage cluster in this application are associated with a shard of kafka, and a node instance in the data storage cluster consumes a shard of kafka. When the cluster expansion is triggered, first determine the target shard that needs to be migrated on the currently started historical physical host; then, start a new physical host, and start the node instance for consuming the target shard on the new physical host; then, set the original node instance on the original historical physical host that originally consumed the target shard and provided services to the outside world as the master node instance, and set the newly started node instance on the new physical host as the slave node instance; finally, when the consumption progress of the slave node instance is consistent with that of the master node instance, take the master node instance offline, and promote the slave node instance to provide services to the outside world. Since the data storage cluster provides services to the outside world normally during the entire expansion process, the business is not aware of it, and the purpose of smooth expansion is achieved.

[0051] In one embodiment, the original node instance on the original historical physical host that originally consumes the target shard and provides external services is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance, including:

[0052] Based on Zookeeper, the original node instance on the original historical physical host that originally consumes the target shard and provides external services is set as the master node instance, and based on Zookeeper, the newly started node instance on the new physical host is set as the slave node instance.

[0053] It should be noted that Zookeeper is an open source distributed application coordination service that provides functions including configuration maintenance, domain name services, distributed synchronization, and group services.

[0054] In the embodiment of the present application, the electronic device registers the node instances started on each physical host in the data storage cluster into Zookeeper, and Zookeeper coordinates the master and slave node instances. For example, when the electronic device starts a node instance on any physical host in the data storage cluster, it registers the started node instance into Zookeeper.

[0055] Accordingly, when dividing the master and slave node instances, the electronic device sets the original node instance on the original historical physical host that originally consumed the target shard and provided services to the outside as the master node instance based on Zookeeper, and sets the newly started node instance on the new physical host as the slave node instance based on Zookeeper, wherein the slave node instance does not provide services to the outside. In short, the electronic device divides the node instances that consume the same shard (i.e., the target shard) into master and slave node instances based on Zookeeper, wherein the node instance started later is divided into a slave node instance.

[0056] In one embodiment, after the slave node instance is promoted to the master node instance to provide external services, the method further includes:

[0057] (1) When the cluster is scaled down, a node instance for consuming the target shard is started on the original historical physical host;

[0058] (2) Based on Zookeeper, the newly started node instance on the original historical physical host is set as the new slave node instance;

[0059] (3) When the consumption progress of the new slave node instance is consistent with that of the master node instance, the new physical host is shut down, and the new slave node instance is promoted to the master node instance based on Zookeeper to provide external services.

[0060] It is understandable that for data storage clusters, in addition to the need to expand capacity, there is also a corresponding need to shrink capacity.

[0061] After the slave node instance on the new physical host is promoted to the master node instance to provide external services, that is, after the smooth expansion of the data storage cluster is completed, when the cluster is triggered to shrink, the electronic device starts the node instance for consuming the aforementioned target shard on the original historical physical host, and registers the newly started node instance for consuming the aforementioned target shard on the original historical physical host to Zookeeper. Then, based on Zookeeper, the electronic device sets the newly started node instance for consuming the aforementioned target shard on the original historical physical host as the new slave node instance.

[0062] It should be noted that the present application does not impose any specific restrictions on the method of triggering cluster expansion. For example, it can be triggered manually by a user or automatically by an electronic device.

[0063] When the consumption progress of the new slave node instance is consistent with that of the current master node instance, the electronic device can take the current master node instance offline, shut down the aforementioned new physical host, and promote the new slave node instance to the master node instance based on Zookeeper to provide external services. At this point, the smooth scaling of the data storage cluster is completed, and the business is unaware of the entire scaling process.

[0064] In one embodiment, the capacity expansion method provided by the present application further includes:

[0065] (1) Quantitatively evaluate the load of the data storage cluster to obtain the load value of the data storage cluster;

[0066] (2) When the load value of the data storage cluster is greater than a preset expansion threshold, cluster expansion is triggered; or,

[0067] (3) When the load value of the data storage cluster is less than a preset shrinking threshold, cluster shrinking is triggered, and the preset shrinking threshold is less than the preset expansion threshold.

[0068] An optional expansion / contraction triggering solution is provided in an embodiment of the present application.

[0069] The electronic device can quantitatively evaluate the overall load of the data storage cluster according to a preset evaluation cycle. For example, the electronic device collects load parameters related to the data storage cluster according to a pre-configured data collection strategy, and then quantitatively evaluates the load of the data storage cluster according to the collected load parameters according to a preset quantitative evaluation strategy to obtain the load value of the data storage cluster.

[0070] It should be noted that the present application is also configured with a preset expansion threshold for triggering cluster expansion and a preset reduction threshold for triggering reduction. After evaluating the load value of the data storage cluster, the electronic device compares the load value of the data storage cluster with the preset expansion threshold / preset reduction threshold. When the load value of the data storage cluster is greater than the preset expansion threshold, cluster expansion is triggered; when the load value of the data storage cluster is less than the preset reduction threshold, cluster reduction is triggered.

[0071] In one embodiment, the load of the data storage cluster is quantitatively evaluated to obtain the load value of the data storage cluster, including:

[0072] (1) Obtain multi-dimensional load parameters of the data storage cluster and normalize the multi-dimensional load parameters;

[0073] (2) Perform weighted summation on the normalized multi-dimensional load parameters, and use the obtained weighted sum value as the load value of the data storage cluster.

[0074] The load parameters of different dimensions are not specifically limited in the embodiments of the present application, and can be configured by a person skilled in the art according to actual needs. For example, the present application uses the number of read and write requests, the number of connections, the CPU utilization, and the disk utilization as load parameters for quantitative evaluation.

[0075] Accordingly, when quantitatively evaluating the load of the data storage cluster, the electronic device first obtains the multidimensional load parameters of the data storage cluster. Since these load parameters belong to different dimensions, the electronic device also normalizes the obtained multidimensional load parameters. Among them, the specific normalization method is not specifically limited in this application, and can be selected by ordinary technicians in this field according to actual needs.

[0076] After completing the normalization processing of the acquired multi-dimensional load parameters, the electronic device further performs weighted summation on the normalized multi-dimensional load parameters to obtain a weighted sum value. The weights corresponding to the load parameters of each dimension can be set by ordinary technicians in this field according to actual needs, and this application does not impose specific restrictions on this.

[0077] In the embodiment of the present application, after the electronic device performs weighted summation on the normalized multi-dimensional load parameters to obtain a weighted sum value, the electronic device uses the weighted sum value as the load value of the data storage cluster.

[0078] In one embodiment, determining a target shard that needs to be migrated on a currently started historical physical host includes:

[0079] (1) Quantitatively evaluate the load of the currently started historical physical hosts to obtain the load value of each historical physical host;

[0080] (2) Some shards on the historical physical host whose load values ​​reach the preset performance threshold are used as target shards.

[0081] Among them, the load quantification evaluation method for historical physical hosts can be implemented accordingly with reference to the load quantification evaluation method for the data storage cluster in the above embodiment. The difference is that one is an overall evaluation of all physical hosts in the data storage cluster, and the other is an evaluation of a single physical host.

[0082] In an embodiment of the present application, when determining the target shard that needs to be migrated on the currently started historical physical host, the electronic device first quantitatively evaluates the load of the currently started historical physical host to obtain the load value of each historical physical host.

[0083] Then, the electronic device determines a historical physical host whose load value reaches a preset performance threshold among the currently started historical physical hosts, and uses the shards consumed by some node instances on the historical physical host whose load value reaches the preset performance threshold as the target shards.

[0084] It should be noted that the specific value of the preset performance threshold is not limited in the embodiments of the present application and can be set by ordinary technicians in this field according to actual needs.

[0085] In one embodiment, determining a target shard that needs to be migrated on a currently started historical physical host includes:

[0086] (1) Quantitatively evaluate the service performance of the currently started historical physical host to obtain the performance value of each historical physical host;

[0087] (2) Some shards on the historical physical host whose performance values ​​are lower than the preset performance threshold are used as target shards.

[0088] For example, the electronic device collects performance parameters related to performance of each historical physical host (such as storage space size, bandwidth size, read and write speed, etc.) according to a pre-configured data collection strategy, and then performs a quantitative evaluation of the performance of each historical physical host based on the collected performance parameters according to a preset quantitative evaluation strategy to obtain the performance value of each historical physical host.

[0089] Then, the electronic device determines a historical physical host whose performance value is lower than a preset performance threshold among the currently started historical physical hosts, and uses the shards consumed by some node instances on the historical physical host whose performance value is lower than the preset performance threshold as the target shards.

[0090] It should be noted that the specific value of the preset performance threshold is not limited in the embodiments of the present application and can be set by ordinary technicians in this field according to actual needs.

[0091] Please refer to Figure 2 The process of the cluster expansion method provided in this application can also be:

[0092] In 201, when cluster expansion is triggered, the electronic device determines a target shard that needs to be migrated on a currently started historical physical host.

[0093] In this application, the data storage cluster uses Kafka as a cache. The data is first written to a topic in Kafka. Each Kafka shard stores data in multiple slots. Then the data storage cluster consumes data from Kafka and writes it to the data storage cluster. In the data storage cluster, multiple process instances are started on a physical host, and one process instance is responsible for consuming a shard of Kafka.

[0094] For example, please refer to Figure 3 , this application has a total of 16384 slots, of which slots 0-4095 are associated with kafka's shard 0, slots 4096-8190 are associated with kafka's shard 1, slots 8191-12285 are associated with kafka's shard 2, and slots 12286-16383 are associated with kafka's shard 3. The data storage cluster currently has only one physical host "physical host A" started, and four node instances are started on physical host A, which are used to consume kafka's shards 0, 1, 2, and 3 respectively.

[0095] Based on the data storage cluster provided by the present application, the present application provides a cluster capacity expansion method applied to the data storage cluster.

[0096] When cluster expansion is triggered, the electronic device records the currently started physical host in the data storage cluster as a historical physical host, and determines the target shard that needs to be migrated on the currently started historical physical host according to a preset data migration strategy.

[0097] It should be noted that this application does not impose any specific restrictions on the method of triggering cluster expansion. For example, it can be triggered manually by a user or automatically by an electronic device. In addition, this application does not impose any specific restrictions on the configuration of data migration strategies. It can be configured by ordinary technicians in this field according to actual needs. For example, for Figure 3 In the data storage cluster shown, the electronic device uses shard 2 and shard 3 consumed on network host A as target shards that need to be migrated.

[0098] In 202, the electronic device starts a new physical host, and starts a node instance for consuming the target shard on the new physical host.

[0099] After determining the target shard to be migrated, the electronic device starts a physical host that has not yet been started in the data storage cluster. To distinguish it from the previously started historical physical host, the newly started physical host is recorded as a new physical host.

[0100] After completing the startup of the new physical host, the electronic device starts a corresponding number of node instances for consuming the target shards on the new physical host according to the determined number of target shards.

[0101] For example, please refer to Figure 3 and Figure 4At this time, the number of target shards determined is two, namely shard 2 and shard 3. Since a node instance in this application is only responsible for consuming one shard, the electronic device will start two node instances on the new physical host "physical host B" accordingly, one of which is responsible for consuming shard 2 and the other is responsible for consuming shard 3.

[0102] In 203, the electronic device sets the original node instance on the original historical physical host that originally consumes the target shard and provides services to the outside as the master node instance based on Zookeeper, and sets the newly started node instance on the new physical host as the slave node instance based on Zookeeper.

[0103] It should be noted that Zookeeper is an open source distributed application coordination service that provides functions including configuration maintenance, domain name services, distributed synchronization, and group services.

[0104] In the embodiment of the present application, the electronic device registers the node instances started on each physical host in the data storage cluster into Zookeeper, and Zookeeper coordinates the master and slave node instances. For example, when the electronic device starts a node instance on any physical host in the data storage cluster, it registers the started node instance into Zookeeper.

[0105] Correspondingly, after starting the node instance for consuming the target shard on the new physical host, the electronic device sets the original node instance on the original historical physical host that originally consumed the target shard and provided services to the outside as the master node instance based on Zookeeper, and sets the newly started node instance on the new physical host as the slave node instance based on Zookeeper, wherein the slave node instance does not provide services to the outside. In short, the electronic device divides the node instances that consume the same shard (i.e., the target shard) into master and slave node instances based on Zookeeper, wherein the node instance started later is divided into a slave node instance.

[0106] For example, please refer to Figure 4 At this time, node instances that consume shards 2 and 3 are started in both physical host A and physical host B, but the two node instances that consume shards 2 and 3 in physical host A are divided into master node instances and continue to provide services to the outside world, while the two node instances that consume shards 2 and 3 in physical host B are divided into slave node instances and do not provide services to the outside world.

[0107] In 204, when the consumption progress of the slave node instance is consistent with that of the master node instance, the electronic device takes the master node instance offline and promotes the slave node instance to the master node instance to provide external services.

[0108] When the consumption progress of the slave node instance is consistent with that of the master node instance, the master node instance on the original historical physical host can be taken offline, and the slave node instance on the new physical host can be promoted to the master node instance to continue providing services to the outside world.

[0109] For example, please refer to Figure 4 and Figure 5 , the two node instances that consume shards 2 and 3 in physical host A are taken offline, while the two node instances that consume shards 2 and 3 in physical host B are promoted to master node instances to provide services externally.

[0110] In 205, when the cluster scaling down is triggered, the electronic device starts a node instance for consuming the target shard on the original historical physical host.

[0111] It is understandable that for data storage clusters, in addition to the need to expand capacity, there is also a corresponding need to shrink capacity.

[0112] After the slave node instance on the new physical host is promoted to the master node instance to provide external services, that is, after the smooth expansion of the data storage cluster is completed, when the cluster is triggered to shrink, the electronic device starts the node instance for consuming the aforementioned target shard on the original historical physical host, and registers the newly started node instance for consuming the aforementioned target shard on the original historical physical host to Zookeeper.

[0113] It should be noted that the present application does not impose any specific restrictions on the method of triggering cluster expansion. For example, it can be triggered manually by a user or automatically by an electronic device.

[0114] In 206 , the electronic device sets the newly started node instance on the original historical physical host as a new slave node instance based on Zookeeper.

[0115] Based on Zookeeper, the electronic device sets the node instance newly started on the original historical physical host for consuming the aforementioned target shard as a new slave node instance.

[0116] In 207, when the consumption progress of the new slave node instance is consistent with that of the master node instance, the electronic device shuts down the new physical host and promotes the new slave node instance to the master node instance based on Zookeeper to provide external services.

[0117] When the consumption progress of the new slave node instance is consistent with that of the current master node instance, the electronic device can take the current master node instance offline, shut down the aforementioned new physical host, and promote the new slave node instance to the master node instance based on Zookeeper to provide external services. At this point, the smooth scaling of the data storage cluster is completed, and the business is unaware of the entire scaling process.

[0118] Please refer to Figure 6 , Figure 6 3 is a schematic diagram of the structure of the cluster expansion device provided in the present application. The cluster expansion device may include: a determination module 301, a start module 302, a division module 303 and a switching module 304.

[0119] The determination module 301 is used to determine the target shards that need to be migrated on the currently started historical physical host when cluster expansion is triggered;

[0120] A startup module 302 is used to start a new physical host and start a node instance for consuming the target shard on the new physical host;

[0121] The partitioning module 303 is used to set the original node instance on the original historical physical host that originally consumes the target shard and provides services to the outside as the master node instance, and set the newly started node instance on the new physical host as the slave node instance;

[0122] The switching module 304 is used to take the master node instance offline and promote the slave node instance to the master node instance to provide external services when the consumption progress of the slave node instance is consistent with that of the master node instance.

[0123] In one embodiment, when the original node instance on the original historical physical host that originally consumes the target shard and provides services to the outside is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance, the partitioning module 303 is used to:

[0124] Based on Zookeeper, the original node instance on the original historical physical host that originally consumes the target shard and provides external services is set as the master node instance, and based on Zookeeper, the newly started node instance on the new physical host is set as the slave node instance.

[0125] In one embodiment, after the slave node instance is promoted to the master node instance to provide external services, the startup module 302 is further used to start the node instance for consuming the target shard on the original historical physical host when cluster scaling down is triggered;

[0126] The partitioning module 303 is also used to set the newly started node instance on the original historical physical host as a new slave node instance based on Zookeeper;

[0127] The switching module 304 is also used to shut down the new physical host when the consumption progress of the new slave node instance is consistent with that of the master node instance, and promote the new slave node instance to the master node instance based on Zookeeper to provide external services.

[0128] In one embodiment, the cluster capacity expansion device provided by the present application further includes a trigger module, which is used to:

[0129] Quantitatively evaluate the load of the data storage cluster to obtain the load value of the data storage cluster;

[0130] When the load value of the data storage cluster exceeds the preset expansion threshold, cluster expansion is triggered; or,

[0131] When the load value of the data storage cluster is less than the preset shrinking threshold, cluster shrinking is triggered, and the preset shrinking threshold is less than the preset expansion threshold.

[0132] In one embodiment, when the load of the data storage cluster is quantitatively evaluated to obtain the load value of the data storage cluster, the trigger module is used to:

[0133] Obtain multi-dimensional load parameters of the data storage cluster and normalize the multi-dimensional load parameters;

[0134] The normalized multi-dimensional load parameters are weighted and summed, and the obtained weighted sum value is used as the load value of the data storage cluster.

[0135] In one embodiment, when determining a target shard that needs to be migrated on a currently started historical physical host, the determination module 301 is used to:

[0136] Quantitatively evaluate the load of the currently started historical physical hosts to obtain the load value of each historical physical host;

[0137] Some shards on the historical physical host whose load value reaches the preset performance threshold are used as target shards.

[0138] In one embodiment, when determining the target shard that needs to be migrated on the currently started historical physical host, the determination module 301 is further used to:

[0139] Quantitatively evaluate the service performance of the currently started historical physical host to obtain the performance value of each historical physical host;

[0140] Some shards on historical physical hosts whose performance values ​​are lower than a preset performance threshold are used as target shards.

[0141] It should be noted that the cluster expansion device provided in the embodiment of the present application and the cluster expansion method in the above embodiment belong to the same concept. Any method provided in the cluster expansion method embodiment can be run on the cluster expansion device. The specific implementation process is detailed in the above cluster expansion method embodiment and will not be repeated here.

[0142] The present application also provides a computer-readable storage medium, which is applied to a distributed data storage cluster, wherein multiple slots of the data storage cluster are associated with a shard of Kafka, and a node instance in the data storage cluster consumes a shard of Kafka, and a computer program is stored on the storage medium, and when the computer program stored therein is executed on a computer, the computer executes the cluster expansion method provided in the embodiment of the present application. The storage medium may be a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0143] The present application also provides an electronic device, which is applied to a distributed data storage cluster, wherein multiple slots of the data storage cluster are associated with a shard of Kafka, and a node instance in the data storage cluster consumes a shard of Kafka. The electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the cluster expansion method provided in the present application by calling the computer program stored in the memory.

[0144] For example, the electronic device may be a mobile terminal such as a tablet computer or a smart phone. Figure 7 , Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0145] The processor 401 is electrically connected to the memory 402 .

[0146] The processor 401 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device, executes various functions of the electronic device and processes data by running or loading computer programs stored in the memory 402 and calling data stored in the memory 402.

[0147] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the computer programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, a computer program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0148] In the embodiment of the present application, the processor 401 in the electronic device will load instructions corresponding to the processes of one or more computer programs into the memory 402 according to the following steps, and the processor 401 will run the computer program stored in the memory 402 to implement various functions, as follows:

[0149] When cluster expansion is triggered, determine the target shards that need to be migrated on the currently started historical physical host;

[0150] Start a new physical host and start a node instance for consuming the target shard on the new physical host;

[0151] Set the original node instance on the original historical physical host that originally consumed the target shard and provided external services as the master node instance, and set the newly started node instance on the new physical host as the slave node instance;

[0152] When the consumption progress of the slave node instance is consistent with that of the master node instance, the master node instance is taken offline, and the slave node instance is promoted to the master node instance to provide external services.

[0153] In one embodiment, when the original node instance on the original historical physical host that originally consumes the target shard and provides services to the outside is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance, the processor 401 is used to execute:

[0154] Based on Zookeeper, the original node instance on the original historical physical host that originally consumes the target shard and provides external services is set as the master node instance, and based on Zookeeper, the newly started node instance on the new physical host is set as the slave node instance.

[0155] In one embodiment, after the slave node instance is promoted to the master node instance to provide external services, the processor 401 is further configured to execute:

[0156] When the cluster is scaled down, a node instance for consuming the target shard is started on the original historical physical host;

[0157] Based on Zookeeper, the newly started node instance on the original historical physical host is set as the new slave node instance;

[0158] When the consumption progress of the new slave node instance is consistent with that of the master node instance, the new physical host is shut down, and the new slave node instance is promoted to the master node instance based on Zookeeper to provide external services.

[0159] In one embodiment, the processor 401 is further configured to execute:

[0160] Quantitatively evaluate the load of the data storage cluster to obtain the load value of the data storage cluster;

[0161] When the load value of the data storage cluster exceeds the preset expansion threshold, cluster expansion is triggered; or,

[0162] When the load value of the data storage cluster is less than the preset shrinking threshold, cluster shrinking is triggered, and the preset shrinking threshold is less than the preset expansion threshold.

[0163] In one embodiment, when the load of the data storage cluster is quantitatively evaluated to obtain the load value of the data storage cluster, the processor 401 is configured to execute:

[0164] Obtain multi-dimensional load parameters of the data storage cluster and normalize the multi-dimensional load parameters;

[0165] The normalized multi-dimensional load parameters are weighted and summed, and the obtained weighted sum value is used as the load value of the data storage cluster.

[0166] In one embodiment, when determining a target shard that needs to be migrated on a currently started historical physical host, the processor 401 is configured to execute:

[0167] Quantitatively evaluate the load of the currently started historical physical hosts to obtain the load value of each historical physical host;

[0168] Some shards on the historical physical host whose load value reaches the preset performance threshold are used as target shards.

[0169] In one embodiment, when determining the target shard that needs to be migrated on the currently started historical physical host, the processor 401 is further configured to execute:

[0170] Quantitatively evaluate the service performance of the currently started historical physical host to obtain the performance value of each historical physical host;

[0171] Some shards on historical physical hosts whose performance values ​​are lower than a preset performance threshold are used as target shards.

[0172] It should be noted that the electronic device provided in the embodiment of the present application belongs to the same concept as the cluster expansion method in the above embodiment. Any method provided in the cluster expansion method embodiment can be run on the electronic device. The specific implementation process is detailed in the feature extraction method embodiment and will not be repeated here.

[0173] It should be noted that, for the cluster expansion method of the embodiment of the present application, ordinary testers in the field can understand that all or part of the process of implementing the cluster expansion method of the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as stored in the memory of an electronic device, and executed by at least one processor in the electronic device, and the execution process may include the process of the embodiment of the cluster expansion method. Among them, the storage medium can be a disk, an optical disk, a read-only memory, a random access memory, etc.

[0174] For the cluster expansion device of the embodiment of the present application, each functional module can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

[0175] The above is a detailed introduction to a cluster expansion method, device, storage medium and electronic device provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A cluster expansion method, applied to a distributed data storage cluster, wherein: A plurality of slots in the data storage cluster are associated with a shard of Kafka, and a node instance in the data storage cluster consumes a shard of Kafka, and the cluster expansion method includes: When cluster expansion is triggered, the target shard to be migrated on the currently started historical physical host is determined, and the target shard is the shard in the kafka; Starting a new physical host, and starting a node instance for consuming the target shard on the new physical host; The original node instance on the original historical physical host that originally consumed the target shard and provided external services is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance. Both the master node instance and the slave node instance consume the target shard, the master node instance provides external services, and the slave node instance does not provide external services; When the consumption progress of the target shard by the slave node instance is consistent with that of the master node instance, the master node instance is taken offline, and the slave node instance is promoted to the master node instance to provide external services.

2. The cluster expansion method according to claim 1, wherein: The step of setting the original node instance on the original historical physical host that originally consumes the target shard and provides external services as the master node instance, and setting the newly started node instance on the new physical host as the slave node instance, includes: Based on Zookeeper, the original node instance on the original historical physical host that originally consumed the target shard and provided services to the outside is set as the master node instance, and based on Zookeeper, the newly started node instance on the new physical host is set as the slave node instance.

3. The cluster expansion method according to claim 2, wherein: After the slave node instance is promoted to the master node instance to provide external services, the method further includes: When cluster scaling is triggered, a node instance for consuming the target shard is started on the original historical physical host; Based on Zookeeper, the newly started node instance on the original historical physical host is set as a new slave node instance; When the consumption progress of the new slave node instance is consistent with that of the master node instance, the new physical host is shut down, and based on Zookeeper, the new slave node instance is promoted to the master node instance to provide external services.

4. The cluster expansion method according to claim 3, wherein: Also includes: Performing quantitative evaluation on the load of the data storage cluster to obtain a load value of the data storage cluster; When the load value of the data storage cluster is greater than a preset expansion threshold, cluster expansion is triggered; or, When the load value of the data storage cluster is less than a preset shrinking threshold, cluster shrinking is triggered, and the preset shrinking threshold is less than the preset expansion threshold.

5. The cluster expansion method according to claim 4, wherein: The quantitative evaluation of the load of the data storage cluster to obtain the load value of the data storage cluster includes: Acquire multidimensional load parameters of the data storage cluster, and perform normalization processing on the multidimensional load parameters; A weighted sum is performed on the normalized multi-dimensional load parameters, and the obtained weighted sum value is used as the load value of the data storage cluster.

6. The cluster expansion method according to claim 1, wherein: The step of determining the target shard to be migrated on the currently started historical physical host includes: Quantitatively evaluate the load of the currently started historical physical hosts to obtain the load value of each historical physical host; Some shards on the historical physical host whose load value reaches a preset performance threshold are used as the target shards.

7. The cluster expansion method according to claim 1, wherein: The step of determining the target shard to be migrated on the currently started historical physical host includes: Quantitatively evaluate the service performance of the currently started historical physical host to obtain the performance value of each historical physical host; Some shards on the historical physical host whose performance value is lower than the preset performance threshold are used as the target shards.

8. A cluster expansion device, applied to a distributed data storage cluster, wherein multiple slots of the data storage cluster are associated with a shard of Kafka, and a node instance in the data storage cluster consumes a shard of Kafka, and the cluster expansion device comprises: A determination module is used to determine the target shard that needs to be migrated on the currently started historical physical host when cluster expansion is triggered, and the target shard is the shard in the kafka; A startup module, used to start a new physical host and start a node instance for consuming the target slice on the new physical host; A partitioning module is used to set the original node instance on the original historical physical host that originally consumed the target shard and provided external services as the master node instance, and set the newly started node instance on the new physical host as the slave node instance, wherein both the master node instance and the slave node instance consume the target shard, the master node instance provides external services, and the slave node instance does not provide external services; The switching module, when the consumption progress of the slave node instance and the master node instance in consuming the target shard is consistent, the master node instance is taken offline, and the slave node instance is promoted to the master node instance to provide external services.

9. The cluster expansion device according to claim 8, wherein: When the original node instance on the original historical physical host that originally consumes the target shard and provides external services is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance, the partitioning module is used to: Based on Zookeeper, the original node instance on the original historical physical host that originally consumed the target shard and provided services to the outside is set as the master node instance, and based on Zookeeper, the newly started node instance on the new physical host is set as the slave node instance.

10. The cluster expansion device according to claim 9, wherein: After the slave node instance is promoted to the master node instance to provide external services, the startup module is further used to start a node instance for consuming the target shard on the original historical physical host when cluster scaling down is triggered; The partitioning module is also used to set the newly started node instance on the original historical physical host as a new slave node instance based on Zookeeper; The switching module is also used to shut down the new physical host when the consumption progress of the new slave node instance is consistent with that of the master node instance, and promote the new slave node instance to the master node instance based on Zookeeper to provide external services.

11. The cluster expansion device according to claim 10, wherein: Also included are trigger modules for: Performing quantitative evaluation on the load of the data storage cluster to obtain a load value of the data storage cluster; When the load value of the data storage cluster is greater than a preset expansion threshold, cluster expansion is triggered; or, When the load value of the data storage cluster is less than a preset shrinking threshold, cluster shrinking is triggered, and the preset shrinking threshold is less than the preset expansion threshold.

12. The cluster expansion device according to claim 11, wherein: When the load of the data storage cluster is quantitatively evaluated to obtain the load value of the data storage cluster, the trigger module is used to: Acquire multidimensional load parameters of the data storage cluster, and perform normalization processing on the multidimensional load parameters; A weighted sum is performed on the normalized multi-dimensional load parameters, and the obtained weighted sum value is used as the load value of the data storage cluster.

13. A storage medium, applied to a distributed data storage cluster, wherein: A plurality of slots in the data storage cluster are associated with a shard of Kafka, and a node instance in the data storage cluster consumes a shard of Kafka. A computer program is stored in the storage medium. When the computer program is run on a computer, the computer executes: When cluster expansion is triggered, the target shard to be migrated on the currently started historical physical host is determined, and the target shard is the shard in the kafka; Starting a new physical host, and starting a node instance for consuming the target shard on the new physical host; The original node instance on the original historical physical host that originally consumed the target shard and provided external services is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance. Both the master node instance and the slave node instance consume the target shard, the master node instance provides external services, and the slave node instance does not provide external services; When the consumption progress of the target shard by the slave node instance is consistent with that of the master node instance, the master node instance is taken offline, and the slave node instance is promoted to the master node instance to provide external services.

14. An electronic device, applied to a distributed data storage cluster, wherein: A plurality of slots in the data storage cluster are associated with a shard of Kafka, and a node instance in the data storage cluster consumes a shard of Kafka, the electronic device comprises a processor and a memory, a computer program is stored in the memory, and the processor is used to execute, by calling the computer program stored in the memory: When cluster expansion is triggered, the target shard to be migrated on the currently started historical physical host is determined, and the target shard is the shard in the kafka; Starting a new physical host, and starting a node instance for consuming the target shard on the new physical host; The original node instance on the original historical physical host that originally consumed the target shard and provided external services is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance. Both the master node instance and the slave node instance consume the target shard, the master node instance provides external services, and the slave node instance does not provide external services; When the consumption progress of the target shard by the slave node instance is consistent with that of the master node instance, the master node instance is taken offline, and the slave node instance is promoted to the master node instance to provide external services.

15. The electronic device according to claim 14, wherein: When the original node instance on the original historical physical host that originally consumed the target shard and provided services externally is set as the master node instance, and the newly started node instance on the new physical host is set as the slave node instance, the processor is used to execute: Based on Zookeeper, the original node instance on the original historical physical host that originally consumed the target shard and provided services to the outside is set as the master node instance, and based on Zookeeper, the newly started node instance on the new physical host is set as the slave node instance.

16. The electronic device according to claim 15, wherein: After promoting the slave node instance to the master node instance to provide external services, the processor is further configured to execute: When cluster scaling is triggered, a node instance for consuming the target shard is started on the original historical physical host; Based on Zookeeper, the newly started node instance on the original historical physical host is set as a new slave node instance; When the consumption progress of the new slave node instance is consistent with that of the master node instance, the new physical host is shut down, and based on Zookeeper, the new slave node instance is promoted to the master node instance to provide external services.

17. The electronic device according to claim 16, wherein: The processor is further configured to execute: Performing quantitative evaluation on the load of the data storage cluster to obtain a load value of the data storage cluster; When the load value of the data storage cluster is greater than a preset expansion threshold, cluster expansion is triggered; or, When the load value of the data storage cluster is less than a preset shrinking threshold, cluster shrinking is triggered, and the preset shrinking threshold is less than the preset expansion threshold.

18. The electronic device according to claim 17, wherein: When the load of the data storage cluster is quantitatively evaluated to obtain the load value of the data storage cluster, the processor is configured to execute: Acquire multidimensional load parameters of the data storage cluster, and perform normalization processing on the multidimensional load parameters; A weighted sum is performed on the normalized multi-dimensional load parameters, and the obtained weighted sum value is used as the load value of the data storage cluster.

19. The electronic device according to claim 14, wherein: When determining a target shard that needs to be migrated on a currently started historical physical host, the processor is used to: Quantitatively evaluate the load of the currently started historical physical hosts to obtain the load value of each historical physical host; Some shards on the historical physical host whose load value reaches a preset performance threshold are used as the target shards.

20. The electronic device according to claim 14, wherein: When determining a target shard that needs to be migrated on a currently started historical physical host, the processor is further configured to: Quantitatively evaluate the service performance of the currently started historical physical host to obtain the performance value of each historical physical host; Some shards on the historical physical host whose performance value is lower than the preset performance threshold are used as the target shards.

Images