Cluster expansion method, device, electronic equipment and storage medium

By obtaining and analyzing the data changes in the message middleware and cluster resource consumption data, combining stress testing, predicting resource consumption data to determine expansion information, the problem of inflexible expansion of cloud clusters is solved, efficient and accurate cluster expansion is achieved, and the Expansion cost.

CN117407158BActive Publication Date: 2024-11-19E-SURFING DIGITAL LIFE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311294592.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-08
Publication Date
2024-11-19
Estimated Expiration
2043-10-08

AI Technical Summary

Technical Problem

The existing cloud cluster expansion method cannot effectively determine the expansion information, resulting in the expansion of too large or too small, requiring multiple expansions, increasing costs, and the expansion of the big data cluster is inflexible.

Method used

By obtaining the data volume change information of the message middleware and the resource consumption data of the cluster machine, combining the stress test results, predicting the resource consumption data, determining the expansion information when the preset threshold is reached, and cluster expansion, including disk and machine expansion.

Benefits of technology

It realizes efficient and accurate capacity expansion of the cluster, avoids high capacity expansion costs, improves the efficiency of capacity expansion of distributed cloud clusters, and ensures flexibility of capacity expansion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117407158B_ABST
    Figure CN117407158B_ABST
Patent Text Reader

Abstract

The present invention discloses a cluster expansion method, device, electronic device and storage medium, the method comprising: obtaining data volume change information of a message middleware and resource consumption data of a cluster machine; then obtaining resource consumption prediction data according to the resource consumption data and in combination with the data volume change information; when the resource consumption prediction data reaches a preset threshold, determining expansion information based on a resource consumption target value; and performing cluster expansion according to the expansion information. The embodiment of the present invention performs logical analysis of the message middleware data change information, and then obtains resource consumption prediction data in combination with the resource consumption data already used by the cluster machine, and then determines the expansion information based on the resource consumption prediction data, so as to quickly and timely perform cluster adaptive expansion, avoid high expansion costs, and improve the efficiency of distributed cloud cluster expansion, and can be widely used in the field of data processing technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cluster expansion method, device, electronic equipment and storage medium. Background Art

[0002] As the scale of product business continues to grow, the requirements for distributed cloud cluster data storage and computing are also increasing. The scale of existing cloud clusters, such as Hadoop and Elasticsearch clusters, can no longer meet the requirements of rapid business growth. When the use of distributed clusters reaches the upper limit of the cluster design capacity, the cloud cluster needs to be expanded.

[0003] Generally, the load of the cluster is monitored to determine whether to expand the capacity. This expansion is based on the change of cluster load, and the expansion information cannot be effectively determined. It is easy to cause the expansion to be too large or too small, requiring multiple expansions, resulting in high expansion costs. The existing big data cluster expansion is not flexible. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, the present invention proposes a cluster expansion method, device, electronic device and storage medium, which can efficiently and accurately expand the cluster.

[0005] On the one hand, an embodiment of the present invention provides a cluster capacity expansion method, including:

[0006] Obtain the data volume change information of the message middleware and the resource consumption data of the cluster machines;

[0007] According to the resource consumption data, combined with the data volume change information, the resource consumption forecast data is obtained;

[0008] When the resource consumption forecast data reaches the preset threshold, the capacity expansion information is determined based on the resource consumption target value;

[0009] Expand the cluster based on the expansion information.

[0010] Optionally, obtain the data volume change information of the message middleware and the resource consumption data of the cluster machines, including:

[0011] Obtain the data volume produced and consumed by all topics of the message middleware within a preset time period as the data volume change information of the message middleware;

[0012] Get the disk capacity consumption data of cluster machines and the CPU usage of cluster machines.

[0013] Optionally, the method further comprises:

[0014] Based on the stress test of the cluster machines, a data list of the relationship between the data volume change information and the newly added resource consumption data is obtained, and preset parameters are determined; wherein the preset parameters include at least one of a preset threshold value and a resource consumption target value.

[0015] Optionally, the resource consumption prediction data is obtained based on the resource consumption data and combined with the data volume change information, including:

[0016] Based on the relational data list, according to the data volume change information, the newly added resource consumption data is obtained; wherein the relational data list is obtained based on the stress test of the cluster machine;

[0017] According to the resource consumption data and the newly added resource consumption data, the resource consumption forecast data is obtained.

[0018] Optionally, the resource consumption data includes disk capacity consumption data and CPU usage, the resource consumption prediction data includes disk capacity consumption prediction data and CPU prediction usage, the preset threshold includes a first threshold and a second threshold, and the resource consumption target value includes a disk capacity consumption target value; when the resource consumption prediction data reaches the preset threshold, based on the resource consumption target value, determining the expansion information includes:

[0019] When the disk capacity consumption prediction data reaches a first threshold, the total disk capacity space is obtained, and the ratio of the disk capacity consumption data to the disk capacity consumption target value is subtracted from the total disk capacity space to obtain disk expansion information;

[0020] And / or, when the CPU usage reaches a second threshold, the number of machines in the cluster is obtained, and the machine expansion information is obtained according to the number of machines and a preset ratio.

[0021] Optionally, the expansion information includes disk expansion information and / or machine expansion information; performing cluster expansion according to the expansion information includes:

[0022] Expand the disk capacity of cluster machines according to the disk expansion information;

[0023] And / or, configure a new machine according to the machine expansion information, add the configured new machine to the cluster machine, and complete the cluster machine expansion.

[0024] Optionally, the method further comprises:

[0025] Get the expansion results of cluster machines after cluster expansion is completed;

[0026] When the expansion operation result reaches the expected operating range, the cluster expansion is completed. Otherwise, the cluster is expanded again based on the expansion operation result.

[0027] On the other hand, an embodiment of the present invention provides a cluster capacity expansion device, including:

[0028] The first module is used to obtain the data volume change information of the message middleware and the resource consumption data of the cluster machine;

[0029] The second module is used to obtain resource consumption prediction data based on resource consumption data and data volume change information;

[0030] The third module is used to determine the expansion information based on the resource consumption target value when the resource consumption prediction data reaches a preset threshold;

[0031] The fourth module is used to expand the cluster according to the expansion information.

[0032] Optionally, the first module is specifically used for:

[0033] Obtain the data volume produced and consumed by all topics of the message middleware within a preset time period as the data volume change information of the message middleware;

[0034] Get the disk capacity consumption data of cluster machines and the CPU usage of cluster machines.

[0035] Optionally, the device further comprises:

[0036] The fifth module is used to obtain a list of relationship data between data volume change information and new resource consumption data based on stress testing of cluster machines, and determine preset parameters; wherein the preset parameters include at least one of a preset threshold value and a resource consumption target value.

[0037] Optionally, the second module is specifically used for:

[0038] Based on the relational data list, according to the data volume change information, the newly added resource consumption data is obtained; wherein the relational data list is obtained based on the stress test of the cluster machine;

[0039] According to the resource consumption data and the newly added resource consumption data, the resource consumption forecast data is obtained.

[0040] Optionally, the resource consumption data includes disk capacity consumption data and CPU usage, the resource consumption prediction data includes disk capacity consumption prediction data and CPU prediction usage, the preset threshold includes a first threshold and a second threshold, and the resource consumption target value includes a disk capacity consumption target value; the third module is specifically used to:

[0041] When the disk capacity consumption prediction data reaches a first threshold, the total disk capacity space is obtained, and the ratio of the disk capacity consumption data to the disk capacity consumption target value is subtracted from the total disk capacity space to obtain disk expansion information;

[0042] And / or, when the CPU usage reaches a second threshold, the number of machines in the cluster is obtained, and the machine expansion information is obtained according to the number of machines and a preset ratio.

[0043] Optionally, the expansion information includes disk expansion information and / or machine expansion information, and the fourth module is specifically used to:

[0044] Expand the disk capacity of cluster machines according to the disk expansion information;

[0045] And / or, configure a new machine according to the machine expansion information, add the configured new machine to the cluster machine, and complete the cluster machine expansion.

[0046] Optionally, the device further comprises:

[0047] The sixth module is used to obtain the expansion operation results of the cluster machines after the cluster expansion is completed;

[0048] The seventh module is used to complete cluster expansion when the expansion operation result reaches the expected operation range, otherwise, the cluster is expanded again based on the expansion operation result.

[0049] On the other hand, an embodiment of the present invention provides an electronic device, including: a processor and a memory; the memory is used to store programs; the processor executes the program to implement the above cluster expansion method.

[0050] On the other hand, an embodiment of the present invention provides a computer storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned cluster expansion method.

[0051] The embodiment of the present invention obtains the data volume change information of the message middleware and the resource consumption data of the cluster machine; then obtains the resource consumption prediction data according to the resource consumption data and the data volume change information; when the resource consumption prediction data reaches a preset threshold, the expansion information is determined based on the resource consumption target value; and the cluster is expanded according to the expansion information. The embodiment of the present invention performs a logical analysis of the message middleware data change information, and then obtains the resource consumption prediction data in combination with the resource consumption data already used by the cluster machine, and then determines the expansion information based on the resource consumption prediction data, so that the cluster can be adaptively expanded quickly and timely, avoiding high expansion costs and improving the efficiency of distributed cloud cluster expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0053] Figure 1This is a schematic diagram of an implementation environment for cluster expansion provided by an embodiment of the present invention;

[0054] Figure 2 It is a flow chart of a cluster expansion method provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of a process for obtaining data volume change information and resource consumption data provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of a process for obtaining resource consumption prediction data provided by an embodiment of the present invention;

[0057] Figure 5 A schematic diagram of a process for determining expansion information provided by an embodiment of the present invention;

[0058] Figure 6 A schematic diagram of the overall process of a cluster capacity expansion method combined with capacity expansion result judgment provided by an embodiment of the present invention;

[0059] Figure 7 A schematic diagram of the overall process of the cluster expansion method provided by an embodiment of the present invention;

[0060] Figure 8 A schematic diagram of the structure of a cluster capacity expansion device provided by an embodiment of the present invention;

[0061] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0062] Fig.10 A block diagram of the computer system structure of an electronic device suitable for implementing the embodiment of the present invention is provided for the embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0064] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] It is understandable that the cluster expansion method provided in the embodiment of the present invention can be applied to any computer device with data processing computing capabilities, and this computer device can be various types of terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this.

[0067] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0068] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0069] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0070] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present invention.

[0071] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a cluster expansion method. The following is an example of applying the cluster expansion method to the server 101 for explanation. It can be understood that the cluster expansion method can also be applied to the terminal 102.

[0072] Reference Figure 2 , Figure 2 The present invention provides a flow chart of a cluster expansion method for a server according to an embodiment of the present invention. The execution subject of the cluster expansion method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method comprises the following steps:

[0073] S100, obtaining data volume change information of the message middleware and resource consumption data of the cluster machine;

[0074] It should be noted that in some embodiments, Figure 3 As shown, step S100 may include: S101, obtaining the data volume produced and consumed by all topics of the message middleware within a preset time period as the data volume change information of the message middleware; and, S102, obtaining the disk capacity consumption data of the cluster machine, and obtaining the CPU usage rate of the cluster machine.

[0075] In some specific embodiments, the data volume change information of the message middleware and the resource consumption data of the cluster machine can be collected by the client agent, wherein the process steps of the client agent collection are as follows:

[0076] First, you need to deploy an agent system on each machine in the cluster, which mainly collects changes in the amount of data in the message middleware, including information on cluster machine load, memory, disk usage, and cluster software operation status information. For example, it collects information such as the amount of data produced every 1 minute, 15 minutes, and 60 minutes on each Partition of all Topics in the message middleware.

[0077] Among them, Topic is the subject of the message, which can be understood as the classification of the message. Multiple topics can be created on the message middleware. Messages in the publish-subscribe mode are centralized and gathered. Different producers send messages to the topic, which are published by the MQ server to different subscribers to realize the broadcast of the message. Topic is a logical concept. A topic can contain multiple partitions. Partition is a physical concept. Kafka stores partitions on the broker disk.

[0078] The collected data is then collected and stored, and the usage information and software operation status information of each machine in the cluster are reported to the database in real time through the client interface. This is mainly used for subsequent analysis modules to analyze and implement subsequent cluster expansion steps.

[0079] S200, obtaining resource consumption prediction data based on resource consumption data and data volume change information;

[0080] It should be noted that, in some embodiments, the method may also include: based on a stress test of the cluster machine, obtaining a list of relationship data between data volume change information and new resource consumption data, and determining preset parameters; wherein the preset parameters include at least one of a preset threshold and a resource consumption target value.

[0081] In some embodiments, such as Figure 4 As shown, step S200 may include: S201, based on the relationship data list, according to the data volume change information, obtain new resource consumption data; wherein the relationship data list is obtained based on the stress test of the cluster machine; S202, according to the resource consumption data and the new resource consumption data, obtain resource consumption prediction data.

[0082] In some specific embodiments, resource consumption prediction data can be obtained through the analysis module, wherein the process steps implemented by the analysis module are as follows: the analysis module collects the data volume and machine resource consumption data produced and consumed by the message middleware, including CPU load, memory usage, disk usage and other operating parameters, and performs stress testing on the cluster machines before going online, obtains the production volume, consumption volume, distributed cluster resource consumption and other data of each stage test from the changes in the message middleware data volume and the distributed cluster resource consumption, and obtains a list of maximum and normal value data that the message middleware can withstand for the data volume produced and consumed and the corresponding cluster. In general, redundancy and system space issues are taken into consideration, and it is also taken into consideration that the disk also requires space during calculation. Generally, the total disk capacity is redundant by 30%. Therefore, the data volume produced and consumed by the message middleware plus the existing load value of the cluster is added, and it is predicted that the cluster disk capacity will exceed 70%, or the CPU utilization rate will reach more than 80%, so the capacity will be expanded.

[0083] S300, when the resource consumption prediction data reaches a preset threshold, based on the resource consumption target value, determining expansion information;

[0084] It should be noted that the resource consumption data includes disk capacity consumption data and CPU usage, the resource consumption prediction data includes disk capacity consumption prediction data and CPU prediction usage, the preset threshold includes a first threshold and a second threshold, and the resource consumption target value includes a disk capacity consumption target value. In some embodiments, Figure 5 As shown, step S300 may include:

[0085] S301, when the disk capacity consumption prediction data reaches a first threshold, the total disk capacity space is obtained, and the ratio of the disk capacity consumption data to the disk capacity consumption target value minus the total disk capacity space is subtracted to obtain disk expansion information;

[0086] And / or, S302, when the CPU usage reaches a second threshold, the number of machines in the cluster is obtained, and machine expansion information is obtained according to the number of machines and a preset ratio.

[0087] In some specific embodiments, based on the resource consumption prediction data and the corresponding preset thresholds determined by the analysis module in the aforementioned steps, the following process logic can be implemented by the analysis module:

[0088] The analysis module will make logical judgments based on the growth of data produced and consumed by the message middleware and whether cluster computing resources and storage resources should be expanded. The expansion methods determined by the analysis module include disk expansion and CPU expansion. Costs should be considered when expanding. Generally, it is sufficient to consider a reasonable value after expansion to avoid wasting resources due to too much expansion at one time. The specific contents are as follows:

[0089] Disk expansion model: Before going online, perform a stress test on the total disk capacity. The relationship between the amount of data produced and consumed by the message middleware and the cluster disk usage rate is calculated in the cluster. The number of cluster disk storage resources required after different data volume changes is obtained, that is, the disk model calculation formula. The analysis module analyzes the increase in the amount of data of the message middleware, including the data volume changes of each Topic partition in the last 1 minute, 15 minutes, and 60 minutes. At the same time, calculate whether the amount of data generated in the above time period will exceed the expansion conditions of the existing cluster, that is, the comprehensive evaluation of the indicators of the data volume changes of all Topic topics of the message middleware and the current operation status of the cluster exceeds the warning value that the cluster can bear. For example, if the amount of data of the message middleware increases, the estimated increase in the cluster disk capacity per day is obtained based on the amount of data produced and consumed by the message middleware. Considering the expansion cost, the estimated increase in the disk capacity of the data within a week is obtained. If the estimated increase in the cluster disk capacity within a week + the existing cluster disk capacity exceeds 70%, the cluster needs to be expanded. After the expansion, the cluster operation level is normal (the total disk capacity usage rate of the cluster after the expansion does not exceed 50% is normal, and it can be changed to an appropriate value according to actual needs).

[0090] Disk model calculation formula: U / (F+X)=Y, used space before expansion (U), total space before expansion (F), storage space to be added this time (X), total space usage after expansion (Y=50%), that is, X=(U / Y)-F.

[0091] CPU expansion model: Perform CPU stress testing before going online, and calculate the relationship between the amount of data produced and consumed by the message middleware and the CPU utilization rate in the cluster to obtain the number of CPU resources required after different data volume changes, that is, the CPU model calculation formula. If the amount of data produced and consumed by the message middleware changes within the 1 minute, 15 minutes, and 60 minutes time period, the analysis module will calculate the CPU utilization rate increase caused by the increase in the data volume within the above time period. If it exceeds 80% of the existing cluster CPU utilization rate, the analysis module will deduce the amount of CPU expansion based on the growth of the message middleware data volume.

[0092] CPU model calculation formula: X=1.5Y, the total number of consumer node machines before expansion is Y, and the total number of consumer node machines after expansion is X=1.5Y, that is, the number of consumer machines before expansion increases by 50%, and the goal is to reduce it to below 20% to ensure stable cluster computing.

[0093] In this module, it is necessary to analyze the changes in the data volume of the Topic of the message middleware. The analysis module calculates the changes in the production and consumption of the message middleware data to trigger the resource value of the expansion. At the same time, the information of this expansion is recorded and sent to the distribution module for cluster expansion.

[0094] The cluster in this module generally refers to the cloud cluster, so the parameters in the expansion model should be adjusted according to the actual composition of the cloud cluster.

[0095] S400. Expand the cluster according to the expansion information.

[0096] It should be noted that the expansion information includes disk expansion information and / or machine expansion information. In some embodiments, step S400 may include: expanding the disk of the cluster machine according to the disk expansion information; and / or configuring a new machine according to the machine expansion information, adding the configured new machine to the cluster machine, and completing the cluster machine expansion.

[0097] In some specific embodiments, the capacity expansion information can be received and implemented through the sending module. For example, the sending module receives the capacity expansion information, updates the network configuration and cluster software configuration of the new machine respectively, and sends the network configuration file and cluster software to the new machine. At the same time, the cluster software is restarted through the client agent. For example, the client agent of the new machine receives the instruction sent by the master control module, and the client agent executes the network configuration information such as the update of the firewall list of the new machine sent by the sending module. Then, the cluster software configuration information sent by the sending module is executed in the next step, the software configuration of the new machine is updated, and the software of the newly added machine is restarted. At the same time, the sending module is notified that the network and cluster software configuration of the new machine are successfully updated, and the cluster configuration is updated by the sending module, and the new machine is added to the cluster.

[0098] Among them, in some embodiments, such as Figure 6 As shown, the method also includes: S500, obtaining the expansion operation result of the cluster machine after the cluster expansion is completed; S600, when the expansion operation result reaches the expected operation range, the cluster expansion is completed, otherwise, the cluster is expanded again based on the expansion operation result.

[0099] It should be noted that the expansion operation result is consistent with the resource consumption data of the aforementioned steps. In some embodiments, the resource consumption target value may also include the CPU target usage rate. When the CPU usage rate in the expansion operation result does not meet the CPU target usage rate, the machine is expanded again. Similarly, when the disk capacity consumption forecast data in the expansion operation result does not meet the disk capacity consumption target value, the disk expansion is checked or the disk expansion is directly performed again. The principle of cluster expansion again is consistent with the cluster expansion principle of steps S300 and S400, and will not be repeated.

[0100] In some specific embodiments, the expansion result is also judged. For example, the client collects the load, memory, disk usage information of the distributed cluster and the operating status information of the cluster software, and the analysis module determines whether the current cluster operation is within the expected normal range. If it is within the normal range, the expansion is completed.

[0101] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0102] First of all, it should be explained that the message middleware usually caches data temporarily, which is the data source for the big data cluster. After consuming the cached data of the message middleware, the data enters the big data cluster. The load of the cluster is monitored to determine whether to expand the capacity. This expansion is based on the change in cluster load. There is a lack of comprehensive analysis of the change in source data volume and cluster load capacity, and it is not possible to predict whether the cluster should be expanded based on the comprehensive analysis. At the same time, due to the change in source data volume, it does not provide how much storage and computing resources need to be expanded to achieve a reasonable operating range of the cluster load. There is a high expansion cost and the expansion of the big data cluster is not flexible.

[0103] In view of this, the present invention first deploys the client agent collection component on all machines in the distributed cluster, and obtains the production and consumption capacity information of all topics of the message middleware through the client agent program, including the changes of each topic partition in the past 1 minute, 15 minutes, and 60 minutes. At the same time, the cluster information is collected and reported to the cluster logic analysis module, and combined with the analysis module to judge: first, based on the data volume changes of all topics of the current message middleware in a unit time period and the indicators of the cluster operation status, a comprehensive assessment is made due to the increase in the amount of data in the message middleware and the current load value of the cluster, and it is judged whether this load value exceeds the warning value that the Hadoop and ElasticSearch clusters can bear. Through analysis and judgment, and recording of information exceeding the warning value, including how much total disk space needs to be increased and how much CPU capacity needs to be expanded, the distribution module is notified to complete the network configuration of the new machine and the dynamic configuration and distribution of the cluster software. The client agent of the new machine receives the cluster software and completes the software restart. At the same time, the assembly module sends instructions to all machines in the cluster, and the distributed cluster updates the information of the newly added machines and completes the expansion of the new machine. In some specific embodiments, Figure 7 As shown, the overall process steps implemented by the method of the present invention are:

[0104] 1. Client agent collection:

[0105] First, you need to deploy an agent system on each machine to collect changes in the amount of data in the message middleware, including information on cluster machine load, memory, disk usage, and cluster software operation status information. For example, you need to collect information on the amount of data produced every 1 minute, 15 minutes, and 60 minutes on each Partition of all Topic in the message middleware.

[0106] Among them, Topic is the subject of the message, which can be understood as the classification of the message. Multiple topics can be created on the message middleware. Messages in the publish-subscribe mode are centralized and gathered. Different producers send messages to the topic, which are published by the MQ server to different subscribers to realize the broadcast of the message. Topic is a logical concept. A topic can contain multiple partitions. Partition is a physical concept. Kafka stores partitions on the broker disk.

[0107] 2. Data collection and storage:

[0108] The client reports the usage information and software running status information of each machine in the cluster to the database in real time through the interface, which is mainly used for subsequent analysis modules.

[0109] 3. Analysis module:

[0110] The analysis module collects the data volume and machine resource consumption data produced and consumed by the message middleware, including operating parameters such as CPU load, memory usage, and disk usage. A stress test is performed before going online. The production volume, consumption volume, distributed cluster resource consumption and other data of each stage test are obtained from the changes in the data volume of the message middleware and the distributed cluster resource consumption. A list of maximum and normal values ​​of the data volume produced and consumed by the message middleware and the corresponding cluster can be obtained. In general, redundancy and system space issues are taken into consideration. At the same time, considering that the disk also requires space for calculations, the total disk capacity is generally 30% redundant. Therefore, the amount of data produced and consumed by the message middleware plus the existing load value of the cluster is predicted to exceed 70%, or the cluster disk capacity will be expanded when the CPU utilization rate reaches more than 80%.

[0111] The analysis module will make logical judgments based on the growth of data produced and consumed by the message middleware and whether cluster computing resources and storage resources should be expanded. The expansion methods determined by the analysis module include disk expansion and CPU expansion. Costs should be considered when expanding. Generally, it is sufficient to consider a reasonable value after expansion to avoid wasting resources due to too much expansion at one time. The specific contents are as follows:

[0112] Disk expansion model: Before going online, perform a stress test on the total disk capacity. The relationship between the amount of data produced and consumed by the message middleware and the cluster disk usage rate is calculated in the cluster. The number of cluster disk storage resources required after different data volume changes is obtained, that is, the disk model calculation formula. The analysis module analyzes the increase in the amount of data of the message middleware, including the data volume changes of each Topic partition in the last 1 minute, 15 minutes, and 60 minutes. At the same time, calculate whether the amount of data generated in the above time period will exceed the expansion conditions of the existing cluster, that is, the comprehensive evaluation of the indicators of the data volume changes of all Topic topics of the message middleware and the current operation status of the cluster exceeds the warning value that the cluster can bear. For example, if the amount of data of the message middleware increases, the estimated increase in the cluster disk capacity per day is obtained based on the amount of data produced and consumed by the message middleware. Considering the expansion cost, the estimated increase in the disk capacity of the data within a week is obtained. If the estimated increase in the cluster disk capacity within a week + the existing cluster disk capacity exceeds 70%, the cluster needs to be expanded. After the expansion, the cluster operation level is normal (the total disk capacity usage rate of the cluster after the expansion does not exceed 50% is normal, and it can be changed to an appropriate value according to actual needs).

[0113] Disk model calculation formula: U / (F+X)=Y, used space before expansion (U), total space before expansion (F), storage space to be added this time (X), total space usage after expansion (Y=50%), that is, X=(U / Y)-F.

[0114] CPU expansion model: Perform CPU stress testing before going online, and calculate the relationship between the amount of data produced and consumed by the message middleware and the CPU utilization rate in the cluster to obtain the number of CPU resources required after different data volume changes, that is, the CPU model calculation formula. If the amount of data produced and consumed by the message middleware changes within the 1 minute, 15 minutes, and 60 minutes time period, the analysis module will calculate the increase in data volume within the above time period and the estimated CPU utilization rate of the existing cluster exceeds 80%. The analysis module will deduce the amount of CPU expansion based on the growth of the message middleware data volume.

[0115] CPU model calculation formula: X=1.5Y, the total number of consumer node machines before expansion is Y, and the total number of consumer node machines after expansion is X=1.5Y, that is, the number of consumer machines before expansion increases by 50%, and the goal is to reduce it to below 20% to ensure stable cluster computing.

[0116] In this module, it is necessary to analyze the changes in the data volume of the Topic of the message middleware. The analysis module calculates the changes in the production and consumption of the message middleware data to trigger the resource value of the expansion. At the same time, the information of this expansion is recorded and sent to the distribution module for cluster expansion.

[0117] The cluster in this module generally refers to the cloud cluster, so the parameters in the expansion model should be adjusted according to the actual composition of the cloud cluster.

[0118] 4. The sending module receives information and implements capacity expansion:

[0119] The sending module receives the expansion information, updates the network configuration and cluster software configuration of the new machine, and sends the network configuration file and cluster software to the new machine.

[0120] 5. The client agent restarts the cluster software:

[0121] The client agent of the new machine receives the instruction from the master control module, and executes the network configuration information such as the update of the firewall list of the new machine issued by the issuing module. Then, in the next step, the cluster software configuration information issued by the issuing module is executed to update the software configuration of the new machine, restart the software of the newly added machine, and notify the issuing module at the same time that the network and cluster software configuration of the new machine have been updated successfully. The issuing module updates the cluster configuration and adds the new machine to the cluster.

[0122] 6. Determination of expansion results:

[0123] The client collects the load, memory, disk usage information of the distributed cluster and the running status information of the cluster software. The analysis module determines whether the current cluster operation is within the expected normal range. If it is within the normal range, the expansion is completed.

[0124] In summary, the present invention analyzes the change in the amount of data in the message middleware, and then analyzes the comprehensive resource usage of the current cluster load, constructs a logical judgment to trigger the relationship between the expansion condition values, and obtains a judgment on whether the cluster needs to be expanded, and then automatically expands the cluster. At the same time, the amount of resources required for the cloud cluster expansion is obtained, and the amount of resources includes the amount of storage and CPU for expansion, thereby avoiding high expansion costs and improving the efficiency of distributed cloud cluster expansion. It should be noted that the message middleware in the current industry is not the main basis for judging expansion. The present invention is based on the data in the message middleware flowing into the distributed cluster. First, according to the change in the amount of data flowing into the message middleware, a logical judgment analysis module is constructed, and then it is judged whether this change in data volume will cause the cluster to expand. At present, the traditional cluster expansion is to determine whether the cluster load exceeds the threshold, and then expand the cluster. However, the present invention uses the change of the amount of data in the message middleware to obtain the resources that need to be expanded by the analysis module, and can automatically issue software and start, saving manpower. Compared with the existing technology, the present invention has at least the following beneficial effects: At present, it is the first application in the industry to expand the distributed cloud cluster based on the change of the amount of data produced and consumed by the message middleware. Through the logical analysis of the production and consumption of the message middleware data, the automatic expansion of the cluster can be carried out quickly and timely, avoiding the high cost of expansion and improving the efficiency of distributed cloud cluster expansion;

[0125] Compared with the prior art, the main creativity lies in:

[0126] 1. Currently, traditional cluster expansion is determined based on cluster load. There is no case of expansion based on changes in the amount of data in the message middleware.

[0127] 2. The present invention provides a method and device for automatic expansion of cloud clusters based on message middleware. The method and device are based on the automated analysis of the production and consumption capabilities of all topics related to the message middleware. Through the analysis, the module can analyze the changes in the amount of data produced and consumed by the message middleware data, and predict in advance whether to expand the capacity, and then automatically expand the cluster, thereby avoiding the inefficiency of manual expansion and improving the efficiency of cluster expansion.

[0128] On the other hand, Figure 8 As shown, an embodiment of the present invention provides a cluster expansion device 800, including: a first module 810, used to obtain data volume change information of the message middleware and resource consumption data of the cluster machine; a second module 820, used to obtain resource consumption prediction data based on the resource consumption data and the data volume change information; a third module 830, used to determine expansion information based on the resource consumption target value when the resource consumption prediction data reaches a preset threshold; a fourth module 840, used to expand the cluster according to the expansion information.

[0129] In some embodiments, the device may also include: a fifth module, used to obtain a list of relationship data between data volume change information and new resource consumption data based on stress testing of cluster machines, and determine preset parameters; wherein the preset parameters include at least one of a preset threshold and a resource consumption target value.

[0130] In some embodiments, the device may further include: a sixth module, used to obtain the expansion operation results of the cluster machines after the cluster expansion is completed; a seventh module, used to complete the cluster expansion when the expansion operation results reach the expected operation range, otherwise, expand the cluster again based on the expansion operation results.

[0131] In some specific embodiments, the device of the embodiment of the present invention is applied to the following system architecture to achieve cluster expansion, and the system architecture mainly includes: client agent data collection and analysis module;

[0132] Client agent data collection: The data comes from the change of data volume in the message middleware and cluster load information, including CPU, memory, disk and other information. The collected data is processed and modeled, and stored in the HDFS of the Hadoop cluster and the MySQL database. It is then processed by Spark MLlib and Python programs and provided to the analysis module.

[0133] Analysis module: In this module, it is necessary to analyze the change in the amount of data in the message middleware and the relationship between the cluster load. A stress test is performed before the cluster is put into operation. The characteristic data such as the production volume, consumption volume, and distributed cluster resource consumption of each stage test are obtained from the change in the amount of data in the message middleware and the distributed cluster resource consumption. The change in the amount of data in the message middleware cluster and the corresponding conditional value for triggering expansion of the distributed cluster are obtained. For example, the algorithm for cluster disk expansion adopts U / (F+X)=Y, where the used value before expansion is (U), the total value before expansion is (F), the value of the resources to be added this time is (X), and the final load value of the total resources after expansion is (Y), that is, X=(U / Y)-F. In this patent, a logical analysis model is constructed based on the change in the amount of data in the message middleware, and the cluster is expanded according to the analysis module.

[0134] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0135] On the other hand, Fig. 9 As shown, an embodiment of the present invention further provides an electronic device 900, which includes at least one processor 910 and at least one memory 920 for storing at least one program; taking one processor 910 and one memory 920 as an example.

[0136] The processor 910 and the memory 920 may be connected via a bus or in other ways.

[0137] The memory 920, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory 920 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 920 may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0138] The electronic device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] Specifically, Fig.10 The structure block diagram of a computer system of an electronic device for implementing an embodiment of the present invention is schematically shown.

[0140] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0141] like Fig.10 As shown, the computer system 1000 includes a central processing unit 1001 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1002 (ROM) or the program loaded from the storage part 1008 to the random access memory 1003 (RAM). Various programs and data required for system operation are also stored in the random access memory 1003. The central processing unit 1001, the read-only memory 1002 and the random access memory 1003 are connected to each other through a bus 1004. The input / output interface 1005 (Input / Output interface, i.e., I / O interface) is also connected to the bus 1004.

[0142] The following components are connected to the input / output interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0143] In particular, according to an embodiment of the present invention, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by the central processor 1001, various functions defined in the system of the present invention are executed.

[0144] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0145] The contents of the method embodiments of the present invention are all applicable to the system embodiments. The functions specifically implemented by the system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method.

[0146] Another aspect of an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.

[0147] The contents of the method embodiments of the present invention are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0148] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] It should be noted that, although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0151] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present invention.

[0152] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0153] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0154] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0155] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by an instruction execution device, apparatus or device (such as a computer-based device, a device including a processor, or other device that can fetch instructions from an instruction execution device, apparatus or device and execute the instructions), or in conjunction with such instruction execution device, apparatus or device. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution device, apparatus or device, or in conjunction with such instruction execution device, apparatus or device.

[0156] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0157] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0158] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0159] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0160] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A cluster expansion method, characterized in that: include: Obtain the data volume change information of the message middleware and the resource consumption data of the cluster machines; Obtain resource consumption prediction data based on the resource consumption data and in combination with the data volume change information; When the resource consumption prediction data reaches a preset threshold, based on the resource consumption target value, determining capacity expansion information; The resource consumption data includes disk capacity consumption data and CPU usage, the resource consumption prediction data includes disk capacity consumption prediction data and CPU prediction usage, the preset threshold includes a first threshold and a second threshold, and the resource consumption target value includes a disk capacity consumption target value; when the resource consumption prediction data reaches the preset threshold, determining the expansion information based on the resource consumption target value includes: When the disk capacity consumption prediction data reaches the first threshold, the total disk capacity space is obtained, and the disk capacity expansion information is obtained by subtracting the total disk capacity space from the ratio of the disk capacity consumption data to the disk capacity consumption target value; and / or, when the CPU usage reaches the second threshold, obtaining the number of machines in the cluster, and obtaining machine expansion information according to the number of machines combined with a preset ratio; The cluster is expanded according to the expansion information.

2. The cluster expansion method according to claim 1, characterized in that: The obtaining of the data volume change information of the message middleware and the resource consumption data of the cluster machine includes: Acquire the data volume produced and consumed by all topics of the message middleware within a preset time period as the data volume change information of the message middleware; Obtain disk capacity consumption data of cluster machines, and obtain CPU usage rates of the cluster machines.

3. The cluster expansion method according to claim 1, characterized in that: The method further comprises: Based on the stress test of the cluster machine, a relationship data list of the data volume change information and the newly added resource consumption data is obtained, and preset parameters are determined; wherein the preset parameters include at least one of the preset threshold and the resource consumption target value.

4. The cluster expansion method according to claim 1, characterized in that: The obtaining resource consumption prediction data according to the resource consumption data and in combination with the data volume change information includes: Based on the relational data list, and according to the data volume change information, newly added resource consumption data is obtained; wherein the relational data list is obtained based on a stress test of the cluster machine; Resource consumption prediction data is obtained based on the resource consumption data and the newly added resource consumption data.

5. The cluster expansion method according to claim 1, characterized in that: The expansion information includes disk expansion information and / or machine expansion information; The performing cluster expansion according to the expansion information includes: Expand the disk capacity of the cluster machine according to the disk expansion information; And / or, configuring a new machine according to the machine expansion information, adding the configured new machine to the cluster machine, and completing the cluster machine expansion.

6. The cluster expansion method according to claim 1, characterized in that: The method further comprises: Obtaining the expansion operation result of the cluster machine after the cluster expansion is completed; When the expansion operation result reaches the expected operation range, the cluster expansion is completed; otherwise, the cluster is expanded again based on the expansion operation result.

7. A cluster expansion device, characterized in that: include: The first module is used to obtain the data volume change information of the message middleware and the resource consumption data of the cluster machine; The second module is used to obtain resource consumption prediction data based on the resource consumption data and the data volume change information; The third module is used to determine the expansion information based on the resource consumption target value when the resource consumption prediction data reaches a preset threshold; The resource consumption data includes disk capacity consumption data and CPU usage, the resource consumption prediction data includes disk capacity consumption prediction data and CPU prediction usage, the preset threshold includes a first threshold and a second threshold, and the resource consumption target value includes a disk capacity consumption target value; when the resource consumption prediction data reaches the preset threshold, determining the expansion information based on the resource consumption target value includes: When the disk capacity consumption prediction data reaches the first threshold, the total disk capacity space is obtained, and the disk capacity expansion information is obtained by subtracting the total disk capacity space from the ratio of the disk capacity consumption data to the disk capacity consumption target value; and / or, when the CPU usage reaches the second threshold, obtaining the number of machines in the cluster, and obtaining machine expansion information according to the number of machines combined with a preset ratio; The fourth module is used to expand the cluster according to the expansion information.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 6.

9. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 6 when executed by the processor.