Method and apparatus for evaluating cluster resources

Abstract

The application discloses a cluster resource evaluation method and device, and relates to the technical field of computers. The method comprises the following steps: acquiring the event processing quantity of a target cluster, and determining the total storage space of the target cluster based on the event processing quantity, wherein the target cluster at least comprises one of a distributed processing cluster, a distributed event exchange cluster and a file search cluster; calculating the storage node quantity of the target cluster according to the total storage space and the node storage quantity of the target cluster; acquiring the event processing quantity peak value of the target cluster, and calculating the computing node quantity of the target cluster based on the event processing quantity peak value of the target cluster and the node calculation quantity; and evaluating the cluster resource quantity of the target cluster based on the storage node quantity and the computing node quantity.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for evaluating cluster resources. Background Technology

[0002] As companies continue to grow, assessing their cluster resources becomes particularly important in order to determine their internal resource situation.

[0003] Currently, the evaluation of cluster resources typically involves analyzing historical data to determine the overall processing capacity of the entire cluster, then calculating the average processing capacity based on multiple assets and devices, and finally arriving at the evaluation result based on simple division. However, in practical applications, the existing evaluation method is too simplistic and crude. If the average processing capacity of the assets and devices or the overall processing capacity of the entire cluster is calculated incorrectly, the accuracy of the evaluation results will be affected, resulting in a low accuracy of current cluster resource evaluations. Summary of the Invention

[0004] This application provides a method and apparatus for evaluating cluster resources, with the main objective of implementing a method for evaluating cluster resources to address the problem of low accuracy in existing cluster resource evaluation results.

[0005] To address the aforementioned technical problems, the embodiments of this application provide the following technical solutions:

[0006] Firstly, this application provides a method for evaluating cluster resources, the method comprising:

[0007] Obtain the number of events processed by the target cluster, and determine the total storage space of the target cluster based on the number of events processed. The target cluster includes at least one of the following: a distributed processing cluster, a distributed event communication cluster, and a file search cluster.

[0008] Based on the total storage space and the node storage capacity of the target cluster, the number of storage nodes in the target cluster is calculated. The node storage capacity is used to characterize the maximum storage capacity of a single node in the target cluster during the data storage process. The node storage capacity is determined based on the performance of the asset equipment corresponding to each node in the cluster.

[0009] The peak number of events processed by the target cluster is obtained, and the number of computing nodes in the target cluster is calculated based on the peak number of events processed by the target cluster and the node computing power. The node computing power is used to characterize the maximum processing volume of a single node in the target cluster during the data processing process. The node computing power is determined based on the performance of the asset equipment corresponding to each node in the cluster.

[0010] The number of cluster resources for the target cluster is assessed based on the number of storage nodes and the number of compute nodes.

[0011] Optionally, the number of events processed includes the daily processing volume;

[0012] The step of obtaining the number of events processed by the target cluster and determining the total storage space of the target cluster based on the number of events processed includes:

[0013] The daily storage capacity of the target cluster is calculated based on the daily processing volume and the event data volume, wherein the event data volume is used to characterize the data volume of each task being processed.

[0014] The total storage space is determined based on the daily storage volume and the target number of days.

[0015] Optionally, the daily processing volume includes daily traffic information and daily security log information; the daily traffic information is used to characterize the number of traffic tasks processed by the target cluster each day; the daily security log information is used to characterize the number of log tasks processed by the target cluster each day; the event data volume includes unit traffic data volume and unit log data volume.

[0016] The step of calculating the daily storage capacity of the target cluster based on the daily processing volume and the event data volume includes:

[0017] The unit traffic data volume and the unit log data volume are obtained respectively;

[0018] Calculate the daily traffic processing volume based on the daily traffic information and the unit traffic data volume;

[0019] Calculate the daily log processing volume based on the daily security log information and the unit log data volume;

[0020] The daily storage volume is determined based on the daily traffic processing volume and the daily log processing volume.

[0021] Optionally, obtaining the unit traffic data volume and the unit log data volume respectively includes:

[0022] The cluster type of the target cluster is determined, and the number of data replicas and the data compression ratio are determined based on the cluster type. The number of data replicas is used to represent the number of backups when storing data, and the data compression ratio is used to represent the ratio of the amount of data after compression to the amount of data before compression when storing data.

[0023] Historical traffic data and historical log data are obtained respectively. The average data volume of the traffic task is determined based on the historical traffic data, and the average data volume of the log task is determined based on the historical log data.

[0024] The unit traffic data volume is calculated based on the number of data replicas, the data compression ratio, and the average data volume of the traffic task.

[0025] The unit log data volume is calculated based on the number of data replicas, the data compression ratio, and the average data volume of the log task.

[0026] Optionally, obtaining the peak event processing count of the target cluster, and calculating the number of computing nodes in the target cluster based on the peak event processing count and the node computing load, includes:

[0027] When it is determined that the target cluster includes the distributed processing cluster and the file search cluster, the node ratio relationship between the distributed processing cluster and the file search cluster is determined. The node ratio relationship includes a first ratio and a second ratio, wherein the first ratio corresponds to the distributed processing cluster and the second ratio corresponds to the file search cluster.

[0028] The sum of the first ratio and the second ratio equals 1;

[0029] Based on the peak number of events processed and the computational load of nodes, calculate the total number of computing nodes in the target cluster.

[0030] The number of computing nodes in the distributed processing cluster is determined based on the total number of computing nodes and the first ratio.

[0031] The number of computing nodes in the file search cluster is determined based on the total number of computing nodes and the second ratio.

[0032] Optionally, the step of evaluating the cluster resource quantity of the target cluster based on the number of storage nodes and the number of computing nodes includes:

[0033] The larger of the number of storage nodes and the number of computing nodes is selected as the number of running nodes corresponding to the target cluster, wherein the number of running nodes is used to characterize the number of asset devices that perform data storage and data processing in the target cluster;

[0034] The number of management nodes is determined based on the number of running nodes, wherein the number of management nodes is used to characterize the number of asset devices used for management in the target cluster, and the number of management nodes is in a specific ratio with the number of running nodes, the specific ratio being determined based on the target cluster;

[0035] The number of cluster resources is determined based on the number of running nodes and the number of management nodes.

[0036] Optionally, the number of cluster resources includes a first cluster resource quantity, a second cluster resource quantity, and a third cluster resource quantity; wherein, the first cluster resource quantity corresponds to the distributed processing cluster; the second cluster resource quantity corresponds to the distributed event communication cluster; and the third cluster resource quantity corresponds to the file search cluster.

[0037] After determining the number of cluster resources based on the number of running nodes and the number of management nodes, the method further includes:

[0038] When it is determined that the target cluster includes all three components: the distributed processing cluster, the distributed event communication cluster, and the file search cluster, the total number of resources in the target cluster is calculated based on the resource quantities of the first cluster, the second cluster, and the third cluster.

[0039] Secondly, this application also provides a cluster resource evaluation device, comprising:

[0040] A determining unit is used to obtain the number of events processed by the target cluster and determine the total storage space of the target cluster based on the number of events processed. The target cluster includes at least one of a distributed processing cluster, a distributed event communication cluster, and a file search cluster.

[0041] The first computing unit is used to calculate the number of storage nodes in the target cluster based on the total storage space and the node storage capacity of the target cluster. The node storage capacity is used to characterize the maximum storage capacity of a single node in the target cluster during the data storage process. The node storage capacity is determined based on the performance of the asset equipment corresponding to each node in the cluster.

[0042] The second computing unit is used to obtain the peak number of events processed by the target cluster, and to calculate the number of computing nodes in the target cluster based on the peak number of events processed by the target cluster and the node computing power. The node computing power is used to characterize the maximum processing volume of a single node in the target cluster during the data processing process. The node computing power is determined based on the performance of the asset equipment corresponding to each node in the cluster.

[0043] An evaluation unit is used to evaluate the number of cluster resources of the target cluster based on the number of storage nodes and the number of computing nodes.

[0044] Optionally, the number of events processed includes the daily processing volume;

[0045] The determining unit is further configured to calculate the daily storage volume of the target cluster based on the daily processing volume and the event data volume; and to determine the total storage space based on the daily storage volume and the target number of days, wherein the event data volume is used to characterize the data volume of each task being processed.

[0046] Optionally, the daily processing volume includes daily traffic information and daily security log information; the daily traffic information is used to characterize the number of traffic tasks processed by the target cluster each day; the daily security log information is used to characterize the number of log tasks processed by the target cluster each day; the event data volume includes unit traffic data volume and unit log data volume.

[0047] The determining unit includes:

[0048] The acquisition module is used to acquire the unit traffic data volume and the unit log data volume respectively;

[0049] The first calculation module is used to calculate the daily traffic processing volume based on the daily traffic information and the unit traffic data volume.

[0050] The second calculation module is used to calculate the daily log processing volume based on the daily security log information and the unit log data volume.

[0051] The determination module is used to determine the daily storage volume based on the daily traffic processing volume and the daily log processing volume.

[0052] Optionally, the acquisition module includes:

[0053] The determination submodule is used to determine the cluster type of the target cluster, and to determine the number of data replicas and the data compression ratio based on the cluster type. The number of data replicas is used to characterize the number of backups when storing data, and the data compression ratio is used to characterize the ratio of the amount of data after compression to the amount of data before compression when storing data.

[0054] The acquisition submodule is used to acquire historical traffic data and historical log data respectively, and determine the average data volume of the traffic task based on the historical traffic data, and determine the average data volume of the log task based on the historical log data.

[0055] The first calculation submodule is used to calculate the unit traffic data volume based on the number of data replicas, the data compression ratio, and the average data volume of the traffic task.

[0056] The second calculation submodule is used to calculate the unit log data volume based on the number of data replicas, the data compression ratio, and the average data volume of the log task.

[0057] Optionally, the second computing unit includes:

[0058] The first determining module is used to determine the node ratio relationship between the distributed processing cluster and the file search cluster when it is determined that the target cluster includes the distributed processing cluster and the file search cluster. The node ratio relationship includes a first ratio and a second ratio, wherein the first ratio corresponds to the distributed processing cluster and the second ratio corresponds to the file search cluster; the sum of the first ratio and the second ratio is equal to 1.

[0059] The calculation module is used to calculate the total number of computing nodes in the target cluster based on the peak number of events processed and the computing load of the nodes.

[0060] The second determining module is used to determine the number of computing nodes in the distributed processing cluster based on the total number of computing nodes and the first ratio.

[0061] The third determining module is used to determine the number of computing nodes in the file search cluster based on the total number of computing nodes and the second ratio.

[0062] Optionally, the evaluation unit includes:

[0063] The selection module is used to select the larger one between the number of storage nodes and the number of computing nodes as the number of running nodes corresponding to the target cluster, wherein the number of running nodes is used to characterize the number of asset devices that perform data storage and data processing in the target cluster;

[0064] The first determining module is used to determine the number of management nodes based on the number of running nodes, wherein the number of management nodes is used to characterize the number of asset devices used for management in the target cluster, and the number of management nodes is in a specific ratio with the number of running nodes, the specific ratio being determined based on the target cluster;

[0065] The second determining module is used to determine the number of cluster resources based on the number of running nodes and the number of management nodes.

[0066] Optionally, the number of cluster resources includes a first cluster resource quantity, a second cluster resource quantity, and a third cluster resource quantity; wherein, the first cluster resource quantity corresponds to the distributed processing cluster; the second cluster resource quantity corresponds to the distributed event communication cluster; and the third cluster resource quantity corresponds to the file search cluster.

[0067] The evaluation unit further includes:

[0068] The calculation module is used to calculate the total number of resources of the target cluster based on the resource quantity of the first cluster, the resource quantity of the second cluster, and the resource quantity of the third cluster when it is determined that the target cluster includes all three: the distributed processing cluster, the distributed event communication cluster, and the file search cluster.

[0069] Thirdly, embodiments of this application provide a storage medium including a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to execute the cluster resource evaluation method of the terminal device described in the first aspect.

[0070] Fourthly, embodiments of this application provide a cluster resource evaluation apparatus, the apparatus including a storage medium; and one or more processors, the storage medium being coupled to the processors, the processors being configured to execute program instructions stored in the storage medium; the program instructions, when executed, perform the cluster resource evaluation method for the terminal device described in the first aspect.

[0071] By employing the above-described technical solution, the technical solution provided in this application has at least the following advantages:

[0072] This application provides a method and apparatus for evaluating cluster resources. The method first obtains the number of events processed by a target cluster and determines the total storage space of the target cluster based on the number of events processed. Then, it calculates the number of storage nodes in the target cluster based on the total storage space and the node storage capacity of the target cluster. Next, it obtains the peak number of events processed by the target cluster and calculates the number of computing nodes in the target cluster based on the peak number of events processed and the node computing capacity. Finally, it evaluates the cluster resources of the target cluster based on the number of storage nodes and the number of computing nodes, thereby realizing the cluster resource evaluation function. Compared with the prior art, this application considers the difference between computing power and storage capacity during the cluster resource evaluation process and evaluates cluster resources based on the number of storage nodes and computing nodes. This effectively avoids the problem of insufficient accuracy in evaluating the processing capacity of each asset device due to differences in processing capacity under different operating conditions. Therefore, it effectively solves the problem of low accuracy in evaluation results caused by the existing relatively simple and crude evaluation methods, and improves the accuracy of the cluster resource evaluation process. Furthermore, since the target cluster includes at least one of a distributed processing cluster, a distributed event communication cluster, and a file search cluster, this application can consider multiple different types of clusters for evaluation, avoiding the problem of poor evaluation accuracy caused by confusing multiple different clusters. Simultaneously, the node storage capacity is used to characterize the maximum storage capacity of a single node in the target cluster during data storage. The node storage capacity is determined based on the performance of the asset devices corresponding to each node in the cluster. The node computing capacity is used to characterize the maximum processing capacity of a single node in the target cluster during data processing. The node computing capacity is determined based on the performance of the asset devices corresponding to each node in the cluster. This ensures that when evaluating cluster resources, the different processing capabilities of each asset device in the storage and computing processes can be determined based on the actual performance of the asset devices corresponding to each node in the cluster. This ensures accurate evaluation of the storage and computing capabilities of asset devices after prolonged use and performance degradation, thereby guaranteeing the accuracy of evaluating the number of resources in the cluster based on storage and computing capabilities.

[0073] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0074] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein:

[0075] Figure 1 A flowchart illustrating a method for evaluating cluster resources provided in an embodiment of this application is shown;

[0076] Figure 2 A flowchart of another cluster resource evaluation method provided in an embodiment of this application is shown;

[0077] Figure 3 A block diagram illustrating the composition of a cluster resource evaluation apparatus provided in an embodiment of this application is shown.

[0078] Figure 4 A block diagram of another cluster resource evaluation apparatus provided in an embodiment of this application is shown. Detailed Implementation

[0079] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0080] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0081] This application provides a method for evaluating cluster resources, specifically as follows: Figure 1 As shown, the method includes:

[0082] 101. Obtain the number of events processed by the target cluster, and determine the total storage space of the target cluster based on the number of events processed.

[0083] The target cluster includes at least one of the following: a distributed processing cluster, a distributed event communication cluster, and a file search cluster.

[0084] In this embodiment, the target cluster can be understood as the cluster whose resources the user wants to confirm. Generally, enterprises typically use distributed networking control of multiple asset devices. In this case, multiple asset devices are equivalent to nodes in the network, which can coordinate and control based on a distributed system management mechanism, jointly utilizing the processing performance of each node in the network to form a cluster.

[0085] In this embodiment, the target cluster can be categorized into several common asset management clusters, including: The first is a distributed processing cluster, namely Hadoop. Hadoop is a common, reliable, scalable, distributed computing open-source software. It allows for distributed processing of large datasets across computer clusters using a simple programming model. Its design scales from a single server to thousands of machines, each providing local computing and storage; each server and specific machine can be considered an enterprise asset. The second is a distributed event communication cluster, namely Kafka. Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, stream analytics, data integration, and mission-critical applications. The third is a document search cluster, namely ES (Elasticsearch, a distributed search engine). ES is an open-source, distributed full-text search engine built on Lucene with a REST (Representational State Transfer) interface. ES is also a distributed document database where each field is indexed and searchable, capable of scaling to hundreds of servers for storage and processing petabyte-level (PB) data.

[0086] Since the assets and devices on each node form a unified whole in any of the aforementioned clusters, the amount of data a cluster can process reflects its data storage capacity. Therefore, in this embodiment, the total storage space of the current target cluster is determined by the number of events processed. Of course, in specific applications, this number of events processed can be the amount of data processed daily or monthly; no specific limitation is made here, and it can be selected based on the user's needs.

[0087] 102. Calculate the number of storage nodes in the target cluster based on the total storage space and the node storage capacity of the target cluster.

[0088] The node storage capacity is used to characterize the maximum storage capacity of a single node in the target cluster during the data storage process. The node storage capacity is determined based on the performance of the asset equipment corresponding to each node in the cluster.

[0089] Based on the description of the steps above, it can be seen that in a cluster, the nodes form a whole through a distributed system. Thus, after determining the total storage space of the entire target cluster, the number of nodes in the current target cluster can be deduced by the data storage capacity of each node.

[0090] In this embodiment, the node storage capacity can be understood as the maximum storage capacity of the asset device corresponding to a single node during the data storage process. Since the node storage capacity can reflect the actual storage capacity of the asset device corresponding to the node during actual application, it can ensure that the node storage capacity determined based on this represents the actual storage capacity, and is different from the storage capacity indicated by the asset device at the factory. This solves the problem of the accuracy of the node storage capacity determination when there is a difference between the storage capacity indicated by the factory and the actual storage capacity after the asset device has been used for a long time. This lays the foundation for the accuracy of the overall cluster resource evaluation results.

[0091] 103. Obtain the peak number of events processed by the target cluster, and calculate the number of computing nodes in the target cluster based on the peak number of events processed by the target cluster and the computing load of the nodes.

[0092] The node computational load is used to characterize the maximum processing volume of a single node in the target cluster during the data processing process. The node computational load is determined based on the performance of the asset equipment corresponding to each node in the cluster.

[0093] In practical applications, the actual functions of different asset devices in a cluster may differ. For example, some asset devices are mainly used for data processing, while others are mainly used for data storage. Therefore, it is necessary to determine the nodes involved in data processing during the evaluation of cluster resources. In this embodiment, the number of computing nodes can be determined based on the peak value of the number of events processed. It should be noted that, as described above, the number of events processed can be divided based on the time period selected by the user. This can lead to fluctuations in the amount of data processed in different time periods. To ensure that all nodes with computing capabilities participate in determining the number of computing nodes and to avoid some nodes not participating in the processing, thus affecting the determination of the number of computing nodes, the method in this embodiment is to select the highest peak value from multiple event processing values ​​over a relatively long period. This ensures that all nodes with computing capabilities participate, thereby guaranteeing the accuracy of the determination of the number of computing nodes.

[0094] Furthermore, the determination of node computational load is based on the performance of the corresponding asset equipment for each node. In other words, similar to the node storage load in the aforementioned steps, the computational load during real-time processing is used, rather than the computational load indicated by the asset equipment at the factory. This ensures that the processing performance of the asset equipment will decrease after a long period of use, thus preventing a difference from the computational load indicated by the factory. This, in turn, lays the foundation for the accuracy of the overall cluster resource assessment results.

[0095] 104. Evaluate the number of cluster resources of the target cluster based on the number of storage nodes and the number of compute nodes.

[0096] Once the number of storage nodes and compute nodes is determined, the total number of asset devices in the target cluster capable of storing and processing data is also determined. Therefore, the resource requirements of the cluster can be assessed based on these two numbers. Because the asset devices on different nodes for processing and storing data are taken into account, the accuracy of the assessment results is ensured.

[0097] This application provides a method for evaluating cluster resources. First, it obtains the number of events processed by a target cluster and determines the total storage space of the target cluster based on the number of events processed. Then, it calculates the number of storage nodes in the target cluster based on the total storage space and the node storage capacity of the target cluster. Next, it obtains the peak number of events processed by the target cluster and calculates the number of computing nodes in the target cluster based on the peak number of events processed and the node computing capacity. Finally, it evaluates the cluster resources of the target cluster based on the number of storage nodes and the number of computing nodes, thereby realizing the cluster resource evaluation function. Compared with existing technologies, this application considers the difference between computing power and storage capacity during the cluster resource evaluation process and evaluates cluster resources based on the number of storage nodes and computing nodes. This effectively avoids the problem of insufficient accuracy in evaluating the processing capacity of each asset device due to differences in processing capacity under different operating conditions. Therefore, it effectively solves the problem of low accuracy in evaluation results caused by existing, relatively simple and crude evaluation methods, and improves the accuracy of the cluster resource evaluation process. Furthermore, since the target cluster includes at least one of a distributed processing cluster, a distributed event communication cluster, and a file search cluster, this application can consider multiple different types of clusters for evaluation, avoiding the problem of poor evaluation accuracy caused by confusing multiple different clusters. Simultaneously, the node storage capacity is used to characterize the maximum storage capacity of a single node in the target cluster during data storage. The node storage capacity is determined based on the performance of the asset devices corresponding to each node in the cluster. The node computing capacity is used to characterize the maximum processing capacity of a single node in the target cluster during data processing. The node computing capacity is determined based on the performance of the asset devices corresponding to each node in the cluster. This ensures that when evaluating cluster resources, the different processing capabilities of each asset device in the storage and computing processes can be determined based on the actual performance of the asset devices corresponding to each node in the cluster. This ensures accurate evaluation of the storage and computing capabilities of asset devices after prolonged use and performance degradation, thereby guaranteeing the accuracy of evaluating the number of resources in the cluster based on storage and computing capabilities.

[0098] To provide a more detailed explanation, this application provides another method for evaluating cluster resources, as detailed below. Figure 2 As shown, the method includes:

[0099] 201. Obtain the number of events processed by the target cluster, and determine the total storage space of the target cluster based on the number of events processed.

[0100] The target cluster includes at least one of the following: a distributed processing cluster, a distributed event communication cluster, and a file search cluster.

[0101] Specifically, since the number of events processed by the target cluster can be selected based on different time periods such as daily, monthly, and yearly, in order to avoid changes in cluster resource transmission due to a long time period selected by the user, this embodiment can adopt daily processing. Thus, the number of events processed can specifically include the daily processing volume.

[0102] Therefore, this step can specifically include the following when executed:

[0103] Step 1: Calculate the daily storage capacity of the target cluster based on the daily processing volume and the event data volume, whereby the event data volume is used to characterize the data volume of each task being processed.

[0104] Step 2: Determine the total storage space based on the daily storage amount and the target number of days.

[0105] In step 1, once the daily processing volume is determined, the number of tasks to be processed each day is also determined. Multiplying the data volume of each task by this daily volume determines the total data storage volume of the cluster each day, i.e., the amount of data stored per unit date (day). Then, in step 2, the total storage volume for a given period of time can be determined based on this daily storage volume and the desired time length. In other words, by combining the daily storage volume with the target number of days, the total storage space of the target cluster can be determined. This total storage space represents the total amount of data that the entire cluster can store within the target number of days, reflecting the overall storage capacity of the cluster.

[0106] Furthermore, since the storage task is divided into two types—traffic and logs—the storage capacity needs to be determined separately for each of these two data types.

[0107] Therefore, the daily processing volume may include daily traffic information and daily security log information; the daily traffic information is used to characterize the number of traffic tasks processed by the target cluster each day; the daily security log information is used to characterize the number of log tasks processed by the target cluster each day; the event data volume includes unit traffic data volume and unit log data volume.

[0108] Based on this, step 1 above, which calculates the daily storage capacity of the target cluster according to the daily processing volume and the event data volume, includes:

[0109] Step A: Obtain the unit traffic data volume and the unit log data volume respectively;

[0110] Step B: Calculate the daily traffic processing volume based on the daily traffic information and the unit traffic data volume;

[0111] Step C: Calculate the daily log processing volume based on the daily security log information and the unit log data volume;

[0112] Step D: Determine the daily storage volume based on the daily traffic processing volume and the daily log processing volume.

[0113] In this embodiment, since the daily storage capacity needs to consider both traffic and log data, the process of determining the daily storage capacity first involves obtaining the data volume corresponding to each traffic task (i.e., the unit traffic data volume) and the data volume corresponding to each log task (i.e., the unit log data volume). Then, based on step B, multiplying the number of traffic tasks processed each day by the unit traffic data volume yields the total data volume for all traffic tasks processed each day. Similarly, multiplying the number of log tasks processed each day by the unit log data volume determines the total data volume for all log tasks processed each day. Finally, the daily storage capacity is obtained by summing the data volume of log tasks processed each day (daily traffic processing volume) and the data volume of traffic tasks processed each day (daily log processing volume).

[0114] Furthermore, in practical applications, different clusters differ in specific details when storing data, such as compression ratio and number of replicas. The number of data replicas represents the number of backups made during data storage, and the data compression ratio represents the ratio of the amount of data after compression to the amount of data before compression.

[0115] Based on this, obtaining the unit traffic data volume and the unit log data volume in step A above includes:

[0116] First, determine the cluster type of the target cluster, and then determine the number of data replicas and the data compression ratio based on the cluster type;

[0117] Then, historical traffic data and historical log data are obtained respectively, and the average data volume of the traffic task is determined based on the historical traffic data, and the average data volume of the log task is determined based on the historical log data.

[0118] Then, the unit traffic data volume is calculated based on the number of data replicas, the data compression ratio, and the average data volume of the traffic task;

[0119] Finally, the unit log data volume is calculated based on the number of data replicas, the data compression ratio, and the average data volume of the log task.

[0120] Specifically, the method described in this step is illustrated in the following example:

[0121] For example, when the target cluster is identified as a distributed processing cluster, specifically Hadoop, since Hadoop requires two backup files during data storage, the number of data replicas can be determined to be three (two backup files plus one original file, totaling three). Furthermore, based on the nature of Hadoop, the compressed data size is one-quarter of the uncompressed size during storage, thus the compression ratio can be determined to be 0.25. Then, if the average log file size is determined to be 10kb based on historical log data, the unit log file size would be: 10kb * 0.25 * 3 = 7.5kb.

[0122] 202. Calculate the number of storage nodes in the target cluster based on the total storage space and the node storage capacity of the target cluster.

[0123] The node storage capacity is used to characterize the maximum storage capacity of a single node in the target cluster during the data storage process. The node storage capacity is determined based on the performance of the asset equipment corresponding to each node in the cluster.

[0124] In calculating the number of storage nodes based on total storage space and node storage capacity, the accuracy of the node storage capacity directly affects the accuracy of the final determined number of storage nodes. Therefore, in this embodiment, the node storage capacity can be understood as the actual performance of the asset device corresponding to each actual node during storage by obtaining its runtime attribute parameters. Thus, the node storage capacity can reflect the actual runtime storage capacity of all nodes in the current cluster. This ensures that the node storage capacity can reflect the actual state of the asset device, avoiding the situation where the storage performance is still based on the factory specifications when the asset device's performance degrades after long-term use, which leads to real-time discrepancies. Consequently, the overall accuracy of the cluster resource quantity assessment can be ensured.

[0125] 203. Obtain the peak number of events processed by the target cluster, and calculate the number of computing nodes in the target cluster based on the peak number of events processed by the target cluster and the computing load of the nodes.

[0126] The node computational load is used to characterize the maximum processing volume of a single node in the target cluster during the data processing process. The node computational load is determined based on the performance of the asset equipment corresponding to each node in the cluster.

[0127] Specifically, this step may include the following during execution:

[0128] When it is determined that the target cluster includes the distributed processing cluster and the file search cluster, the node ratio relationship between the distributed processing cluster and the file search cluster is determined. The node ratio relationship includes a first ratio and a second ratio, wherein the first ratio corresponds to the distributed processing cluster and the second ratio corresponds to the file search cluster; the sum of the first ratio and the second ratio is equal to 1.

[0129] Based on the peak number of events processed and the computational load of nodes, calculate the total number of computing nodes in the target cluster.

[0130] The number of computing nodes in the distributed processing cluster is determined based on the total number of computing nodes and the first ratio.

[0131] The number of computing nodes in the file search cluster is determined based on the total number of computing nodes and the second ratio.

[0132] Since Hadoop and Elasticsearch are used collaboratively in practical applications, a certain proportional relationship exists between them when determining the number of nodes. Therefore, in this step, when calculating the total number of computing nodes in the target cluster based on the peak number of events processed and the computing load per node, it is necessary to consider the ratio between the two in determining the overall data processing capacity of the cluster. In other words, the specific number of nodes in the two types of clusters (Hadoop and Elasticsearch) should be determined according to this proportional relationship. Generally, Hadoop can account for 3 / 5 of the total number of computing nodes, while Elasticsearch can account for the remaining 2 / 5.

[0133] In addition, when the cluster type is a distributed event communication cluster, since this distributed communication cluster, i.e. Kafka, is used independently, the total number of compute nodes can be directly determined as the number of compute nodes in the Kafka cluster.

[0134] 204. Evaluate the number of cluster resources of the target cluster based on the number of storage nodes and the number of compute nodes.

[0135] In practical applications, not all nodes perform both storage and data processing during the evaluation. This means there will be a discrepancy between the determined number of storage nodes and the number of compute nodes. Due to the nature of computer systems, some nodes may only perform processing (compute nodes), while others may only perform storage (storage nodes), and still others may perform both (both storage and compute nodes). Therefore, in determining the cluster resource quantity of the target cluster based on the number of storage nodes and compute node data, it is necessary to select the larger of these two quantities, which is closer to the actual cluster resource quantity. Therefore, this step may include:

[0136] The larger of the number of storage nodes and the number of computing nodes is selected as the number of running nodes corresponding to the target cluster, wherein the number of running nodes is used to characterize the number of asset devices that perform data storage and data processing in the target cluster;

[0137] The number of management nodes is determined based on the number of running nodes, wherein the number of management nodes is used to characterize the number of asset devices used for management in the target cluster, and the number of management nodes is in a specific ratio with the number of running nodes, the specific ratio being determined based on the target cluster;

[0138] The number of cluster resources is determined based on the number of running nodes and the number of management nodes.

[0139] In practical applications, as a type of distributed system, any of the above-mentioned clusters not only have nodes that perform the aforementioned storage and data processing (i.e., running nodes), but also nodes that manage each node (i.e., management nodes). There is a certain relationship between the number of management nodes and the number of running nodes. For example, when the number of running nodes is 20, the number of management nodes is generally 2 to 3, and the number of management nodes will not approach or exceed the number of running nodes.

[0140] Therefore, in this step, by determining the number of management nodes based on the number of running nodes, and by determining the number of cluster resources based on the number of running nodes and the number of management nodes, we can ensure that the asset devices corresponding to the management nodes are not overlooked, thereby further improving the accuracy of assessing the number of cluster resources.

[0141] Furthermore, in practical applications, the three types of clusters mentioned above—Hadoop clusters, Kafka clusters, and Elasticsearch clusters—may not be used in isolation, but rather all three may be used simultaneously. In such cases, the evaluation of cluster resource quantity requires determining the cluster resource quantity of each of these three clusters separately and then summing them up to obtain the overall evaluation result of the cluster resource quantity of all clusters in the actual enterprise.

[0142] Based on this, the number of cluster resources includes a first cluster resource quantity, a second cluster resource quantity, and a third cluster resource quantity; wherein, the first cluster resource quantity corresponds to the distributed processing cluster; the second cluster resource quantity corresponds to the distributed event communication cluster; and the third cluster resource quantity corresponds to the file search cluster.

[0143] After determining the number of cluster resources by considering the number of running nodes and the number of management nodes in the aforementioned steps, the method further includes:

[0144] When it is determined that the target cluster includes all three components: the distributed processing cluster, the distributed event communication cluster, and the file search cluster, the total number of resources in the target cluster is calculated based on the resource quantities of the first cluster, the second cluster, and the third cluster.

[0145] For example, the result of this step can be: the total number of enterprise cluster resources is actually the sum of the number of Hadoop cluster resources (first cluster resources), the number of Kafka cluster resources (second cluster resources), and the number of ES cluster resources (third cluster resources).

[0146] Furthermore, as a response to the above Figure 1 and Figure 2 In addition to the implementation of the method shown, another embodiment of this application also provides a cluster resource evaluation device. This cluster resource evaluation device embodiment corresponds to the foregoing method embodiment. For ease of reading, this cluster resource evaluation device embodiment will not repeat the details of the foregoing method embodiment, but it should be understood that the device in this embodiment can correspondingly implement all the contents of the foregoing method embodiment. Specifically, as follows... Figure 3 As shown, the cluster resource evaluation device includes:

[0147] The determining unit 31 can be used to obtain the number of events processed by the target cluster and determine the total storage space of the target cluster based on the number of events processed. The target cluster includes at least one of a distributed processing cluster, a distributed event communication cluster, and a file search cluster.

[0148] The first computing unit 32 can be used to calculate the number of storage nodes in the target cluster based on the total storage space and the node storage amount of the target cluster. The node storage amount can be used to characterize the maximum storage amount of a single node in the target cluster during the data storage process. The node storage amount is determined based on the performance of the asset equipment corresponding to each node in the cluster.

[0149] The second computing unit 33 can be used to obtain the peak number of events processed by the target cluster, and calculate the number of computing nodes of the target cluster based on the peak number of events processed by the target cluster and the node computing power. The node computing power can be used to characterize the maximum processing volume of a single node in the target cluster during the data processing process. The node computing power is determined based on the performance of the asset equipment corresponding to each node in the cluster.

[0150] The evaluation unit 34 can be used to evaluate the number of cluster resources of the target cluster based on the number of storage nodes and the number of computing nodes.

[0151] Furthermore, such as Figure 4 As shown, the number of events processed includes the daily processing volume;

[0152] The determining unit 31 can also be used to calculate the daily storage volume of the target cluster based on the daily processing volume and the event data volume; and to determine the total storage space based on the daily storage volume and the target number of days, wherein the event data volume can be used to characterize the data volume of each task being processed.

[0153] Furthermore, such as Figure 4 As shown, the daily processing volume includes daily traffic information and daily security log information; the daily traffic information is used to characterize the number of traffic tasks processed by the target cluster each day; the daily security log information can be used to characterize the number of log tasks processed by the target cluster each day; the event data volume includes unit traffic data volume and unit log data volume.

[0154] The determining unit 31 includes:

[0155] The acquisition module 311 can be used to acquire the unit traffic data volume and the unit log data volume respectively;

[0156] The first calculation module 312 can be used to calculate the daily traffic processing volume based on the daily traffic information and the unit traffic data volume.

[0157] The second calculation module 313 can be used to calculate the daily log processing volume based on the daily security log information and the unit log data volume.

[0158] The determination module 314 can be used to determine the daily storage volume based on the daily traffic processing volume and the daily log processing volume.

[0159] Furthermore, such as Figure 4 As shown, the acquisition module 311 includes:

[0160] The determination submodule 3111 can be used to determine the cluster type of the target cluster, and determine the number of data replicas and the data compression ratio based on the cluster type. The number of data replicas is used to characterize the number of backups when storing data, and the data compression ratio can be used to characterize the ratio of the amount of data after compression to the amount of data before compression when storing data.

[0161] The acquisition submodule 3112 can be used to acquire historical traffic data and historical log data respectively, and determine the average data volume of the traffic task based on the historical traffic data, and determine the average data volume of the log task based on the historical log data.

[0162] The first calculation submodule 3113 can be used to calculate the unit traffic data volume based on the number of data replicas, the data compression ratio, and the average data volume of the traffic task.

[0163] The second calculation submodule 3114 can be used to calculate the unit log data volume based on the number of data replicas, the data compression ratio, and the average data volume of the log task.

[0164] Furthermore, such as Figure 4 As shown, the second computing unit 33 includes:

[0165] The first determining module 331 can be used to determine the node ratio relationship between the distributed processing cluster and the file search cluster when it is determined that the target cluster includes the distributed processing cluster and the file search cluster. The node ratio relationship includes a first ratio and a second ratio, wherein the first ratio corresponds to the distributed processing cluster and the second ratio corresponds to the file search cluster; the sum of the first ratio and the second ratio is equal to 1.

[0166] The calculation module 332 can be used to calculate the total number of computing nodes in the target cluster based on the peak number of events processed and the computing volume of nodes.

[0167] The second determining module 333 can be used to determine the number of computing nodes in the distributed processing cluster based on the total number of computing nodes and the first ratio.

[0168] The third determining module 334 can be used to determine the number of computing nodes in the file search cluster based on the total number of computing nodes and the second ratio.

[0169] Furthermore, such as Figure 4 As shown, the evaluation unit 34 includes:

[0170] The selection module 341 can be used to select the larger one between the number of storage nodes and the number of computing nodes as the number of running nodes corresponding to the target cluster, wherein the number of running nodes can be used to characterize the number of asset devices that perform data storage and data processing in the target cluster.

[0171] The first determining module 342 can be used to determine the number of management nodes based on the number of running nodes, wherein the number of management nodes can be used to characterize the number of asset devices used for management in the target cluster, and the number of management nodes has a specific ratio with the number of running nodes, the specific ratio being determined based on the target cluster;

[0172] The second determining module 343 can be used to determine the number of cluster resources based on the number of running nodes and the number of management nodes.

[0173] Furthermore, such as Figure 4 As shown, the number of cluster resources includes a first cluster resource quantity, a second cluster resource quantity, and a third cluster resource quantity; wherein, the first cluster resource quantity corresponds to the distributed processing cluster; the second cluster resource quantity corresponds to the distributed event communication cluster; and the third cluster resource quantity corresponds to the file search cluster.

[0174] The evaluation unit 34 further includes:

[0175] The calculation module 344 can be used to calculate the total number of resources of the target cluster based on the resource quantity of the first cluster, the resource quantity of the second cluster, and the resource quantity of the third cluster when it is determined that the target cluster includes all three: the distributed processing cluster, the distributed event communication cluster, and the file search cluster.

[0176] This application provides a method and apparatus for evaluating cluster resources. The method first obtains the number of events processed by a target cluster and determines the total storage space of the target cluster based on the number of events processed. Then, it calculates the number of storage nodes in the target cluster based on the total storage space and the node storage capacity of the target cluster. Next, it obtains the peak number of events processed by the target cluster and calculates the number of computing nodes in the target cluster based on the peak number of events processed and the node computing capacity. Finally, it evaluates the cluster resources of the target cluster based on the number of storage nodes and the number of computing nodes, thereby realizing the cluster resource evaluation function. Compared with the prior art, this application considers the difference between computing power and storage capacity during the cluster resource evaluation process and evaluates cluster resources based on the number of storage nodes and computing nodes. This effectively avoids the problem of insufficient accuracy in evaluating the processing capacity of each asset device due to differences in processing capacity under different operating conditions. This effectively solves the problem of low accuracy in evaluation results caused by the existing relatively simple and crude evaluation methods, and improves the accuracy of the cluster resource evaluation process. Furthermore, since the target cluster includes at least one of a distributed processing cluster, a distributed event communication cluster, and a file search cluster, this application can consider multiple different types of clusters for evaluation, avoiding the problem of poor evaluation accuracy caused by confusing multiple different clusters. Simultaneously, the node storage capacity is used to characterize the maximum storage capacity of a single node in the target cluster during data storage. The node storage capacity is determined based on the performance of the asset devices corresponding to each node in the cluster. The node computing capacity is used to characterize the maximum processing capacity of a single node in the target cluster during data processing. The node computing capacity is determined based on the performance of the asset devices corresponding to each node in the cluster. This ensures that when evaluating cluster resources, the different processing capabilities of each asset device in the storage and computing processes can be determined based on the actual performance of the asset devices corresponding to each node in the cluster. This ensures accurate evaluation of the storage and computing capabilities of asset devices after prolonged use and performance degradation, thereby guaranteeing the accuracy of evaluating the number of resources in the cluster based on storage and computing capabilities.

[0177] This application provides a storage medium that includes a stored program, wherein the program, when running, controls the device where the storage medium is located to execute the cluster resource evaluation method described above.

[0178] Storage media may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0179] This application embodiment also provides a cluster resource evaluation apparatus, the apparatus including a storage medium; and one or more processors, the storage medium being coupled to the processors, the processors being configured to execute program instructions stored in the storage medium; the program instructions, when executed, perform the cluster resource evaluation method described above.

[0180] This application provides an apparatus including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: obtaining the number of events processed by a target cluster, and determining the total storage space of the target cluster based on the number of events processed, wherein the target cluster includes at least one of a distributed processing cluster, a distributed event communication cluster, and a file search cluster; calculating the number of storage nodes in the target cluster based on the total storage space and the node storage capacity of the target cluster, wherein the node storage capacity characterizes the maximum storage capacity of a single node in the target cluster during data storage, and is determined based on the performance of the asset equipment corresponding to each node in the cluster; obtaining the peak number of events processed by the target cluster, and calculating the number of computing nodes in the target cluster based on the peak number of events processed by the target cluster and the node computing capacity, wherein the node computing capacity characterizes the maximum processing capacity of a single node in the target cluster during data processing, and is determined based on the performance of the asset equipment corresponding to each node in the cluster; and evaluating the number of cluster resources of the target cluster based on the number of storage nodes and the number of computing nodes.

[0181] Furthermore, the number of events processed includes the daily processing volume;

[0182] The step of obtaining the number of events processed by the target cluster and determining the total storage space of the target cluster based on the number of events processed includes:

[0183] The daily storage capacity of the target cluster is calculated based on the daily processing volume and the event data volume, wherein the event data volume is used to characterize the data volume of each task being processed.

[0184] The total storage space is determined based on the daily storage volume and the target number of days.

[0185] Furthermore, the daily processing volume includes daily traffic information and daily security log information; the daily traffic information is used to characterize the number of traffic tasks processed by the target cluster each day; the daily security log information is used to characterize the number of log tasks processed by the target cluster each day; the event data volume includes unit traffic data volume and unit log data volume.

[0186] The step of calculating the daily storage capacity of the target cluster based on the daily processing volume and the event data volume includes:

[0187] The unit traffic data volume and the unit log data volume are obtained respectively;

[0188] Calculate the daily traffic processing volume based on the daily traffic information and the unit traffic data volume;

[0189] Calculate the daily log processing volume based on the daily security log information and the unit log data volume;

[0190] The daily storage volume is determined based on the daily traffic processing volume and the daily log processing volume.

[0191] Furthermore, obtaining the unit traffic data volume and the unit log data volume respectively includes:

[0192] The cluster type of the target cluster is determined, and the number of data replicas and the data compression ratio are determined based on the cluster type. The number of data replicas is used to represent the number of backups when storing data, and the data compression ratio is used to represent the ratio of the amount of data after compression to the amount of data before compression when storing data.

[0193] Historical traffic data and historical log data are obtained respectively. The average data volume of the traffic task is determined based on the historical traffic data, and the average data volume of the log task is determined based on the historical log data.

[0194] The unit traffic data volume is calculated based on the number of data replicas, the data compression ratio, and the average data volume of the traffic task.

[0195] The unit log data volume is calculated based on the number of data replicas, the data compression ratio, and the average data volume of the log task.

[0196] Furthermore, obtaining the peak event processing count of the target cluster, and calculating the number of computing nodes in the target cluster based on the peak event processing count and the node computing load, includes:

[0197] When it is determined that the target cluster includes the distributed processing cluster and the file search cluster, the node ratio relationship between the distributed processing cluster and the file search cluster is determined. The node ratio relationship includes a first ratio and a second ratio, wherein the first ratio corresponds to the distributed processing cluster and the second ratio corresponds to the file search cluster.

[0198] The sum of the first ratio and the second ratio equals 1;

[0199] Based on the peak number of events processed and the computational load of nodes, calculate the total number of computing nodes in the target cluster.

[0200] The number of computing nodes in the distributed processing cluster is determined based on the total number of computing nodes and the first ratio.

[0201] The number of computing nodes in the file search cluster is determined based on the total number of computing nodes and the second ratio.

[0202] Furthermore, the step of evaluating the cluster resource quantity of the target cluster based on the number of storage nodes and the number of computing nodes includes:

[0203] The larger of the number of storage nodes and the number of computing nodes is selected as the number of running nodes corresponding to the target cluster, wherein the number of running nodes is used to characterize the number of asset devices that perform data storage and data processing in the target cluster;

[0204] The number of management nodes is determined based on the number of running nodes, wherein the number of management nodes is used to characterize the number of asset devices used for management in the target cluster, and the number of management nodes is in a specific ratio with the number of running nodes, the specific ratio being determined based on the target cluster;

[0205] The number of cluster resources is determined based on the number of running nodes and the number of management nodes.

[0206] Furthermore, the number of cluster resources includes a first cluster resource quantity, a second cluster resource quantity, and a third cluster resource quantity; wherein, the first cluster resource quantity corresponds to the distributed processing cluster; the second cluster resource quantity corresponds to the distributed event communication cluster; and the third cluster resource quantity corresponds to the file search cluster.

[0207] After determining the number of cluster resources based on the number of running nodes and the number of management nodes, the method further includes:

[0208] When it is determined that the target cluster includes all three components: the distributed processing cluster, the distributed event communication cluster, and the file search cluster, the total number of resources in the target cluster is calculated based on the resource quantities of the first cluster, the second cluster, and the third cluster.

[0209] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing program code with the following initialization steps: obtaining the number of events processed by a target cluster, and determining the total storage space of the target cluster based on the number of events processed, wherein the target cluster includes at least one of a distributed processing cluster, a distributed event communication cluster, and a file search cluster; calculating the number of storage nodes in the target cluster based on the total storage space and the node storage capacity of the target cluster, wherein the node storage capacity is used to characterize the maximum storage capacity of a single node in the target cluster during data storage, and the node storage capacity is determined based on the performance of the asset equipment corresponding to each node in the cluster; obtaining the peak number of events processed by the target cluster, and calculating the number of computing nodes in the target cluster based on the peak number of events processed by the target cluster and the node computing capacity, wherein the node computing capacity is used to characterize the maximum processing capacity of a single node in the target cluster during data processing, and the node computing capacity is determined based on the performance of the asset equipment corresponding to each node in the cluster; and evaluating the number of cluster resources of the target cluster based on the number of storage nodes and the number of computing nodes.

[0210] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0211] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0212] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0213] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0214] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0215] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0216] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0217] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0218] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0219] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for evaluating cluster resources, characterized in that, include: Obtain the number of events processed by the target cluster, and determine the total storage space of the target cluster based on the number of events processed. The target cluster includes at least one of the following: a distributed processing cluster, a distributed event communication cluster, and a file search cluster. Based on the total storage space and the node storage capacity of the target cluster, the number of storage nodes in the target cluster is calculated. The node storage capacity is used to characterize the maximum storage capacity of a single node in the target cluster during the data storage process. The node storage capacity is determined by obtaining the attribute parameters of the asset device corresponding to the node during operation. It is the maximum storage capacity of the asset device during the data storage process, which is used to reflect the true storage capacity when the actual storage capacity of the asset device differs from the factory-marked storage capacity after a long period of use. The peak number of events processed by the target cluster is obtained, and the number of computing nodes in the target cluster is calculated based on the peak number of events processed by the target cluster and the node computing power. The node computing power is used to characterize the maximum processing power of a single node in the target cluster during the data processing process. The node computing power is determined by obtaining the attribute parameters of the asset device corresponding to the node during operation, and is the maximum processing power of the asset device during the data processing process. It is used to reflect the true computing power when the actual processing power of the asset device after a long period of use differs from the computing power indicated at the factory. The number of cluster resources for the target cluster is assessed based on the number of storage nodes and the number of compute nodes.

2. The method according to claim 1, characterized in that, The number of events processed includes the daily processing volume; The step of obtaining the number of events processed by the target cluster and determining the total storage space of the target cluster based on the number of events processed includes: The daily storage capacity of the target cluster is calculated based on the daily processing volume and the event data volume, wherein the event data volume is used to characterize the data volume of each task being processed. The total storage space is determined based on the daily storage volume and the target number of days.

3. The method according to claim 2, characterized in that, The daily processing volume includes daily traffic information and daily security log information; the daily traffic information is used to characterize the number of traffic tasks processed by the target cluster each day. The daily security log information is used to characterize the number of log tasks processed by the target cluster each day; the event data volume includes the unit traffic data volume and the unit log data volume. The step of calculating the daily storage capacity of the target cluster based on the daily processing volume and the event data volume includes: The unit traffic data volume and the unit log data volume are obtained respectively; Calculate the daily traffic processing volume based on the daily traffic information and the unit traffic data volume; Calculate the daily log processing volume based on the daily security log information and the unit log data volume; The daily storage volume is determined based on the daily traffic processing volume and the daily log processing volume.

4. The method according to claim 3, characterized in that, The steps of obtaining the unit traffic data volume and the unit log data volume respectively include: The cluster type of the target cluster is determined, and the number of data replicas and the data compression ratio are determined based on the cluster type. The number of data replicas is used to represent the number of backups when storing data, and the data compression ratio is used to represent the ratio of the amount of data after compression to the amount of data before compression when storing data. Historical traffic data and historical log data are obtained respectively. The average data volume of the traffic task is determined based on the historical traffic data, and the average data volume of the log task is determined based on the historical log data. The unit traffic data volume is calculated based on the number of data replicas, the data compression ratio, and the average data volume of the traffic task. The unit log data volume is calculated based on the number of data replicas, the data compression ratio, and the average data volume of the log task.

5. The method according to any one of claims 1-4, characterized in that, The step of obtaining the peak event processing count of the target cluster and calculating the number of computing nodes in the target cluster based on the peak event processing count and the node computing load includes: When it is determined that the target cluster includes the distributed processing cluster and the file search cluster, the node ratio relationship between the distributed processing cluster and the file search cluster is determined. The node ratio relationship includes a first ratio and a second ratio, wherein the first ratio corresponds to the distributed processing cluster and the second ratio corresponds to the file search cluster. The sum of the first ratio and the second ratio equals 1; Based on the peak number of events processed and the computational load of nodes, calculate the total number of computing nodes in the target cluster. The number of computing nodes in the distributed processing cluster is determined based on the total number of computing nodes and the first ratio. The number of computing nodes in the file search cluster is determined based on the total number of computing nodes and the second ratio.

6. The method according to any one of claims 1-4, characterized in that, The process of evaluating the cluster resources of the target cluster based on the number of storage nodes and the number of compute nodes includes: The larger of the number of storage nodes and the number of computing nodes is selected as the number of running nodes corresponding to the target cluster, wherein the number of running nodes is used to characterize the number of asset devices that perform data storage and data processing in the target cluster; The number of management nodes is determined based on the number of running nodes, wherein the number of management nodes is used to characterize the number of asset devices used for management in the target cluster, and the number of management nodes is in a specific ratio with the number of running nodes, the specific ratio being determined based on the target cluster; The number of cluster resources is determined based on the number of running nodes and the number of management nodes.

7. The method according to claim 6, characterized in that, The number of cluster resources includes a first cluster resource quantity, a second cluster resource quantity, and a third cluster resource quantity; wherein, the first cluster resource quantity corresponds to the distributed processing cluster; the second cluster resource quantity corresponds to the distributed event communication cluster; and the third cluster resource quantity corresponds to the file search cluster. After determining the number of cluster resources based on the number of running nodes and the number of management nodes, the method further includes: When it is determined that the target cluster includes all three components: the distributed processing cluster, the distributed event communication cluster, and the file search cluster, the total number of resources in the target cluster is calculated based on the resource quantities of the first cluster, the second cluster, and the third cluster.

8. A device for evaluating cluster resources, characterized in that, include: A determining unit is used to obtain the number of events processed by the target cluster and determine the total storage space of the target cluster based on the number of events processed. The target cluster includes at least one of a distributed processing cluster, a distributed event communication cluster, and a file search cluster. The first calculation unit is used to calculate the number of storage nodes in the target cluster based on the total storage space and the node storage capacity of the target cluster. The node storage capacity is used to characterize the maximum storage capacity of a single node in the target cluster during the data storage process. The node storage capacity is determined by obtaining the attribute parameters of the asset device corresponding to the node during operation. It is the maximum storage capacity of the asset device during the data storage process, which is used to reflect the true storage capacity when the actual storage capacity of the asset device differs from the factory-marked storage capacity after a long period of use. The second computing unit is used to obtain the peak number of events processed by the target cluster, and to calculate the number of computing nodes in the target cluster based on the peak number of events processed by the target cluster and the node computing power. The node computing power is used to characterize the maximum processing power of a single node in the target cluster during the data processing process. The node computing power is determined by obtaining the attribute parameters of the asset device corresponding to the node during operation, and is the maximum processing power of the asset device during the data processing process. It is used to reflect the true computing power when the actual processing power of the asset device after a long period of use differs from the factory-marked computing power. An evaluation unit is used to evaluate the number of cluster resources of the target cluster based on the number of storage nodes and the number of computing nodes.

9. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the cluster resource evaluation method according to any one of claims 1 to 7.

10. A device for evaluating cluster resources, characterized in that, The apparatus includes a storage medium; and one or more processors, the storage medium being coupled to the processors, the processors being configured to execute program instructions stored in the storage medium; the program instructions, when executed, perform the cluster resource evaluation method according to any one of claims 1 to 7.

Images