Cluster performance statistical data processing method and device and medium

Abstract

The invention discloses a cluster performance statistical data processing method and device and a medium, relates to the technical field of data monitoring, is used for processing performance data obtained through monitoring, and provides the cluster performance statistical data processing method for solving the problem that the performance data acquired through monitoring at present occupies too large space of a system disk. Firstly, the precision of performance data is reduced through data aggregation, that is, the performance data within a certain time range is aggregated, multiple pieces of performance data are aggregated into historical data, and the purpose of reducing occupation of storage resources of a system disk is achieved. And then, carrying out retention duration monitoring on the aggregated historical data, clearing the historical data with overlong retention duration in the database, and further releasing the storage space of the system disk. Namely, the method can reduce the total amount of the performance data on the basis of ensuring that the historical data with enough duration can be checked, so that the pressure on the cluster and the occupation of system disk resources are reduced.

Description

Technical Field

[0001] The present invention relates to the field of data monitoring technology, and in particular to a cluster performance statistical data processing method, device and medium. Background Art

[0002] In the context of digital transformation, the amount of data continues to grow. Distributed storage systems have been widely used in various fields due to their advantages of high scalability and high reliability. However, with the expansion of the amount of data in cluster nodes, in order to monitor the operation of the cluster in a timely and accurate manner, it is usually necessary to monitor various performance indicators of each node in the cluster, such as bandwidth, the number of read and write operations per second (Input / Output Operations Per Second, IOPS), resource utilization, hard disk information, network card information and other performance indicators. The more monitoring indicators there are and the longer the cluster runs, the more performance statistics will be generated.

[0003] However, since the performance statistics of the cluster are generally stored locally in the cluster node or in the database, they occupy the space of the system hard disk. Considering the cost of the system hard disk, the capacity of the system hard disk is generally not very large, so the space resources allocated to store monitoring data are relatively limited. Therefore, excessive historical performance indicator data will greatly increase the pressure on the cluster and the occupation of system disk resources.

[0004] Therefore, technical personnel in this field are in urgent need of a cluster performance statistical data processing method that can ensure that performance indicators with a sufficient time span can be viewed as data basis for analyzing the system operation status, while also reducing the pressure of historical data on the cluster and the occupancy of system disk resources. Summary of the invention

[0005] The purpose of the present invention is to provide a cluster performance statistical data processing method, device and medium, so as to ensure that performance indicators can be viewed as data basis for analyzing system operation status, while also reducing the pressure of historical data on the cluster and the occupation of system disk resources.

[0006] In order to solve the above technical problems, the present invention provides a cluster performance statistical data processing method, which is applied to a management platform side. The management platform runs in a management node of a cluster in the form of a container. The method includes:

[0007] Obtain performance data corresponding to each performance indicator;

[0008] According to a preset aggregation precision strategy, the performance data corresponding to each performance indicator is aggregated according to the corresponding aggregation precision, and multiple performance data within the time range corresponding to the aggregation precision are aggregated into a historical data, and the historical data is saved in a database; wherein the aggregation precision strategy includes multiple different aggregation precisions, and each performance indicator uniquely corresponds to a certain aggregation precision;

[0009] Determining the storage duration of each of the historical data stored in the database;

[0010] According to a preset data cleaning strategy, the historical data whose storage time exceeds the corresponding cleaning time limit is cleared; wherein the data cleaning strategy includes a plurality of cleaning time limits, and the cleaning time limits correspond to the performance indicators one by one.

[0011] In a possible embodiment, according to a preset aggregation precision strategy, the performance data corresponding to each performance indicator is aggregated according to a corresponding aggregation precision, and a plurality of the performance data within a time range corresponding to the aggregation precision are aggregated into one historical data, including:

[0012] For each of the performance indicators, obtaining from the database the corresponding aggregation timestamp of the most recently aggregated historical data;

[0013] According to the current system time, subtract a duration corresponding to the aggregation precision to obtain the aggregation time limit;

[0014] In a time range with the aggregation timestamp plus a duration corresponding to the aggregation accuracy as the starting time and the aggregation time limit as the ending time, the aggregation time period is divided according to the corresponding aggregation accuracy; wherein, the first aggregation time period takes the aggregation timestamp as the starting time; if multiple aggregation time periods can be divided, the ending time of the previous aggregation time period is the starting time of the next aggregation time period, the duration of each aggregation time period is equal, and the time range that is less than one aggregation time period does not participate in the division of the aggregation time period;

[0015] The average value of the performance data in each aggregation period is used as the historical data after aggregation in the current aggregation period, and the aggregation timestamp of the historical data is the starting time of the aggregation period.

[0016] In a possible embodiment, before aggregating the performance data corresponding to each performance indicator according to the corresponding aggregation precision according to the preset aggregation precision strategy, the method further includes:

[0017] Determine whether the aggregation switch is in the on state;

[0018] If yes, then enter the step of aggregating the performance data corresponding to each performance indicator according to the corresponding aggregation precision according to the preset aggregation precision strategy;

[0019] If not, proceed to the step of determining the storage duration of each of the historical data stored in the database.

[0020] In a possible embodiment, the clearing of the historical data whose storage time exceeds the corresponding clearing time limit according to the preset data clearing strategy includes:

[0021] Obtaining performance model annotations; wherein the performance model annotations include model annotation items corresponding to each performance indicator, and the model annotation items include: table name, timestamp column name, performance value column name, and label column name;

[0022] Acquire the historical data corresponding to each of the performance indicators from a database according to the performance model annotation;

[0023] Determine the storage duration of each of the historical data according to the current system time and the aggregated timestamp of the historical data;

[0024] Determine whether the storage time of each of the historical data exceeds the corresponding cleaning time limit;

[0025] If it exceeds, generating a cleanup strategy configuration class according to the model annotation item of the historical data corresponding to the performance indicator, and saving the cleanup strategy configuration class in the table cleanup strategy;

[0026] A delete command statement is generated according to the table cleanup strategy, and the historical data whose storage time exceeds the corresponding cleanup time limit is cleared by executing the delete command statement.

[0027] In a possible embodiment, the data cleaning strategy includes: a plurality of pre-configured default cleaning time limits, the default cleaning time limits corresponding to the performance indicators one by one;

[0028] Before determining whether the storage time of each of the historical data exceeds the corresponding cleaning time limit, the method further includes:

[0029] Determine whether a unified cleanup time limit is configured on the user side;

[0030] If not, the cleaning time limit corresponding to each performance indicator is: the default cleaning time limit pre-configured in the data cleaning strategy and corresponding to the performance indicator;

[0031] If so, the cleaning time limit corresponding to each of the performance indicators is the unified cleaning time limit.

[0032] In a possible embodiment, the management platform includes a plurality of threads, and the threads are used to perform the aggregation task of the performance data and the deletion task of the historical data by calling a timed task cycle;

[0033] Among them, the thread uniquely corresponds to one aggregation precision, and is used to implement the aggregation task and the deletion task of one or more performance indicators corresponding to the aggregation precision; and the timing duration of the scheduled task is the same as the duration corresponding to the aggregation precision.

[0034] In a possible embodiment, it further includes:

[0035] When any of the threads fails to execute the aggregation task or the deletion task, the failed aggregation task or the deletion task is terminated, and the scheduled task is maintained to re-trigger the aggregation task and the deletion task in the next cycle of the scheduled task.

[0036] In order to solve the above technical problems, the present invention further provides a cluster performance statistical data processing device, which is applied to a management platform side. The management platform runs in a management node of a cluster in the form of a container. The device includes:

[0037] A collection module is used to obtain performance data corresponding to each performance indicator;

[0038] An aggregation module, configured to aggregate the performance data corresponding to each performance indicator according to a corresponding aggregation precision according to a preset aggregation precision strategy, aggregate multiple performance data within a time range corresponding to the aggregation precision into one historical data, and save the historical data into a database; wherein the aggregation precision strategy includes multiple different aggregation precisions, and each performance indicator uniquely corresponds to a certain aggregation precision;

[0039] A statistical module, used to determine the storage time of each of the historical data stored in the database;

[0040] A clearing module is used to clear the historical data whose storage time exceeds the corresponding clearing time limit according to a preset data clearing strategy; wherein the data clearing strategy includes multiple clearing time limits, and the clearing time limits correspond to the performance indicators one by one.

[0041] In order to solve the above technical problems, the present invention further provides a cluster performance statistical data processing device, comprising:

[0042] Memory for storing computer programs;

[0043] The processor is used to implement the steps of the cluster performance statistical data processing method as described above when executing the computer program.

[0044] To solve the above technical problem, the present invention further provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the cluster performance statistical data processing method as described above are implemented.

[0045] The present invention provides a cluster performance statistical data processing method. After monitoring and collecting the data of each performance indicator, the performance data is first aggregated according to a preset aggregation accuracy to convert high-precision (or high-granularity) performance data into historical data with lower precision for persistent storage, so that it can be called as a data reference by subsequent relevant personnel when needed. In addition, for the low-precision historical data after aggregation, the method also monitors the storage time of each historical data in the database through a preset data cleaning strategy. If the storage time exceeds the corresponding cleaning time limit, the historical data is considered expired and deleted from the database, thereby freeing up the storage space of the system disk.

[0046] In summary, the present invention provides a two-layer performance data simplification solution: the first layer reduces the accuracy of performance data through data aggregation, that is, the performance data within a certain time range is aggregated to reduce the amount of performance data within the time range, so as to reduce the occupation of system disk storage resources. The second layer is to monitor the storage time of the aggregated historical data. For historical data that has been retained in the database for too long, it is considered that it is of little help in the analysis of the system's operating status, and it is cleared from the database to further free up the storage space of the system disk. Therefore, this method can reduce the total amount of performance data on the basis of ensuring that historical data of sufficient length can be viewed, thereby reducing the pressure on the cluster and the occupation of system disk resources, and better meeting the needs of actual distributed storage cluster applications. In addition, the method is deployed in the cluster in the form of a container, which makes the method easier to transplant and deploy in actual applications, and facilitates version updates and upgrades.

[0047] The cluster performance statistics data processing device and non-volatile storage medium provided by the present invention correspond to the above method and have the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] FIG1 is a flow chart of a cluster performance statistics data processing method provided by an embodiment of the present invention;

[0050] FIG2 is a flow chart of a method for reducing the precision of performance data aggregation provided by an embodiment of the present invention;

[0051] FIG3 is a flow chart of a method for clearing expired historical data provided by an embodiment of the present invention;

[0052] FIG4 is a structural diagram of a cluster performance statistics data processing device provided by an embodiment of the present invention;

[0053] FIG. 5 is a structural diagram of another cluster performance statistical data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] The core of the present invention is to provide a cluster performance statistical data processing method, device and medium.

[0056] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0057] In cluster applications, in order to monitor the operation of the cluster, it is necessary to monitor and collect data on various performance indicators of the cluster. The above performance indicators include but are not limited to: bandwidth, the number of read / write operations per second (Input / Output Operations Per Second, IOPS), resource utilization, hard disk information, network card information, etc.

[0058] These performance indicators collect data at a set collection period, and save the collected performance data to the local cluster node or database, which will occupy a certain amount of system hard disk space. In particular, since the sampling period of each performance indicator is not necessarily the same, some performance indicators may be collected every 10 seconds, some performance indicators may be collected every 1 minute, and other performance indicators may be collected every 1 hour or longer. For these performance indicators with different sampling periods, when analyzing the cluster operation status based on performance data, it is often necessary for the performance data to cover a longer time span (for example, 6 months). Therefore, for performance indicators with a shorter sampling period, retaining performance data for a longer time span will occupy a large amount of system disk space, thereby causing excessive pressure on the cluster and being detrimental to the long-term stable operation of the cluster.

[0059] To solve the above problems, the present invention provides a cluster performance statistics data processing method, which is applied to the management platform side. The management platform runs in the management node of the cluster in the form of a container (docker). As shown in Figure 1, the method includes:

[0060] S1: Obtain performance data corresponding to each performance indicator.

[0061] S2: According to the preset aggregation precision strategy, the performance data corresponding to each performance indicator is aggregated according to the corresponding aggregation precision, multiple performance data within the time range corresponding to the aggregation precision are aggregated into a historical data, and the historical data is saved in the database.

[0062] The aggregation precision strategy includes a plurality of different aggregation precisions, and each performance indicator uniquely corresponds to a certain aggregation precision.

[0063] S3: Determine the storage duration of each historical data stored in the database.

[0064] S4: According to the preset data cleaning strategy, clear the historical data whose storage time exceeds the corresponding cleaning time limit.

[0065] The data cleaning strategy includes multiple cleaning time limits, and the cleaning time limits correspond to performance indicators one by one.

[0066] First, this method can be deployed in the management node of the cluster in the form of a docker container, which facilitates cross-platform deployment and porting of this method in various clusters. At the same time, deployment in the form of a docker container can also easily manage and distribute application components of different versions, making it easier to update and upgrade versions.

[0067] Specifically, this embodiment provides a possible implementation scheme, and this method is implemented through the master service of the cluster management node. The master service manages: a monitor (Listener) component, an aggregation task (Down Sampletask) component, and a cleanup task (Cleanup task) component.

[0068] Among them, the Listener component is used to listen to the performance data reported by each node and persist it in the database for storage, which corresponds to the above-mentioned step S1. The Down Sample task component is used to perform down-precision aggregation processing on the performance data listened to by the Listener component, which corresponds to the above-mentioned step S2. The Cleanup task component is used to implement the scheduled cleaning of historical data after the aggregation processing of the DownSample task component, determine the retention time of historical data in the database, and clear the historical data that exceeds the cleaning time limit to release the storage space of the system disk, which corresponds to the above-mentioned steps S3 and S4.

[0069] Based on the above, this embodiment manages the three components of Listener, Down Sample task and Cleanup task through the master service of the cluster, and packages these three components into a container for deployment or transplantation in various platforms, so as to implement this method conveniently and quickly, while meeting the user's demand for viewing historical data for a long time, and reducing the resource pressure of the system.

[0070] Furthermore, the acquisition of performance data in step S1 can be implemented by an agent service deployed in each node in the cluster. The agent service deployed in each node is responsible for collecting performance data of performance indicators of the node and reporting the performance data to the management node. Corresponding to the above-mentioned embodiment of the containerized application, that is, the master service reported to the management node is monitored by the Listener component.

[0071] Afterwards, for step S2. It should be noted that in the preset aggregation accuracy strategies of different performance indicators, the aggregation accuracy of different performance indicators can be the same or different, and can be set according to actual needs. The aggregation accuracy is also the time granularity of the aggregated historical data. For example, if there is a performance indicator with a sampling period of 15s, that is, it is collected once every 15s. And the aggregation accuracy of the performance indicator is 1h, then for the aggregation of the performance indicator, 240 performance data within the 1h interval (such as 00:00:00~00:59:60) are aggregated to obtain 1 performance indicator, that is, one aggregation of the performance indicator is completed. Based on the time span of the performance data collected from the performance indicator, one or more aggregations as described above are performed to complete the performance data aggregation of the performance indicator.

[0072] It should also be noted that each performance indicator corresponds to at most one aggregation precision, and the time span corresponding to the aggregation precision is usually larger than the time span corresponding to the collection period of the performance indicator. The aggregation precision can be freely set according to actual needs. In a possible example, the aggregation precision can be divided into four types: 1 minute, 1 hour, 12 hours, and 1 day. For some performance indicators, their collection period (such as once a day) may be equal to or exceed the maximum aggregation precision. For these performance indicators, there is no need to aggregate the performance data, and there is no corresponding aggregation precision. However, this part of the performance indicators can still be retained for a long time according to the method corresponding to steps S3 and S4 of this method. At this time, for this part of the performance indicators that do not need to be aggregated, the performance data collected can be directly used as the historical data targeted by steps S3 and S4.

[0073] Step S3 is a step of comparing the storage time of each historical data in the database with the corresponding cleaning time limit to determine whether the historical data is expired. Expired historical data is cleared in step S4. It should be noted that, similar to the above-mentioned aggregation accuracy, the clearing time limits corresponding to different performance indicators can be the same or different, and this embodiment does not limit this.

[0074] It should also be noted that since the performance indicator monitoring in the cluster is a continuous process, that is, the performance data for each performance indicator is increasing over time, so this method is actually triggered periodically. This method can be triggered to execute as an overall cycle, but it should be noted that the above steps S1, S2, and steps S3 and S4 correspond to different tasks (collection, aggregation and cleaning), and the time parameters they are based on are also different (sampling period, aggregation accuracy, cleaning time limit), so the above steps can also be executed regularly through different cycles.

[0075] In summary, the cluster performance statistical data processing method provided by the present invention provides a two-layer data simplification solution. First, the accuracy of the performance data is reduced through data aggregation, that is, the performance data within a certain time range is aggregated, and multiple performance data are aggregated into a historical data, so as to achieve the purpose of reducing the occupation of system disk storage resources. After that, the retention time of the aggregated historical data is monitored, and the historical data that has been retained in the database for too long is cleared, further freeing up the storage space of the system disk. In other words, this method can reduce the total amount of performance data on the basis of ensuring that historical data of sufficient length can be viewed, thereby reducing the pressure on the cluster and the occupation of system disk resources, and better meeting the needs of actual distributed storage cluster applications.

[0076] Furthermore, for the performance data aggregation corresponding to the above step S2, this embodiment also provides a possible implementation scheme, as shown in FIG2 , where the above step S2 specifically includes:

[0077] S21: For each performance indicator, obtain the corresponding aggregation timestamp of the most recently aggregated historical data from the database.

[0078] S22: Subtract a duration corresponding to the aggregation precision from the current system time to obtain an aggregation time limit.

[0079] S23: within a time range with the aggregation timestamp plus a duration corresponding to the aggregation precision as the starting time and the aggregation time limit as the ending time, divide the aggregation time period according to the corresponding aggregation precision.

[0080] The first aggregation period starts with the aggregation timestamp. If multiple aggregation periods can be divided, the end time of the previous aggregation period is the start time of the next aggregation period. The duration of each aggregation period is equal, and the time range that is less than one aggregation period does not participate in the division of the aggregation period.

[0081] S24: The average value of the performance data in each aggregation period is used as the aggregated historical data of the current aggregation period, and the aggregation timestamp of the historical data is the start time of the aggregation period.

[0082] From the above, it can be seen that the aggregation task corresponding to step S2 can be triggered periodically, but the triggering period does not necessarily need to be equal to the aggregation accuracy. That is, for step S2, aggregation can be performed once every time range that reaches the aggregation accuracy requirement (that is, the triggering period of step S2 is equal to the duration of the aggregation accuracy), but it can also be achieved in one aggregation. The aggregation of time ranges with multiple aggregation accuracy requirements (that is, the triggering period of step S2 is greater than or equal to twice the duration of the aggregation accuracy). At this time, since the aggregation period is more flexible and changeable, it is not easy to predetermine how many time ranges corresponding to the aggregation accuracy are included in the time span of the performance data targeted at each aggregation. Therefore, in this embodiment, the time range targeted by this aggregation is determined by steps S21 and S22.

[0083] To more clearly illustrate the solution of this embodiment, the process of the solution provided by this embodiment is further described below with reference to an example:

[0084] For example, assume that the aggregation accuracy of performance indicator A is 1 hour, and when its aggregation task is triggered (assuming the current time is t), 4.5 hours (t-4.5h) have passed since the aggregation timestamp of the last aggregated historical event. Since the starting time of the aggregation period is used as the aggregation timestamp of the historical data in this embodiment, and an aggregation period is 1 hour, the aggregation period corresponding to the most recently aggregated historical data is (t-4.5h, t-3.5h). In other words, the time range of the performance data that has not been aggregated until time t is (t-3.5h, t), that is, the starting time t1 is determined to be t-3.5h.

[0085] In this method, performance data that is less than one aggregation precision time is not aggregated (i.e., the aggregation time limit is t2, t2=t-1h). Therefore, the time range (t1, t2) of the performance data targeted by this aggregation is actually (t-3.5h, t-1h), a total of 2.5h long time range.

[0086] Among them, 2 hours of performance data can be divided into two aggregation periods, and two aggregated historical data can be obtained. However, it should be noted that these two aggregation periods start from the past, that is, the two aggregation periods are (t1, t1+1h) and (t1+1h, t1+2h), and the remaining 0.5h (t1+2h, t2) will not be processed in this aggregation, and will be aggregated when the next step S2 is triggered. At this time, the timestamp of the historical data aggregated in the time period (t1, t1+1h) is the starting time of the time period, that is, t1. The timestamp of the historical data aggregated in the time period (t1+1h, t1+2h) is the starting time of the time period, that is, t1+1h.

[0087] Therefore, when the next step S2 is triggered, the aggregation timestamp of the most recent historical data is t1+1h, and the starting time of the aggregation period determined based on this is t1+2h, which is the starting time of the performance data that has not been aggregated in this aggregation. It can be seen that the aggregation method provided in this embodiment can ensure that when the aggregation is triggered by any period, it can ensure that no performance data is missed and can be aggregated into historical data and saved in the database.

[0088] In addition, it is particularly important to note that, according to step S22, this embodiment does not aggregate the performance data starting from the current moment and pushing forward a time corresponding to the aggregation precision. That is, in this embodiment, the high precision of the most recent performance data is retained to meet the user's needs for a more fine-grained analysis of the recent operation status of the cluster. Based on the solution provided by this embodiment, the complete life cycle of the data of each performance indicator in the cluster includes: being collected (at this time, the precision of the performance data is the time granularity of the sampling period) - from the time when it is collected to an aggregation precision (at this time, the performance data is not aggregated, and the precision is still the time granularity of the sampling period) - exceeding the time corresponding to an aggregation precision, to before the cleaning time limit (at this time, the performance data is aggregated, and the precision is the aggregation precision) - exceeding the cleaning time limit and being cleaned.

[0089] It can be seen that this embodiment provides richer performance data retention accuracy in the cluster to meet the different needs of users. For performance data that has just been collected and has not yet reached a corresponding aggregation accuracy, no aggregation is performed, and the initial high accuracy is retained to meet the user's high-precision analysis requirements for the recent operation status of the cluster. For performance data that exceeds an aggregation accuracy time, aggregation processing is performed to reduce the accuracy of the data to reduce the occupation of system disk space. For historical data that exceeds the cleanup time limit, it is directly cleared to further reduce the occupation of system disk space.

[0090] On the other hand, this embodiment also provides a possible implementation scheme. Before step S2, the method further includes:

[0091] S5: Determine whether the polymerization switch is in the on state; if so, proceed to step S2; if not, go to step S3.

[0092] That is, in this method, a control scheme for turning on and off the aggregation function is provided, and the aggregation function can be turned on and off by the aggregation switch to meet the needs of users who do not want any reduction in the accuracy of performance data in special scenarios.

[0093] Furthermore, the aggregation switch in this embodiment can be a visual switch set on the cluster front-end visual interface. Specifically, the management platform to which this method is applied has a front-end visual interface, on which the aggregation switch is displayed for the user to control opening and closing.

[0094] On the other hand, this embodiment also provides a possible implementation scheme for the cleaning task corresponding to step S4, as shown in FIG3 , the above step S4 specifically includes:

[0095] S41: Obtain performance model annotations.

[0096] The performance model annotation includes model annotation items corresponding to each performance indicator, and the model annotation items include: table name, timestamp column name, performance value column name and label column name. That is, the model annotation item is used to determine the historical data of the unique corresponding performance indicator from the database.

[0097] It should be noted that, according to the aggregation scheme provided in the above embodiment, the above embodiment retains the performance data that does not reach an aggregation precision time without aggregation, and this part of the performance data also needs to be persisted in the database. Correspondingly, this part of the performance data can also be found and obtained from the database through the performance model annotation.

[0098] S42: Obtain historical data corresponding to each performance indicator from the database according to the performance model annotation.

[0099] S43: Determine the storage duration of each historical data according to the current system time and the aggregated timestamp of the historical data.

[0100] S44: Determine whether the storage time of each historical data exceeds the corresponding cleaning time limit. If so, go to step S45.

[0101] The cleaning time limits corresponding to the historical data may be the same or different, and this embodiment does not impose any limitation on this.

[0102] S45: Generate a cleanup strategy configuration class according to the model annotation items of the performance indicators corresponding to the historical data, and save the cleanup strategy configuration class in the table cleanup strategy.

[0103] From the above, we can know that the model annotation item can be used to uniquely identify the historical data of a performance indicator in the database. Therefore, generating a cleanup strategy configuration class based on the model annotation item can indicate the deletion of the historical data of a performance indicator in the database.

[0104] S46: Generate a delete command statement according to the table cleanup strategy, and execute the delete command statement to clear historical data whose storage time exceeds the corresponding cleanup time limit.

[0105] The above-mentioned delete command statement is determined according to the delete command supported by the database. For example, for a common relational database system, the above-mentioned delete command can be implemented using Structured Query Language (SQL). That is, the above-mentioned delete command is a SQL statement that indicates deletion.

[0106] In summary, this embodiment provides a possible specific implementation scheme for historical data cleaning tasks. The overall cleaning process is simple and easy to implement, and no additional changes are required to the cluster and database, and it can be implemented using existing command tools. This achieves the purpose of cleaning up expired historical data to free up system disk storage space.

[0107] Furthermore, in the above embodiment, it is not limited whether the cleaning time limits corresponding to the various performance indicators are the same. For this, this embodiment also provides a possible implementation scheme:

[0108] The data cleaning strategy includes: multiple pre-configured default cleaning time limits, and the default cleaning time limits correspond to performance indicators one by one.

[0109] Before step S44, the method further includes:

[0110] S47: Determine whether a unified cleaning time limit is configured on the user side; if not, the cleaning time limit corresponding to each performance indicator is: the default cleaning time limit pre-configured in the data cleaning strategy and corresponding to the performance indicator; if so, the cleaning time limit corresponding to each performance indicator is a unified cleaning time limit.

[0111] That is, in this embodiment, two schemes are provided for setting the cleaning time limit, one of which is the default configuration scheme, that is, the corresponding default cleaning time limit is configured for each performance indicator in advance. The default cleaning time limits of various performance indicators can be the same or different, and are determined according to actual needs. The second is to uniformly configure the same cleaning time limit for all performance indicators. The two schemes can be implemented in combination, and a certain scheme can be selected by using a switch. Specifically, when there is a unified cleaning time limit configuration, the authority of this configuration takes precedence, and each performance indicator adopts a unified cleaning time limit. When there is no unified cleaning time limit configuration, the default cleaning time limit configuration of each performance indicator is adopted to achieve targeted historical data cleaning, making the cleaning of historical data more flexible.

[0112] On the other hand, the above embodiments do not limit the implementation of the above aggregation tasks and cleanup tasks. This embodiment also provides a possible specific implementation plan:

[0113] The management platform includes multiple threads, which are used to perform the aggregation task of performance data and the deletion task of historical data by calling the scheduled task cycle.

[0114] Among them, the thread uniquely corresponds to an aggregation precision, and is used to implement the aggregation task and deletion task of one or more performance indicators corresponding to the aggregation precision, and the timing duration of the scheduled task is the same as the duration of the corresponding aggregation precision.

[0115] In this embodiment, specific aggregation tasks and cleanup tasks are implemented through threads. The threads periodically execute aggregation tasks and cleanup tasks by calling timed tasks. The timing period of the timed task can be freely set, for example, it can be set to the duration corresponding to the aggregation accuracy. At this time, each triggering of the timed task corresponds to the aggregation of an aggregation period. There are no special requirements for the triggering period of the cleanup task, so the cleanup task can share the same timing period with the aggregation task. That is, after the thread is triggered to execute the timed task, it can continue to trigger the execution of the cleanup task.

[0116] In addition, in a possible embodiment, a thread may correspond to only one performance indicator, that is, one timing task (the aggregation accuracy corresponding to the performance indicator), one aggregation task and one cleanup task, so as to ensure the accuracy of the thread's processing of the performance data and avoid confusion.

[0117] However, due to the high aggregation accuracy of some performance indicators, such as 1 hour or 1 day, if the thread only continuously calls the scheduled task and performs aggregation and cleanup tasks once every 1 hour or 1 day, it will undoubtedly lead to a waste of resources. Especially if there are a large number of performance indicators that need to be monitored in the cluster, the solution of one performance indicator corresponding to one thread will cause a large number of threads to reside in the management platform, thus bringing unnecessary pressure to the cluster.

[0118] Based on this, for these performance indicators with greater aggregation accuracy, a solution in which multiple performance indicators correspond to one thread can be adopted, which is the solution described above in this embodiment. However, it should be noted that since thread triggering aggregation and cleanup are both implemented by timed tasks, a thread generally only calls one timed task to avoid confusion. Therefore, if a thread corresponds to multiple performance indicators, the aggregation accuracy of these performance indicators must be consistent.

[0119] In addition, this embodiment also does not limit how many performance indicators a thread specifically corresponds to, and it should be determined based on the actual situation of thread data processing. Exemplarily, a thread processes the data of each corresponding performance indicator in turn, including aggregation of performance data and cleaning of historical data. And the thread needs to ensure that all corresponding performance indicator data processing tasks (aggregation and cleaning) are completed before the timing cycle of the next scheduled task arrives. In addition, this embodiment has no other restrictions. In other words, if it is assumed that the time required for data processing of a performance indicator is certain, then the thread with greater aggregation accuracy can correspond to more performance indicators.

[0120] It should also be noted that for some performance indicators with higher aggregation accuracy, if a thread cannot complete the data processing tasks of multiple threads within the time corresponding to one aggregation accuracy, the thread may only correspond to one performance indicator to ensure that the data processing tasks of each performance indicator can meet the priority of timing task cycle trigger execution.

[0121] In particular, when the implementation scheme of this embodiment is adopted, when a thread needs to process data for multiple performance indicators. Since the processing has a sequence, the triggering time of the aggregation and cleanup tasks for some performance indicators is no longer an integer time (that is, a time that can be divided by the aggregation precision duration). That is, corresponding to an aggregation implementation scheme in the above embodiment, when the time range based on the aggregation timestamp plus a duration corresponding to the aggregation precision as the starting time and the aggregation time limit as the ending time cannot be divided by the aggregation precision (that is, it cannot be divided into an integer number of aggregation time periods), it can be started from the past and only the divided aggregation time periods can be aggregated, and the subsequent time range that is less than an aggregation time period can be processed at the next aggregation.

[0122] On the other hand, the above-mentioned aggregation and cleaning both need to interact with the database, so if the connection with the database is interrupted during task execution, the task execution will fail. Based on this, this embodiment also provides a corresponding implementation plan, and the method also includes:

[0123] When any thread fails to execute an aggregation task or a deletion task, the failed aggregation task or the deletion task is terminated, and the scheduled task is maintained to re-trigger the aggregation task and the deletion task in the next cycle of the scheduled task.

[0124] As can be seen from the above, the processing of performance data and historical data of performance indicators can be implemented through threads. For threads, which are commonly used to execute tasks, there are currently many mature solutions for detecting whether thread processing tasks have failed, so this embodiment will not elaborate on how to determine whether a thread fails to execute an aggregation task or a deletion task. It can be seen from a specific aggregation scheme and cleaning scheme provided by the above embodiment that both aggregation and cleaning use the aggregation timestamp of historical data to determine the data object targeted by this task. Whether in aggregation or cleaning, the aggregation timestamp is updated at the end of the task (aggregation is to generate new historical data, corresponding to the generation of a new aggregation timestamp, and cleaning is to delete old historical data, corresponding to the deletion of the old aggregation timestamp). And the above aggregation and cleaning processes are all to process the historical data currently available in the database. Therefore, even if any aggregation task or cleaning task fails to process, it will not change the historical data stored in the database and its aggregation timestamp. That is, it is equivalent to that the failed aggregation task or cleaning task has never occurred from beginning to end (for the historical data finally stored in the database), then when the aggregation task and cleaning task are triggered by the next scheduled task, the data not processed by the previous task can be processed. That is, when the thread fails to execute an aggregation task or deletion task due to reasons such as disconnection with the database, it only needs to terminate the aggregation task or deletion task without repeating it. The data of the failed task can be reprocessed in the next scheduled task cycle, and no data will be missed or not processed.

[0125] In the above embodiment, a cluster performance statistics data processing method is described in detail, and the present invention also provides a corresponding embodiment of a cluster performance statistics data processing device. It should be noted that the present invention describes the embodiment of the device part from two perspectives, one is based on the functional module perspective, and the other is based on the hardware perspective.

[0126] From the perspective of functional modules, as shown in FIG4 , this embodiment provides a cluster performance statistical data processing device, which is applied to a management platform side. The management platform runs in a management node of a cluster in the form of a container. The device includes:

[0127] The acquisition module 11 is used to obtain the performance data corresponding to each performance indicator;

[0128] Aggregation module 12, used for aggregating the performance data corresponding to each performance indicator according to the corresponding aggregation precision according to the preset aggregation precision strategy, a plurality of performance data within the time range corresponding to the aggregation precision are aggregated into a historical data, and the historical data is saved in the database; wherein the aggregation precision strategy includes a plurality of different aggregation precisions, and each performance indicator uniquely corresponds to a certain aggregation precision;

[0129] The statistics module 13 is used to determine the storage time of each historical data stored in the database;

[0130] The clearing module 14 is used to clear historical data whose storage time exceeds the corresponding clearing time limit according to a preset data clearing strategy; wherein the data clearing strategy includes multiple clearing time limits, and the clearing time limits correspond to the performance indicators one by one.

[0131] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0132] FIG5 is a structural diagram of a cluster performance statistical data processing device provided by another embodiment of the present invention. As shown in FIG5 , a cluster performance statistical data processing device includes: a memory 20 for storing computer programs;

[0133] The processor 21 is configured to implement the steps of a cluster performance statistical data processing method in the above embodiment when executing a computer program.

[0134] The cluster performance statistical data processing device provided in this embodiment may include but is not limited to a mobile terminal, a personal computer, a workstation, etc.

[0135] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0136] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of a cluster performance statistical data processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. Data 203 may include, but is not limited to, a cluster performance statistical data processing method, etc.

[0137] In some embodiments, a cluster performance statistical data processing device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0138] Those skilled in the art will appreciate that the structure shown in FIG. 5 does not constitute a limitation on a cluster performance statistical data processing device, and may include more or fewer components than those shown in the figure.

[0139] An embodiment of the present invention provides a cluster performance statistical data processing device, including a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a cluster performance statistical data processing method.

[0140] In addition to the embodiments of a cluster performance statistics data processing method and device provided in the above embodiments, the present invention also provides an embodiment corresponding to a computer program product. A computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the cluster performance statistics data processing method described in any of the above embodiments can be implemented.

[0141] Since the embodiments of the computer program product part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the computer program product part, which will not be repeated here.

[0142] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0143] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0144] The above is a detailed introduction to a cluster performance statistics data processing method, device and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0145] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A cluster performance statistical data processing method, characterized in that: Applied to the management platform side, the management platform runs in the management node of the cluster in the form of a container, and the method includes: Obtain performance data corresponding to each performance indicator; According to a preset aggregation precision strategy, the performance data corresponding to each performance indicator is aggregated according to the corresponding aggregation precision, and multiple performance data within the time range corresponding to the aggregation precision are aggregated into a historical data, and the historical data is saved in a database; wherein the aggregation precision strategy includes multiple different aggregation precisions, and each performance indicator uniquely corresponds to a certain aggregation precision; Determining the storage duration of each of the historical data stored in the database; According to a preset data cleaning strategy, the historical data whose storage time exceeds the corresponding cleaning time limit is cleared; wherein the data cleaning strategy includes a plurality of cleaning time limits, and the cleaning time limits correspond to the performance indicators one by one.

2. The cluster performance statistical data processing method according to claim 1, characterized in that: According to a preset aggregation precision strategy, the performance data corresponding to each performance indicator is aggregated according to the corresponding aggregation precision, and a plurality of the performance data within a time range corresponding to the aggregation precision are aggregated into one historical data including: For each of the performance indicators, obtaining from the database the corresponding aggregation timestamp of the most recently aggregated historical data; According to the current system time, subtract a duration corresponding to the aggregation precision to obtain the aggregation time limit; In a time range with the aggregation timestamp plus a duration corresponding to the aggregation accuracy as the starting time and the aggregation time limit as the ending time, the aggregation time period is divided according to the corresponding aggregation accuracy; wherein, the first aggregation time period takes the aggregation timestamp as the starting time; if multiple aggregation time periods can be divided, the ending time of the previous aggregation time period is the starting time of the next aggregation time period, the duration of each aggregation time period is equal, and the time range that is less than one aggregation time period does not participate in the division of the aggregation time period; The average value of the performance data in each aggregation period is used as the historical data after aggregation in the current aggregation period, and the aggregation timestamp of the historical data is the starting time of the aggregation period.

3. The cluster performance statistical data processing method according to claim 2, characterized in that: Before aggregating the performance data corresponding to each performance indicator according to the corresponding aggregation precision according to the preset aggregation precision strategy, the method further includes: Determine whether the aggregation switch is in the on state; If yes, then enter the step of aggregating the performance data corresponding to each performance indicator according to the corresponding aggregation precision according to the preset aggregation precision strategy; If not, proceed to the step of determining the storage duration of each of the historical data stored in the database.

4. The cluster performance statistical data processing method according to claim 2, characterized in that: The clearing of the historical data whose storage time exceeds the corresponding clearing time limit according to the preset data clearing strategy includes: Obtaining performance model annotations; wherein the performance model annotations include model annotation items corresponding to each performance indicator, and the model annotation items include: table name, timestamp column name, performance value column name, and label column name; Acquire the historical data corresponding to each of the performance indicators from a database according to the performance model annotation; Determine the storage duration of each of the historical data according to the current system time and the aggregated timestamp of the historical data; Determine whether the storage time of each of the historical data exceeds the corresponding cleaning time limit; If it exceeds, generating a cleanup strategy configuration class according to the model annotation item of the historical data corresponding to the performance indicator, and saving the cleanup strategy configuration class in the table cleanup strategy; A delete command statement is generated according to the table cleanup strategy, and the historical data whose storage time exceeds the corresponding cleanup time limit is cleared by executing the delete command statement.

5. The cluster performance statistical data processing method according to claim 4, characterized in that: The data cleaning strategy includes: a plurality of pre-configured default cleaning time limits, the default cleaning time limits corresponding to the performance indicators one by one; Before determining whether the storage time of each of the historical data exceeds the corresponding cleaning time limit, the method further includes: Determine whether a unified cleanup time limit is configured on the user side; If not, the cleaning time limit corresponding to each performance indicator is: the default cleaning time limit pre-configured in the data cleaning strategy and corresponding to the performance indicator; If so, the cleaning time limit corresponding to each of the performance indicators is the unified cleaning time limit.

6. The cluster performance statistical data processing method according to claim 4, characterized in that: The management platform includes a plurality of threads, and the threads are used to perform the aggregation task of the performance data and the deletion task of the historical data by calling a scheduled task cycle; Among them, the thread uniquely corresponds to one aggregation precision, and is used to implement the aggregation task and the deletion task of one or more performance indicators corresponding to the aggregation precision; and the timing duration of the scheduled task is the same as the duration corresponding to the aggregation precision.

7. The cluster performance statistical data processing method according to claim 6, characterized in that: Also includes: When any of the threads fails to execute the aggregation task or the deletion task, the failed aggregation task or the deletion task is terminated, and the scheduled task is maintained to re-trigger the aggregation task and the deletion task in the next cycle of the scheduled task.

8. A cluster performance statistical data processing device, characterized in that: Applied to the management platform side, the management platform runs in the management node of the cluster in the form of a container, and the device includes: A collection module is used to obtain performance data corresponding to each performance indicator; An aggregation module, configured to aggregate the performance data corresponding to each performance indicator according to a corresponding aggregation precision according to a preset aggregation precision strategy, aggregate multiple performance data within a time range corresponding to the aggregation precision into one historical data, and save the historical data into a database; wherein the aggregation precision strategy includes multiple different aggregation precisions, and each performance indicator uniquely corresponds to a certain aggregation precision; A statistical module, used to determine the storage time of each of the historical data stored in the database; A clearing module is used to clear the historical data whose storage time exceeds the corresponding clearing time limit according to a preset data clearing strategy; wherein the data clearing strategy includes multiple clearing time limits, and the clearing time limits correspond to the performance indicators one by one.

9. A cluster performance statistical data processing device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the cluster performance statistical data processing method as described in any one of claims 1 to 7 when executing the computer program.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the cluster performance statistical data processing method according to any one of claims 1 to 7 are implemented.

Images