Computing cluster and data collection method, device and storage medium thereof

Abstract

This application provides a computing cluster and its data acquisition method, device, and storage medium. In this application embodiment, in a computing cluster scenario, the acquisition frequency of performance indicator data is adaptively changed based on changes in performance indicator data. This ensures acquisition accuracy and guarantees the accuracy of performance analysis and decision-making based on the analysis results, while also reducing the overhead of performance indicator data acquisition and processing. During the adaptive change of acquisition frequency, for at least two computing nodes executing the same task, the master node is responsible for the adaptive change processing of the acquisition frequency and synchronizes it to other computing nodes when necessary. Other computing nodes do not need to be responsible for the adaptive change processing of the acquisition frequency, thus reducing their processing burden.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, and particularly relates to a computing cluster, a data collection method and device thereof and a storage medium. BACKGROUND

[0002] High Performance Computing (HPC) refers to a computing system and environment that usually uses many processors (as part of a single machine) or several computers organized in a cluster (as a single computing resource). There are many types of HPC systems, ranging from large clusters of standard computers to highly specialized hardware. Most cluster-based HPC systems use high-performance networks to interconnect computers.

[0003] Among them, performance monitoring and performance analysis are indispensable parts of building an HPC system. For an HPC system, the processing overhead of performance indicator data is a challenge faced by the HPC system in performance monitoring and performance analysis, including data transmission overhead, data processing overhead, data storage overhead and data analysis overhead.

[0004] In order to reduce the processing overhead of performance indicator data, the usual practice is to use a lower collection frequency to reduce the amount of performance indicator data, but a lower collection frequency may cause the loss or distortion of performance indicator data, affecting the accuracy of performance analysis results, leading to decision-making errors based on performance analysis results. SUMMARY

[0005] Aspects of the present application provide a computing cluster, a data collection method and device thereof and a storage medium to solve the contradiction between the processing overhead of performance indicator data and the accuracy of performance analysis results, ensure the collection accuracy of performance indicators, guarantee the accuracy of performance analysis, and reduce the collection and processing overhead of performance indicator data.

[0006] The embodiment of the present application provides a computing cluster, comprising: a management node and a plurality of computing nodes, each computing node is deployed with a plurality of collectors, different collectors are used for collecting different performance indexes; the management node is used for deploying a same job task on at least two computing nodes in the plurality of computing nodes, and controlling the at least two computing nodes to execute the job task; each computing node is used for starting at least two target collectors related to the job task in the process of executing the job task, so that the at least two target collectors collect at least two kinds of performance index data of the computing node where the at least two target collectors are located at a current collection frequency; in the case that it is determined that the computing node is a master node in the at least two computing nodes, the collection frequency of the at least two target collectors is adjusted according to variation information of the at least two kinds of performance index data; the at least two computing nodes other than the computing node are informed to adjust the collection frequency of the at least two target collectors on the at least two computing nodes other than the computing node, so that the at least two target collectors on the at least two computing nodes other than the computing node continue to collect the at least two kinds of performance index data of the computing node where the at least two target collectors are located at the adjusted collection frequency.

[0007] The embodiment of the present application also provides a data collection method for a computing cluster, comprising: in the process of executing a job task, starting at least two target collectors related to the job task, so that the at least two target collectors collect at least two kinds of performance index data of the computing node where the at least two target collectors are located at a current collection frequency; in the case that it is determined that the computing node is a master node in the at least two computing nodes, the collection frequency of the at least two target collectors is adjusted according to variation information of the at least two kinds of performance index data; the at least two computing nodes other than the computing node are informed to adjust the collection frequency of the at least two target collectors deployed on the at least two computing nodes other than the computing node, so that the at least two target collectors on the at least two computing nodes other than the computing node continue to collect the at least two kinds of performance index data of the computing node where the at least two target collectors are located at the adjusted collection frequency.

[0008] The embodiment of the present application also provides another data collection method for a computing cluster, comprising: in the process of executing a job task, starting at least two target collectors related to the job task, so that the at least two target collectors collect at least two kinds of performance index data of the computing node where the at least two target collectors are located at a current collection frequency; according to the relevance of performance indexes collected by the at least two target collectors, the at least two target collectors are divided into at least two associated collector groups; according to variation information of performance index data collected by target collectors in each associated collector group, the collection frequency of the target collectors in each associated collector group is adjusted respectively, so that the target collectors continue to collect the at least two kinds of performance index data of the computing node where the target collectors are located at the adjusted collection frequency.

[0009] The embodiment of the present application further provides a data collection device, which is applied to any computing node in a computing cluster and comprises: a starting module, configured to start at least two target collectors related to a job task during execution of the job task, so that the at least two target collectors collect at least two kinds of performance index data of the computing node where the at least two target collectors are located at a current collection frequency; an adjusting module, configured to, if it is determined that the computing node is a master node among the at least two computing nodes, adjust the collection frequency of the at least two target collectors according to variation information of the at least two kinds of performance index data; and a notifying module, configured to notify other computing nodes among the at least two computing nodes to adjust the collection frequency of the at least two target collectors deployed on the other computing nodes, so that the at least two target collectors on the other computing nodes continue to collect the at least two kinds of performance index data of the computing node where the at least two target collectors are located at the adjusted collection frequency.

[0010] The embodiment of the present application further provides a computing node, which can be applied to a computing cluster, and the computing node comprises: a memory and a processor; the memory is configured to store a computer program; and the processor is coupled with the memory and is configured to execute the computer program, so as to execute the steps in the above method.

[0011] The embodiment of the present application further provides a computer readable storage medium storing a computer program, when the computer program is executed by a processor, the processor can implement the steps in the above method.

[0012] In the embodiment of the present application, in a computing cluster scenario, the collection frequency of performance index data is adaptively changed according to variation information of the performance index data, which can not only guarantee collection accuracy and ensure accuracy of performance analysis based on the performance index data and decision-making based on analysis results, but also reduce collection and processing overhead of the performance index data; in the process of adaptively changing the collection frequency, for at least two computing nodes executing the same job task, a master node among the at least two computing nodes is responsible for adaptive change processing of the collection frequency and synchronizes to other computing nodes if the collection frequency needs to be changed, and the other computing nodes do not need to be responsible for the adaptive change processing of the collection frequency, so that the processing burden of the other computing nodes can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and the explanation of the present application, and do not constitute improper limitations on the present application. In the drawings:

[0014] Figure 1a A structural schematic diagram of a computing cluster is provided for an illustrative embodiment of the present application;

[0015] Figure 1bA schematic diagram of multiple frequency groups provided for an exemplary embodiment of the present application;

[0016] Figure 2 A flowchart of a data collection method of a computing cluster provided for another exemplary embodiment of the present application;

[0017] Figure 3 A flowchart of a data collection method of a computing cluster provided for another exemplary embodiment of the present application;

[0018] Figure 4 A structural diagram of a data collection device provided for another exemplary embodiment of the present application;

[0019] Figure 5 A structural diagram of a data collection device provided for another exemplary embodiment of the present application;

[0020] Figure 6 A structural diagram of a computing node provided for another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in conjunction with the specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0022] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the drawings.

[0023] Figure 1a A structural diagram of a computing cluster 100 provided for an exemplary embodiment of the present application. The computing cluster 100 of the present embodiment can be implemented as a large-scale computing platform, or an HPC system, or one or more machine rooms, an Internet data center (IDC), or a cloud computing system, etc., and the present embodiment does not limit the specific implementation form of the computing cluster 100. As shown in the figure, the computing cluster 100 includes a management and control node 101 and multiple computing nodes 102. The management and control node 101 and the multiple computing nodes 102, and the multiple computing nodes 102 can be communicatively connected. Figure 1a

[0024] ​In the embodiment, the communication connection described above can be a wired or wireless communication connection. Alternatively, in the case of a wireless communication connection, the nodes can be connected through a mobile network, and the network mode of the mobile network can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), 5G, WiMax, or a new network mode to be introduced in the future, etc. Alternatively, the nodes can also be located in the same local area network, and in the case of a wireless communication connection, the nodes can also be connected through Bluetooth, WiFi, infrared, zigbee, NFC, or the like.

[0025] In the embodiment, the implementation form of the management node 101 and the computing node 102 is not limited. For example, the management node 101 can be deployed on a virtual machine, a cloud server, a cloud host, or a physical machine. Alternatively, the management node 101 can be centrally deployed on one physical machine or one virtual machine, or can be distributedly deployed on multiple physical machines or multiple virtual machines, and is not limited thereto. Correspondingly, the computing node 102 can be any device form with certain computing and communication capabilities, such as a virtual machine, a physical machine (such as a server, a computer device), a cloud server, a cloud host, a virtual center, a server array, or a database, etc.

[0026] The management node 101 can provide a man-machine interface to the user on one hand, and receive the submitted job task from the user through the man-machine interface, and on the other hand, can perform various management and control on the computing cluster 100, for example, deploying the job task on the multiple computing nodes 102, controlling the computing nodes 102 to execute the job task, managing the task execution state of the computing nodes 102, etc. In actual application, one job task can be deployed on one computing node 102, or can also be deployed on at least two computing nodes 102, depending on the type and performance requirement of the job task. For a job task with large computing amount or high computing efficiency requirement, it can be deployed on at least two computing nodes 102 simultaneously, and executed in parallel by the at least two computing nodes 102, to improve the computing efficiency. Based on this, the management node 101 can be specifically used to deploy the same job task on at least two computing nodes 102 of the multiple computing nodes 102, and control the at least two computing nodes 102 to execute the job task. The way of deploying the job task on the computing node 102 can be, but is not limited to: issuing the data related to the job task to the computing node 102, or issuing the task instruction to the computing node 102, and carrying the identification information of the job task in the task instruction, so that the computing node 102 obtains the data related to the job task from the task database according to the identification information of the job task. The way of controlling the computing node 102 to execute the job task can be, but is not limited to: sending the start instruction to the computing node 102, to instruct the computing node 102 to start executing the job task; or issuing the job instruction parameter to the computing node 102, which includes the execution time of the job task, for example, starting the job task after 10 minutes, or starting the job task at xxx minutes of specified time xxx, etc.

[0027] In the embodiment, each computing node 102 acts as a task execution node, can receive a job task deployed by the management node 101, and executes the job task according to the control of the management node 101. In addition, a plurality of collectors are deployed on each computing node 102, each collector is responsible for collecting one kind of performance index data, and different collectors are responsible for collecting different performance index data. The collector can be a program code with a data collection function, which can be a plug-in, SDK relying on a main program, or an independent software function module, and is not limited in this regard. Each collector can collect performance index data at a certain collection frequency, and the number of performance index data is directly related to the size of the collection frequency; the higher the collection frequency, the more performance index data collected, the higher the accuracy of performance analysis and performance monitoring based on the performance index data, and of course, the larger the data transmission, storage and calculation overheads; the lower the collection frequency, the less performance index data collected, the relatively lower the accuracy of performance analysis and performance monitoring based on the performance index data, and of course, the smaller the data transmission, storage and calculation overheads.

[0028] Based on the above, in addition to executing the job task according to the control of the management node 101, each computing node 102 in the embodiment can also start at least two collectors related to the job task during the execution of the job task, so that the started at least two collectors collect at least two kinds of performance index data at the current collection frequency. In order to facilitate description and distinction, the at least two collectors started by the computing node 102 during the execution of the job task and related to the job task are referred to as target collectors, the number of target collectors is at least two, and the target collectors started for different job tasks will be different, which is determined according to the requirements of the job task on the performance index.

[0029] The performance index data in the embodiment includes, but is not limited to, CPU utilization of the computing node 102, memory utilization, memory remaining amount, size of network bandwidth, CPU resource occupied by a job task, memory resource occupied by the job task, bandwidth resource consumed by the job task, and the like. According to the performance index data, performance analysis and performance monitoring can be performed from the dimensions of the computing node 102 and / or the job task. For example, according to the performance index data, performance properties of the computing node 102 can be analyzed or monitored, and the performance properties that can be analyzed or monitored include, but are not limited to, task load condition, network state, and current available resource amount, and the current available resource amount at least includes the remaining amount of CPU or memory of the computing node 102. Further, the management node 101 can obtain the performance properties of the computing node 102, and can determine whether a new job task can be assigned to the computing node 102 and whether the resource amount of the computing node 102 needs to be dynamically adjusted, such as increasing CPU resource or network bandwidth resource, according to the performance properties. For example, according to the performance index data, the running state of the job task, resource consumption condition, and quality of service (QoS) corresponding to the job task can be analyzed or monitored. Further, the management node 101 can obtain the running state of the job task, resource consumption condition, and QoS corresponding to the job task, and can determine whether the computing node 102 needs to be increased or decreased for the job task according to the information, so as to achieve reasonable utilization of node resources as much as possible while guaranteeing the running state and QoS of the job task.

[0030] Based on the above analysis, as shown in FIG. 1, the computer cluster of the embodiment further includes a performance analysis node 104, which is communicatively connected with the management node 101 and the plurality of computing nodes. Figure 1a The performance analysis node 104 is responsible for receiving the performance index data reported by each computing node 102, and performing performance analysis and performance monitoring on the computing node 102 and / or the job task according to the performance index data reported by each computing node 102. Performance analysis and monitoring are necessary parts of protective maintenance of the computing cluster 100, which facilitates operation and maintenance personnel to understand the running condition of the entire cluster and observe the resource use efficiency of the cluster.

[0031] Specifically, the performance analysis node 104 can analyze or monitor the performance properties of the computing nodes 102 according to the performance indicator data reported by the computing nodes 102, such as the CPU utilization, memory utilization, memory remaining amount, network bandwidth size, etc. of the computing nodes 102, and the performance properties that can be analyzed or monitored include but are not limited to: task load condition, network state, and current available resource amount, etc. The performance properties of the computing nodes 102 are provided to the management and control node 101 for further decision making. And / or, the performance analysis node 104 can analyze or monitor the running state, resource consumption condition, and QoS performance data corresponding to the job task of the job task according to the performance indicator data reported by the computing nodes 102, such as the CPU utilization, memory utilization, CPU resource amount, memory resource amount, and consumed bandwidth resource of the job task of the computing nodes 102, and provide various performance data of the job task to the management and control node 101 for further decision making. For example, the management and control node 101 can analyze the behavior characteristics of the job task at runtime, and adapt the optimal resource configuration for the job task according to the behavior characteristics of the job task at runtime, and the resource configuration at least includes the number of computing nodes 102, CPU, memory, network, etc. on each computing node 102.

[0032] Based on the above, for the case of deploying the same job task on at least two computing nodes 102, the performance analysis node 104 can be specifically used to analyze the latest performance properties of the at least two computing nodes 102 according to the at least two kinds of performance indicator data respectively reported by the at least two computing nodes 102, and provide the latest performance properties to the management and control node 101 for further decision making, so as to form a closed loop in performance management and control of the entire computer cluster.

[0033] In the embodiments of the present application, each computing node 102 can execute a job task and can start at least two target collectors related to the job task during the execution of the job task, so that the at least two target collectors collect at least two kinds of performance indicator data of the computing node 102 where the target collectors are located at the current collection frequency. In addition, during the collection of the performance indicator data by the at least two target collectors, each computing node 102 can also adaptively change the collection frequency used by the at least two target collectors to collect the performance indicator data according to the change information of the performance indicator data, so as to realize variable frequency collection of the performance indicator data, which can not only guarantee the collection accuracy and the accuracy of performance analysis and decision making based on the performance indicator data, but also reduce the collection and processing overhead of the performance indicator data.

[0034] In the embodiment, since at least two target collectors on each computing node 102 can need to do frequency conversion during collection, if the collection frequency of each computing node 102 is calculated and adjusted one by one for the at least two target collectors, the data processing amount is too large, which is time-consuming and laborious, especially in the supercomputing scenario, the number of computing nodes 102 and target collectors is large, the calculation amount is large, and the overall performance of the computing node 102 is affected. In order to reduce the data processing amount caused by dynamic adjustment of the collection frequency and improve the adjustment efficiency of the frequency of the target collector, in the embodiment of the application, for at least two computing nodes 102 deploying the same job task, a master node 103 can be selected from the at least two computing nodes 102, the master node 103 adjusts the collection frequency of the at least two target collectors according to the change information of the at least two performance indicator data of the master node 103 collected by the at least two target collectors, and notifies other computing nodes 102 in the at least two computing nodes 102 to adjust the collection frequency of the at least two target collectors on the at least two computing nodes 102, so that the at least two target collectors on each computing node 102 can continue to collect the at least two performance indicator data at the adjusted collection frequency. It should be noted that the at least two target collectors on each computing node are responsible for collecting the at least two performance indicator data of the computing node. In the process, only the master node 103 is responsible for performing the data processing operation related to the frequency adjustment, and the other computing nodes 102 do not need to perform the data processing operation related to the collection frequency adjustment, and can directly adjust the collection frequency of the at least two target collectors on the at least two computing nodes 102 according to the notification of the master node 103, which can reduce the data processing amount caused by dynamic adjustment of the collection frequency, and can improve the adjustment efficiency of the collection frequency of the target collector.

[0035] In the above or below embodiments of the application, each computing node 102 starts at least two target collectors related to the job task during the execution of the task, and a specific implementation manner is as follows: at least two target collectors corresponding to at least two performance indicators related to the task executed by itself are determined; then the identification information of the at least two target collectors is stored locally, and a start instruction is sent to the corresponding at least two target collectors; after receiving the start instruction, the at least two target collectors start running and collect the corresponding performance indicator data.

[0036] Further, the at least two collectors send the collected performance indicator data to the corresponding computing node 102, and the corresponding computing node 102 adjusts the collection frequency of the at least two target collectors according to the change information of the at least two kinds of performance indicator data collected by the at least two target collectors. Since the collection frequencies of the target collectors in the execution of the same job task can be the same or similar, in order to reduce the task amount of the computing nodes 102 for adjusting the collection frequencies of the target collectors, one computing node 102 can be selected as the master node 103 from the at least two computing nodes 102, and the master node 103 determines the adjusted collection frequency according to the change information of the at least two kinds of performance indicator data collected by the at least two target collectors on the master node 103, adjusts the collection frequencies of the at least two target collectors on the master node 103, and notifies the other computing nodes 102, and the other computing nodes 102 directly adjust the collection frequencies of the at least two target collectors on the other computing nodes 102 according to the notification. Based on this, for the at least two computing nodes 102 executing the same job task, each computing node 102 needs to determine whether it is the master node 103, and in the case of determining that it is the master node 103, adjusts the collection frequencies of the at least two target collectors according to the change information of the at least two kinds of performance indicator data collected by the at least two target collectors on the computing node 102, and notifies the other computing nodes 102 to adjust the collection frequencies of the at least two target collectors on the other computing nodes 102. Further, for each computing node 102, in the case of determining that it is not the master node 103, the computing node 102 can wait for the notification of the master node 103, before receiving the notification of the master node 103, the at least two target collectors collect the at least two kinds of performance indicator data of the computing node 102 at the current collection frequency, and after receiving the notification of the master node 103, the computing node 102 adjusts the collection frequencies of the at least two target collectors, and the at least two target collectors continue to collect the at least two kinds of performance indicator data of the computing node 102 at the adjusted collection frequency.

[0037] In the embodiments of the present application, the selection manner of the master node 103 is not limited, and the following manners can be used but are not limited to the following manners:

[0038] Manner A1: The master node 103 is selected by the management and control node 101. Specifically, the management and control node 101 is further configured to select the master node 103 from the at least two computing nodes 102 according to the attribute information of the at least two computing nodes 102, and send a notification message to the master node 103. For the at least two computing nodes 102 executing the same job task, each computing node 102 can determine whether it is the master node 103 according to whether the notification message sent by the management and control node 101 is received, and if the notification message is received, it is determined that the computing node 102 is the master node 103, and if the notification message is not received, it is determined that the computing node 102 is not the master node 103.

[0039] In the embodiment, the at least two computing nodes 102 can execute one or more job tasks, and the workloads of different job tasks can be different in size, required network bandwidth and used resource amount. Based on this, the management node 101 selects the master node 103 from the at least two computing nodes 102 according to the attribute information of the at least two computing nodes 102. An optional implementation manner of selecting the master node 103 from the at least two computing nodes 102 is as follows: selecting the master node 103 from the at least two computing nodes 102 according to at least one performance attribute of task load, network state and available resource amount of the at least two computing nodes 102. The task load represents the size of the load of the task executed by the computing node 102; the network state represents the size of the network bandwidth of the computing node 102 when executing the task; and the available resource amount represents the current remaining amount of CPU, memory and the like of the computing node 102.

[0040] In an optional embodiment, several specific implementation manners of selecting the master node 103 from the at least two computing nodes 102 are as follows: selecting the master node 103 from the at least two computing nodes 102 according to the task load of the at least two computing nodes 102; or selecting the master node 103 from the at least two computing nodes 102 according to the network state of the at least two computing nodes 102; or selecting the master node 103 from the at least two computing nodes 102 according to the available resource amount of the at least two computing nodes 102; or selecting the master node 103 from the at least two computing nodes 102 according to the task load and the network state of the at least two computing nodes 102; or selecting the master node 103 from the at least two computing nodes 102 according to the task load and the available resource amount of the at least two computing nodes 102; or selecting the master node 103 from the at least two computing nodes 102 according to the network state and the available resource amount of the at least two computing nodes 102; or selecting the master node 103 from the at least two computing nodes 102 according to the task load, the network state and the available resource amount of the at least two computing nodes 102.

[0041] Further, in the optional embodiment above, when the master node 103 is selected from the at least two computing nodes 102 according to the task load of the at least two computing nodes 102, the computing node 102 with the smaller task load is selected as the master node 103; when the master node 103 is selected from the at least two computing nodes 102 according to the network status of the at least two master nodes 103, the computing node 102 with the better network status is selected as the master node 103; when the master node 103 is selected from the at least two computing nodes 102 according to the available resource amount of the at least two computing nodes 102, the computing node 102 with the larger available resource amount is selected as the master node 103; when the master node 103 is selected from the at least two computing nodes 102 according to the task load and the network status of the at least two computing nodes 102, the computing node 102 with the smaller task load and the better network status is selected as the master node 103; when the master node 103 is selected from the at least two computing nodes 102 according to the task load and the available resource amount of the at least two computing nodes 102, the computing node 102 with the smaller task load and the larger available resource amount is selected as the master node 103; when the master node 103 is selected from the at least two computing nodes 102 according to the network status and the available resource amount of the at least two computing nodes 102, the computing node 102 with the better network status and the larger available resource amount is selected as the master node 103; or when the master node 103 is selected from the at least two computing nodes 102 according to the task load, the network status and the available resource amount of the at least two computing nodes 102, the computing node 102 with the smaller task load, the better network status and the larger available resource amount is selected as the master node 103.

[0042] It should be noted that the above-mentioned manner of determining the master node 103 from the at least two computing nodes 102 is only exemplary and is not limited thereto.

[0043] Further, in the embodiment, considering that the performance attribute of the master node 103 can be dynamically changed, in order to facilitate the execution efficiency of dynamically adjusting the collection frequency, a new master node 103 can be dynamically replaced. Based on this, the performance analysis node 104 can analyze the latest performance attribute of the at least two computing nodes 102 according to the at least two kinds of performance index data respectively reported by the at least two computing nodes 102, and provide the latest performance attribute to the management and control node 101; the management and control node 101 is further configured to: according to the latest performance attribute of the at least two computing nodes 102, reselect a new master node 103 from the at least two computing nodes 102, and send a notification message to the new master node 103. Further, when the new master node 103 receives the notification message, the new master node 103 can automatically become the new master node 103. Further, the management and control node 101 can also send indication information to the original master node 103 to become a non-master node 103, and the original master node 103 closes the function of the master node 103 when receiving the indication information. It should be noted that in the embodiment, the performance analysis node 104 can also be configured to analyze the latest performance attribute of the at least two computing nodes 102 according to the at least two kinds of performance index data respectively reported by the at least two computing nodes 102, and provide the latest performance attribute to the management and control node 101. Figure 1aThe way in which the management and control node 101 selects the master node 103 and dynamically updates the master node 103 is shown in FIG. 1.

[0044] In a specific embodiment, for each computing node 102 performing the same job task, whether the computing node 102 is the master node 103 can be determined according to the specified attribute information of at least two computing nodes 102 (i.e. itself and other computing nodes 102) and in combination with a pre-set condition that should be satisfied when selecting the master node 103 according to the specified attribute information. The specified attribute information can be the device number or IP address of the computing node 102, and the condition that should be satisfied when selecting the master node 103 according to the specified attribute information can be that the node with the largest device number or IP address is the master node 103, or that the node with the smallest device number or IP address is the master node 103. Based on this, one way of determining whether the computing node 102 is the master node 103 according to the device number or IP address of at least two computing nodes 102 and in combination with the pre-set condition that should be satisfied when selecting the master node 103 according to the specified attribute information includes: each computing node 102 compares its own device number or IP address with the device number or IP address of other computing devices; if the device number or IP address of the computing node 102 is the largest, then the computing node 102 is determined to be the master node 103, otherwise the computing node 102 is determined not to be the master node 103. In another optional embodiment, each computing node 102 compares its own device number or IP address with the device number or IP address of other computing devices; if the device number or IP address of the computing node 102 is the smallest, then the computing node 102 is determined to be the master node 103, otherwise the computing node 102 is determined not to be the master node 103. It should be noted that in the above embodiments, the computing node 102 can be the master node 103 only when the computing node 102 is the only computing node 102 performing the same job task, or when the computing node 102 is one of the computing nodes 102 performing the same job task. Figure 1a The way in which the computing node 102 autonomously negotiates the master node 103 is shown in FIG. 1.

[0045] In the above or below embodiments of the present application, the master node 103 can adjust the collection frequencies of the at least two target collectors according to the change information of the at least two kinds of performance indicator data collected by the at least two target collectors. The specific implementation is as follows: first, the at least two target collectors are divided into at least two associated collector groups according to the correlation of the performance indicators collected by the at least two target collectors; and then, the collection frequencies of the target collectors in each associated collector group are adjusted according to the change information of the performance indicator data collected by the target collectors in each associated collector group. In this embodiment, the collectors are grouped, and the collection frequencies of the collectors with strong correlation are adjusted as a group, that is, the collection frequencies of the target collectors in the same associated collector group are the same after adjustment, which is beneficial to further simplify the calculation resources consumed by adjusting the collection frequencies and improve the overall adjustment efficiency of the collection frequencies.

[0046] In an optional embodiment, the target collectors with the correlation of the performance indicators collected by the target collectors being greater than a preset threshold can be divided into at least two associated collector groups. For example, the target collector for collecting CPU utilization and the target collector for collecting CPU floating point operation efficiency can be divided into the same associated collector group, the target collector for collecting memory utilization and the target collector for collecting read / write bandwidth resources can be divided into the same associated collector group, and the target collector for collecting network receiving / transmitting bandwidth resources and the target collector for collecting network receiving / transmitting packet rate can be divided into the same associated collector group, but the present application is not limited thereto.

[0047] Further, in an optional embodiment, for each associated collector group, before adjusting the collection frequencies of the target collectors in the associated collector group, it can be further judged whether the current collection frequencies of the target collectors in the associated collector group are the same, and if not, the current collection frequencies of the target collectors in the associated collector group are adjusted to the same collection frequency. The current collection frequency refers to the collection frequency currently used by the target collectors.

[0048] In the above optional embodiment, in the case that the current collection frequencies of the target collectors in the associated collector group are different, the current collection frequencies of the target collectors in the associated collector group are adjusted to the same collection frequency, which can be implemented in the following optional ways: the current collection frequencies of the target collectors in the associated collector group are all adjusted to the average of the current collection frequencies of the target collectors in the associated collector group; or the current collection frequencies of the target collectors in the associated collector group are all adjusted to the maximum collection frequency among the current collection frequencies of the target collectors in the associated collector group; or the current collection frequencies of the target collectors in the associated collector group are all adjusted to the minimum collection frequency among the current collection frequencies of the target collectors in the associated collector group. The above ways of adjusting the current collection frequencies of the target collectors in the associated collector group to the same collection frequency are only exemplary and are not limited thereto.

[0049] Similarly, in the above optional embodiment, after the current collection frequencies of the target collectors in the associated collector group are adjusted to the same collection frequency, the collection frequencies of the target collectors in each associated collector group are adjusted according to the change information of the performance index data collected by the target collectors in each associated collector group, which can be implemented in the following ways: for each associated collector group, the change information of the performance index data collected by the target collectors in the associated collector group is determined to determine the frequency variation direction corresponding to the associated collector group; then the current collection frequency of the target collectors in the associated collector group is adjusted to the preset frequency closest to the frequency variation direction among a plurality of preset frequencies, wherein the plurality of preset frequencies are arranged in ascending order. In this embodiment, the frequency variation direction includes three kinds of increasing frequency, keeping unchanged and decreasing frequency, but is not limited thereto. The frequency variation granularity of increasing frequency and decreasing frequency can be refined to obtain more frequency variation directions. In this embodiment, a plurality of frequencies are preset, and the plurality of preset frequencies are arranged in ascending order and are not the same. Assuming that the plurality of preset frequencies are f1, f2, f3, f4 and f5 in ascending order, and the current collection frequency is f2, and the frequency variation direction is increasing frequency, the preset frequency closest to the frequency variation direction among the plurality of preset frequencies is f3; similarly, in the case that the frequency variation direction is decreasing frequency, the preset frequency closest to the frequency variation direction among the plurality of preset frequencies is f1.

[0050] In the above embodiment, a plurality of frequency groups are preset, and each frequency group corresponds to a preset frequency. The preset frequencies are arranged in ascending order and are arranged in descending order of collection frequency, for example, Figure 1bAs shown, the frequency groups include, for example, a high frequency group m-high frequency group 1, a base frequency group, and a low frequency group 1-low frequency group n; where m and n are positive integers. For the master node 103, at the beginning, at least two target collectors can be initialized to the base frequency, and these target collectors are uniformly added to the base frequency group for management; then, according to the relevance of the performance indicators collected by the target collectors, at least two target collectors are divided into different associated collector groups; for each associated collector group, according to the change information of the performance indicator data collected by the target collectors in the associated collector group, the frequency direction corresponding to the associated collector group is determined; and according to the frequency direction, the target collectors in the associated collector group are adjusted from the current frequency group to the frequency group closest in the frequency direction.

[0051] Specifically, for the associated collector group with the frequency direction of increasing frequency, the target collectors in the associated collector group are moved from the base frequency group to the high frequency group 1; for the associated collector group with the frequency direction of decreasing frequency, the target collectors in the associated collector group are moved from the base frequency group to the low frequency group 1; and for the associated collector group with the frequency direction of keeping unchanged, the target collectors in the associated collector group continue to remain in the base frequency group. Over time, the frequency groups of the target collectors in each associated collector group can be continuously adjusted according to the similar frequency conversion mode, and the target collectors located in a certain frequency group use the preset frequency corresponding to the frequency group to collect performance indicator data.

[0052] In an optional embodiment, for each associated collector group, according to the change information of the performance indicator data collected by the target collectors in the associated collector group, the frequency direction corresponding to the associated collector group is determined, and a specific implementation is as follows: first, for each associated collector group, the key performance indicator data collected by the key collectors in the associated collector group is obtained, the key collectors are target collectors responsible for collecting key performance indicators, and the key indicator data is part of the indicator data that can be collected by the target collectors in the associated collector group, for example, the key performance indicator data can be one or more indicator data with high importance ranking; then, according to the set statistical interval, the change rates of the key performance indicator data are calculated, and a global change rate is generated according to the change rates of the key performance indicator data; further, according to the global change rate, the frequency direction corresponding to the associated collector group is determined, and the frequency direction corresponding to the associated collector group can be any one of increasing frequency, decreasing frequency, and keeping unchanged.

[0053] In the process of calculating the change rate of each key performance indicator data, the change rate of each key performance indicator data within a statistical interval can be calculated according to the set statistical interval. The present embodiment does not limit the statistical interval. The current collection period corresponding to the current collection frequency of the target collector can be used as the statistical interval. For example, if the collection period corresponding to the current collection frequency is 1 s, the statistical interval is 1 s, that is, one key performance indicator data is collected every 1 s, and the change rate of the key performance indicator data collected in adjacent two times is calculated. Alternatively, a plurality of collection periods can be used as the statistical interval. For example, 10 collection periods can be used as the statistical interval, that is, the statistical interval is 10 s, and the change rate of the key performance indicator data is calculated every 10 s. The change rate of the key performance indicator data within 10 s can be calculated according to 10 key performance indicator data collected within 10 s.

[0054] For ease of description, Pi is used to represent each key performance indicator, Δ(Pi) is used to represent the change amount of the performance value of the key indicator Pi in adjacent two statistical intervals, different weights are given to different key performance indicators, and (W1, W2,..., Wn) is used to represent the weights. Wherein, Wi represents the weight of the i th key performance indicator. The global change rate generated according to the change rate of each key performance indicator data can be represented as: It should be noted that the present embodiment sets a statistical threshold for each key performance indicator. The statistical threshold is represented by (PT1, PT2,..., PTn), wherein PTi represents the minimum change threshold of the i th key performance indicator. Based on this, whether the change amount of each key performance indicator within a judgment period exceeds the threshold can be judged. If the change amount exceeds the threshold, the change rate of each key performance indicator data is calculated. Otherwise, if the change amount of the key performance indicator data of two adjacent periods does not exceed the corresponding threshold, it indicates that the change of the key performance indicator data is small, and the change direction can be directly set to 0, indicating that the frequency needs not to be adjusted, that is, the frequency direction remains unchanged.

[0055] In the case of calculating the change rate of each key performance indicator data, the global change rate is generated according to the above weighted sum formula. Further, according to the global change rate, the variable frequency strategy is used to determine the variable frequency direction corresponding to the associated collector group, and one specific embodiment is as follows: the global change rate is taken as the input of the variable frequency strategy, and the variable frequency direction corresponding to the associated collector group is determined according to the output value. Wherein, when the output value is 0, it means that the frequency of the target collector is not changed; when the output value is 1, it means that the collection frequency of the target collector needs to be increased; when the output value is -1, it means that the collection frequency of the target collector needs to be reduced. Further optionally, the output value is determined by using the variable frequency strategy, and one specific embodiment is as follows: the weighted change rate of the key indicator data in the associated collector group is calculated, and for the convenience of description, the weighted change rate of the key indicator data is represented as KeyDelta, and the upper and lower limits of the change rate threshold are (β1, β2), wherein β1≤β2, when KeyDelta>β2, the collection frequency needs to be increased, and the output value is 1; when KeyDelta<β1, the collection frequency needs to be reduced, and the output value is -1; when β1≤KeyDelta≤β2, the original collection frequency is maintained, and the output value is 0.

[0056] In an optional embodiment, in order to further improve the accuracy of the variable frequency direction, for each associated collector group, the variable frequency direction corresponding to the associated collector group is determined according to the change information of the performance indicator data collected by the target collector in the associated collector group. Another specific embodiment is as follows: for each associated collector group, the variable frequency direction corresponding to the associated collector group is determined according to the change information of the performance indicator data collected by the target collector in the associated collector group and the performance analysis result obtained by the performance indicator data collected by at least two target collectors last time. Further optionally, for each associated collector group, the first variable frequency direction corresponding to the associated collector group is determined according to the change information of the performance indicator data collected by the target collector in the associated collector group, and the second variable frequency direction corresponding to the associated collector group is determined according to the analysis result obtained by the performance indicator data collected by at least two target collectors last time. If the first variable frequency direction and the second variable frequency direction are the same, the first variable frequency direction is determined as the variable frequency direction corresponding to the associated collector group; if the first variable frequency direction and the second variable frequency direction are not the same, the current collection frequency can not be adjusted temporarily, and the first variable frequency direction in a plurality of statistical intervals can be continuously counted, and the variable frequency direction corresponding to the associated collector group is finally determined according to the first variable frequency direction in the continuous plurality of statistical intervals.

[0057] It should be noted that, in the process of adjusting the collection frequency of the target collector in the associated collector group, if the current collection frequency of the target collector in the associated collector group is the maximum preset frequency, and the frequency conversion direction is to increase the frequency, the current collection frequency is kept unchanged; if the current collection frequency of each target collector in the associated collector group is the minimum preset frequency, and the frequency conversion direction is to decrease the frequency, the current collection frequency is kept unchanged.

[0058] It is explained that, in the embodiment, each computing node 102 can group at least two target collectors locally, and each computing node 102 groups the target collectors according to the same standard. In this way, the master node 103 can send a notification message to other computing nodes 102 when the frequency conversion direction of a certain associated collector group is to increase the frequency or decrease the frequency, and the notification message carries the indication information of increasing the frequency or decreasing the frequency. In this way, other computing nodes 102 can increase or decrease the sampling frequency of the target collectors in the corresponding associated collector group according to the indication information. Specifically, the sampling frequency can be increased or decreased to the preset frequency closest to the frequency conversion direction. Further optionally, after dividing the at least two target collectors into at least two associated collector groups, each computing node 102 can also uniformly adjust the current collection frequency of each target collector in each associated collector group in the same way. In this way, the current collection frequencies of the same associated collector groups on different computing nodes 102 are the same, so that the target collectors in the same associated collector group on each computing node 102 can be increased or decreased at the same time, ensuring that the same collection frequency is used, ensuring the consistency of the collection frequencies of the same collectors on different computing nodes 102, and facilitating subsequent analysis and processing of the same performance index data.

[0059] In the above embodiments of the present application, in the computing cluster 100 scenario, the collection frequency of the performance index data is adaptively changed according to the change information of the performance index data, which can not only ensure the collection accuracy and the accuracy of the performance analysis based on the performance index data and the decision based on the analysis result, but also reduce the collection overhead. In the process of adaptively changing the collection frequency, the master node 103 in the at least two computing nodes 102 is responsible for the adaptive change of the collection frequency and synchronizes the change to other computing nodes 102 when necessary, and other computing nodes 102 do not need to be responsible for the adaptive change of the collection frequency, which can reduce the collection overhead of other computing nodes 102 and further reduce the overall collection overhead.

[0060] In the above embodiment, the case where the same job task is deployed on at least two computing nodes 102 is described, and how the computing node 102 acting as the master node 103 implements the function of adjusting the collection frequency. In addition to this, the same job task can also be deployed on one computing node 102, in which case, no master node 103 needs to be selected, and the computing node 102 executing the job task can adjust the collection frequency of the local target collector by itself in the following manner, which specifically includes: during the execution of the job task, starting at least two target collectors related to the job task, so that the at least two target collectors collect at least two kinds of performance indicator data of the computing node where the target collectors are located at the current collection frequency; according to the correlation of the performance indicators collected by the at least two target collectors, dividing the at least two target collectors into at least two associated collector groups; according to the change information of the performance indicator data collected by the target collectors in each associated collector group, respectively adjusting the collection frequency of the target collectors in each associated collector group, so that the target collectors in each associated collector group continue to collect at least two kinds of performance indicator data of the computing node where the target collectors are located at the adjusted collection frequency. The detailed implementation of each step can be referred to the description of the foregoing embodiment, which will not be described here.

[0061] Figure 2 A flowchart of a data collection method for a computing cluster 100 is provided for an exemplary embodiment of the present application.

[0062] As shown in Figure 2 , the method comprises:

[0063] 201. During the execution of a job task, at least two target collectors related to the job task are started, so that the at least two target collectors collect at least two kinds of performance indicator data of the computing node where the target collectors are located at the current collection frequency, and the job task is deployed on at least two computing nodes in a computing cluster 100;

[0064] 202. In the case where it is determined that the computing node is a master node among the at least two computing nodes, the collection frequency of the at least two target collectors is adjusted according to the change information of the at least two kinds of performance indicator data;

[0065] 203. The other computing nodes among the at least two computing nodes are notified to adjust the collection frequency of the at least two target collectors deployed thereon, so that the at least two target collectors on the other computing nodes continue to collect at least two kinds of performance indicator data of the computing node where the target collectors are located at the adjusted collection frequency.

[0066] In the embodiment, the adjustment of the collection frequencies of the at least two target collectors according to the change information of the at least two performance index data comprises: dividing the at least two target collectors into at least two associated collector groups according to the relevance of the performance indexes collected by the at least two target collectors; and adjusting the collection frequencies of the target collectors in each associated collector group respectively according to the change information of the performance index data collected by the target collectors in each associated collector group.

[0067] In an optional embodiment, the adjustment of the collection frequencies of the target collectors in each associated collector group according to the change information of the performance index data collected by the target collectors in each associated collector group comprises: for each associated collector group, determining a frequency variation direction corresponding to the associated collector group according to the change information of the performance index data collected by the target collectors in the associated collector group; and adjusting the current collection frequencies of the target collectors in the associated collector group to a preset frequency closest to the current collection frequencies of the target collectors in the associated collector group in the frequency variation direction, wherein the preset frequencies are arranged in ascending order.

[0068] Further optionally, for each associated collector group, in the process of adjusting the collection frequencies of the target collectors in the associated collector group, further comprises: if the current collection frequencies of the target collectors in the associated collector group are the maximum preset frequency and the frequency variation direction is to increase the frequency, keeping the current collection frequencies unchanged; or if the current collection frequencies of the target collectors in the associated collector group are the minimum preset frequency and the frequency variation direction is to decrease the frequency, keeping the current collection frequencies unchanged.

[0069] Further optionally, for each associated collector group, before the determination of the frequency variation direction corresponding to the associated collector group according to the change information of the performance index data collected by the target collectors in the associated collector group, further comprises: for each associated collector group, if the current collection frequencies of the target collectors in the associated collector group are different, adjusting the current collection frequencies of the target collectors in the associated collector group to the same collection frequency.

[0070] In an optional embodiment, the adjustment of the current collection frequencies of the target collectors in the associated collector group to the same collection frequency comprises: adjusting the current collection frequencies of the target collectors in the associated collector group to the average of the current collection frequencies of the target collectors in the associated collector group; or adjusting the current collection frequencies of the target collectors in the associated collector group to the maximum collection frequency among the current collection frequencies of the target collectors in the associated collector group; or adjusting the current collection frequencies of the target collectors in the associated collector group to the minimum collection frequency among the current collection frequencies of the target collectors in the associated collector group.

[0071] In an optional embodiment, for each associated collector group, the variable frequency direction corresponding to the associated collector group is determined according to the change information of the performance indicator data collected by the target collector in the associated collector group, including: for each associated collector group, obtaining the key performance indicator data collected by the key collector in the associated collector group, the key collector being the target collector responsible for collecting the key performance indicator; according to the set statistical interval, the change rate of each key performance indicator data is calculated, and a global change rate is generated according to the change rate of each key performance indicator data; according to the global change rate, the variable frequency direction corresponding to the associated collector group is determined, and the variable frequency direction includes any one of increasing frequency, reducing frequency and keeping unchanged.

[0072] In an optional embodiment, for each associated collector group, the variable frequency direction corresponding to the associated collector group is determined according to the change information of the performance indicator data collected by the target collector in the associated collector group, including: for each associated collector group, according to the change information of the performance indicator data collected by the target collector in the associated collector group, and the performance analysis result obtained by the performance indicator data collected by the at least two target collectors last time, the variable frequency direction corresponding to the associated collector group is determined.

[0073] It should be noted that the principles of the specific implementation of each step in the data collection method for computing cluster 100 provided in this embodiment can be referred to the corresponding content in the computing cluster 100 embodiment described above, which will not be repeated here.

[0074] Figure 3 Another flowchart of a data collection method for computing cluster 100 is provided for an exemplary embodiment of the present application. As shown in Figure 3 The method comprises:

[0075] 301、In the process of executing a job task, at least two target collectors related to the job task are started to make the at least two target collectors collect at least two performance indicator data of the computing node where they are located at the current collection frequency;

[0076] 302、According to the correlation of the performance indicators collected by the at least two target collectors, the at least two target collectors are divided into at least two associated collector groups;

[0077] 303、According to the change information of the performance indicator data collected by the target collector in each associated collector group, the collection frequency of the target collector in each associated collector group is adjusted respectively, so that the target collector continues to collect at least two performance indicator data of the computing node where it is located at the adjusted collection frequency.

[0078] It should be noted that the principle of the specific implementation of each step in the data collection method for computing cluster 100 provided in this embodiment can be referred to the corresponding content in the computing cluster 100 embodiment described above, which will not be repeated here.

[0079] Figure 4 The structural schematic diagram of the data collection device provided in an exemplary embodiment of the present application is shown in FIG. 1. As shown in the figure, the device comprises: Figure 4

[0080] The starting module 41 is configured to start at least two target collectors related to a job task during execution of the job task, so that the at least two target collectors collect at least two kinds of performance indicator data of the computing nodes where the at least two target collectors are located at a current collection frequency, and the job task is deployed on at least two computing nodes in the computing cluster 100.

[0081] The adjusting module 42 is configured to, in a case where it is determined that the self is a master node in the at least two computing nodes, adjust the collection frequency of the at least two target collectors according to the change information of the at least two kinds of performance indicator data.

[0082] The notification module 43 is configured to notify other computing nodes in the at least two computing nodes to adjust the collection frequency of the at least two target collectors deployed thereon, so that the at least two target collectors on the other computing nodes continue to collect at least two kinds of performance indicator data of the computing nodes where the at least two target collectors are located at the adjusted collection frequency.

[0083] In an optional embodiment, when the adjusting module 42 is configured to adjust the collection frequency of the at least two target collectors according to the change information of the at least two kinds of performance indicator data, it is specifically configured to: divide the at least two target collectors into at least two associated collector groups according to the relevance of the performance indicators collected by the at least two target collectors; and adjust the collection frequency of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in each associated collector group, respectively.

[0084] In an optional embodiment, when the adjusting module 42 is configured to adjust the collection frequency of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in each associated collector group, it is specifically configured to: for each associated collector group, determine a frequency change direction corresponding to the associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group; and adjust the current collection frequency of the target collectors in the associated collector group to a preset frequency closest to the frequency change direction in a plurality of preset frequencies, wherein the plurality of preset frequencies are arranged in ascending order. ​

[0085] Further optionally, the adjusting module 42, when used for adjusting the collection frequencies of the target collectors in each associated collector group, is further used for: if the current collection frequency of the target collectors in the associated collector group is the maximum preset frequency, and the frequency conversion direction is to increase the frequency, keeping the current collection frequency unchanged; if the current collection frequencies of the target collectors in the associated collector group are the minimum preset frequency, and the frequency conversion direction is to decrease the frequency, keeping the current collection frequency unchanged.

[0086] Further optionally, the adjusting module 42, when used for determining the frequency conversion direction corresponding to each associated collector group according to the change information of the performance index data collected by the target collectors in the associated collector group, is further used for: for each associated collector group, if the current collection frequencies of the target collectors in the associated collector group are different, adjusting the current collection frequencies of the target collectors in the associated collector group to the same collection frequency.

[0087] In an optional embodiment, when the adjusting module 42 is used for adjusting the current collection frequencies of the target collectors in the associated collector group to the same collection frequency, it is specifically used for: adjusting the current collection frequencies of the target collectors in the associated collector group to the average of the current collection frequencies of the target collectors in the associated collector group; or adjusting the current collection frequencies of the target collectors in the associated collector group to the maximum collection frequency among the current collection frequencies of the target collectors in the associated collector group; or adjusting the current collection frequencies of the target collectors in the associated collector group to the minimum collection frequency among the current collection frequencies of the target collectors in the associated collector group.

[0088] In an optional embodiment, when the adjusting module 42 is used for determining the frequency conversion direction corresponding to each associated collector group according to the change information of the performance index data collected by the target collectors in the associated collector group, it is specifically used for: for each associated collector group, obtaining the key performance index data collected by the key collector in the associated collector group, the key collector being the target collector responsible for collecting the key performance index; according to a set statistical interval, calculating the change rates of the key performance index data, and generating a global change rate according to the change rates of the key performance index data; and determining the frequency conversion direction corresponding to the associated collector group according to the global change rate, the frequency conversion direction including any one of increasing the frequency, decreasing the frequency, and keeping unchanged.

[0089] In an optional embodiment, when the adjustment module 42 determines the frequency conversion direction corresponding to each associated collector group based on the change information of the performance index data collected by the target collectors within the associated collector group, it is specifically used to: determine the frequency conversion direction corresponding to each associated collector group based on the change information of the performance index data collected by the target collectors within the associated collector group and the performance analysis result obtained most recently based on the performance index data collected by the at least two target collectors.

[0090] It should be noted that the data acquisition device provided in this embodiment can achieve the above-mentioned... Figure 2 The technical solutions described in the method embodiments, and the specific implementation principles of each module or unit mentioned above, can be found in the above. Figure 1a The computing cluster 100 embodiment shown and Figure 2 The relevant content in the method embodiments will not be repeated here.

[0091] Figure 5 This is a schematic diagram of the structure of another data acquisition device provided as an exemplary embodiment of this application. For example... Figure 5 As shown, the device includes:

[0092] The startup module 51 is used to start at least two target collectors related to the job task during the execution of the job task, so that the at least two target collectors collect at least two performance index data of the computing node where they are located at the current collection frequency.

[0093] Grouping module 52 is used to divide the at least two target collectors into at least two associated collector groups based on the correlation of the performance indicators collected by the at least two target collectors.

[0094] The adjustment module 53 is used to adjust the acquisition frequency of the target acquisition device in each associated acquisition device group according to the change information of the performance index data collected by the target acquisition device in each associated acquisition device group, so that the target acquisition device continues to collect at least two performance index data of its computing node at the adjusted acquisition frequency.

[0095] It should be noted that the data acquisition device provided in this embodiment can achieve the above-mentioned... Figure 3 The technical solutions described in the method embodiments, and the specific implementation principles of each module or unit mentioned above, can be found in the above. Figure 1a The computing cluster 100 embodiment shown and Figure 3 The relevant content in the method embodiments will not be repeated here.

[0096] Figure 6This is a schematic diagram of a computing node provided as an exemplary embodiment of this application. This computing node can serve as any computing node in the aforementioned computing cluster 100, such as… Figure 6 As shown, the computing node includes: a memory 60a and a processor 60b; the memory 60a is used to store computer programs; the processor 60b, coupled to the memory 60a, is used to execute the computer programs for:

[0097] During the execution of a job task, at least two target collectors related to the job task are activated so that the at least two target collectors collect at least two performance index data of the computing nodes where they are located at the current collection frequency. The job task is deployed on at least two computing nodes in the computing cluster 100.

[0098] If it is determined that it is the master node among the at least two computing nodes, the acquisition frequency of the at least two target collectors is adjusted according to the change information of the at least two performance index data.

[0099] The other computing nodes among the at least two computing nodes are notified to adjust the acquisition frequency of the at least two target acquisition devices deployed on them, so that the at least two target acquisition devices on the other computing nodes continue to acquire at least two performance index data of their respective computing nodes at the adjusted acquisition frequency.

[0100] In an optional embodiment, when the processor 60b adjusts the acquisition frequency of the at least two target collectors based on the change information of the at least two performance index data, it is specifically used to: divide the at least two target collectors into at least two associated collector groups based on the correlation of the performance indexes collected by the at least two target collectors; and adjust the acquisition frequency of the target collectors in each associated collector group based on the change information of the performance index data collected by the target collectors in each associated collector group.

[0101] In an optional embodiment, when the processor 60b adjusts the acquisition frequency of the target acquisition device in each associated acquisition device group based on the change information of the performance index data collected by the target acquisition device in each associated acquisition device group, it is specifically used to: determine the frequency conversion direction corresponding to the associated acquisition device group based on the change information of the performance index data collected by the target acquisition device in the associated acquisition device group for each associated acquisition device group; and adjust the acquisition frequency currently used by the target acquisition device in the associated acquisition device group to the preset frequency closest to the frequency conversion direction among a plurality of preset frequencies, wherein the plurality of preset frequencies are arranged from small to large.

[0102] In an optional embodiment, the processor 60b is further configured to, for each associated collector group, keep the current collection frequency unchanged if the current collection frequency of the target collector in the associated collector group is the maximum preset frequency and the frequency conversion direction is to increase the frequency, or keep the current collection frequency unchanged if the current collection frequencies of the target collectors in the associated collector group are the minimum preset frequency and the frequency conversion direction is to decrease the frequency, in the process of adjusting the collection frequency of the target collector in the associated collector group.

[0103] In an optional embodiment, the processor 60b is further configured to, for each associated target collector, determine the frequency conversion direction corresponding to the associated collector group according to the change information of the performance index data collected by the target collectors in the associated collector group, before adjusting the current collection frequencies of the target collectors in the associated collector group to the same collection frequency.

[0104] In an optional embodiment, the processor 60b is further configured to, in the process of adjusting the current collection frequencies of the target collectors in the associated collector group to the same collection frequency, adjust the current collection frequencies of the target collectors in the associated collector group to the average of the current collection frequencies of the target collectors in the associated collector group, or to the maximum collection frequency among the current collection frequencies of the target collectors in the associated collector group, or to the minimum collection frequency among the current collection frequencies of the target collectors in the associated collector group.

[0105] In an optional embodiment, the processor 60b is further configured to, for each associated collector group, obtain the key performance index data collected by the key collector in the associated collector group, the key collector being the target collector responsible for collecting the key performance index, and determine the frequency conversion direction corresponding to the associated collector group according to the change rate of each key performance index data and a global change rate generated according to the change rate of each key performance index data, in the process of determining the frequency conversion direction corresponding to the associated collector group according to the change information of the performance index data collected by the target collectors in the associated collector group.

[0106] In an optional embodiment, when the processor 60b determines the frequency variation direction corresponding to each associated collector group according to the variation information of the performance indicator data collected by the target collector in the associated collector group, for each associated collector group, the processor 60b is specifically configured to: determine the frequency variation direction corresponding to the associated collector group according to the variation information of the performance indicator data collected by the target collector in the associated collector group, and the performance analysis result obtained last time according to the performance indicator data collected by the at least two target collectors.

[0107] In the above embodiments, the case where the same job task is deployed on at least two computing nodes is described, and how the computing node serving as the master node implements the function of performing collection frequency adjustment. In addition to this, the same job task can also be deployed on one computing node, in which case, no master node needs to be selected, and the processor 60b of the computing node executing the job task can also perform the following operations:

[0108] During execution of the job task, starting at least two target collectors related to the job task, so that the at least two target collectors collect at least two kinds of performance indicator data of the computing node where the target collectors are located at a current collection frequency;

[0109] According to the correlation of the performance indicators collected by the at least two target collectors, the at least two target collectors are divided into at least two associated collector groups;

[0110] According to the variation information of the performance indicator data collected by the target collector in each associated collector group, the collection frequency of the target collector in each associated collector group is adjusted respectively, so that the target collector continues to collect at least two kinds of performance indicator data of the computing node where the target collector is located at the adjusted collection frequency.

[0111] Further, as shown in Figure 6 , the electronic device further includes a communication component 60c, a power supply component 60d, and other components. Figure 6 Only some components are shown schematically in the figure, and it does not mean that the electronic device only includes Figure 6 the components shown in the figure.

[0112] It should be noted that the computing node provided in the embodiment can implement the technical solutions described in the above Figure 2 or Figure 3 method embodiments, and the principles of implementation of the above modules or units can be referred to the corresponding content in the above Figure 1a computing cluster 100 embodiment, which will not be described here again.

[0113] An example embodiment of the present application provides a computer-readable storage medium storing computer programs / instructions that, when executed by a processor, enable the processor to implement the steps of the above-described methods, which will not be repeated here.

[0114] An example embodiment of the present application provides a computer program product including computer programs / instructions that, when executed by a processor, enable the processor to implement the steps of the above-described methods, which will not be repeated here.

[0115] The communication component in the above-described embodiments is configured to facilitate wired or wireless communication between the node where the communication component is located and other nodes. The node where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G, or the like, or a combination thereof. In an example embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0116] The display in the above-described embodiments includes a screen, which can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect a duration and a pressure associated with the touching or the sliding action.

[0117] The power supply component in the above-described embodiments provides power to various components of the node where the power supply component is located. The power supply component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the node where the power supply component is located.

[0118] The audio component in the above-described embodiments can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) that is configured to receive an external audio signal when the node where the audio component is located is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in a memory or transmitted via the communication component. In some embodiments, the audio component further includes a speaker for outputting audio signals.

[0119] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0120] The present application is described in reference to the flowchart and / or block diagrams of the method, node (system), and computer program product according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing node to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing node, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0121] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing node to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing node, such that a series of operational steps are carried out on the computer or other programmable node to produce a computer implemented process such that the instructions which execute on the computer or other programmable node provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0123] In one typical configuration, the computing node includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0124] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0125] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage nodes, or any other non-transmission medium that can be used to store information that can be accessed by a computing node. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0126] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or node including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or node. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or node including the element.

[0127] The above only describes the embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various changes and modifications to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A computing cluster, characterized by, The method comprises: controlling the at least two computing nodes to execute the job task; in a case where it is determined that the computing node is a master node in the at least two computing nodes, dividing the at least two target collectors into at least two associated collector groups according to the correlation of the performance indicators collected by the at least two target collectors; and adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group; adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group. The method comprises: in a case where it is determined that the computing node is a master node in the at least two computing nodes, dividing the at least two target collectors into at least two associated collector groups according to the correlation of the performance indicators collected by the at least two target collectors; and adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group; adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group.

2. A data collection method for computing clusters, applied to any computing node in a computing cluster, characterized in that, The method comprises: in a case where it is determined that the computing node is a master node in the at least two computing nodes, dividing the at least two target collectors into at least two associated collector groups according to the correlation of the performance indicators collected by the at least two target collectors; and adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group; adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group. The method comprises:

3. The method of claim 2, wherein, in a case where it is determined that the computing node is a master node in the at least two computing nodes, dividing the at least two target collectors into at least two associated collector groups according to the correlation of the performance indicators collected by the at least two target collectors; and adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group; adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group. The method comprises:

4. The method of claim 3, wherein, in a case where it is determined that the computing node is a master node in the at least two computing nodes, dividing the at least two target collectors into at least two associated collector groups according to the correlation of the performance indicators collected by the at least two target collectors; and adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group; adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group. The method comprises: in a case where it is determined that the computing node is a master node in the at least two computing nodes, dividing the at least two target collectors into at least two associated collector groups according to the correlation of the performance indicators collected by the at least two target collectors; and adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group; adjusting the collection frequencies of the target collectors in each associated collector group according to the change information of the performance indicator data collected by the target collectors in the associated collector group. If the current collection frequency of the target collector in the associated collector group is the maximum preset frequency, and the frequency conversion direction is to increase the frequency, the current collection frequency is kept unchanged; If the current collection frequency of each target collector in the associated collector group is the minimum preset frequency, and the frequency conversion direction is to decrease the frequency, the current collection frequency is kept unchanged.

5. The method of claim 3, wherein, Before determining the frequency conversion direction corresponding to each associated collector group according to the change information of the performance index data collected by the target collectors in the associated collector group, the method further comprises: For each associated collector group, if the current collection frequencies of the target collectors in the associated collector group are different, adjusting the current collection frequencies of the target collectors in the associated collector group to the same collection frequency.

6. The method of claim 5, wherein, Adjusting the current collection frequencies of the target collectors in the associated collector group to the same collection frequency comprises: adjusting the current collection frequencies of the target collectors in the associated collector group to the average value of the current collection frequencies of the target collectors in the associated collector group; or adjusting the current collection frequencies of the target collectors in the associated collector group to the maximum collection frequency among the current collection frequencies of the target collectors in the associated collector group; or adjusting the current collection frequencies of the target collectors in the associated collector group to the minimum collection frequency among the current collection frequencies of the target collectors in the associated collector group.

7. The method of claim 3, wherein, For each associated collector group, determining the frequency conversion direction corresponding to the associated collector group according to the change information of the performance index data collected by the target collectors in the associated collector group comprises: For each associated collector group, obtaining the key performance index data collected by the key collectors in the associated collector group, the key collectors being the target collectors responsible for collecting the key performance index; According to the set statistical interval, the change rates of the key performance index data are calculated, and a global change rate is generated according to the change rates of the key performance index data; According to the global change rate, the frequency conversion direction corresponding to the associated collector group is determined, the frequency conversion direction including any one of increasing frequency, decreasing frequency and keeping unchanged.

8. The method according to any one of claims 3 to 7, characterized in that, For each associated collector group, determining the frequency conversion direction corresponding to the associated collector group according to the change information of the performance index data collected by the target collectors in the associated collector group comprises: For each associated collector group, determining the frequency conversion direction corresponding to the associated collector group according to the change information of the performance index data collected by the target collectors in the associated collector group and the performance analysis result obtained last time according to the performance index data collected by the at least two target collectors.

9. A data collection method for a computing cluster, applied to any computing node in the computing cluster, characterized in that, The method comprises: During the execution of a job task, starting at least two target collectors related to the job task to enable the at least two target collectors to collect at least two kinds of performance index data of the computing node where the at least two target collectors are located at a current collection frequency; According to the association of the performance indexes responsible for being collected by the at least two target collectors, the at least two target collectors are divided into at least two associated collector groups; According to the change information of the performance index data collected by the target collectors in each associated collector group, the collection frequency of each target collector in each associated collector group is adjusted respectively, so that the target collectors continue to collect at least two kinds of performance index data of the computing nodes where the target collectors are located at the adjusted collection frequency.

10. A data collection apparatus applied to any computing node in a computing cluster, characterized in that, The device comprises: a starting module, configured to start at least two target collectors related to a job task during execution of the job task, so that the at least two target collectors collect at least two kinds of performance index data of the computing nodes where the target collectors are located at a current collection frequency, the job task being deployed on at least two computing nodes in a computing cluster; an adjusting module, configured to, in a case where it is determined that a self node is a master node of the at least two computing nodes, divide the at least two target collectors into at least two associated collector groups according to the association of the performance indexes collected by the at least two target collectors, and adjust the collection frequency of each target collector in each associated collector group according to change information of the performance index data collected by the target collectors in each associated collector group; a notifying module, configured to notify other computing nodes of the at least two computing nodes to adjust the collection frequency of the at least two target collectors deployed on the other computing nodes, so that the at least two target collectors on the other computing nodes continue to collect at least two kinds of performance index data of the computing nodes where the target collectors are located at the adjusted collection frequency.

11. A data collection apparatus applied to any computing node in a computing cluster, characterized in that, The device comprises: a starting module, configured to start at least two target collectors related to a job task during execution of the job task, so that the at least two target collectors collect at least two kinds of performance index data of the computing nodes where the target collectors are located at a current collection frequency, the job task being deployed on at least two computing nodes in a computing cluster; a grouping module, configured to divide the at least two target collectors into at least two associated collector groups according to the association of the performance indexes collected by the at least two target collectors; an adjusting module, configured to adjust the collection frequency of each target collector in each associated collector group according to change information of the performance index data collected by the target collectors in each associated collector group, so that the target collectors continue to collect at least two kinds of performance index data of the computing nodes where the target collectors are located at the adjusted collection frequency.

12. A computing node applicable to a computing cluster, characterized in that, The computing node comprises a memory and a processor; the memory is configured to store a computer program; the processor is coupled to the memory and is configured to execute the computer program to perform the steps in the method of any one of claims 2-9.

13. A computer readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the processor is enabled to perform the steps in the method of any one of claims 2-9.

Images