A collection item updating method and device of a node, an electronic device, and a readable medium

Abstract

Embodiments of the present application provide a node collection item updating method and device, electronic equipment and readable medium, a collection item updating system acquires collection item information, the collection item information records node type attributes, and acquires the template configuration file corresponding to the node type attribute, generates the target collection item configuration file of the master node and the slave node by using the collection item information and the template configuration file, distributes the master node target collection item configuration file and the slave node target collection item configuration file to the master node, and distributes the slave node target collection item configuration file to the slave node. The method of the present application dynamically generates the collection item configuration file and completes the distribution of the collection item configuration file, realizes the automatic updating of the collection item under the large-scale server cluster, and improves the efficiency of the large-scale server collection item configuration updating. At the same time, since too much manual operation is avoided, the accuracy of the collection item configuration updating is also improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for updating data collection items of a node, a device for updating data collection items of a node, an electronic device, and a computer-readable medium. Background Technology

[0002] The AI ​​training platform offers a wealth of monitoring features, including monitoring of various dimensions such as the host, network, memory, and accelerator card. The monitoring data is collected through the open-source component Telegraf. Telegraf collects various monitoring metrics data through plugins, and then outputs the collected monitoring data to the InfluxDB time-series database for storage via an output component.

[0003] In an AI training platform, to achieve comprehensive monitoring, in addition to relying on Telegraf's built-in data collection plugin, it is also necessary to develop custom data collection scripts based on business needs, so as to achieve monitoring data collection that better meets actual requirements.

[0004] During the use of an AI training platform, users may adjust the data collection items, such as the collection interval, based on the characteristics of the cluster. They may also add new data collection items to the system according to business needs, thereby achieving the monitoring and collection of specific business performance data. However, manually adjusting and customizing data collection items is inefficient, and the possibility of errors during manual operation is relatively high. Summary of the Invention

[0005] In view of the above problems, embodiments of the present invention are proposed to provide a node acquisition item update method, a node acquisition item update device, an electronic device, and a computer-readable medium to overcome or at least partially solve the above problems.

[0006] This invention discloses a method for updating data collection items of nodes, applied to a data collection item update system. The data collection item update system is connected to at least one server cluster, and the nodes in the server cluster are divided into at least one master node and at least one slave node. The method includes:

[0007] Obtain custom collection item information; the collection item information records node type attributes; the node type attributes include master node type and / or slave node type;

[0008] Obtain the preset template configuration file corresponding to the node type attribute; the preset template configuration file includes at least a master node template configuration file and a slave node template configuration file;

[0009] Using the preset template configuration file corresponding to the collected item information and the node type attribute, generate the main node target collected item configuration file and / or the slave node target collected item configuration file;

[0010] The master node target collection item configuration file and / or the slave node target collection item configuration file are distributed to the master node, and the slave node target collection item configuration file is distributed to the slave node, so that the master node uses the master node target collection item configuration file and / or the slave node target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node uses the slave node target collection item configuration file to overwrite the original collection item configuration file in the slave node;

[0011] The master node target collection item configuration file and the slave node target collection item configuration file each include at least one collection item, and the master node and the slave node are used to collect performance index data according to the collection item.

[0012] Optionally, the step of generating a master node target collection item configuration file and / or a slave node target collection item configuration file using a preset template configuration file corresponding to the collection item information and the node type attribute includes:

[0013] The master node template configuration file and / or the slave node template configuration file are read into memory through file streams to obtain the master node basic configuration file and / or the slave node basic configuration file.

[0014] The collected item information is added to the master node base configuration file and / or the slave node base configuration file corresponding to the node type attribute to obtain the master node target collected item configuration file and / or the slave node target collected item configuration file.

[0015] Optionally, the step of distributing the master node target collection item configuration file and / or the slave node target collection item configuration file to the master node, and distributing the slave node target collection item configuration file to the slave node, so that the master node uses the master node target collection item configuration file and / or the slave node target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node uses the slave node target collection item configuration file to overwrite the original collection item configuration file in the slave node, includes:

[0016] Based on the configuration directory information recorded in the master node target acquisition item configuration file and the slave node target acquisition item configuration file, the master node target acquisition item configuration file and / or the slave node target acquisition item configuration file are distributed to the configuration directory corresponding to the master node, and the slave node target acquisition item configuration file is distributed to the configuration directory corresponding to the slave node.

[0017] Optionally, the method further includes:

[0018] The master node target acquisition item configuration file and the slave node target acquisition item configuration file also include acquisition time interval and output configuration information, so that the master node and slave node acquire the performance index data according to the acquisition time interval and store it in the database corresponding to the output configuration information.

[0019] Optionally, the method further includes:

[0020] If the target acquisition item configuration file of the slave node is lost during the distribution to the slave node, the master node will distribute the received target acquisition item configuration file of the slave node to the slave node.

[0021] Optionally, after the steps of distributing the master node target collection item configuration file and / or the slave node target collection item configuration file to the master node, and distributing the slave node target collection item configuration file to the slave node, so that the master node uses the master node target collection item configuration file and / or the slave node target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node uses the slave node target collection item configuration file to overwrite the original collection item configuration file in the slave node, the method further includes:

[0022] Monitor the distribution progress of each of the aforementioned nodes;

[0023] Based on the distribution progress of each node, the distribution status of each node is determined; the distribution status includes distribution execution completed, distribution in progress, and distribution execution failed.

[0024] Based on the distribution progress and distribution status of each node, a distribution log is generated for each node.

[0025] Optionally, the method further includes:

[0026] If the distribution status of the node is distribution execution failure, obtain the distribution log of the node;

[0027] Based on the distribution failure reasons recorded in the distribution log of the node, determine whether the reason for the node's distribution execution failure is a network failure;

[0028] If so, then the master node target acquisition item configuration file and / or slave node target acquisition item configuration file are redistributed to the node.

[0029] This invention discloses a node data collection item update device, applied to a data collection item update system. The data collection item update system is connected to at least one server cluster, and the nodes in the server cluster are divided into at least one master node and at least one slave node. The device includes:

[0030] The data acquisition module is used to acquire custom data acquisition information; the data acquisition information includes at least one performance indicator; the data acquisition information records a node type attribute; the node type attribute includes a master node type and / or a slave node type;

[0031] The template acquisition module is used to acquire the preset template configuration file corresponding to the node type attribute; the preset template configuration file includes at least the master node template configuration file and the slave node template configuration file;

[0032] The generation module is used to generate a main node target collection item configuration file and / or a slave node target collection item configuration file by using the collection item information and the node type attribute of the preset template configuration file.

[0033] A distribution module is used to distribute the target collection item configuration file of the master node and / or the target collection item configuration file of the slave node to the master node, and to distribute the target collection item configuration file of the slave node to the slave node, so that the master node uses the target collection item configuration file of the master node and / or the target collection item configuration file of the slave node to overwrite the original collection item configuration file in the master node, and the slave node uses the target collection item configuration file of the slave node to overwrite the original collection item configuration file in the slave node;

[0034] The master node target collection item configuration file and the slave node target collection item configuration file each include at least one collection item, and the master node and the slave node are used to collect performance index data according to the collection item.

[0035] Optionally, the generation module includes:

[0036] The reading submodule is used to read the master node template configuration file and / or the slave node template configuration file into memory through file streams to obtain the master node basic configuration file and / or the slave node basic configuration file.

[0037] Add a submodule to add the collected item information to the master node base configuration file and / or the slave node base configuration file corresponding to the node type attribute, so as to obtain the master node target collected item configuration file and / or the slave node target collected item configuration file.

[0038] Optionally, the distribution module includes:

[0039] The distribution submodule is used to distribute the target collection item configuration file of the master node and / or the target collection item configuration file of the slave node to the configuration directory corresponding to the master node, and to distribute the target collection item configuration file of the slave node to the configuration directory corresponding to the slave node, according to the configuration directory information recorded in the target collection item configuration file of the master node and the target collection item configuration file of the slave node.

[0040] Optionally, the device further includes:

[0041] The master node target acquisition item configuration file and the slave node target acquisition item configuration file also include acquisition time interval and output configuration information, so that the master node and slave node acquire the performance index data according to the acquisition time interval and store it in the database corresponding to the output configuration information.

[0042] Optionally, the device further includes:

[0043] The master node distribution module is used to distribute the received target collection item configuration file of the slave node to the slave node if the slave node target collection item configuration file is lost during the distribution process to the slave node.

[0044] Optionally, the device further includes:

[0045] The monitoring module is used to monitor the distribution progress of each node.

[0046] The determination module is used to determine the distribution status of each node based on the distribution progress of each node; the distribution status includes distribution execution completed, distribution in progress, and distribution execution failed.

[0047] The log generation module is used to generate distribution logs for each node based on the distribution progress and distribution status of each node.

[0048] Optionally, the device further includes:

[0049] The log acquisition module is used to acquire the distribution log of the node if the distribution status of the node is distribution execution failure;

[0050] The judgment module is used to determine whether the reason for the failure of the node's distribution execution is a network failure, based on the reason for the distribution failure recorded in the node's distribution log.

[0051] The retransmission module is used to re-distribute the master node target acquisition item configuration file and / or the slave node target acquisition item configuration file to the node if the condition is met.

[0052] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0053] The memory is used to store computer programs;

[0054] When the processor executes the program stored in the memory, it implements the node acquisition item update method as described in the embodiments of the present invention.

[0055] This invention also discloses one or more computer-readable media storing instructions that, when executed by one or more processors, cause the processors to perform the node acquisition item update method as described in this invention.

[0056] The embodiments of the present invention have the following advantages:

[0057] In this embodiment of the invention, the data collection item update system obtains customized data collection item information, which records node type attributes, and obtains a preset template configuration file corresponding to the node type attributes. Using the data collection item information and the preset template configuration file corresponding to the node type attributes, it generates a target data collection item configuration file for the master node and / or a target data collection item configuration file for the slave node. The system then distributes the target data collection item configuration file for the master node and / or the target data collection item configuration file for the slave node, so that the master node uses the target data collection item configuration file for the master node and / or the target data collection item configuration file for the slave node to overwrite the original data collection item configuration file in the master node, and the slave node uses the target data collection item configuration file for the slave node to overwrite the original data collection item configuration file in the slave node. This method of the present invention, by dynamically generating and distributing data collection item configuration files, achieves automatic updates of data collection items in a large-scale server cluster, eliminating the need for manual login to each cluster node. This improves the efficiency of data collection item configuration updates on large-scale servers, and also improves the accuracy of data collection item configuration updates by avoiding excessive manual operations. Attached Figure Description

[0058] Figure 1 This is a flowchart of the steps of a node data collection item update method provided in an embodiment of the present invention;

[0059] Figure 2 This is a flowchart of another node collection item update method provided in an embodiment of the present invention;

[0060] Figure 3 This is a flowchart of generating a target collection item configuration file provided in an embodiment of the present invention;

[0061] Figure 4 This is a flowchart of an update node collection item provided in an embodiment of the present invention;

[0062] Figure 5 This is a structural block diagram of a node data collection item update device provided in an embodiment of the present invention;

[0063] Figure 6 This is a block diagram of an electronic device provided in an embodiment of the present invention;

[0064] Figure 7 This is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. Detailed Implementation

[0065] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] Reference Figure 1 The diagram illustrates a flowchart of a node data collection item update method provided in an embodiment of the present invention, which may specifically include the following steps:

[0067] Step 101: Obtain custom collection item information; the collection item information records node type attributes; the node type attributes include master node type and / or slave node type;

[0068] In this embodiment of the invention, a data collection item update system can be set up. This system can be connected to at least one server cluster to manage the updates of data collection items on the servers in the server cluster. The nodes in the server cluster can be divided into at least one master node and at least one slave node, and the master node can manage the slave nodes.

[0069] When it is necessary to update the data collection items on server nodes, users can customize the data collection item information and store it in a relational database. The data collection item update system of this invention can automatically retrieve the user-defined data collection item information from the relational database and automatically update the data collection items on nodes in a large-scale server cluster using the customized data collection item information, thereby eliminating the need for manual login to each cluster node.

[0070] In this embodiment of the invention, the data collection item update system can obtain customized data collection item information from a relational database. The data collection item information may include at least one data collection item. Nodes can collect performance indicator data based on the data collection item. For example, data collection items may include system information, CPU information, memory information, CPU temperature information, disk I / O information, accelerator card temperature, container performance data, high-speed network card information, container partition usage information, etc. The data collection item information may also record node type attributes, which may include master node type and / or slave node type. The data collection item update system can determine which data collection item to update—the master node or the slave node—based on the node type attributes.

[0071] Step 102: Obtain the preset template configuration file corresponding to the node type attribute; the preset template configuration file includes at least a master node template configuration file and a slave node template configuration file;

[0072] In this embodiment of the invention, performance indicator data acquisition software can be installed on the nodes. This software can be used to collect performance indicator data based on the collected items. The performance indicator data acquisition software can have preset template configuration files, which can include at least a master node template configuration file and a slave node template configuration file.

[0073] Template configuration can include basic configuration attributes, basic collection items, and output configuration information. Basic configuration attributes can be global configuration information, such as collection time interval and timeout. Basic collection items can be commonly used collection items for nodes, such as node system information, node CPU information, node memory information, node CPU temperature information, and node disk I / O information. Output configuration information can be the storage path for performance index data collected by the performance index data collection software, such as connection configuration information for time series databases, port number, username, and password.

[0074] After determining the node type attribute, the data collection update system can obtain the preset template configuration file corresponding to the node type attribute. Specifically, when the node attribute type is a master node type, the master node template configuration file can be obtained; when the node attribute type is a slave node type, the slave node template configuration file can be obtained; when the node attribute type is both a master node type and a slave node type, both the master node template configuration file and the slave node template configuration file can be obtained.

[0075] Step 103: Using the preset template configuration file corresponding to the collection item information and the node type attribute, generate the main node target collection item configuration file and / or the slave node target collection item configuration file.

[0076] After obtaining the collection item information and template configuration file, the target collection item configuration file can be generated using the template configuration file corresponding to the collection item information and node type attributes.

[0077] Specifically, when the template configuration file is the master node template configuration file, the target collection item configuration file for the master node is generated using the collection item information and the master node template configuration file; when the template configuration file is the slave node template configuration file, the target collection item configuration file for the slave node is generated using the collection item information and the slave node template configuration file; when the template configuration file is both the master node template configuration file and the slave node template configuration file, the target collection item configuration files for the master node and the slave node are generated using the collection item information and the master node template configuration file and the slave node template configuration file. Combining the template configuration file of the performance indicator data collection software Telegraf and the collection item configuration information stored in the relational database, the collection configuration file applied to Telegraf is dynamically generated, realizing the dynamic generation of Telegraf configuration files. This makes the configuration of cluster collection items more flexible and efficient, and effectively reduces the problems of misoperation and low efficiency caused by manually modifying configuration files in the past.

[0078] Step 104: Distribute the master node target collection item configuration file and / or the slave node target collection item configuration file to the master node, and distribute the slave node target collection item configuration file to the slave node, so that the master node uses the master node target collection item configuration file and / or the slave node target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node uses the slave node target collection item configuration file to overwrite the original collection item configuration file in the slave node.

[0079] After generating the target collection item configuration files for the master node and / or slave nodes, these files can be distributed to the master node. This prevents the loss of generated configuration files due to VIP migration in monitoring services or high-availability cluster environments. Simultaneously, the collection item update system can distribute the target collection item configuration files from slave nodes to slave nodes. This allows the master node to overwrite its original collection item configuration file with the master node's target collection item configuration file and / or the slave node's target collection item configuration file, and vice versa. This updates the collection item configuration files on the updated nodes, effectively updating the collection items on those nodes.

[0080] By employing multi-threading to simultaneously distribute the generated target data collection item configuration file to nodes within the cluster, the data collection items of multiple nodes within the cluster can be updated concurrently. This achieves batch processing of data collection item configuration updates for nodes within the cluster, enabling dynamic updates of cluster data collection items that take effect in real time, avoiding the previous requirement of logging into each cluster node separately to modify the configuration file and restart the service.

[0081] After restarting the performance metrics data acquisition software, it can then collect performance metrics data based on the updated collection items, enabling it to collect performance metrics data corresponding to the real-time updated collection items.

[0082] The node collection item update method of this invention involves a collection item update system acquiring custom collection item information, which records node type attributes. The system also acquires a preset template configuration file corresponding to the node type attributes. Using the collection item information and the preset template configuration file corresponding to the node type attributes, a target collection item configuration file for a master node and / or a target collection item configuration file for a slave node are generated. The master node and / or slave node target collection item configuration files are then distributed to the master node, and the slave node target collection item configuration files are distributed to the slave nodes. This allows the master node to overwrite its original collection item configuration file with the master node target collection item configuration file and / or the slave node target collection item configuration file, and vice versa. This method, by dynamically generating and distributing collection item configuration files, enables automatic updates of collection items in large-scale server clusters. It eliminates the need for manual login to each cluster node, improving the efficiency of large-scale server collection item configuration updates. Furthermore, by avoiding excessive manual operations, it also improves the accuracy of collection item configuration updates.

[0083] Reference Figure 2 The diagram illustrates a flowchart of another node data collection item update method provided in this embodiment of the invention, which may specifically include the following steps:

[0084] Step 201: Obtain custom collection item information; the collection item information records node type attributes; the node type attributes include master node type and / or slave node type;

[0085] Step 202: Obtain the preset template configuration file corresponding to the node type attribute; the preset template configuration file includes at least a master node template configuration file and a slave node template configuration file;

[0086] In one embodiment of the present invention, the method further includes:

[0087] S11, the master node target acquisition item configuration file and the slave node target acquisition item configuration file also include acquisition time interval and output configuration information, so that the master node and slave node acquire the performance index data according to the acquisition time interval and store it in the database corresponding to the output configuration information.

[0088] The target acquisition item configuration file for the master node and the target acquisition item configuration file for the slave node can also include acquisition time intervals and output configuration information. The performance indicator data acquisition software on the master node and the slave node can acquire performance indicator data according to the acquisition time intervals and store the performance indicator data in the time series database corresponding to the output configuration information.

[0089] Step 203: Read the master node template configuration file and / or the slave node template configuration file into memory through file streams to obtain the master node basic configuration file and / or the slave node basic configuration file.

[0090] After obtaining the template configuration file, the master node template configuration file and / or slave node template configuration file can be read into memory through file streams to obtain the master node basic configuration file and / or slave node basic configuration file, thereby initializing the master node template configuration file and / or slave node template configuration file.

[0091] Step 204: Add the collected item information to the master node basic configuration file and / or the slave node basic configuration file corresponding to the node type attribute to obtain the master node target collected item configuration file and / or the slave node target collected item configuration file;

[0092] After initializing the master node template configuration file and / or slave node template configuration file to obtain the master node base configuration file and / or slave node base configuration file, the collection item information can be added to the master node base configuration file and / or slave node base configuration file corresponding to the node type attribute to obtain the master node target collection item configuration file and / or slave node target collection item configuration file.

[0093] Specifically, when the template configuration file is the master node template configuration file, the collection item information is added to the master node template configuration file to obtain the master node target collection item configuration file; when the template configuration file is the slave node template configuration file, the collection item information is added to the slave node template configuration file to obtain the slave node target collection item configuration file; when the template configuration file is both the master node template configuration file and the slave node template configuration file, the collection item information is added to the master node template configuration file and the slave node template configuration file respectively to obtain the master node target collection item configuration file and the slave node target collection item configuration file.

[0094] Reference Figure 3The diagram illustrates a flowchart of a method for generating a target acquisition item configuration file, as provided in an embodiment of the present invention.

[0095] 1. First, you can obtain the master node template configuration file and slave node template configuration file of the performance indicator data collection software Telegraf. The template configuration file may include basic collection attributes, basic collection items, and output configuration information.

[0096] 2. Next, you can obtain custom collection item information stored in the relational database. Custom collection items can include accelerator card monitoring information collection, Docker container monitoring information collection, k8s monitoring information collection, and disk hang count monitoring information collection.

[0097] 3. Furthermore, the template configuration file can be read into memory via file stream, that is, the basic collection attributes, output configuration information, and basic collection items can be read into memory to obtain the basic configuration file, and the custom collection item information can be added to the basic configuration file to generate the main node target collection item configuration file and / or the slave node target collection item configuration file.

[0098] Step 205: Distribute the master node target collection item configuration file and / or the slave node target collection item configuration file to the master node, and distribute the slave node target collection item configuration file to the slave node, so that the master node uses the master node target collection item configuration file and / or the slave node target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node uses the slave node target collection item configuration file to overwrite the original collection item configuration file in the slave node.

[0099] In one embodiment of the present invention, the steps of distributing the master node target collection item configuration file and / or the slave node target collection item configuration file to the master node, and distributing the slave node target collection item configuration file to the slave node, so that the master node uses the master node target collection item configuration file and / or the slave node target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node uses the slave node target collection item configuration file to overwrite the original collection item configuration file in the slave node, include:

[0100] S21, based on the configuration directory information recorded in the master node target acquisition item configuration file and the slave node target acquisition item configuration file, distribute the master node target acquisition item configuration file and / or the slave node target acquisition item configuration file to the configuration directory corresponding to the master node, and distribute the slave node target acquisition item configuration file to the configuration directory corresponding to the slave node.

[0101] The target collection item configuration files for both master and slave nodes can contain configuration directory information. After generating the master and / or slave target collection item configuration files, the `kubectl get nodes` command in Kubernetes can be used to obtain the list of master and slave nodes in the cluster. Based on the configuration directory information recorded in the master and slave target collection item configuration files, the configuration files are distributed to the corresponding configuration directories on the master nodes and vice versa. This allows the master node to overwrite its original collection item configuration file with the master and / or slave target collection item configuration files, and the slave nodes to overwrite their original collection item configuration files with the slave target collection item configuration files, thus updating the collection item configuration files on the nodes, i.e., updating the collection items on the nodes.

[0102] In one embodiment of the present invention, after the steps of distributing the master node target collection item configuration file and / or the slave node target collection item configuration file to the master node, and distributing the slave node target collection item configuration file to the slave node, so that the master node uses the master node target collection item configuration file and / or the slave node target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node uses the slave node target collection item configuration file to overwrite the original collection item configuration file in the slave node, the method further includes:

[0103] S31, Monitor the distribution progress of each node;

[0104] During the process of distributing the target collection item configuration files of the master node and the target collection item configuration files of the slave node to the master node and slave node, the collection item update system can monitor the distribution progress of each node and record the node name and node distribution progress so as to further determine the distribution status of the node based on the node distribution progress.

[0105] Optionally, the data collection item update system can also monitor the type of this data collection item update operation, the total distribution progress of all nodes, the distribution status of all nodes, and generate a log for this distribution operation. The type of this data collection item update operation can include modification, addition, and deletion; the distribution status can be distribution completed, distribution in progress, or distribution failed.

[0106] S32, determine the distribution status of each node according to the distribution progress of each node; the distribution status includes distribution execution completed, distribution in progress, and distribution execution failed;

[0107] After determining the distribution progress of each node, the distribution status of the node can be determined based on the distribution progress. If the distribution progress of a node is 100%, the distribution execution of the node can be determined to be complete; if the distribution progress of a node is 0%-100% (excluding the endpoint value of 100%), the distribution status can be determined to be in progress; if the distribution progress remains unchanged within a preset time period, the distribution status can be determined to be distribution execution failure.

[0108] S33, Based on the distribution progress and distribution status of each node, generate a distribution log for each node.

[0109] After determining the distribution progress and status of each node, a distribution log can be generated for each node based on this information. The distribution log records the entire distribution process, and if distribution fails, it also records the reason for the failure, allowing for subsequent improvements based on the cause of the failure.

[0110] In one embodiment of the present invention, the method further includes:

[0111] S41, if the distribution status of the node is distribution execution failure, obtain the distribution log of the node;

[0112] During the process of distributing the target collection item configuration files of the master node and the target collection item configuration files of the slave node to the master node and slave node, if it is determined that the distribution status of a certain node is that the distribution execution has failed, the reason for the distribution execution failure can be determined so as to make improvements. Therefore, the distribution log of the node that failed to distribute can be obtained first.

[0113] S42, Based on the distribution failure reasons recorded in the distribution log of the node, determine whether the reason for the node's distribution execution failure is a network failure;

[0114] After obtaining the distribution logs of the nodes that failed to distribute the data, the cause of the failure can be determined based on the reasons recorded in the logs. Once the cause is determined, it can be judged whether the failure was due to a network failure. If the failure was caused by a network failure, it can be overcome by redistributing the data. If the failure was due to other issues, appropriate improvement measures can be implemented.

[0115] S43, if so, then redistribute the master node target acquisition item configuration file and / or slave node target acquisition item configuration file to the node.

[0116] When the failure to distribute data to a node is determined to be due to a network failure, the data collection item update system can redistribute the master node target data collection item configuration file and / or the slave node target data collection item configuration file to that node. Specifically, if the node that failed to distribute data is a master node, the master node target data collection item configuration file and the slave node target data collection item configuration file can be redistributed to the master node; if the node that failed to distribute data is a slave node, the slave node target data collection item configuration file can be redistributed to the slave node.

[0117] By monitoring the distribution results of each node during the distribution process, the distribution progress of each node in the cluster can be tracked in real time. When a node update failure is detected, the failure log of the corresponding node can be viewed, and the cluster node configuration can be re-distributed through the re-distribution function. This allows for targeted re-distribution of distribution failures caused by network failures or other reasons.

[0118] In one embodiment of the present invention, the method further includes:

[0119] S51, if the target acquisition item configuration file of the slave node is lost during the distribution to the slave node, the master node will distribute the received target acquisition item configuration file of the slave node to the slave node.

[0120] To prevent the inability to collect performance metrics data if the target collection item configuration file of a slave node is lost, the collection item update system in this embodiment of the invention can also distribute the target collection item configuration file of the slave node to the master node. By utilizing the master node's function of managing slave nodes, in the event that the target collection item configuration file of a slave node is lost, the master node can distribute the target collection item configuration file of the slave node to the slave node. This eliminates the need for the slave node to obtain the target collection item configuration file from the collection item update system, reducing the complexity of the process of obtaining the target collection item configuration file of the slave node.

[0121] Reference Figure 4 The diagram illustrates a flowchart of an update node collection item provided in an embodiment of the present invention.

[0122] 1. First, you can obtain the master node template configuration file and slave node template configuration file of the performance index data collection software Telegraf, initialize the template configuration file to obtain the basic configuration file, and add the custom collection item information to the basic configuration file to generate the master node target collection item configuration file and / or slave node target collection item configuration file.

[0123] 2. Next, the target collection item configuration file of the master node and / or the target collection item configuration file of the slave node can be sent to the master node, and the target collection item configuration file of the slave node can be sent to the slave node.

[0124] 3. During the distribution operation, the distribution task status of the node can be monitored. If the distribution task fails, it can be re-executed; if the distribution task is completed, the distribution task of the node can be deleted.

[0125] The node collection item update method of this invention involves a collection item update system acquiring custom collection item information, which records node type attributes. The system also acquires a preset template configuration file corresponding to the node type attributes. Using the collection item information and the preset template configuration file corresponding to the node type attributes, a target collection item configuration file for a master node and / or a target collection item configuration file for a slave node are generated. The master node and / or slave node target collection item configuration files are then distributed to the master node, and the slave node target collection item configuration files are distributed to the slave nodes. This allows the master node to overwrite its original collection item configuration file with the master node target collection item configuration file and / or the slave node target collection item configuration file, and vice versa. This method, by dynamically generating and distributing collection item configuration files, enables automatic updates of collection items in large-scale server clusters. It eliminates the need for manual login to each cluster node, improving the efficiency of large-scale server collection item configuration updates. Furthermore, by avoiding excessive manual operations, it also improves the accuracy of collection item configuration updates.

[0126] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0127] Reference Figure 5 The diagram illustrates a structural block diagram of a node data collection item update device provided in an embodiment of the present invention, which may specifically include the following modules:

[0128] The data acquisition module 501 is used to acquire custom data acquisition information; the data acquisition information includes at least one performance indicator; the data acquisition information records a node type attribute; the node type attribute includes a master node type and / or a slave node type;

[0129] The template acquisition module 502 is used to acquire the preset template configuration file corresponding to the node type attribute; the preset template configuration file includes at least the master node template configuration file and the slave node template configuration file;

[0130] The generation module 503 is used to generate a main node target collection item configuration file and / or a slave node target collection item configuration file by using a preset template configuration file corresponding to the collection item information and the node type attribute;

[0131] The distribution module 504 is used to distribute the master node target collection item configuration file and / or the slave node target collection item configuration file to the master node, and to distribute the slave node target collection item configuration file to the slave node, so that the master node uses the master node target collection item configuration file and / or the slave node target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node uses the slave node target collection item configuration file to overwrite the original collection item configuration file in the slave node;

[0132] The master node target collection item configuration file and the slave node target collection item configuration file each include at least one collection item, and the master node and the slave node are used to collect performance index data according to the collection item.

[0133] Optionally, the generation module includes:

[0134] The reading submodule is used to read the master node template configuration file and / or the slave node template configuration file into memory through file streams to obtain the master node basic configuration file and / or the slave node basic configuration file.

[0135] Add a submodule to add the collected item information to the master node base configuration file and / or the slave node base configuration file corresponding to the node type attribute, so as to obtain the master node target collected item configuration file and / or the slave node target collected item configuration file.

[0136] Optionally, the distribution module includes:

[0137] The distribution submodule is used to distribute the target collection item configuration file of the master node and / or the target collection item configuration file of the slave node to the configuration directory corresponding to the master node, and to distribute the target collection item configuration file of the slave node to the configuration directory corresponding to the slave node, according to the configuration directory information recorded in the target collection item configuration file of the master node and the target collection item configuration file of the slave node.

[0138] Optionally, the device further includes:

[0139] The master node target acquisition item configuration file and the slave node target acquisition item configuration file also include acquisition time interval and output configuration information, so that the master node and slave node acquire the performance index data according to the acquisition time interval and store it in the database corresponding to the output configuration information.

[0140] Optionally, the device further includes:

[0141] The master node distribution module is used to distribute the received target collection item configuration file of the slave node to the slave node if the slave node target collection item configuration file is lost during the distribution process to the slave node.

[0142] Optionally, the device further includes:

[0143] The monitoring module is used to monitor the distribution progress of each node.

[0144] The determination module is used to determine the distribution status of each node based on the distribution progress of each node; the distribution status includes distribution execution completed, distribution in progress, and distribution execution failed.

[0145] The log generation module is used to generate distribution logs for each node based on the distribution progress and distribution status of each node.

[0146] Optionally, the device further includes:

[0147] The log acquisition module is used to acquire the distribution log of the node if the distribution status of the node is distribution execution failure;

[0148] The judgment module is used to determine whether the reason for the failure of the node's distribution execution is a network failure, based on the reason for the distribution failure recorded in the node's distribution log.

[0149] The retransmission module is used to re-distribute the master node target acquisition item configuration file and / or the slave node target acquisition item configuration file to the node if the condition is met.

[0150] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0151] In addition, embodiments of the present invention also provide an electronic device, such as... Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0152] Memory 603 is used to store computer programs;

[0153] When processor 601 executes a program stored in memory 603, it performs the following steps:

[0154] Obtain custom collection item information; the collection item information records node type attributes; the node type attributes include master node type and / or slave node type;

[0155] Obtain the preset template configuration file corresponding to the node type attribute; the preset template configuration file includes at least a master node template configuration file and a slave node template configuration file;

[0156] Using the preset template configuration file corresponding to the collected item information and the node type attribute, generate the main node target collected item configuration file and / or the slave node target collected item configuration file;

[0157] The master node target collection item configuration file and / or the slave node target collection item configuration file are distributed to the master node, and the slave node target collection item configuration file is distributed to the slave node, so that the master node uses the master node target collection item configuration file and / or the slave node target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node uses the slave node target collection item configuration file to overwrite the original collection item configuration file in the slave node;

[0158] The master node target collection item configuration file and the slave node target collection item configuration file each include at least one collection item, and the master node and the slave node are used to collect performance index data according to the collection item.

[0159] Optionally, the step of generating a master node target collection item configuration file and / or a slave node target collection item configuration file using a preset template configuration file corresponding to the collection item information and the node type attribute includes:

[0160] The master node template configuration file and / or the slave node template configuration file are read into memory through file streams to obtain the master node basic configuration file and / or the slave node basic configuration file.

[0161] The collected item information is added to the master node base configuration file and / or the slave node base configuration file corresponding to the node type attribute to obtain the master node target collected item configuration file and / or the slave node target collected item configuration file.

[0162] Optionally, the step of distributing the master node target collection item configuration file and / or the slave node target collection item configuration file to the master node, and distributing the slave node target collection item configuration file to the slave node, so that the master node uses the master node target collection item configuration file and / or the slave node target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node uses the slave node target collection item configuration file to overwrite the original collection item configuration file in the slave node, includes:

[0163] Based on the configuration directory information recorded in the master node target acquisition item configuration file and the slave node target acquisition item configuration file, the master node target acquisition item configuration file and / or the slave node target acquisition item configuration file are distributed to the configuration directory corresponding to the master node, and the slave node target acquisition item configuration file is distributed to the configuration directory corresponding to the slave node.

[0164] Optionally, the method further includes:

[0165] The master node target acquisition item configuration file and the slave node target acquisition item configuration file also include acquisition time interval and output configuration information, so that the master node and slave node acquire the performance index data according to the acquisition time interval and store it in the database corresponding to the output configuration information.

[0166] Optionally, the method further includes:

[0167] If the target acquisition item configuration file of the slave node is lost during the distribution to the slave node, the master node will distribute the received target acquisition item configuration file of the slave node to the slave node.

[0168] Optionally, after the steps of distributing the master node target collection item configuration file and / or the slave node target collection item configuration file to the master node, and distributing the slave node target collection item configuration file to the slave node, so that the master node uses the master node target collection item configuration file and / or the slave node target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node uses the slave node target collection item configuration file to overwrite the original collection item configuration file in the slave node, the method further includes:

[0169] Monitor the distribution progress of each of the aforementioned nodes;

[0170] Based on the distribution progress of each node, the distribution status of each node is determined; the distribution status includes distribution execution completed, distribution in progress, and distribution execution failed.

[0171] Based on the distribution progress and distribution status of each node, a distribution log is generated for each node.

[0172] Optionally, the method further includes:

[0173] If the distribution status of the node is distribution execution failure, obtain the distribution log of the node;

[0174] Based on the distribution failure reasons recorded in the distribution log of the node, determine whether the reason for the node's distribution execution failure is a network failure;

[0175] If so, then the master node target acquisition item configuration file and / or slave node target acquisition item configuration file are redistributed to the node.

[0176] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0177] The communication interface is used for communication between the aforementioned terminal and other devices.

[0178] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0179] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0180] like Figure 7As shown, in another embodiment of the present invention, a computer-readable storage medium 701 is also provided, which stores instructions that, when run on a computer, cause the computer to execute the node acquisition item update method described in the above embodiment.

[0181] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the node acquisition item update method described in the above embodiments.

[0182] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0183] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0184] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0185] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for updating a collection item of a node, characterized in that, The application is applied to a collection item updating system connected with at least one server cluster, nodes in the server cluster are divided into at least one master node and at least one slave node, and the method comprises: obtaining customized collection item information; the collection item information records node type attributes; the node type attributes include master node types and / or slave node types; obtaining preset template configuration files corresponding to the node type attributes; the preset template configuration files at least include master node template configuration files and slave node template configuration files; generating master node target collection item configuration files and / or slave node target collection item configuration files by using the collection item information and the preset template configuration files corresponding to the node type attributes; distributing the master node target collection item configuration files and / or the slave node target collection item configuration files to the master node, and distributing the slave node target collection item configuration files to the slave node, so that the master node uses the master node target collection item configuration files and / or the slave node target collection item configuration files to overwrite original collection item configuration files in the master node, and the slave node uses the slave node target collection item configuration files to overwrite original collection item configuration files in the slave node; wherein the master node target collection item configuration files and the slave node target collection item configuration files respectively include at least one collection item, and the master node and the slave node are used to collect performance index data according to the collection item.

2. The method of claim 1, wherein, The step of generating master node target collection item configuration files and / or slave node target collection item configuration files by using the collection item information and the preset template configuration files corresponding to the node type attributes comprises: reading the master node template configuration files and / or the slave node template configuration files into memory through file streams respectively to obtain master node basic configuration files and / or slave node basic configuration files; adding the collection item information into the master node basic configuration files and / or the slave node basic configuration files corresponding to the node type attributes to obtain the master node target collection item configuration files and / or the slave node target collection item configuration files.

3. The method of claim 1, wherein, The step of distributing the master node target collection item configuration files and / or the slave node target collection item configuration files to the master node, and distributing the slave node target collection item configuration files to the slave node, so that the master node uses the master node target collection item configuration files and / or the slave node target collection item configuration files to overwrite original collection item configuration files in the master node, and the slave node uses the slave node target collection item configuration files to overwrite original collection item configuration files in the slave node comprises: distributing the master node target collection item configuration files and / or the slave node target collection item configuration files to configuration directories corresponding to the master node according to configuration directory information recorded in the master node target collection item configuration files and the slave node target collection item configuration files, and distributing the slave node target collection item configuration files to configuration directories corresponding to the slave node.

4. The method of claim 1, wherein, The method further comprises: The master node target collection item configuration file and the slave node target collection item configuration file further comprise collection time intervals and output configuration information, so that the master node and the slave node collect the performance index data according to the collection time intervals and store the performance index data in the databases corresponding to the output configuration information.

5. The method of claim 1, wherein, The method further comprises: If the slave node target collection item configuration file is lost in the process of being distributed to the slave node, the master node distributes the received slave node target collection item configuration file to the slave node.

6. The method of claim 1, wherein, After the step of distributing the master node target collection item configuration file and / or the slave node target collection item configuration file to the master node and distributing the slave node target collection item configuration file to the slave node, so that the master node replaces the original collection item configuration file in the master node with the master node target collection item configuration file and / or the slave node target collection item configuration file, and the slave node replaces the original collection item configuration file in the slave node with the slave node target collection item configuration file, the method further comprises: Monitoring the distribution progress of each node; According to the distribution progress of each node, determining the distribution state of each node; the distribution state comprises distribution execution completion, distribution execution in progress, and distribution execution failure; Based on the distribution progress and the distribution state of each node, generating a distribution log of each node.

7. The method of claim 6, wherein, The method further comprises: If the distribution state of the node is distribution execution failure, obtaining the distribution log of the node; According to the distribution failure reason recorded in the distribution log of the node, determining whether the reason for the distribution execution failure of the node is network failure; If yes, re-distributing the master node target collection item configuration file and / or the slave node target collection item configuration file to the node.

8. An acquisition item updating apparatus of a node, characterized by comprising: The device is applied to a collection item updating system, the collection item updating system is connected with at least one server cluster, nodes in the server cluster are divided into at least one master node and at least one slave node, and the device comprises: A collection item information acquisition module is configured to acquire self-defined collection item information; the collection item information comprises at least one performance index; the collection item information records a node type attribute; the node type attribute comprises a master node type and / or a slave node type; A template acquisition module is configured to acquire a preset template configuration file corresponding to the node type attribute; the preset template configuration file at least comprises a master node template configuration file and a slave node template configuration file; A generation module is configured to generate a master node target collection item configuration file and / or a slave node target collection item configuration file by using the collection item information and the preset template configuration file corresponding to the node type attribute. a distribution module, configured to distribute the master target collection item configuration file and / or the slave target collection item configuration file to the master node, and distribute the slave target collection item configuration file to the slave node, so that the master node adopts the master target collection item configuration file and / or the slave target collection item configuration file to overwrite the original collection item configuration file in the master node, and the slave node adopts the slave target collection item configuration file to overwrite the original collection item configuration file in the slave node; wherein the master target collection item configuration file and the slave target collection item configuration file each comprise at least one collection item, and the master node and the slave node are configured to collect performance index data according to the collection items.

9. An electronic device, comprising: comprise a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; the memory is configured to store a computer program; the processor is configured to execute the program stored on the memory, and implement the collection item updating method of the node according to any one of claims 1-7.

10. One or more computer readable media having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the collection item updating method of the node according to any one of claims 1-7.

Images