Performance data acquisition and processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a performance data collecting and processing method and device, electronic equipment and a storage medium. The method comprises the following steps: generating a data acquisition instruction, and obtaining main node performance data, standby node performance data and expansion cabinet performance data; transmitting the performance data of the expansion cabinet to a standby node, and storing all the performance data to a shared database; in response to the received performance data processing request, controlling the master node to generate a data processing task, and storing the data processing task to a shared database; controlling the main node to obtain a to-be-executed task in the shared database, and selecting a to-be-executed standby node from the standby nodes; distributing a to-be-executed task to the to-be-executed standby node; and controlling the standby node to be executed to analyze all the performance data, generating a data analysis result, and storing the data analysis result in the shared database. By adopting the method, the load of the main node can be reduced, and the efficiency and performance of the whole storage system are improved.

Description

Technical Field

[0001] The present application relates to the technical field of storage systems, and in particular to a method, device, electronic device and storage medium for collecting and processing performance data. Background Art

[0002] As the scale and complexity of storage systems continue to increase, the burden on the master node is also increasing. In traditional architectures, the master node is usually responsible for collecting the performance data of the master node, the performance data of the standby node, and the performance data of the expansion cabinet, as well as processing these performance data. However, this performance data collection and processing mode has the following disadvantages: centralized collection and processing makes the master node overloaded, and as the number of storage nodes increases, the amount of data that the master node needs to process increases sharply, resulting in a significant increase in the pressure on the master node in the collection and processing of performance data. This not only increases the memory and CPU usage of the master node, but may also cause performance bottlenecks, affecting the overall response speed and stability of the system, and node expansion is also limited.

[0003] Therefore, there is an urgent need to propose a performance data collection and processing method, device, electronic device and storage medium that can reduce the burden on the master node and improve the efficiency and performance of the entire storage system. Summary of the invention

[0004] Based on this, it is necessary to provide a performance data collection and processing method, device, electronic device and storage medium that can reduce the burden on the master node and improve the efficiency and performance of the entire storage system in response to the above technical problems.

[0005] In a first aspect, a performance data collection and processing method is provided, which is applied to a distributed storage system, wherein the distributed storage system includes a master node, a standby node, an expansion cabinet, and a shared database, wherein the master node is communicatively connected to the standby node, the standby node is communicatively connected to the expansion cabinet, and the shared database is communicatively connected to the master node and the standby node, respectively, and the method includes:

[0006] Generate data collection instructions according to a preset sampling period;

[0007] According to the data collection instruction, the main node, the standby node and the expansion cabinet are respectively controlled to collect data to obtain the main node performance data, the standby node performance data and the expansion cabinet performance data;

[0008] Transmitting the expansion cabinet performance data to the standby node, and saving the master node performance data on the master node and the standby node performance data and expansion cabinet performance data on the standby node to a shared database;

[0009] In response to receiving a performance data processing request, controlling the master node to generate a data processing task, and saving the data processing task to the shared database;

[0010] According to a preset task execution order, control the master node to obtain the tasks to be executed in the shared database;

[0011] According to a preset task allocation rule, the master node is controlled to select a standby node to be executed from the standby nodes, and the task to be executed is allocated to the standby node to be executed;

[0012] In response to the standby node to be executed receiving the task to be executed, the standby node to be executed is controlled to analyze the master node performance data, standby node performance data and expansion cabinet performance data in the shared database, generate data analysis results, and save the data analysis results to the shared database.

[0013] In one embodiment, the step of saving the data processing task to a shared database includes:

[0014] Classifying the data processing tasks according to task execution time to obtain task classification results, wherein the task classification results include immediately executed tasks and scheduled executed tasks;

[0015] In response to the data processing task being an immediately executed task, sequentially saving the data processing task to a to-be-executed task queue in the shared database;

[0016] In response to the data processing task being a scheduled execution task, the data processing task is sequentially saved in a scheduled task table of the shared database;

[0017] In response to the scheduled execution task reaching the scheduled execution time, the scheduled execution task is inserted into the to-be-executed queue as a to-be-executed task.

[0018] In one embodiment, controlling the master node to obtain the tasks to be executed in the shared database according to a preset task execution order includes:

[0019] Determine whether the to-be-executed task queue contains the scheduled execution task;

[0020] In response to the scheduled execution task being included in the to-be-executed task queue, the scheduled execution task is sequentially used as the first to-be-executed task in the order in which the scheduled execution tasks are generated, and the immediate execution task is sequentially used as the second to-be-executed task in the order in which the immediate execution tasks are generated;

[0021] In response to the queue of tasks to be executed not including the scheduled execution task, the master node is controlled to sequentially obtain the tasks to be executed according to the generation order of the tasks to be executed immediately.

[0022] In one embodiment, controlling the master node to select a standby node to be executed from the standby nodes according to a preset task allocation rule includes:

[0023] Obtaining an estimated execution time of the data processing task;

[0024] Normalizing the estimated execution time according to a preset time normalization method to obtain a normalized execution time, wherein the time normalization method is to divide the estimated execution time by the maximum time;

[0025] Obtaining an estimated memory usage of the data processing task;

[0026] Normalizing the estimated memory usage according to a preset memory normalization method to obtain a normalized memory usage, wherein the memory normalization method is to divide the estimated memory usage by the total memory specification;

[0027] According to the normalized execution time and the normalized memory usage, the IO load weight value and the memory utilization weight value of the standby node are calculated according to a preset weight value calculation method;

[0028] Obtaining a relative value of the IO load and a relative value of the memory utilization of the standby node;

[0029] Based on the IO load weight value, the memory utilization weight value, the IO load relative value, and the memory utilization relative value, the idle rate score of the standby node is calculated according to a preset idle rate calculation method;

[0030] A standby node to be executed among the standby nodes is determined according to the idle rate score.

[0031] In one embodiment, the IO load weight value and the memory utilization weight value of the standby node are calculated according to the normalized execution time and the normalized memory usage according to a preset weight value calculation method, including:

[0032] Determine whether the normalized execution time of the data processing task exceeds a preset time threshold;

[0033] In response to the normalized execution time exceeding a preset time threshold, the data processing task is determined to be a long-time task, and the IO load weight value and the memory utilization weight value of the standby node are calculated according to the weight value calculation method of the long-time task;

[0034] The method of calculating the weight value of the IO load and the memory utilization weight value of the standby node according to the weight value calculation method of the long time-consuming task includes:

[0035] The IO load weight value of the standby node is calculated according to IO load weight value = IO load weight coefficient of the long-time task × normalized execution time;

[0036] The memory utilization weight value of the standby node is calculated according to memory utilization weight value=memory utilization weight coefficient of the long-time task×normalized memory occupancy.

[0037] In one embodiment, obtaining the relative value of the IO load and the relative value of the memory utilization of the standby node includes:

[0038] Obtaining an IO load reference value, an IO load maximum value, and a current IO load value of the standby node;

[0039] According to IO load relative value=(current IO load value-IO load reference value) / (IO load maximum value-IO load reference value), calculate and obtain the IO load relative value of the standby node;

[0040] Obtaining a memory utilization baseline value, a memory utilization maximum value, and a current memory utilization of the standby node;

[0041] The relative value of memory utilization of the standby node is obtained by calculation according to relative value of memory utilization=(current memory utilization-reference value of memory utilization) / (maximum value of memory utilization-reference value of memory utilization).

[0042] In one embodiment, the idle rate score of the standby node is calculated based on the IO load weight value, the memory utilization weight value, the IO load relative value, and the memory utilization relative value according to a preset idle rate calculation method, including:

[0043] The idle rate score of the standby node is calculated according to the idle rate score=(IO load relative value×IO load weight value)+(memory utilization relative value×memory utilization weight value);

[0044] The determining, according to the idle rate score, a standby node to be executed among the standby nodes comprises:

[0045] Sorting the standby nodes from high to low according to the idle rate scores to obtain an idle rate sorting result;

[0046] According to the idle rate sorting result, the standby node ranked first is determined as the standby node to be executed.

[0047] In a second aspect, a performance data collection and processing device is provided, the device comprising a master node, a standby node, an expansion cabinet, and a shared database, the master node is communicatively connected to the standby node, the standby node is communicatively connected to the expansion cabinet, and the shared database is communicatively connected to the master node and the standby node, respectively.

[0048] The master node collects and obtains its own master node performance data according to the received data collection instruction, and saves the master node performance data to the shared database;

[0049] The standby node collects and obtains its own standby node performance data according to the received data collection instruction;

[0050] The expansion cabinet collects and obtains its own expansion cabinet performance data according to the received data collection instruction, and transmits the expansion cabinet performance data to the standby node;

[0051] The standby node receives the expansion cabinet performance data, and saves the standby node performance data and the expansion cabinet performance data to the shared database;

[0052] The master node generates a data processing task and saves the data processing task to the shared database;

[0053] The master node obtains the tasks to be executed in the shared database according to a preset task execution order;

[0054] The master node selects a standby node to be executed from the standby nodes according to a preset task allocation rule, and allocates the task to be executed to the standby node to be executed;

[0055] The standby node to be executed analyzes the master node performance data, the standby node performance data and the expansion cabinet performance data in the shared database according to the received task to be executed, generates a data analysis result, and saves the data analysis result to the shared database.

[0056] In a third aspect, an electronic device is provided, comprising one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0057] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any one of the first aspects are performed.

[0058] In a fifth aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method described in any one of the above-mentioned first aspects are implemented.

[0059] The above-mentioned performance data collection and processing method, device, electronic device and storage medium transmit the expansion cabinet performance data to the backup node, and save the main node performance data on the main node and the backup node performance data and expansion cabinet performance data on the backup node to a shared database, thereby reducing the burden on the main node and improving data processing efficiency. By executing data processing tasks through the expansion cabinet and the backup node, effective management and utilization of storage system resources are achieved, and the overall performance of the system is improved. According to the data analysis results, the performance of the main node, the backup node, the expansion cabinet and the entire storage system can be understood. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of a flow chart of a method for collecting and processing performance data in one embodiment;

[0061] Figure 2 A system flow chart of a method for collecting and processing performance data in one embodiment;

[0062] Figure 3 is a structural block diagram of a device for collecting and processing performance data in one embodiment;

[0063] Figure 4 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] Embodiment 1

[0066] In one embodiment, Figure 1 , Figure 2 As shown, a method for collecting and processing performance data is provided, which is applied to a distributed storage system, wherein the distributed storage system includes a master node, a standby node, an expansion cabinet, and a shared database, wherein the master node is communicatively connected to the standby node, the standby node is communicatively connected to the expansion cabinet, and the shared database is communicatively connected to the master node and the standby node, respectively, and the method includes:

[0067] Generate data collection instructions according to a preset sampling period;

[0068] According to the data collection instruction, the main node, the standby node and the expansion cabinet are respectively controlled to collect data to obtain the main node performance data, the standby node performance data and the expansion cabinet performance data;

[0069] Transmitting the expansion cabinet performance data to the standby node, and saving the master node performance data on the master node and the standby node performance data and expansion cabinet performance data on the standby node to a shared database;

[0070] In response to receiving a performance data processing request, controlling the master node to generate a data processing task, and saving the data processing task to the shared database;

[0071] According to a preset task execution order, control the master node to obtain the tasks to be executed in the shared database;

[0072] According to a preset task allocation rule, the master node is controlled to select a standby node to be executed from the standby nodes, and the task to be executed is allocated to the standby node to be executed;

[0073] In response to the standby node to be executed receiving the task to be executed, the standby node to be executed is controlled to analyze the master node performance data, standby node performance data and expansion cabinet performance data in the shared database, generate data analysis results, and save the data analysis results to the shared database.

[0074] Specifically, the performance data of the expansion cabinet is transmitted to the standby node, and the main node performance data on the main node, the standby node performance data on the standby node, and the expansion cabinet performance data are saved to the shared database, which reduces the burden on the main node and improves the data processing efficiency. By executing data processing tasks through the expansion cabinet and the standby node, the effective management and utilization of storage system resources are achieved, and the overall performance of the system is improved. According to the data analysis results, the performance of the main node, the standby node, the expansion cabinet, and the entire storage system can be understood.

[0075] In one embodiment, the step of saving the data processing task to a shared database includes:

[0076] Classifying the data processing tasks according to task execution time to obtain task classification results, wherein the task classification results include immediately executed tasks and scheduled executed tasks;

[0077] In response to the data processing task being an immediately executed task, sequentially saving the data processing task to a to-be-executed task queue in the shared database;

[0078] In response to the data processing task being a scheduled execution task, the data processing task is sequentially saved in a scheduled task table of the shared database;

[0079] In response to the scheduled execution task reaching the scheduled execution time, the scheduled execution task is inserted into the to-be-executed queue as a to-be-executed task.

[0080] Specifically, tasks that are executed immediately: such tasks do not require waiting; tasks that are executed at a scheduled time: such tasks need to wait for a specific time to be executed, and data processing tasks are saved in a shared database so that subsequent backup nodes can execute the data processing tasks, thereby improving the distributed processing capabilities of the system.

[0081] In one embodiment, controlling the master node to obtain the tasks to be executed in the shared database according to a preset task execution order includes:

[0082] Determine whether the to-be-executed task queue contains the scheduled execution task;

[0083] In response to the scheduled execution task being included in the to-be-executed task queue, the scheduled execution task is sequentially used as the first to-be-executed task in the order in which the scheduled execution tasks are generated, and the immediate execution task is sequentially used as the second to-be-executed task in the order in which the immediate execution tasks are generated;

[0084] In response to the queue of tasks to be executed not including the scheduled execution task, the master node is controlled to sequentially obtain the tasks to be executed according to the generation order of the tasks to be executed immediately.

[0085] Specifically, when the queue of tasks to be executed includes both scheduled tasks and immediate tasks, the scheduled tasks are executed first and then the immediate tasks, and they are executed in sequence according to the order in which the tasks are generated; when the queue of tasks to be executed only includes immediate tasks, the immediate tasks are executed in sequence according to the order in which the tasks are generated; when the queue of tasks to be executed only includes scheduled tasks, the scheduled tasks are executed in sequence according to the order in which the tasks are generated.

[0086] In one embodiment, controlling the master node to select a standby node to be executed from the standby nodes according to a preset task allocation rule includes:

[0087] Obtaining an estimated execution time of the data processing task;

[0088] Normalizing the estimated execution time according to a preset time normalization method to obtain a normalized execution time, wherein the time normalization method is to divide the estimated execution time by the maximum time;

[0089] Obtaining an estimated memory usage of the data processing task;

[0090] Normalizing the estimated memory usage according to a preset memory normalization method to obtain a normalized memory usage, wherein the memory normalization method is to divide the estimated memory usage by the total memory specification;

[0091] According to the normalized execution time and the normalized memory usage, the IO load weight value and the memory utilization weight value of the standby node are calculated according to a preset weight value calculation method;

[0092] Obtaining a relative value of the IO load and a relative value of the memory utilization of the standby node;

[0093] Based on the IO load weight value, the memory utilization weight value, the IO load relative value, and the memory utilization relative value, the idle rate score of the standby node is calculated according to a preset idle rate calculation method;

[0094] A standby node to be executed among the standby nodes is determined according to the idle rate score.

[0095] Specifically, the maximum duration is greater than or equal to the maximum value of the historical estimated execution duration. Duration normalization: convert the estimated execution duration of the data processing task into a value between 0 and 1. Memory normalization: convert the estimated memory usage of the data processing task into a value between 0 and 1. According to the relative value of the IO load and the relative value of the memory utilization of each standby node, combined with the normalized execution duration and the normalized memory usage, the idle rate score of each standby node is calculated, so as to select the standby node with the highest idle rate as the standby node to be executed.

[0096] In one embodiment, the IO load weight value and the memory utilization weight value of the standby node are calculated according to the normalized execution time and the normalized memory usage according to a preset weight value calculation method, including:

[0097] Determine whether the normalized execution time of the data processing task exceeds a preset time threshold;

[0098] In response to the normalized execution time exceeding a preset time threshold, the data processing task is determined to be a long-time task, and the IO load weight value and the memory utilization weight value of the standby node are calculated according to the weight value calculation method of the long-time task;

[0099] The method of calculating the weight value of the IO load and the memory utilization weight value of the standby node according to the weight value calculation method of the long time-consuming task includes:

[0100] The IO load weight value of the standby node is calculated according to IO load weight value = IO load weight coefficient of the long-time task × normalized execution time;

[0101] The memory utilization weight value of the standby node is calculated according to memory utilization weight value=memory utilization weight coefficient of the long-time task×normalized memory occupancy.

[0102] Specifically, according to the task characteristics of the data processing task, such as long execution time or large memory requirements, the weights of IO load and memory utilization are dynamically allocated. When the normalized execution time exceeds the preset time threshold, the data processing task is determined to be a long-time task; when the normalized memory usage exceeds the preset memory threshold, the data processing task is determined to be a large memory requirement task. Among them, the time threshold and memory threshold can be set to 0.5, or can be flexibly set according to actual conditions.

[0103] Assume that for a long-time task, the IO load weight value = 0.5 × estimated task duration, and the memory utilization weight value = 0.5 × estimated memory usage; for a task with large memory requirements, the IO load weight value = the IO load weight coefficient of the task with large memory requirements × estimated task duration, and the memory utilization weight value = the memory utilization weight coefficient of the task with large memory requirements × estimated memory usage, for example, the IO load weight value = 0.4 × estimated task duration, and the memory utilization weight value = 0.6 × estimated memory usage. The weight value coefficient can be flexibly set according to actual conditions.

[0104] According to the weight value calculation formula, the IO load weight value and memory utilization weight value of the standby node are calculated respectively, so as to subsequently calculate the idle rate score of the standby node.

[0105] In one embodiment, obtaining the relative value of the IO load and the relative value of the memory utilization of the standby node includes:

[0106] Obtaining an IO load reference value, an IO load maximum value, and a current IO load value of the standby node;

[0107] According to IO load relative value=(current IO load value-IO load reference value) / (IO load maximum value-IO load reference value), calculate and obtain the IO load relative value of the standby node;

[0108] Obtaining a memory utilization baseline value, a memory utilization maximum value, and a current memory utilization of the standby node;

[0109] The relative value of memory utilization of the standby node is obtained by calculation according to relative value of memory utilization=(current memory utilization-reference value of memory utilization) / (maximum value of memory utilization-reference value of memory utilization).

[0110] Specifically, assuming that the IO load baseline value is 0 and the IO load maximum value is 1000000, the IO load relative value = current IO load value / 1000000. Assuming that the memory utilization baseline value is 0 and the memory utilization maximum value is 100, the memory utilization relative value = current memory utilization / 100. According to the relative value calculation formula, the IO load relative value and memory utilization relative value of the standby node are calculated, so as to subsequently calculate the idle rate score of the standby node.

[0111] In one embodiment, the idle rate score of the standby node is calculated based on the IO load weight value, the memory utilization weight value, the IO load relative value, and the memory utilization relative value according to a preset idle rate calculation method, including:

[0112] The idle rate score of the standby node is calculated according to the idle rate score=(IO load relative value×IO load weight value)+(memory utilization relative value×memory utilization weight value);

[0113] The determining, according to the idle rate score, a standby node to be executed among the standby nodes comprises:

[0114] Sorting the standby nodes from high to low according to the idle rate scores to obtain an idle rate sorting result;

[0115] According to the idle rate sorting result, the standby node ranked first is determined as the standby node to be executed.

[0116] Specifically, according to the idle rate calculation formula, the idle rate score of the standby node is calculated, and the standby node with the highest idle rate is selected to execute the task to be executed, thereby improving data processing efficiency.

[0117] In one embodiment, the data analysis result includes task monitoring data. After the data analysis result is saved to the shared database, the method further includes:

[0118] In response to receiving a task completion signal, controlling the master node to generate a data read instruction, and sending the data read instruction to the shared database;

[0119] In response to the shared database receiving the data reading instruction, controlling the shared database to transmit the task monitoring data to the master node, wherein the task monitoring data includes the actual execution time and actual memory usage of the task to be executed;

[0120] Obtaining a task classification result of the task to be executed;

[0121] In response to the task classification result being an immediately executed task, using the actual execution duration as an estimated execution duration of the next immediately executed task, and using the actual memory usage as an estimated memory usage of the next immediately executed task;

[0122] In response to the task classification result being a scheduled execution task, the actual execution duration is used as the estimated execution duration of the next scheduled execution task, and the actual memory usage is used as the estimated memory usage of the next scheduled execution task.

[0123] Specifically, by acquiring task monitoring data, the performance of the storage system can be monitored, analyzed and optimized.

[0124] It should be understood that although Figure 1 , Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 , Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0125] Embodiment 2

[0126] In one embodiment, Figure 3 As shown, a performance data collection and processing device is provided, the device includes a main node, a standby node, an expansion cabinet and a shared database, the main node is connected to the standby node in communication, the standby node is connected to the expansion cabinet in communication, and the shared database is connected to the main node and the standby node in communication, respectively.

[0127] The master node collects and obtains its own master node performance data according to the received data collection instruction, and saves the master node performance data to the shared database;

[0128] The standby node collects and obtains its own standby node performance data according to the received data collection instruction;

[0129] The expansion cabinet collects and obtains its own expansion cabinet performance data according to the received data collection instruction, and transmits the expansion cabinet performance data to the standby node;

[0130] The standby node receives the expansion cabinet performance data, and saves the standby node performance data and the expansion cabinet performance data to the shared database;

[0131] The master node generates a data processing task and saves the data processing task to the shared database;

[0132] The master node obtains the tasks to be executed in the shared database according to a preset task execution order;

[0133] The master node selects a standby node to be executed from the standby nodes according to a preset task allocation rule, and allocates the task to be executed to the standby node to be executed;

[0134] The standby node to be executed analyzes the master node performance data, the standby node performance data and the expansion cabinet performance data in the shared database according to the received task to be executed, generates data analysis results, and saves the data analysis results to the shared database.

[0135] Specifically, each expansion cabinet is equipped with an independent controller, which contains CPU and memory resources, enabling it to run programs autonomously. Each controller collects its own CPU usage, memory usage, disk I / O and other performance data through built-in performance monitoring tools such as iostat (I / O statistics, performance analysis tool) and vmstat (Virtual Meomory Statistics, virtual memory statistics) in each sampling cycle.

[0136] Using distributed storage solutions such as NFS (Network File System) or Ceph (a distributed storage system), a storage pool space is mapped to the file system of each control cabinet (including the primary node and the backup node). In this way, each control cabinet can directly access and operate the shared database, improving the parallelism and efficiency of data access.

[0137] For the specific definition of the performance data collection and processing device, please refer to the definition of the performance data collection and processing method above, which will not be repeated here. Each module in the above-mentioned performance data collection and processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0138] Embodiment 3

[0139] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0140] Generate data collection instructions according to a preset sampling period;

[0141] According to the data collection instruction, the main node, the standby node and the expansion cabinet are respectively controlled to collect data to obtain the main node performance data, the standby node performance data and the expansion cabinet performance data;

[0142] Transmitting the expansion cabinet performance data to the standby node, and saving the master node performance data on the master node and the standby node performance data and expansion cabinet performance data on the standby node to a shared database;

[0143] In response to receiving a performance data processing request, controlling the master node to generate a data processing task, and saving the data processing task to the shared database;

[0144] According to a preset task execution order, control the master node to obtain the tasks to be executed in the shared database;

[0145] According to a preset task allocation rule, the master node is controlled to select a standby node to be executed from the standby nodes, and the task to be executed is allocated to the standby node to be executed;

[0146] In response to the standby node to be executed receiving the task to be executed, the standby node to be executed is controlled to analyze the master node performance data, standby node performance data and expansion cabinet performance data in the shared database, generate data analysis results, and save the data analysis results to the shared database.

[0147] When the program instructions are read and executed by the one or more processors, the program instructions may also perform operations corresponding to the steps in the above method embodiments, and the above description may be referred to, which will not be repeated here. Figure 4 , which exemplarily shows the architecture of the electronic device, which may specifically include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420 may be communicatively connected via a communication bus 430.

[0148] Among them, the processor 410 can be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in this application.

[0149] The memory 420 can be implemented in the form of a read-only memory (ROM), a random access memory (RAM), a static storage device, a dynamic storage device, etc. The memory 420 can store an operating system 421 for controlling the operation of the electronic device 400, and a basic input and output system (BIOS) 422 for controlling the low-level operation of the electronic device 400. In addition, a web browser 423, a data storage manager 424, and an icon font processing system 425, etc. can also be stored. The above-mentioned icon font processing system 425 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0150] The input / output interface 413 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0151] The network interface 414 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0152] The bus 430 comprises a pathway for transmitting information between the various components of the device (eg, the processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420).

[0153] In addition, the electronic device 400 can also obtain information on specific collection conditions from the virtual resource object collection condition information database 441 for use in condition judgment, etc.

[0154] It should be noted that, although the electronic device 400 only shows a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, a memory 420, a bus 430, etc., in the specific implementation process, the electronic device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0155] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling an electronic device (which can be a personal computer, a cloud server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0156] Embodiment 4

[0157] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0158] Generate data collection instructions according to a preset sampling period;

[0159] According to the data collection instruction, the main node, the standby node and the expansion cabinet are respectively controlled to collect data to obtain the main node performance data, the standby node performance data and the expansion cabinet performance data;

[0160] Transmitting the expansion cabinet performance data to the standby node, and saving the master node performance data on the master node and the standby node performance data and expansion cabinet performance data on the standby node to a shared database;

[0161] In response to receiving a performance data processing request, controlling the master node to generate a data processing task, and saving the data processing task to the shared database;

[0162] According to a preset task execution order, control the master node to obtain the tasks to be executed in the shared database;

[0163] According to a preset task allocation rule, the master node is controlled to select a standby node to be executed from the standby nodes, and the task to be executed is allocated to the standby node to be executed;

[0164] In response to the standby node to be executed receiving the task to be executed, the standby node to be executed is controlled to analyze the master node performance data, standby node performance data and expansion cabinet performance data in the shared database, generate data analysis results, and save the data analysis results to the shared database.

[0165] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0166] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0167] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

[0168] Embodiment 5

[0169] In one embodiment, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0170] Generate data collection instructions according to a preset sampling period;

[0171] According to the data collection instruction, the main node, the standby node and the expansion cabinet are respectively controlled to collect data to obtain the main node performance data, the standby node performance data and the expansion cabinet performance data;

[0172] Transmitting the expansion cabinet performance data to the standby node, and saving the master node performance data on the master node and the standby node performance data and expansion cabinet performance data on the standby node to a shared database;

[0173] In response to receiving a performance data processing request, controlling the master node to generate a data processing task, and saving the data processing task to the shared database;

[0174] According to a preset task execution order, control the master node to obtain the tasks to be executed in the shared database;

[0175] According to a preset task allocation rule, the master node is controlled to select a standby node to be executed from the standby nodes, and the task to be executed is allocated to the standby node to be executed;

[0176] In response to the standby node to be executed receiving the task to be executed, the standby node to be executed is controlled to analyze the master node performance data, standby node performance data and expansion cabinet performance data in the shared database, generate data analysis results, and save the data analysis results to the shared database.

[0177] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a computer program product. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0178] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for collecting and processing performance data, applied to a distributed storage system, wherein the distributed storage system comprises a master node, a standby node, an expansion cabinet, and a shared database, wherein the master node is communicatively connected to the standby node, the standby node is communicatively connected to the expansion cabinet, and the shared database is communicatively connected to the master node and the standby node, respectively, and characterized in that: The method comprises: Generate data collection instructions according to a preset sampling period; According to the data collection instruction, the main node, the standby node and the expansion cabinet are respectively controlled to collect data to obtain the main node performance data, the standby node performance data and the expansion cabinet performance data; Transmitting the expansion cabinet performance data to the standby node, and saving the master node performance data on the master node and the standby node performance data and expansion cabinet performance data on the standby node to a shared database; In response to receiving a performance data processing request, controlling the master node to generate a data processing task, and saving the data processing task to the shared database; According to a preset task execution order, control the master node to obtain the tasks to be executed in the shared database; According to a preset task allocation rule, the master node is controlled to select a standby node to be executed from the standby nodes, and the task to be executed is allocated to the standby node to be executed; In response to the standby node to be executed receiving the task to be executed, the standby node to be executed is controlled to analyze the master node performance data, standby node performance data and expansion cabinet performance data in the shared database, generate data analysis results, and save the data analysis results to the shared database.

2. The method according to claim 1, characterized in that: The step of saving the data processing task to a shared database comprises: Classifying the data processing tasks according to task execution time to obtain task classification results, wherein the task classification results include immediately executed tasks and scheduled executed tasks; In response to the data processing task being an immediately executed task, sequentially saving the data processing task to a to-be-executed task queue in the shared database; In response to the data processing task being a scheduled execution task, the data processing task is sequentially saved in a scheduled task table of the shared database; In response to the scheduled execution task reaching the scheduled execution time, the scheduled execution task is inserted into the to-be-executed queue as a to-be-executed task.

3. The method according to claim 2, characterized in that: The controlling the master node to obtain the tasks to be executed in the shared database according to the preset task execution order includes: Determine whether the to-be-executed task queue contains the scheduled execution task; In response to the scheduled execution task being included in the to-be-executed task queue, the scheduled execution task is sequentially used as the first to-be-executed task in the order in which the scheduled execution tasks are generated, and the immediate execution task is sequentially used as the second to-be-executed task in the order in which the immediate execution tasks are generated; In response to the queue of tasks to be executed not including the scheduled execution task, the master node is controlled to sequentially obtain the tasks to be executed according to the generation order of the tasks to be executed immediately.

4. The method according to claim 1, characterized in that: The step of controlling the master node to select a standby node to be executed from the standby nodes according to a preset task allocation rule includes: Obtaining an estimated execution time of the data processing task; Normalizing the estimated execution time according to a preset time normalization method to obtain a normalized execution time, wherein the time normalization method is to divide the estimated execution time by the maximum time; Obtaining an estimated memory usage of the data processing task; Normalizing the estimated memory usage according to a preset memory normalization method to obtain a normalized memory usage, wherein the memory normalization method is to divide the estimated memory usage by the total memory specification; According to the normalized execution time and the normalized memory usage, the IO load weight value and the memory utilization weight value of the standby node are calculated according to a preset weight value calculation method; Obtaining a relative value of the IO load and a relative value of the memory utilization of the standby node; Based on the IO load weight value, the memory utilization weight value, the IO load relative value, and the memory utilization relative value, the idle rate score of the standby node is calculated according to a preset idle rate calculation method; A standby node to be executed among the standby nodes is determined according to the idle rate score.

5. The method according to claim 4, characterized in that: The step of calculating the IO load weight value and the memory utilization weight value of the standby node according to the normalized execution time and the normalized memory usage according to a preset weight value calculation method includes: Determine whether the normalized execution time of the data processing task exceeds a preset time threshold; In response to the normalized execution time exceeding a preset time threshold, the data processing task is determined to be a long-time task, and the IO load weight value and the memory utilization weight value of the standby node are calculated according to the weight value calculation method of the long-time task; The method of calculating the weight value of the IO load and the memory utilization weight value of the standby node according to the weight value calculation method of the long time-consuming task includes: The IO load weight value of the standby node is calculated according to IO load weight value = IO load weight coefficient of the long-time task × normalized execution time; The memory utilization weight value of the standby node is calculated according to memory utilization weight value=memory utilization weight coefficient of the long-time task×normalized memory occupancy.

6. The method according to claim 5, characterized in that: The obtaining of the relative value of the IO load and the relative value of the memory utilization of the standby node includes: Obtaining an IO load reference value, an IO load maximum value, and a current IO load value of the standby node; According to IO load relative value=(current IO load value-IO load reference value) / (IO load maximum value-IO load reference value), calculate and obtain the IO load relative value of the standby node; Obtaining a memory utilization baseline value, a memory utilization maximum value, and a current memory utilization of the standby node; The relative value of memory utilization of the standby node is obtained by calculation according to relative value of memory utilization=(current memory utilization-reference value of memory utilization) / (maximum value of memory utilization-reference value of memory utilization).

7. The method according to claim 6, characterized in that: The calculating the idle rate score of the standby node according to a preset idle rate calculation method based on the IO load weight value, the memory utilization weight value, the IO load relative value, and the memory utilization relative value includes: The idle rate score of the standby node is calculated according to the idle rate score=(IO load relative value×IO load weight value)+(memory utilization relative value×memory utilization weight value); The determining, according to the idle rate score, a standby node to be executed among the standby nodes comprises: Sorting the standby nodes from high to low according to the idle rate scores to obtain an idle rate sorting result; According to the idle rate sorting result, the standby node ranked first is determined as the standby node to be executed.

8. A performance data collection and processing device, characterized in that: The device includes a master node, a standby node, an expansion cabinet and a shared database, wherein the master node is connected to the standby node in communication, the standby node is connected to the expansion cabinet in communication, and the shared database is connected to the master node and the standby node in communication, respectively. The master node collects and obtains its own master node performance data according to the received data collection instruction, and saves the master node performance data to the shared database; The standby node collects and obtains its own standby node performance data according to the received data collection instruction; The expansion cabinet collects and obtains its own expansion cabinet performance data according to the received data collection instruction, and transmits the expansion cabinet performance data to the standby node; The standby node receives the expansion cabinet performance data, and saves the standby node performance data and the expansion cabinet performance data to the shared database; The master node generates a data processing task and saves the data processing task to the shared database; The master node obtains the tasks to be executed in the shared database according to a preset task execution order; The master node selects a standby node to be executed from the standby nodes according to a preset task allocation rule, and allocates the task to be executed to the standby node to be executed; The standby node to be executed analyzes the master node performance data, the standby node performance data and the expansion cabinet performance data in the shared database according to the received task to be executed, generates data analysis results, and saves the data analysis results to the shared database.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images