A performance data collection processing method and device, electronic equipment and storage medium

Abstract

The application relates to a performance data collection processing method and device, electronic equipment and a storage medium. The method comprises the following steps: generating a data collection instruction, obtaining main node performance data, backup node performance data and extension cabinet performance data; transmitting the extension cabinet performance data to the backup node and saving all the performance data to a shared database; in response to receiving a performance data processing request, controlling the main node to generate a data processing task and saving the data processing task to the shared database; controlling the main node to acquire a to-be-executed task in the shared database, selecting a to-be-executed backup node from the backup node; assigning the to-be-executed task to the to-be-executed backup node; controlling the to-be-executed backup node to analyze all the performance data, generating a data analysis result and saving the data analysis result to the shared database. The method can reduce the burden of the main node and improve the efficiency and performance of the whole storage system.

Description

Technical Field

[0001] This application relates to the field of storage system technology, and in particular to a method, apparatus, electronic device, and storage medium for acquiring and processing performance data. Background Technology

[0002] As storage systems become increasingly large and complex, the burden on master nodes becomes heavier. In traditional architectures, master nodes are typically responsible for collecting and processing performance data from master nodes, standby nodes, and expansion cabinets. However, this performance data collection and processing model has the following drawbacks: centralized collection and processing overburdens the master node, and as the number of storage nodes increases, the amount of data the master node needs to process rises sharply, significantly increasing the pressure on it for performance data collection and processing. This not only increases the master node's memory and CPU usage but may also cause performance bottlenecks, affecting the overall system response speed and stability. Furthermore, node expansion is also limited.

[0003] Therefore, there is an urgent need for a method, device, electronic equipment, and storage medium for acquiring and processing performance data that can reduce the burden on the master node and improve the efficiency and performance of the entire storage system. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, electronic device, and storage medium for acquiring and processing performance data that can reduce the burden on the master node and improve the efficiency and performance of the entire storage system, in order to address the above-mentioned technical problems.

[0005] Firstly, a method for collecting and processing performance data is provided, applied to a distributed storage system. The distributed storage system includes a master node, a standby node, expansion cabinets, and a shared database. The master node and the standby node are communicatively connected, the standby node and the expansion cabinet are communicatively connected, and the shared database is communicatively connected to both the master node and the standby node. The method includes:

[0006] Data acquisition instructions are generated according to the preset sampling period;

[0007] According to the data acquisition instructions, the main node, standby node and expansion cabinet are controlled to acquire data respectively, so as to obtain the performance data of the main node, the performance data of the standby node and the performance data of the expansion cabinet.

[0008] The performance data of the expansion cabinet is transmitted to the backup node, and the performance data of the master node on the master node, the performance data of the backup node on the backup node, and the performance data of the expansion cabinet are saved to the shared database.

[0009] In response to receiving a performance data processing request, the master node is controlled to generate a data processing task and the data processing task is saved to the shared database;

[0010] According to the preset task execution order, the master node is controlled to obtain the tasks to be executed from the shared database;

[0011] According to the preset task allocation rules, the master node is controlled to select a backup node to be executed from the backup nodes, and the task to be executed is assigned to the backup node to be executed.

[0012] In response to the standby node receiving the task to be executed, the standby node is controlled to analyze the performance data of the primary node, the standby node, and the expansion cabinet in the shared database, generate data analysis results, and save the data analysis results to the shared database.

[0013] In one embodiment, saving the data processing task to a shared database includes:

[0014] The data processing tasks are classified according to their execution time to obtain task classification results, wherein the task classification results include tasks to be executed immediately and tasks to be executed on a scheduled basis.

[0015] In response to the fact that the data processing task is an immediately executable task, the data processing task is sequentially saved to the queue of tasks to be executed in the shared database;

[0016] In response to the fact that the data processing task is a scheduled task, the data processing task is saved sequentially to the scheduled task table of the shared database;

[0017] In response to the scheduled execution time of the scheduled task, the scheduled task is inserted into the queue of tasks to be executed.

[0018] In one embodiment, controlling the master node to obtain tasks to be executed from the shared database according to a preset task execution order includes:

[0019] Determine whether the scheduled task is present in the queue of tasks to be executed;

[0020] In response to the presence of the scheduled task in the queue of tasks to be executed, the scheduled task is sequentially designated as the first task to be executed according to the generation order of the scheduled task, and the immediately executed task is sequentially designated as the second task to be executed according to the generation order of the immediately executed task;

[0021] In response to the absence of the scheduled task in the queue of tasks to be executed, the master node is controlled to sequentially acquire the tasks to be executed according to the generation order of the immediately executed tasks.

[0022] In one embodiment, controlling the master node to select a backup node to be executed from the backup nodes according to a preset task allocation rule includes:

[0023] Obtain the estimated execution time of the data processing task;

[0024] The estimated execution time is normalized according to a preset time normalization method to obtain a normalized execution time, wherein the time normalization method is to divide the estimated execution time by the maximum time value.

[0025] Obtain the estimated memory usage of the data processing task;

[0026] The estimated memory usage is normalized according to a preset memory normalization method to obtain a normalized memory usage, wherein the memory normalization method is to divide the estimated memory usage by the total memory specification.

[0027] Based on the normalized execution time and normalized memory usage, the IO load weight value and memory utilization weight value of the backup node are calculated according to the preset weight value calculation method.

[0028] Obtain the relative values ​​of IO load and memory utilization of the backup node;

[0029] Based on the IO load weight value, memory utilization weight value, IO load relative value, and memory utilization relative value, the idle rate score of the standby node is calculated according to the preset idle rate calculation method.

[0030] The standby nodes to be executed among the standby nodes are determined based on the idle rate score.

[0031] In one embodiment, the step of calculating the IO load weight value and memory utilization weight value of the standby node according to the normalized execution time and normalized memory usage, and using a preset weight value calculation method, includes:

[0032] Determine whether the normalized execution time of the data processing task exceeds a preset time threshold;

[0033] In response to the normalized execution time exceeding a preset time threshold, the data processing task is determined to be a long-running task, and the IO load weight value and memory utilization weight value of the backup node are calculated according to the weight value calculation method for long-running tasks.

[0034] The calculation of the IO load weight value and memory utilization weight value of the backup node according to the weight value calculation method for long-duration tasks includes:

[0035] The IO load weight value of the backup node is calculated based on the formula: IO load weight value = IO load weight coefficient of long-running task × normalized execution time.

[0036] The memory utilization weight value of the backup node is calculated based on the formula: memory utilization weight value = memory utilization weight coefficient of long-running task × normalized memory usage.

[0037] In one embodiment, obtaining the relative values ​​of the IO load and memory utilization of the standby node includes:

[0038] Obtain the baseline IO load, maximum IO load, and current IO load value of the standby node;

[0039] The relative IO load of the standby node is calculated based on the formula: IO load relative value = (current IO load value - IO load baseline value) / (maximum IO load value - IO load baseline value).

[0040] Obtain the baseline memory utilization value, maximum memory utilization value, and current memory utilization value of the standby node;

[0041] The relative memory utilization value of the standby node is calculated based on the formula: relative memory utilization value = (current memory utilization value - baseline memory utilization value) / (maximum memory utilization value - baseline memory utilization value).

[0042] In one embodiment, the step of calculating the idle rate score of the standby node based on the IO load weight value, memory utilization weight value, relative IO load value, and relative memory utilization value according to a preset idle rate calculation method includes:

[0043] The idle rate score of the standby node is calculated based on the idle rate score = (relative value of IO load × weight value of IO load) + (relative value of memory utilization × weight value of memory utilization).

[0044] The step of determining the standby nodes to be executed from the standby nodes based on the idle rate score includes:

[0045] The spare nodes are sorted from high to low according to their idle rate scores to obtain the idle rate sorting results;

[0046] Based on the idle rate sorting results, the first-ranked standby node is determined as the standby node to be executed.

[0047] Secondly, a performance data acquisition and processing device is provided. The device includes a master node, a standby node, an expansion cabinet, and a shared database. The master node is communicatively connected to the standby node, the standby node is communicatively connected to the expansion cabinet, and the shared database is communicatively connected to both the master node and the standby node.

[0048] The master node collects its own master node performance data according to the received data collection instruction, and saves the master node performance data to the shared database;

[0049] The backup node collects its own backup node performance data according to the received data collection instructions;

[0050] The expansion cabinet collects its own expansion cabinet performance data according to the received data acquisition command, and transmits the expansion cabinet performance data to the backup node;

[0051] The backup node receives the performance data of the expansion cabinet and saves the backup node performance data and the expansion cabinet performance data to the shared database.

[0052] The master node generates a data processing task and saves the data processing task to the shared database;

[0053] The master node obtains the tasks to be executed from the shared database according to the preset task execution order;

[0054] The master node selects a backup node to be executed from the backup nodes according to a preset task allocation rule, and assigns the task to be executed to the backup node to be executed;

[0055] The standby node to be executed analyzes the performance data of the primary node, the standby node, and the expansion cabinet in the shared database according to the received task to be executed, generates data analysis results, and saves the data analysis results to the shared database.

[0056] Thirdly, an electronic device is provided, the electronic device including one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method as described in any one of the first aspects above.

[0057] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the steps of the method as described in any one of the first aspects above.

[0058] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method as described in any one of the first aspects above.

[0059] The aforementioned performance data acquisition and processing methods, devices, electronic equipment, and storage media transmit expansion cabinet performance data to the standby node and save the primary node performance data on the primary node, the standby node performance data on the standby node, and the expansion cabinet performance data to a shared database. This reduces the burden on the primary node and improves data processing efficiency. By performing data processing tasks through expansion cabinets and standby nodes, effective management and utilization of storage system resources are achieved, improving the overall system performance. Data analysis results provide insights into the performance of the primary node, standby nodes, expansion cabinets, and the entire storage system. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating a performance data acquisition and processing method in one embodiment;

[0061] Figure 2 This is a system flowchart of a performance data acquisition and processing method in one embodiment;

[0062] Figure 3 This is a structural block diagram of a performance data acquisition and processing device in one embodiment;

[0063] Figure 4 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] Example 1

[0066] In one embodiment, such as Figure 1 , Figure 2 As shown, a method for collecting and processing performance data is provided, applied to a distributed storage system. The distributed storage system includes a master node, a standby node, an expansion cabinet, and a shared database. The master node and the standby node are communicatively connected, the standby node and the expansion cabinet are communicatively connected, and the shared database is communicatively connected to both the master node and the standby node. The method includes:

[0067] Data acquisition instructions are generated according to the preset sampling period;

[0068] According to the data acquisition instructions, the main node, standby node and expansion cabinet are controlled to acquire data respectively, so as to obtain the performance data of the main node, the performance data of the standby node and the performance data of the expansion cabinet.

[0069] The performance data of the expansion cabinet is transmitted to the backup node, and the performance data of the master node on the master node, the performance data of the backup node on the backup node, and the performance data of the expansion cabinet are saved to the shared database.

[0070] In response to receiving a performance data processing request, the master node is controlled to generate a data processing task and the data processing task is saved to the shared database;

[0071] According to the preset task execution order, the master node is controlled to obtain the tasks to be executed from the shared database;

[0072] According to the preset task allocation rules, the master node is controlled to select a backup node to be executed from the backup nodes, and the task to be executed is assigned to the backup node to be executed.

[0073] In response to the standby node receiving the task to be executed, the standby node is controlled to analyze the performance data of the primary node, the standby node, and the expansion cabinet in the shared database, generate data analysis results, and save the data analysis results to the shared database.

[0074] Specifically, performance data from the expansion cabinet is transferred to the standby node, and the performance data of the primary node on the primary node, the standby node on the standby node, and the expansion cabinet are saved to a shared database. This reduces the burden on the primary node and improves data processing efficiency. By performing data processing tasks through the expansion cabinets and standby nodes, effective management and utilization of storage system resources are achieved, improving the overall system performance. Data analysis results provide insights into the performance of the primary node, standby nodes, expansion cabinets, and the entire storage system.

[0075] In one embodiment, saving the data processing task to a shared database includes:

[0076] The data processing tasks are classified according to their execution time to obtain task classification results, wherein the task classification results include tasks to be executed immediately and tasks to be executed on a scheduled basis.

[0077] In response to the fact that the data processing task is an immediately executable task, the data processing task is sequentially saved to the queue of tasks to be executed in the shared database;

[0078] In response to the fact that the data processing task is a scheduled task, the data processing task is saved sequentially to the scheduled task table of the shared database;

[0079] In response to the scheduled execution time of the scheduled task, the scheduled task is inserted into the queue of tasks to be executed.

[0080] Specifically, tasks that can be executed immediately do not require waiting; tasks that can be executed on a schedule require waiting for a specific time. Saving data processing tasks to a shared database allows backup nodes to execute the data processing tasks later, thus improving the system's distributed processing capabilities.

[0081] In one embodiment, controlling the master node to obtain tasks to be executed from the shared database according to a preset task execution order includes:

[0082] Determine whether the scheduled task is present in the queue of tasks to be executed;

[0083] In response to the presence of the scheduled execution task in the queue of tasks to be executed, the scheduled execution task is sequentially designated as the first task to be executed according to the generation order of the scheduled execution task, and the immediately executed task is sequentially designated as the second task to be executed according to the generation order of the immediately executed task;

[0084] In response to the absence of the scheduled task in the queue of tasks to be executed, the master node is controlled to sequentially acquire the tasks to be executed according to the generation order of the immediately executed tasks.

[0085] Specifically, when the queue of tasks to be executed includes both scheduled tasks and immediate tasks, the scheduled tasks are executed first, followed by the immediate tasks, and they are executed in the order they were generated. When the queue of tasks to be executed contains only immediate tasks, the immediate tasks are executed in the order they were generated. When the queue of tasks to be executed contains only scheduled tasks, the scheduled tasks are executed in the order they were generated.

[0086] In one embodiment, controlling the master node to select a backup node to be executed from the backup nodes according to a preset task allocation rule includes:

[0087] Obtain the estimated execution time of the data processing task;

[0088] The estimated execution time is normalized according to a preset time normalization method to obtain a normalized execution time, wherein the time normalization method is to divide the estimated execution time by the maximum time value.

[0089] Obtain the estimated memory usage of the data processing task;

[0090] The estimated memory usage is normalized according to a preset memory normalization method to obtain a normalized memory usage, wherein the memory normalization method is to divide the estimated memory usage by the total memory specification.

[0091] Based on the normalized execution time and normalized memory usage, the IO load weight value and memory utilization weight value of the backup node are calculated according to the preset weight value calculation method.

[0092] Obtain the relative values ​​of IO load and memory utilization of the backup node;

[0093] Based on the IO load weight value, memory utilization weight value, IO load relative value, and memory utilization relative value, the idle rate score of the standby node is calculated according to the preset idle rate calculation method.

[0094] The standby nodes to be executed among the standby nodes are determined based on the idle rate score.

[0095] Specifically, the maximum execution time is greater than or equal to the maximum historical estimated execution time. Duration normalization converts the estimated execution time of the data processing task into a value between 0 and 1. Memory normalization converts the estimated memory usage of the data processing task into a value between 0 and 1. Based on the relative values ​​of IO load and memory utilization of each standby node, combined with the normalized execution time and normalized memory usage, an idle rate score is calculated for each standby node to select the standby node with the highest idle rate for execution.

[0096] In one embodiment, the step of calculating the IO load weight value and memory utilization weight value of the standby node according to the normalized execution time and normalized memory usage, using a preset weight value calculation method, includes:

[0097] Determine whether the normalized execution time of the data processing task exceeds a preset time threshold;

[0098] In response to the normalized execution time exceeding a preset time threshold, the data processing task is determined to be a long-running task, and the IO load weight value and memory utilization weight value of the backup node are calculated according to the weight value calculation method for long-running tasks.

[0099] The calculation of the IO load weight value and memory utilization weight value of the backup node according to the weight value calculation method for long-duration tasks includes:

[0100] The IO load weight value of the backup node is calculated based on the formula: IO load weight value = IO load weight coefficient of long-running task × normalized execution time.

[0101] The memory utilization weight value of the backup node is calculated based on the formula: memory utilization weight value = memory utilization weight coefficient of long-running task × normalized memory usage.

[0102] Specifically, based on the characteristics of the data processing task, such as long execution time or large memory requirements, the weights of I / O load and memory utilization are dynamically allocated. When the normalized execution time exceeds a preset time threshold, the data processing task is identified as a long-running task; when the normalized memory usage exceeds a preset memory threshold, the data processing task is identified as a large memory-requirement task. The time threshold and memory threshold can be set to 0.5, or flexibly adjusted according to actual needs.

[0103] For long-running tasks, the IO load weight is calculated as 0.5 × estimated task duration, and the memory utilization weight is calculated as 0.5 × estimated memory usage. For tasks with high memory requirements, the IO load weight is calculated as the IO load weight coefficient for tasks with high memory requirements × estimated task duration, and the memory utilization weight is calculated as the memory utilization weight coefficient for tasks with high memory requirements × estimated memory usage. For example, the IO load weight could be 0.4 × estimated task duration, and the memory utilization weight could be 0.6 × estimated memory usage. These weight coefficients can be flexibly set according to the actual situation.

[0104] Based on the weighting formula, the IO load weight value and memory utilization weight value of the standby node are calculated respectively, so as to calculate the idle rate score of the standby node in the future.

[0105] In one embodiment, obtaining the relative values ​​of the IO load and memory utilization of the standby node includes:

[0106] Obtain the baseline IO load, maximum IO load, and current IO load value of the standby node;

[0107] The relative IO load of the standby node is calculated based on the formula: IO load relative value = (current IO load value - IO load baseline value) / (maximum IO load value - IO load baseline value).

[0108] Obtain the baseline memory utilization value, maximum memory utilization value, and current memory utilization value of the standby node;

[0109] The relative memory utilization value of the standby node is calculated based on the formula: relative memory utilization value = (current memory utilization value - baseline memory utilization value) / (maximum memory utilization value - baseline memory utilization value).

[0110] Specifically, assuming the baseline IO load is 0 and the maximum IO load is 1,000,000, then the relative IO load value = current IO load value / 1,000,000. Similarly, assuming the baseline memory utilization is 0 and the maximum memory utilization is 100, then the relative memory utilization value = current memory utilization / 100. Based on these relative value calculation formulas, the relative IO load value and relative memory utilization value of the standby node are calculated to subsequently calculate the standby node's idle rate score.

[0111] In one embodiment, calculating the idle rate score of the standby node based on the IO load weight value, memory utilization weight value, relative IO load value, and relative memory utilization value according to a preset idle rate calculation method includes:

[0112] The idle rate score of the standby node is calculated based on the idle rate score = (relative value of IO load × weight value of IO load) + (relative value of memory utilization × weight value of memory utilization).

[0113] The step of determining the standby nodes to be executed from the standby nodes based on the idle rate score includes:

[0114] The spare nodes are sorted from high to low according to their idle rate scores to obtain the idle rate sorting results;

[0115] Based on the idle rate sorting results, the first-ranked standby node is determined as the standby node to be executed.

[0116] Specifically, based on the idle rate calculation formula, the idle rate score of the standby node is calculated, and the standby node with the highest idle rate is selected to execute the task to be executed, thereby improving data processing efficiency.

[0117] In one embodiment, the data analysis results include task monitoring data, and after saving the data analysis results to the shared database, the method further includes:

[0118] In response to receiving a task completion signal, the master node is controlled to generate a data read instruction and send the data read instruction to the shared database;

[0119] In response to the shared database receiving the data read instruction, the shared database is controlled to transmit the task monitoring data to the master node, wherein the task monitoring data includes the actual execution time and actual memory usage of the task to be executed;

[0120] Obtain the task classification result of the task to be executed;

[0121] In response to the task classification result being an immediately executed task, the actual execution time is used as the estimated execution time of the next immediately executed task, and the actual memory usage is used as the estimated memory usage of the next immediately executed task.

[0122] In response to the task classification result being a scheduled task, the actual execution time is used as the estimated execution time of the next scheduled task, and the actual memory usage is used as the estimated memory usage of the next scheduled task.

[0123] Specifically, by acquiring task monitoring data, the performance of the storage system can be monitored, analyzed, and optimized.

[0124] It should be understood that, although Figure 1 , Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 , Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0125] Example 2

[0126] In one embodiment, such as Figure 3 As shown, a performance data acquisition and processing device is provided. The device includes a master node, a standby node, an expansion cabinet, and a shared database. The master node and the standby node are communicatively connected, the standby node is communicatively connected to the expansion cabinet, and the shared database is communicatively connected to both the master node and the standby node.

[0127] The master node collects its own master node performance data according to the received data collection instruction, and saves the master node performance data to the shared database;

[0128] The backup node collects its own backup node performance data according to the received data collection instructions;

[0129] The expansion cabinet collects its own expansion cabinet performance data according to the received data acquisition command, and transmits the expansion cabinet performance data to the backup node;

[0130] The backup node receives the performance data of the expansion cabinet and saves the backup node performance data and the expansion cabinet performance data to the shared database.

[0131] The master node generates a data processing task and saves the data processing task to the shared database;

[0132] The master node obtains the tasks to be executed from the shared database according to the preset task execution order;

[0133] The master node selects a backup node to be executed from the backup nodes according to a preset task allocation rule, and assigns the task to be executed to the backup node to be executed;

[0134] The standby node to be executed analyzes the performance data of the primary node, the standby node, and the expansion cabinet in the shared database according to the received task to be executed, generates data analysis results, and saves the data analysis results to the shared database.

[0135] Specifically, each expansion cabinet is equipped with an independent controller containing CPU and memory resources, enabling it to run programs autonomously. During each sampling period, each controller collects its own performance data, such as CPU usage, memory consumption, and disk I / O, through built-in performance monitoring tools, such as iostat (I / O statistics) and vmstat (Virtual Memory Statistics).

[0136] By utilizing distributed storage solutions such as NFS (Network File System) or Ceph (a distributed storage system), a storage pool space is mapped to the file system of each control rack (including primary and standby nodes). This allows each control rack to directly access and manipulate the shared database, improving the parallelism and efficiency of data access.

[0137] Specific limitations regarding the performance data acquisition and processing device can be found in the limitations of the performance data acquisition and processing method described above, and will not be repeated here. Each module in the aforementioned performance data acquisition and processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independently of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of each module.

[0138] Example 3

[0139] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0140] Data acquisition instructions are generated according to the preset sampling period;

[0141] According to the data acquisition instructions, the main node, standby node and expansion cabinet are controlled to acquire data respectively, so as to obtain the performance data of the main node, the performance data of the standby node and the performance data of the expansion cabinet.

[0142] The performance data of the expansion cabinet is transmitted to the backup node, and the performance data of the master node on the master node, the performance data of the backup node on the backup node, and the performance data of the expansion cabinet are saved to the shared database.

[0143] In response to receiving a performance data processing request, the master node is controlled to generate a data processing task and the data processing task is saved to the shared database;

[0144] According to the preset task execution order, the master node is controlled to obtain the tasks to be executed from the shared database;

[0145] According to the preset task allocation rules, the master node is controlled to select a backup node to be executed from the backup nodes, and the task to be executed is assigned to the backup node to be executed.

[0146] In response to the standby node receiving the task to be executed, the standby node is controlled to analyze the performance data of the primary node, the standby node, and the expansion cabinet in the shared database, generate data analysis results, and save the data analysis results to the shared database.

[0147] When the program instructions are read and executed by the one or more processors, they can also perform operations corresponding to the steps in the above method embodiments, as described above, and will not be repeated here. (Reference) Figure 4 This exemplifies the architecture of an electronic device, which may include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, video display adapter 411, disk drive 412, input / output interface 413, network interface 414, and memory 420 can communicate with each other via a communication bus 430.

[0148] The processor 410 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in this application.

[0149] The memory 420 can be implemented as a read-only memory (ROM), random access memory (RAM), static storage device, dynamic storage device, etc. The memory 420 can store the operating system 421 for controlling the operation of the electronic device 400, and the basic input / output system (BIOS) 422 for controlling the low-level operations of the electronic device 400. Additionally, it can store a web browser 423, data storage management 424, and an icon / font processing system 425, etc. The aforementioned icon / font processing system 425 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 420 and executed by the processor 410.

[0150] Input / output interface 413 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0151] Network interface 414 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0152] Bus 430 includes a pathway for transmitting information between various components of the device, such as processor 410, video display adapter 411, disk drive 412, input / output interface 413, network interface 414, and memory 420.

[0153] In addition, the electronic device 400 can also obtain information on specific acquisition conditions from the virtual resource object acquisition condition information database 441 for condition judgment, etc.

[0154] It should be noted that although the above-described electronic device 400 only shows a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, a memory 420, and a bus 430, in specific implementations, the electronic device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0155] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause an electronic device (which may be a personal computer, cloud server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0156] Example 4

[0157] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0158] Data acquisition instructions are generated according to the preset sampling period;

[0159] According to the data acquisition instructions, the main node, standby node and expansion cabinet are controlled to acquire data respectively, so as to obtain the performance data of the main node, the performance data of the standby node and the performance data of the expansion cabinet.

[0160] The performance data of the expansion cabinet is transmitted to the backup node, and the performance data of the master node on the master node, the performance data of the backup node on the backup node, and the performance data of the expansion cabinet are saved to the shared database.

[0161] In response to receiving a performance data processing request, the master node is controlled to generate a data processing task and the data processing task is saved to the shared database;

[0162] According to the preset task execution order, the master node is controlled to obtain the tasks to be executed from the shared database;

[0163] According to the preset task allocation rules, the master node is controlled to select a backup node to be executed from the backup nodes, and the task to be executed is assigned to the backup node to be executed.

[0164] In response to the standby node receiving the task to be executed, the standby node is controlled to analyze the performance data of the primary node, the standby node, and the expansion cabinet in the shared database, generate data analysis results, and save the data analysis results to the shared database.

[0165] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0166] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0167] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

[0168] Example 5

[0169] In one embodiment, a computer program product is provided, on which a computer program is stored, the computer program performing the following steps when executed by a processor:

[0170] Data acquisition instructions are generated according to the preset sampling period;

[0171] According to the data acquisition instructions, the main node, standby node and expansion cabinet are controlled to acquire data respectively, so as to obtain the performance data of the main node, the performance data of the standby node and the performance data of the expansion cabinet.

[0172] The performance data of the expansion cabinet is transmitted to the backup node, and the performance data of the master node on the master node, the performance data of the backup node on the backup node, and the performance data of the expansion cabinet are saved to the shared database.

[0173] In response to receiving a performance data processing request, the master node is controlled to generate a data processing task and the data processing task is saved to the shared database;

[0174] According to the preset task execution order, the master node is controlled to obtain the tasks to be executed from the shared database;

[0175] According to the preset task allocation rules, the master node is controlled to select a backup node to be executed from the backup nodes, and the task to be executed is assigned to the backup node to be executed.

[0176] In response to the standby node receiving the task to be executed, the standby node is controlled to analyze the performance data of the primary node, the standby node, and the expansion cabinet in the shared database, generate data analysis results, and save the data analysis results to the shared database.

[0177] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer program product, and when the computer program is executed, it can include the processes of the embodiments of the above methods.

[0178] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for acquiring and processing performance data, applied to a distributed storage system, the distributed storage system comprising a master node, a standby node, an expansion cabinet, and a shared database, wherein the master node is communicatively connected to the standby node, the standby node is communicatively connected to the expansion cabinet, and the shared database is communicatively connected to both the master node and the standby node, characterized in that... The method includes: Data acquisition instructions are generated according to the preset sampling period; According to the data acquisition instructions, the main node, standby node and expansion cabinet are controlled to acquire data respectively, so as to obtain the performance data of the main node, the performance data of the standby node and the performance data of the expansion cabinet. The performance data of the expansion cabinet is transmitted to the backup node, and the performance data of the master node on the master node, the performance data of the backup node on the backup node, and the performance data of the expansion cabinet are saved to the shared database. In response to receiving a performance data processing request, the master node is controlled to generate a data processing task and the data processing task is saved to the shared database; According to the preset task execution order, the master node is controlled to obtain the tasks to be executed from the shared database; According to the preset task allocation rules, the master node is controlled to select a backup node to be executed from the backup nodes, and the task to be executed is assigned to the backup node to be executed. In response to the standby node receiving the task to be executed, the standby node is controlled to analyze the performance data of the primary node, the standby node, and the expansion cabinet in the shared database, generate data analysis results, and save the data analysis results to the shared database; The step of controlling the master node to select a backup node to be executed from the backup nodes according to a preset task allocation rule includes: Obtain the estimated execution time of the data processing task; The estimated execution time is normalized according to a preset time normalization method to obtain a normalized execution time, wherein the time normalization method is to divide the estimated execution time by the maximum time value. Obtain the estimated memory usage of the data processing task; The estimated memory usage is normalized according to a preset memory normalization method to obtain a normalized memory usage, wherein the memory normalization method is to divide the estimated memory usage by the total memory specification. Based on the normalized execution time and normalized memory usage, the IO load weight value and memory utilization weight value of the backup node are calculated according to the preset weight value calculation method. Obtain the relative values ​​of IO load and memory utilization of the backup node; Based on the IO load weight value, memory utilization weight value, IO load relative value, and memory utilization relative value, the idle rate score of the standby node is calculated according to the preset idle rate calculation method. The standby nodes to be executed among the standby nodes are determined based on the idle rate score; The step of calculating the IO load weight value and memory utilization weight value of the backup node according to the normalized execution time and normalized memory usage, and using a preset weight value calculation method, includes: Determine whether the normalized execution time of the data processing task exceeds a preset time threshold, and whether the normalized memory usage of the data processing task exceeds a preset memory threshold. In response to the normalized execution time exceeding a preset time threshold, the data processing task is determined to be a long-running task, and the IO load weight value and memory utilization weight value of the backup node are calculated according to the weight value calculation method for long-running tasks. In response to the normalized memory usage exceeding a preset memory threshold, the data processing task is determined to be a large memory demand task, and the IO load weight value and memory utilization weight value of the backup node are calculated according to the weight value calculation method for large memory demand tasks. The calculation of the IO load weight value and memory utilization weight value of the backup node according to the weight value calculation method for long-duration tasks includes: The IO load weight value of the backup node is calculated based on the formula: IO load weight value = IO load weight coefficient of long-running task × normalized execution time. The memory utilization weight value of the backup node is calculated based on the formula: memory utilization weight value = memory utilization weight coefficient of long-running task × normalized memory usage. The calculation of the IO load weight value and memory utilization weight value of the backup node according to the weight value calculation method for tasks with large memory requirements includes: The IO load weight value of the backup node is calculated based on the formula: IO load weight value = IO load weight coefficient of large memory demand task × normalized execution time. The memory utilization weight value of the backup node is calculated based on the formula: memory utilization weight value = memory utilization weight coefficient of large memory demand task × normalized memory usage.

2. The method according to claim 1, characterized in that, Saving the data processing task to the shared database includes: The data processing tasks are classified according to their execution time to obtain task classification results, wherein the task classification results include tasks to be executed immediately and tasks to be executed on a scheduled basis. In response to the fact that the data processing task is an immediately executable task, the data processing task is sequentially saved to the queue of tasks to be executed in the shared database; In response to the fact that the data processing task is a scheduled task, the data processing task is saved sequentially to the scheduled task table of the shared database; In response to the scheduled execution time of the scheduled task, the scheduled task is inserted as a task to be executed into the task queue.

3. The method according to claim 2, characterized in that, The step of controlling the master node to obtain tasks to be executed from the shared database according to a preset task execution order includes: Determine whether the scheduled task is present in the queue of tasks to be executed; In response to the presence of the scheduled task in the queue of tasks to be executed, the scheduled task is sequentially designated as the first task to be executed according to the generation order of the scheduled task, and the immediately executed task is sequentially designated as the second task to be executed according to the generation order of the immediately executed task; In response to the absence of the scheduled task in the queue of tasks to be executed, the master node is controlled to sequentially acquire the tasks to be executed according to the generation order of the immediately executed tasks.

4. The method according to claim 1, characterized in that, The process of obtaining the relative values ​​of IO load and memory utilization of the backup node includes: Obtain the baseline IO load, maximum IO load, and current IO load value of the standby node; The relative IO load of the standby node is calculated based on the formula: IO load relative value = (current IO load value - IO load baseline value) / (maximum IO load value - IO load baseline value). Obtain the baseline memory utilization value, maximum memory utilization value, and current memory utilization value of the standby node; The relative memory utilization value of the standby node is calculated based on the formula: relative memory utilization value = (current memory utilization value - baseline memory utilization value) / (maximum memory utilization value - baseline memory utilization value).

5. The method according to claim 4, characterized in that, The idle rate score of the standby node is calculated based on the IO load weight value, memory utilization weight value, relative IO load value, and relative memory utilization value, according to a preset idle rate calculation method, including: The idle rate score of the standby node is calculated based on the idle rate score = (relative value of IO load × weight value of IO load) + (relative value of memory utilization × weight value of memory utilization). The step of determining the standby nodes to be executed from the standby nodes based on the idle rate score includes: The spare nodes are sorted from high to low according to their idle rate scores to obtain the idle rate sorting results; Based on the idle rate sorting results, the first-ranked standby node is determined as the standby node to be executed.

6. A device for acquiring and processing performance data to implement the method of claim 1, characterized in that, The device includes a master node, a backup node, an expansion cabinet, and a shared database. The master node is communicatively connected to the backup node, the backup node is communicatively connected to the expansion cabinet, and the shared database is communicatively connected to both the master node and the backup node. The master node collects its own master node performance data according to the received data collection instruction, and saves the master node performance data to the shared database; The backup node collects its own backup node performance data according to the received data collection instructions; The expansion cabinet collects its own expansion cabinet performance data according to the received data acquisition command, and transmits the expansion cabinet performance data to the backup node; The backup node receives the performance data of the expansion cabinet and saves the backup node performance data and the expansion cabinet performance data to the shared database. The master node generates a data processing task and saves the data processing task to the shared database; The master node obtains the tasks to be executed from the shared database according to the preset task execution order; The master node selects a backup node to be executed from the backup nodes according to a preset task allocation rule, and assigns the task to be executed to the backup node to be executed; The standby node to be executed analyzes the performance data of the primary node, the standby node, and the expansion cabinet in the shared database according to the received task to be executed, generates data analysis results, and saves the data analysis results to the shared database.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images